In Q2 2009, we congratulated Dow Jones on taking a full five additional days merely to confirm our VC funding numbers. We’ve also drawn attention to the absurd historical disparities between Dow Jones’ and Thomson’s figures, asking some inconvenient questions about a topic which these two data giants prefer to ignore. In short, we’ve never been shy about shining a critical light on the traditional and historically unchallenged providers of venture capital information.
It’s now Q3 2009, and Dow Jones has once again released its funding numbers several days after ChubbyBrain. Some things don’t change. Except this time the two figures paint divergent stories on the health of the venture capital asset class (we posted $6.1B and Dow Jones $5.1B). This makes for an excellent opportunity to explain what drives the differences between our approach and theirs.
But first let’s give credit where credit is due: Dow Jones VentureSource has a rigorous process, employs dozens of researchers, and has been doing the same thing for 20 years. When we talk to VCs, it’s clear that they are undisputedly the leading data providers in this space. It is also clear that despite significant media coverage of the MoneyTree Report numbers, Thomson’s VenturExpert has virtually no presence. (Note: Thomson research underlies the PwC MoneyTree Report, whose puzzling method and data continues to perplex).
Even less disputable than the venerability of Dow Jones, however, is the simple reality that times change. Technologies advance, attitudes and behaviors evolve, and better processes are introduced. Furthermore, it is seldom that the market leaders are the ones driving this change. Lulled by their dominant position, brand power, and legacy thinking, they prefer clinging to the past and decry the new methods. And yet we all know the outcome to this tale of disruptive innovation. Better technology and processes trump history.
So what about the delta in our Q3 2009 numbers? It is attributable to two primary differences:
Definitional differences may drive some disparity – they may include deals we don’t or we may attribute a deal to Q3 which they will put into Q4 because of differences in closing vs announcement dates. But definitional reasons aren’t sufficient to account for the entire delta. We may monitor the innovation economy more holistically, but our definition of what is in vs. out and is simply not that dissimilar to Dow Jones’s. You can take a look at our detailed definitions here. As for VentureSource, here is what they include: “These statistics represent equity investments into early-stage, innovative companies only and do not include companies receiving funding solely from corporate, individual, and/or government investors, or from buyout or other non-VC investment firms.” (Update: see our post on claims by PwC MoneyTree’s Tracy Lefteroff that Dow Jones VentureSource includes debt financings in their quarterly venture capital numbers)
The larger driver, therefore, is one of methodology. Dow Jones describes its methodology as follows: “The investment figures included in this release were collected by surveying professional venture capital firms, through in-depth interviews with portfolio company CEOs and CFOs, and from a number of secondary sources.”
The survey methodology is simply an anachronism today. It wasn’t always that way: 20 years ago, when Dow Jones started, it made perfect sense. Information was much less accessible. In many cases, obtaining SEC or other official documents required actually visiting Washington DC or the states in which they originated. And so well-capitalized companies like Dow Jones developed a highly labor-intensive, relationship-driven model to collect this disparate information. Their products, like VentureSource, were uniquely positioned to provide data-driven insights into the worlds of venture capital and beyond.
But times have changed on multiple fronts:
- Information availability – Historic information silos continue to be broken down. Regulatory documents and other reliable information sources are now more accessible.
- Technology – Through the thoughtful application of technology, we can extract this information efficiently and create mechanisms by which it is structured, verified and added to our platform.
- Proactive user contribution – We get inputs directly from VCs and their companies about funding events. Of course, all such contributions must be confirmed, but this is the product of a recent behavioral shift which the web has enabled and encouraged (think mass collaboration).
The above technological and behavioral changes allow ChubbyBrain not just to get data which is more comprehensive and correct, but also to do so more quickly. And therein lies a significant part of the Q3 differences – differences which we will continue to see over time as our approach gets better and better. Remember, we are only seven months old at this point!
With some of the underlying trends and our capabilities understood, it’s also important to make a few obvious points about the limitations of Dow Jones VentureSource’s surveying method:
- Sampling – Who is included? How complete is the surveyed universe? Is there a bias in the sample? The truth is that Dow Jones cannot possibly sample every single member of the venture capital and entrepreneur population. The ChubbyBrain method offers access to more data points with less screening bias. Surveying is used as a means to complement data & research – not as our primary means.
- Cooperation / Response Rates – What percent of the surveyed population cooperates? Some may refuse, others may forget. Many don’t find the time to send in the questionnaire by due date. Venture capitalists are busy people! Is there a natural bias in those who decided to share information? (We’ve actually noticed some interesting trends about likely survey response rates based on size of firm.) Our method offers less dependency on respondent cooperation and as a result, we capture more funding events.
- False responses / distortion – Can this really be avoided? Either intentional or unintentional distortion can occur, not to mention manual data entry errors. With the ChubbyBrain method, everything is verifiable through information sources we employ. These data points are then confirmed with the investor or company as required. This solves for the intentional / unintentional distortion issue as well as data entry issue.
- Time / Timing – Data preparation takes time (logging the data in; checking for accuracy; taxonomizing and categorizing) which drives response deadline requirements and leads to reduced response rates. Beyond non-responses, late responses jeopardize accuracy. Our method offers faster, more efficient, real-time data aggregation and entry. As a result, our team of three FTEs is able to deliver insights not just faster, but with increased rigor and analytics (see our 44 page Fast Company report here) than the team of 50+ researchers at VentureSource.
In our first seven months of being in public beta, better technology and processes have helped us develop the ability to deliver more comprehensive, timely data about venture capital in a more efficient manner. And we plan to only get better. The Q3 2009 venture capital activity numbers tell a story, but the larger story behind these venture capital numbers is one of new vs. old models in information and data gathering.
More to come…
photo credit: Flickr