Big Data and the Art of Failing Better

July 10, 2013 - by Rachel Balik

In the past, a leader was someone who could get you to do stuff in the absence of information. Now it’s the person who can ask the best question about what’s going on, and find an answer. - Alistair Croll, author of Lean Analytics.

The subject has been beaten to death and approached from myriad angles, but it’s still one we all (especially those of us in the startup community) love to contemplate: failure. More specifically, we think about the fact that anyone who’s really innovating should expect to fail multiple times along the way. Naturally, insofar as we’ve embraced failure as a necessary but sufficient element of success, we’re also putting some thought into how to fail well.

One of the ways to fail well, or more specifically, to “fail cheaper” is rigorous data analysis, said Quentin Hardy yesterday on the NY Times Bits blog. Expensive failure - the kind where your entire business goes under - still happens - but as our analytics get better, so does our ability to experiment, make informed decision and/or pivot. And it’s not just any analytics, it’s primarily big data analytics that can serve us particularly well. Of course, there are many definitions of big data, but at the heart of it is "diversity of information," Hardy says.

We read this post and immediately thought, “Manta.” (And not in the same way we’re thinking about Manta when we brush our teeth and tie our shoes.) Hardy notes that the cost of risk has decreased because so many other standard business costs have dropped over the past decade, however, one cost that hasn’t really dropped is that of running queries on data.

Until now.

A few weeks ago, we wrote that “the future of big data is asking dumb questions,” and made the case that not only in business, but also in data science itself, a solid track record of trial and error is critical to harnessing the power of data. The problem, of course, is that when every query is costly both in terms of time and money, most business decision makers will claim that in fact, there is such thing as a stupid question.

And it’s not just questions on a one-off basis that are deemed too costly. Bryan Cantrill pointed out that in the world of big data today, if you’re a small startup or team within an enterprise and decide to launch a data analysis project, you might spend the first three months simply setting up the technology necessary to ask questions in the first place. You need to make a very convincing case for why it’s a good use of money, and you need to boost morale and maintain momentum for three whole months before you can even deliver results.

Sounds like fun, right? Bryan and Mark Cavage didn’t think so either, which is why they took this IM conversation, added a few lines of code and produced Manta.* As Bryan says, Manta does for big data what Heroku did for web development. With Heroku, instead of spending months to launch a website, you can be up in a few days. Just as Heroku is a Platform-as-as-Service, Manta is Data-as-a-Service, meaning that anyone can make queries quickly and cheaply. It gives the freedom to access and analyze that “diversity of information.” The implications of that diversity are broad and multifaceted, but at a high level, the concept is pretty simple: More data at a low cost means more innovation and better business decisions. Hypergrowth requires experimentation, but experimentation isn’t all that valuable if it can’t be measured and internalized quickly.

And the potential of Data-as-a-Service doesn’t stop there. What if data scientists could securely gain access to the data of others, making that pool of information even more diverse? What if there was an easy and inexpensive way to learn not just from our own failures, but those of others? These are the kinds of questions we’re asking all the time here, and thanks to quite of bit of trial and error, we’re pretty close to an answer. We invite you to add some questions and answers of your own by giving Manta a try yourself.

*Not actually how Manta was built. For more on the technology behind Manta, see this, this and this.