What Could You Accomplish With 1,000 Computers?

October 17, 2012 Editor 0

An interview with Frederic Lalonde and Chris Lynch, serial entrepreneurs and founders of Hack/Reduce,”Boston’s Big Data Hacker Space.” Fred Lalonde is founder and CEO of travel start-up Hopper and former vice president at Expedia. Chris Lynch was most recently SVP & GM of HP’s Data Analytics Business Unit.

HBR: You recently launched Hack/Reduce — a sort of Big Data playground — in collaboration with MIT, Harvard, and other Boston-area universities. What was the inspiration behind it?

Fred Lalonde: We started Hack/Reduce in response to two big barriers to using Big Data. One is scalability. You can’t do Big Data on your laptop. You need 1,000 computers. The second is finding people who have the training, knowledge and expertise to work with Big Data. Even in a city like Boston, with the depth of software talent, there was literally nowhere where people with an idea could boot up enough computing power to try it.

We started with a one-day hack-a-thon and invited people to come over. The premise was you say to geeks: “We’re booting up all the computing power you need to come and do whatever you want.”

HBR: What types of experiments have people tried?

FL: We had people working on genomic sequencing to find cancer markers. People were analyzing real-time traffic patterns. A group of people determined who the friends of U.S. Congressmen and Congresswomen are by analyzing Twitter graphs. One group looked at best places to pick up a bikeshare in Montreal. (Hint: Nobody returns bikes to the stations at the tops of hills.) It’s amazing what people will do in a day with computing power and data sets. We’ve seen amazing things happen in five hours. Imagine what could happen if these people had five months.

HBR: Hack/Reduce started out as a side project for you, but it sounds like it has real legs?

FL: It’s now a full-blown 503c, established in partnership with the State of Massachusetts.

HBR: You’re also launching a travel company, Hopper.com, that’s being built from the ground-up on Big Data. Tell us about the opportunity you saw in the travel space.

FL: All of the existing travel sites work well if you know what you’re looking for — a Chinese restaurant near a train station, or a hotel — but planning a trip is an aspirational process, and there’s a whole element of discovery that’s not covered anywhere.

This month, we’ll crawl a billion pages about travel using our own hardware and our own data center. We’re parsing this billion+ sea of travel information and creating structure out of it. To put it in context, there are about 2 million travel-related blog posts created every day. There are about 56 million blogs on WordPress right now and travel is the fifth most often used tag — it’s bigger than fashion and sports. This doesn’t even include social networking posts. There is more super-interesting, high-quality, first-hand user experience information outside of the travel sites than there is inside travel sites. But, when we plan a trip today, the search engines are always returning the same user-review sites. The size of the data being generated in the wild is immense and it’s just sitting out there. Part of what we’re doing is trying to understand the blogosphere as a whole, finding the relevant travel data, and adding it to a catalog to make it consumable.

When we started looking at the problem, we realized that the travel catalog is broken. To use a music analogy, imagine that you didn’t have a unique catalog: you heard a song, but didn’t know the title, and maybe you could find the artist online. If there was no structured data set for music, you could still download an MP3, but value-added services like Pandora and Spotify couldn’t exist. If you look at travel, it’s an equally large data set — and I’m not talking about flights and hotels, because no one travels for the flight and hotel. We travel for the destination, the experience. Imagine a complete database of every experience on earth, and every experience that’s been had in that place — that’s the type of travel experience you would have with Big Data, and that’s what we set out to build with Hopper.

HBR: How do you know which data to trust?

FL: The ability to do statistical analysis — the patterns that occur at the scale of Big Data — is where the truth lies. We’re taking a holistic view that only a group of computers can do.

HBR: What type of employee thrives in a company that’s founded on Big Data?

FL: Almost everyone in our company has written code at some point. More generally, Big Data requires people who like complexity and solving big problems.

HBR: What’s the potential of Big Data from your point of view?

Chris Lynch: The potential is as disruptive as the internet and mobile technologies have been. Big data is changing curriculums at universities, creating new roles and professions (e.g. Data Scientist), and highlighting the fact that we are all content creators. The availability of the data and access to tools (in many ways thanks to the open source community) has never been greater; I anticipate the role big data analytics plays within organizations to increase dramatically over the next several years. And while Big Data may be overhyped, I do think it’s going to solve a lot of the world’s problems.

HBR: What’s your advice to HBR readers who are just beginning to think about the potential of Big Data in their own companies?

FL: If you’re an exec thinking about Big Data, it probably feels vague and confusing. You need to get your product people to sit down with the people who are solving data problems, and you need to start working with your own data. It’s shocking what most companies are sitting on right now. The established companies are going to fall into two categories — the ones who are leveraging data, and the ones who aren’t. And the upside will be staggering.

_____________________

 

Go to Source

Categories: HBR, Insights