Need speed for big data? Think in-memory data management
January 27, 2013 Editor 0
In a forthcoming report we will highlight technologies and solutions that take advantage of the decline in prices of RAM, the popularity of distributed and cloud computing systems, and the need for faster queries on large, distributed data stores. Established technology companies have had interesting offerings, but what initially caught our attention were open source projects that started gaining traction last year.
An example we frequently hear about is the demand for tools that support interactive query performance. Faster query response times translate to more engaged and productive analysts, and real-time reports. Over the past two years several in-memory solutions emerged to deliver 5X-100X faster response times. A recent paper from Microsoft Research noted that even in this era of big data and Hadoop, many MapReduce jobs fit in the memory of a single server. To scale to extremely large datasets several new systems use a combination of distributed computing (in-memory grids), compression, and (columnar) storage technologies.
Another interesting aspect of in-memory technologies is that they seem to be everywhere these days. We’re looking at tools aimed at analysts (Tableau, Qlikview, Tibco Spotfire, Platfora), databases that target specific workloads or data types (VoltDB, SAP HANA, Hekaton, Redis, Druid, Kognitio, and Yarcdata), frameworks for analytics (Spark/Shark, GraphLab, GridGain, Asterix/Hyracks), and the data center (RAMCloud, memory Iocality).
We’ll be talking to companies and hackers to get a sense of how in-memory solutions fit into their planning. Along these lines, we would love to hear what you think about the rise of these technologies, as well as applications, companies and projects we should look at. Feel free to reach out to us on Twitter (Ben is @bigdata and Roger is @rogerm) or leave a comment on this post.
- First measurements taken of South Africa’s Iron Age magnetic field history
- Intuition and Deliberation for Better Decision-Making
- Predicting loan repayment default among second tier borrowers in Ghana
- Open Innovation Key for Modern Local Authorities
- Tackling youth unemployment – whose job is it?
- Boost Your Innovation Confidence
Subscribe to our stories
- The Strategic Role of Design in Driving Digital Innovation June 10, 2021
- Correction to: Hybrid mosquitoes? Evidence from rural Tanzania on how local communities conceptualize and respond to modified mosquitoes as a tool for malaria control June 10, 2021
- BRIEF FOCUS: Optimal spacing for groundnuts in smallholder farming systems June 9, 2021
- COVID-19 pandemic: impacts on the achievements of Sustainable Development Goals in Africa June 9, 2021
- Explicit knowledge networks and their relationship with productivity in SMEs May 30, 2021