Ascent Index: Big Data

Big data is a hot topic, especially here in Massachusetts – arguably the Big Data capital of the world with 12,000 people employed in big data jobs in more than 100 companies across the state, an unmatched talent pool from top universities, and home to EMC Corp.

While data analytics has been around for many years, recently the term ‘Big Data’ has exploded. Since our April launch of the Ascent Enterprise IT Index, which measures social buzz around trend-setting topics such as virtualization, BYOD, cloud security and big data, we found big data to be the consistent winner with the most mentions – nearly 250,000.

But within those conversations, what’s interesting is that there has been a noticeable decline in the talk about extracting data and an increase in chatter about the scalability of it.

Of note, a Wall Street Journal piece on the recent buyout of Digg, one of the first social aggregation sites, explained that four years after being founded in 2004, Digg was valued at more than $160 million, but was sold last week for a mere $500,000. Why? For one thing, Digg was not able to effectively house the data entering their system every day.

During their re-launch in 2010, Digg realized they had problems with the scalability of their existing MySQL database software, and started a shift over to another open source system, Cassandra. But the new Digg launched before the new database was effectively installed, and their team was unable to fix the problems. This turned out to be a killer.

The data explosion of recent years has forced the hands of organizations everywhere, not just Digg. The ability to scale your internal database is no longer a nice-to-have; it’s a necessity. The companies that can store and manage vast amounts of data seamlessly will have a major advantage; those that do not, will struggle and may even perish.