Five Ways to Advance your Career with Modern Data Architecture

Guest post by Todd Christy, Slalom Consulting and member of Ascent’s Technical Advisory Board.

What do flat tires and Hadoop have in common? If you’re one of the 75 million drivers covered by the largest US roadside services provider you may experience the real value of “big data” when your tow truck arrives promptly to swap your tire or fill your tank. They are using a modern data architecture to select the highest-quality service providers from among thousands, and predict driver assistance needs based on a complex blend of traffic patterns, weather forecasts, and incident history. What if you’ve spent millions building a low-latency trading platform to power your asset management business, but your system monitoring tools detect problems at lethargic one-minute intervals? Minutes away when milliseconds count! One global investment firm built a real-time modern data architecture in the Amazon cloud, using a mix of open source and commercial tools like Apache HBase, Storm, Kafka, and Esper to provide sub-second system diagnostics and performance analysis.

Whether filling tanks or filling orders, a new breed of data processing, analysis, and visualization tools are powering today’s business innovators. Traditional relational databases (RDBMS) form the core of the vast majority of business systems, and have fueled the success of some of the world’s largest software companies. But IDC estimates that the “big data” market will reach $16B this year, and is growing six-times faster than the overall IT market. So what is “big data” and what makes your data architecture “modern”? Here are 3 ways to know if you’re leveraging a modern data architecture:

High-Value Data: The term “big data” implies large volumes of data. While high-volume is often the case, “high-value” is the critical measure of business impact. Modern data architectures employ advanced techniques to derive high business value from data that was previously ignored, under-utilized, or poorly understood. Predictive analytics, in-memory data processors, and data platforms that can analyze large volumes of complex structured and unstructured data each can unleash captive value within a company’s data reserves.

Real-Time Processing: To many businesses the challenges of processing data hourly or via nightly batch runs is daunting enough. Today’s modern data architecture demands more sophisticated, real-time results. Social networking, ultra-efficient pricing, real-time offer management, and a workforce and customer base enabled with powerful mobile apps — these represent just a few of the opportunities for innovation leaders to differentiate and create new demand. The critical need for real-time processing is exemplified by the “speed layer” defined by Twitter’s Nathan Marz in his Lambda Architecture design, and by the popularity of open source tools such as Apache Storm and Kafka, and commercial variants like AWS Kinesis.

Flexible and Interactive: Processing large volumes of complex data to distill valuable conclusions is part of the big data challenge, but delivering those results in a usable form to key decision makers is the last mile of the modern data architecture. Usability may come in the form of advanced analytics like predictive models, driving businesses toward more likely valuable outcomes and market segments. The market for rich, interactive data visualization tools has exploded in recent years, with innovators like Qlik, Tableau, and Good Data challenging slower-moving incumbents like IBM/Cognos and SAP/Business Objects. These tools allow end-users to interact with data in an intuitive way, producing charts and dashboards that direct business decision making. Users are further enabled by the late schema binding of tools like Splunk and Elasticsearch and powerful visualization tools like D3 and Kibana, fostering data experimentation that yields material business value.

“So I’m just an IT guy…what does all of this modern data stuff have to do with me?” Well, here are five suggestions that might just get you thinking about how these technologies might benefit your business and advance your career:

1. Mine some data: Identify business data that exists across a mix of data feeds and formats that has thus far been of little use; derive a new business insight by combining this data in unexpected ways
   Possible tools: Hadoop, MapR, AWS Elastic MapReduce
2. Analyze a social feed: Incorporate an active but little-used stream of social data that affects your business from among the many public conversations available — Facebook, Twitter, Yelp, Google Play, Apple App Store, Instagram, LinkedIn, StackOverflow, Quora, Livefyre, Disqus
   Possible tools: Google Analytics, Moz Analytics, Salesforce Marketing Cloud, Adobe Social
3. Introduce self-service data discovery: Today’s hottest data visualization & discovery tools often gain entry when individual business users install them to avoid onerous IT processes and delays; foster this culture of discovery by introducing modern data discovery tools that promote self service and end-user engagement
   Possible tools: Qlik, Tableau, Good Data, Splunk, Loggly
4. Predict something: Build a basic predictive model and introduce proactive decision making into a business area that has traditionally been reactive
   Possible tools: SAS, RapidMiner, Alteryx, MATLAB
5. Replace a batch process: Accelerate a critical business operation that has depended on batch processing by introducing real-time insight; enable business users to make same-day decisions where they were previously waiting days or weeks
   Possible tools: AWS Redshift or Kinesis, Apache Storm, Splunk

According to IBM, 90% of the world’s data was produced in the past two years. With no slowing in site, data architectures are evolving faster than ever before to keep pace. Opportunities abound for startups, acquisitive incumbents, service providers, and ambitious IT workers to capitalize on the demand for innovation that this growth creates. It will require modern data architectures to capture optimal value, and lead the innovation race.

Todd Christy is Managing Director of Technology for Slalom Consulting’s Boston office. He was the founding CTO of Pyxis Mobile, an enterprise mobility software company. toddc@slalom.com