Real-Time Analytics in the Cloud

Being data-driven is no longer a nice-to-have characteristic, but a necessary one. If you don’t use data to drive your decision-making in today's competitive world, you are bound to fall behind your competitors. Real-time analytics gets you the information you need as soon as you need it. But what does “real-time” really mean? Does it mean being able to visualize data instantly? Or is it a relative term?  

The biggest benefits of being data-driven

At the start of the discussion, attendees were asked about the most significant benefits of being data-driven. A CISO started off by saying that intelligence-driven cybersecurity helps them in the early detection of anomalies, which is a crucial aspect of their cyber-defense strategy. An analyst added that data analytics could enable you to make informed decisions and benchmark outcomes based on predictions if you have solid numbers. A director of product management considered being able to analyze customer behavior and preferences as the most significant benefit of being data-driven. A CTO shared that data engineering is all about empowering people with the right information and identifying avenues of improvement. Lastly, an executive from the education industry told the audience that data helps them predict which students are likelier to succeed in their school.

What (really) is real-time?

When talking about real-time analytics, a speaker said that real-time is not about how fast you get but about how fast you get good, insightful data. Real-time can mean days, hours, minutes, seconds, or even microseconds for different use-cases. It depends on the SLA that gives you the experience that you need. For example, if you implement context switching in an operating system, real-time could mean nanoseconds. But if you are processing historical data to create your marketing strategy for the next month, real-time could mean hours and not weeks.

The evolution of data analytics

A participant remarked that in the past, we would sometimes have to wait 24-48 hours for SQL queries to be complete before a management decision could be made. Today, we don’t have these constraints because of cloud and communication services. Real-time analytics empowers us to present decisions to management in real-time “within the order of human perception.” They further added that emerging use-cases of real-time analytics are in autonomous vehicles and vehicle safety. Real-time camera feed gets sent to edge computing environments where artificial intelligence is used to detect anomalies in real-time and alert the user before the incident can occur.  

Deciding which data you really need

With data coming in from virtually everywhere, how do you decide what to keep and discard? An executive posited that if you start to gather everything, you will overwhelm yourself and your systems. Everyone else agreed. To decide which data to store, you need to contextualize it. What does this data really represent? Does it deliver any meaningful metrics that can help me make better business decisions? If you start asking intelligent questions before loading and processing data, you will have much more refined and well-formatted datasets.

Known knowns, known unknowns, unknown knowns, and unknown unknowns…

An attendee talked about the importance of understanding how much you trust and understand various kinds of data. The known knowns are your business KPIs. You already know everything about them, and can use them to determine business health and generate actionable insights. The unknown knowns represent the use-cases for which you know the questions, but don’t have the data and/or the analysis. Known unknowns are when you have heaps of data but are still looking to find patterns or insights. The unknown unknowns are datasets that you don’t know exist.

To make data make insights evident to you, you must get it in front of the right people at the right time. You must ask the right questions like, “Do I have the right data?” “Is it in the format I want it to be?” “What exactly am I looking for?” Generating insights from data takes effort, patience, and creativity.

Building a data culture

Multiple contributors agreed that building a data culture in an organization is pivotal to becoming truly data-driven. However, not all employees are receptive to change. One contributor said that some people in their workforce do not want to “change the way they have always dealt with data.” There is usually a bell-curve distribution where some employees are super excited to explore the new tools, some “laggards” never adapt and resist change, and the broad middle represents the rest who do not really care.  

One way to get people interested is by showing them the true potential of modern data analytics and how it can improve their lives. Getting the higher management onboard early can also help drive change and introduce a cultural shift.

‍