Architecting Data-Intensive Applications

In today’s fast-paced world, using data to your advantage is no longer a nice-to-have but a necessity. If you don’t collect and process data, you will not know where you are lacking or how your competition is leaving you in the dust. Being data-driven means learning from historical data, generating actionable insights from seemingly disparate information, and making calculated business decisions. But what does it take to become data-driven? What are the different stages along the journey, and how can you progress to the next one?

‍

Why should you take a data-driven approach?

During the discussion, attendees were asked about their organizational motivators to be more data-driven. A CIO started the conversation by saying that since they are going through a merger, it’s essential for them to have all the right customer data consolidated in one place. A senior executive of an architectural engineering firm added that data related to the usability of spaces, air quality, and occupancy helps them make better design decisions. The CISO of an educational institution told the audience how they use data to identify factors that can contribute to student success, failure, and happiness. Lastly, a CTO mentioned that they want to extract meaningful insights from heaps of unprocessed data, which could help them improve operational workflows.

‍

What does being data-driven mean?

A speaker exclaimed that “data-driven” could mean different things for different companies, depending on factors like: data size, volume, complexity, variability, sources, concurrency, and structure. Another differentiating factor can be the speed at which the consumers want the data to be processed. Some analysts may require insights to be generated in real-time, while others may be okay with waiting for a few days.

A participant remarked that if you don’t make data-driven decisions in today's competitive world, you put yourself at a disadvantage. Only if you aggregate and process data can you react as quickly as the market needs you to. Data analysis can help you determine customer sentiments about a product, generate competitor insights, adjust pricing, and run targeted marketing campaigns.

Multiple executives also commented on the use-case of data in the distribution/logistics industry. Data can let you know where a truck is, who is driving it, how much load it’s carrying, where it’s headed, what route it is taking (and can it be optimized), and if there’s going to be a delay. This use case is especially relevant if you have thousands of trucks in your fleet.

What is data-driven vs. insight-driven?

An attendee claimed that simply having data isn’t enough if you can’t generate actionable insights from it. Organizations should shift from being data-driven to being more insight-driven. You shouldn’t have to dig through heaps of data to figure out what you are looking for. Instead, the system should be able to automatically classify an insight as important and present it to you.

How does privacy intersect with being data-driven?

PII data must be stored and processed with care. It should be encrypted at rest and in transit. Data masking should be applied wherever required, and access policies must be compliant with the principle of least privilege. National laws and regulatory bodies may require you to follow specific guidelines and/or achieve compliance with some security and privacy frameworks.  Achieving all this often requires new tools and processes.

‍

What are the different stages of the data-driven journey?

An executive shared that organizations go through different stages in the data-driven journey. The first stage is the historical model, where they aggregate and process historical data to decide what to do in the future. The second phase involves real-time data analysis, enabling all employees of an organization to make informed decisions. The third phase is predictive analysis, which tells you what’s going on right now and empowers you to identify potential opportunities and avoid problems. The final phase involves harnessing artificial intelligence to directly tie the predictive analysis to the operational systems and take the human out of the equation.

‍

How does culture play a part in being data-driven?

Various attendees agreed that organizational culture can play a pivotal part in the journey to become data-driven. Different functions in an organization may have different skill-sets and different reactions to change. It may be hard to achieve a data-driven shift in an organization without exclusive support from higher management. However, if you can get some quick, early wins, it can help you win over people and find change advocates. Once the message and tangible benefits reach various parts of the organization, people and functions can become more receptive to change.