The Future Enterprise: Data-Driven Decision Making

August 21, 2018 | Debiprasad Banerjee

The enterprise of the future will thrive on data-driven decision making. You have probably heard and reiterated everywhere by futurists, industry analysts, and data science professionals. Every company generates and owns vast amounts of data, even more so with the digital revolution sweeping the globe. What better opportunity than to put all this data to good use by deploying Artificial Intelligence (AI) based systems that can derive deep insights from all this data and provide necessary inputs to make winning business decisions? Sounds fantastic!

While that could be largely true, what gets lost in the fine print is the journey that one needs to undertake from being an enterprise that owns (or has access to) a lot of data to a successful company that uses this data to make business decisions that impact actual business metrics – like those related to revenue (top line, bottom line or both), customers (acquisition, retention, etc.), organizational (operations, HR, etc.), or others.

Data – The Lifeblood of Organizations

What makes this different and more interesting than other transformational journeys enterprises undertake is that here ‘Data’ takes center stage. Earlier, the data was used only as an input when the business logic was independent of the data. Later, the data is codified separately into the application, crunched, and generated outputs that businesses could consume. Now, the data is the input, and the business logic – if one can even call it that – is hidden at some place within the data as invisible patterns to be revealed using complex AI algorithms. The output is more data that businesses can consume only after further post-processing and delivery in human-friendly formats. So, the entire process is driven by data, with the AI algorithm sitting in between as a black box. It is the new paradigm of AI-driven applications and data-driven enterprises.

Data Cleansing and AI Training

What immediately follows from this is that the data becomes the lifeblood of these organizations and thus requires a lot of attention. One of the main aspects of data is its quality. And that can mean a lot, but broadly measures how readily usable the data is. The essential thing to remember is that until now, data quality has been defined and measured by its ability to be consumed by humans or existing (non-AI) systems. It was therefore, captured, prepared, and stored accordingly, often with additional metadata and other forms of derived data. What happens when we feed this existing data, as is, to the AI systems? To understand this, we need to take a quick look under the hood of AI applications. Before an AI application is put into production and starts working with actual data, it should ‘learn’ how to do so. This activity is carried out in the lab and is called ‘Training’. During the training phase, we should feed the AI applications the required data in a specific form with fields and values in a particular format and order. It should be devoid of additional information that is not essential for the learning process. Without these constraints, the AI system will ‘learn’ incorrectly and derive incorrect or inconsistent meanings from the input data. Picking the correct AI algorithm for the given data set and curing the data to choose the appropriate fields, their types, forms, frequency, etc., becomes essential to getting the desired business output. It is as much an art as science and primarily what data scientists do. Consequently, once these systems go live and start working with the actual data, we must put in place a pre-processing mechanism to cleanse and prep the data for consumption by the AI system to keep getting correct and consistent outputs.

There is considerable work to be done in this step, and according to some estimates, almost 60% of the work that data scientists do is related to cleansing and prepping the data. The complexity and scale of this task are often not understood and almost always underestimated when allocating time, budget, and resources to the enterprises.


In the following discussion, we will take a closer look at some of these challenges in data preparation, how they impact the adoption of AI, and some thoughts on how enterprises should approach this subject.

If you are working in this area and have encountered things discussed here, please share your experience in the comments section.

Related Articles

Want to explore all the ways you can start, run & grow your business?