Deploying AI Applications

December 17, 2018 | Debiprasad Banerjee

Deep learning models are now ubiquitous and analyze massive amounts of audio, image, video, text, and graph data, with applications in all industry segments across the value chain. While significant work has been done on the training side to standardize and automate by using frameworks and tools, deployment of these models into production, especially at scale, remains largely bespoke and exceedingly effort intensive.

Three main components successfully deploy AI-powered business applications in an enterprise.

  1. Data Strategy – Includes the data collation and training of the AI model
  2. The AI algorithm itself
  3. The deployment of the AI model into production with live data

There is a fourth and final part, without which we cannot realize AI’s full benefit. That is the continuous learning loop of the AI application from live data to improve the accuracy/performance of the model. Ideally, the deployment platform and tools need to enable the prediction stream anomaly detection and provide data for the retraining of the model. However, as the scale and complexity of deployed models increase, this may have to be considered and planned as a separate workload. We will address this in a separate discussion in the future. For data strategy discussion, please see my previous posts on this topic.

Discussions about AI tend to focus on data (rightly so), feature engineering, algorithms, and model development. No doubt all that is necessary for successfully using AI is available; however, insufficient attention is paid to the equally important aspect of AI applications’ deployment to production. Managing AI applications in production is so essential that you can say the actual work only begins after you deployed the AI in production with live data! Welcome to the emerging field – ML Ops, AI Ops, or AI Deployments. Not to be confused with using AI/ML for improving IT Operations processes.

Setting up environments for AI pipelines and deploying them can be long and exhausting. Some common hurdles that we face in this process are:

  • Mapping an AI algorithm to the correct hardware for deployment needs thorough research and deep expertise.
  • The availability of multiple AI libraries/frameworks for similar tasks reduces community sharing and the reusability of models.
  • AI Frameworks focus primarily on building models, so testing and deployment strategies need to be planned separately and well in advance.
  • The choice of framework, dataset, and tools made during the planning and modeling stage directly impacts deployment, which cannot always be foreseen.

So, how do we tackle these problems and create a reliable, scalable, and repeatable process for deploying AI applications into production?

This simplest solution, and the one being used by most companies today, is to go with a hosted provider like AWS (there are several others with more customized PaaS offerings like or, where a significant part of the deployment process can be abstracted and a reasonably consistent user (deployment) experience is available for a large cross-section of load vs performance patterns. While there are trade-offs in areas like flexibility and optimization of the hardware specific to the AI models being used, the cost of running an enterprise-size load on the cloud may also become a deterrent to the adoption of AI within the enterprise.

For the rest, unfortunately, the path is still relatively uncharted. There are a few tools from the major players and some from smaller boutique outfits, but they all tend to be very specific to a particular set of framework-model-hardware combinations. Even while using such tools, notable number of bespoke customizations may be required, for which a pool of smart engineers needs to be readily available.

None of these problems are unsurmountable, and the situation is not as grim as it may sound. The tools landscape is evolving every day, and considerable work is being done by leaders like Google, Facebook, Microsoft, etc., and a group of brilliant start-ups. With awareness of the problem and proper planning at the initial phases (think of a process akin to ‘DevOps for AI’), AI applications can be deployed in production and scaled to meet the growing need for better and more reliable enterprise automation.

Related Articles

Want to explore all the ways you can start, run & grow your business?