From DevOps to MLOps: Building End-to-End Pipeline for Fast, Simple & Reliable Deployment of ML in Retail
While most companies are now familiar with the requirements of a good DevOps process, the situation is still not on the same level of maturity for end-to-end MLOps processes that are required to train, test, deploy, run and monitor ML models in production.
Generally, MLOps can be considered as an extension of DevOps. The same principles that are valid for professional software development, also apply in the context of ML development, but the complexity is much higher.
For complete reproducibility of the status of an ML product, at least two artefacts are needed in MLOps (versioned code & model), and ideally versioned data as a third artefact, while in DevOps only one is required (versioned code).
Additionally, model training and optimization needs special tools for experiment tracking. And finally, the behaviour of productive models needs to be monitored with new concepts and tools, in contrast to traditional application monitoring.
At BSH we designed a concept for a generic, modular end-to-end pipeline that provides components for all the necessary steps from initial data preparation, over training, experiment tracking, testing, deployment, running and monitoring in production.
About: After receiving his Degree In Mathematics and Computer Science with focus on Machine Learning and AI, Eric worked as a Senior Data Scientist, ML Engineer and Consultant for several years. He designed concepts to automate the development of AI products by creating End-to-End MLOps Pipelines, as well as strategies to help companies to become data- and AI-driven. Before that, he worked as a Software Engineer for over a decade and later as a Lead Developer on many projects.
Eric is currently a Lead Architect & Advisor for AI & Big Data infrastructure, helping BSH to shape a strategy to become a data and AI driven company. His responsibilities also encompass designing Data Lakes of BSH, with his main focus on concepts for automating data ingestion, ETL, data quality and productionization of ML/AI processes (MLOps) on AWS.
He is the author of the upcoming book “AI Business Transformation” https://book.ejl.ai/ , which will be published by the end of the year. LinkedIn: https://www.linkedin.com/in/achimliese/