15 - 16 June 2022

MLOps Summit MLOps Summit schedule

MLOps Summit San Francisco



Download PDF
  • 08:00

    Coffee & Registration

  • 09:00
    Korri Jones

    MLOps Stage: Chair Welcome

    Korri Jones - Senior Lead Machine Learning Engineer - Chick-fil-A

    Down arrow blue

    Korri Jones is a Sr Lead Machine Learning Engineer and Innovation Coach at Chick-fil-A, Inc. in Atlanta, Georgia where he is focused on MLOps. Prior to his work at Chick-fil-A, he worked as a Business Analyst and Product Trainer for NavMD, Inc., was an adjunct professor at Roane State Community College, and instructor for the Project GRAD summer program at Pellissippi State Community College and the University of Tennessee, Knoxville. His accolades are just as diverse, and he was in the inaugural 40 under 40 for the University of Tennessee in 2021, Volunteer of the year with the Urban League of Greater Atlanta with over 1000 hours in a single calendar year and has received the “Looking to the Future” award within his department at Chick-fil-A among many others, including best speaker awards in business case competitions.

    Twitter Linkedin
  • 09:15
    Adam Kraft

    Where Are We With AutoML?

    Adam Kraft - Machine Learning Engineer - Google Brain

    Down arrow blue

    Where are we With AutoML?

    AutoML aims to help everyone achieve state-of-the-art AI for their specific problems. Techniques such as Neural Architecture Search (NAS) push the boundary of finding efficient and high quality models. How is AutoML being used today and where is the field headed in the future? This talk gives an overview of AutoML techniques, exploring how they work and the challenges of applying them across different AI tasks and settings.

    Adam Kraft is a machine learning engineer on the Google Brain Team, working on AutoML for a wide variety of AI tasks. Before Google, Adam spent eight years in computer vision and machine learning, working with satellite imagery at Orbital Insight and helping customers shop with their camera phones at Amazon.

    Twitter Linkedin
  • 09:45
    Emily Curtin

    Making Code and Humans GPU-Capable at Mailchimp

    Emily Curtin - Senior Machine Learning Engineer - Mailchimp

    Down arrow blue

    Making Code and Humans GPU-Capable at Mailchimp

    What happens when you have a bunch of data scientists, a bunch of new and old projects, a big grab-bag of runtime environments, and you need to get all those humans and all that code access to GPUs? Come see how the ML Eng team at Mailchimp wrestled first with connecting abstract containerized processes to very-not-abstract hardware, then scaled that process across tons of humans and projects. We’ll talk through the technical how-to with Docker, Nvidia, and Kubernetes, but all good ML Engineers know that wrangling the tech is only half the battle and the human factors can be the trickiest part.

    3 Key Takeaways • An overview of the call stack from container, orchestration framework, OS, and all the way down to real GPU hardware • How ML Eng at Mailchimp provides GPU-compatible dev environments for many different projects and data scientists • An experienced take on how to balance data scientist’s human needs against heavy system optimization (spoiler alert: favor the humans)

    Emily May Curtin is a Senior Machine Learning Platform Engineer at Mailchimp, which is definitely what she thought she’d be doing back when she went to film school. She combines her wealth of experience in DevOps, data engineering, distributed systems, and “cloud stuff” to enable Data Scientists at Mailchimp to do their best work. Truthfully, she’d rather be at her easel painting hurricanes and UFOs. Emily lives (and paints) in her hometown of Atlanta, GA, the best city in the world, with her husband Ryan who’s a pretty cool guy.

    Twitter Linkedin
  • 10:15
    Raymond Hitti

    Sama Go: A New Self-Service Data Platform with Machine Assisted Annotation

    Raymond Hitti - Product Manager - Sama

    Down arrow blue

    Sama Go: A New Self-Service Data Platform with Machine Assisted Annotation

    When it comes to AI, your model is only as good as the data it’s trained on. This is why Sama is launching Sama Go, its first self-service Data Platform. Using Sama Go, your team will be able to easily annotate datasets and improve predictions to help you improve your model. Sama Go is tightly integrated with Sama’s workforce, allowing you to hire our team of experts with the click of a button. During this session, Raymond Hitti, Product Manager at Sama, will introduce Machine Assisted Annotation in Sama Go, our newest self-service data platform.

    Raymond Hitti, Product Manager at Sama, is in charge of Sama Go, a new Self Service Data Platform for Machine Learning (ML) practitioners. After his studies in mechanical engineering and computer science at the American University of Beirut, Raymond worked as a software engineer in multiple AI, Robotics and Property startups. Experienced in both product and engineering, Raymond is a proven leader who combines empathy and experience to build useful products. When not at work, you can usually find him racing triathlons or enjoying the Canadian nature with friends.

  • 10:40

    Morning Break

  • 11:00

    Models as Business Assets

  • Mac Macoy

    SPEAKER

    Mac Macoy - Senior Software Engineer - MLOps - Chick-fil-A

    Down arrow blue

    Models as Business Assets

    Linkedin
  • Jimmy Simmons

    SPEAKER

    Jimmy Simmons - Principal Team Leader, Machine Learning Operations and Big Data - Chick-fil-A

    Linkedin
  • 11:30
    Gideon Mendels

    ML System Design for Continuous Experimentation

    Gideon Mendels - CEO and Founder - Comet

    Down arrow blue

    ML System Design for Continuous Experimentation

    While ML model development is a challenging process, the management of these models becomes even more complex once they're in production. Shifting data distributions, upstream pipeline failures, and model predictions that affect the very datasets they’re trained on can create thorny feedback loops between development and production.

    In this talk, Gideon will: • Examine naive ML workflows that don’t take the development-production feedback loop into account and explore why they break down • Showcase system design principles that will help manage these feedback loops more effectively • Share industry case studies where teams have applied these principles to their production ML systems

    Gideon Mendels is CEO and founder of Comet, an ML platform provider. He led a team that trained and deployed more than 50 NLP models in 15 languages as founder of GroupWize. He also worked on hate-speech and deception detection at Google, and he trained and put into production deep learning classifiers for 500 languages at Columbia University jointly with IBM Research.

    Linkedin
  • 12:00

    Panel Discussion: Rolling Out the Practice of MLOps Organizationally

  • Lakshmi Ravi

    MODERATOR

    Lakshmi Ravi - Applied Scientist - Amazon

    Down arrow blue

    Selecting ML Algorithms and Validating

    ML Practitioners often have a dilemma in identifying the right ML Model for the problem space. In this talk, I will be going over the common questions that will help in narrowing down the right next step. Developed model will have to meet certain validation metrics. The next common question is how the validation metrics proposed by scientists will have to be explained to business leaders and help them decide if the model is eligible to deployed. The next step is to find mechanisms to develop and study the online validation metrics. Often online metrics of an ML model launched will require studying the results in Treatment-Control fashion. In this talk, I will describe common development practices that helps in A/B testing of experiments.

    Lakshmi is an Applied Scientist with Amazon.She has been working with Amazon Machine Learning teams for the last 4.5 years. She had the chance to be part of Alexa's NLP team, Behavior Analytics (a causal Inference division in Amazon) and Amazon Music teams (improving the voice experience in Alexa).

    Linkedin
  • Geetha Ganesan

    PANELIST

    Geetha Ganesan - Software Development Manager - Amazon

    Linkedin
  • Korri Jones

    PANELIST

    Korri Jones - Senior Lead Machine Learning Engineer - Chick-fil-A

    Down arrow blue

    Korri Jones is a Sr Lead Machine Learning Engineer and Innovation Coach at Chick-fil-A, Inc. in Atlanta, Georgia where he is focused on MLOps. Prior to his work at Chick-fil-A, he worked as a Business Analyst and Product Trainer for NavMD, Inc., was an adjunct professor at Roane State Community College, and instructor for the Project GRAD summer program at Pellissippi State Community College and the University of Tennessee, Knoxville. His accolades are just as diverse, and he was in the inaugural 40 under 40 for the University of Tennessee in 2021, Volunteer of the year with the Urban League of Greater Atlanta with over 1000 hours in a single calendar year and has received the “Looking to the Future” award within his department at Chick-fil-A among many others, including best speaker awards in business case competitions.

    Twitter Linkedin
  • Supreet Kaur

    PANELIST

    Supreet Kaur - Assistant Vice President - Morgan Stanley

    Down arrow blue

    MLOps/Trusted AI Summit: Closing General Session: Complexity vs Simplicity in ML and AI Projects

    Women in AI Reception: Pivoting into AI

    Supreet is an AVP at Morgan Stanley. Prior to Morgan Stanley, she was a management consultant at ZS Associates where she automated different workflows and built data driven solutions for fortune 500 clients. She is extremely passionate about technology and AI and hence started her own community called DataBuzz where she engages the audience by sharing the latest AI and Tech trends and also mentors people who want to pivot in this field.

  • 12:45

    Lunch

  • 13:45
    Johnson D'Souza

    ML Systems and Production Challenges

    Johnson D'Souza - Sr. Engineering Manager, Machine Learning - Course Hero

    Down arrow blue

    ML Systems and Production Challenges

    Linkedin
  • 14:20

    Round Table Discussions

  • Shilpi Agarwal

    Round Table Topic Leader: Data Ethics in Business - The Cornerstone of Customer Trust

    Shilpi Agarwal - Founder & Chief Data Ethics Officer - DataEthics4All

    Down arrow blue

    Reducing AI’s BMI

    Do you know what is AI’s BMI today? You guessed it. It's really high!! So, how do we fix this? Do you think Big Tech needs to go on a DIET? What diet would that be? Keto? Paleo? Mediterranean or Vegetarian? Come, Join us for this fun talk and learn the right diet for AI Companies.

    Shilpi Agarwal is a Data Philanthropist, Adjunct Faculty at Stanford and MIT $100K Launch Mentor.

    Armed with the technical skills from her Bachelor of Engineering in Computer Science, design thinking skills from her Masters in Design, combined with 20+ years of Business and Marketing know-how by working as a Marketing Consultant for some really big and some small brands, Shilpi started DataEthics4All, troubled with the unethical use of data around her on social media, in business and in political campaigns.

    DataEthics4All is a Community bringing the STEAM in AIᵀᴹ Movement for Youth and celebrating Ethics 1stᵀᴹ Champions of today and tomorrow pledging to help 5 Million economically disadvantaged students in the next 5 years by breaking barriers of entry in tech and creating awareness on the ethical use of data in data science and artificial intelligence in enterprise, working towards a better Data and AI World.

    Twitter Linkedin
  • Ban Kawas

    Round Table Topic Leader: Explainable AI (XAI) and Its Role in Building Trusted AI

    Ban Kawas - Senior Research Scientist - Reinforcement Learning - Meta

    Down arrow blue

    Ban is a Senior AI Research Scientist at Meta. She is working on democratizing Reinforcement Learning and enabling its use in the real world, spanning several application areas from compiler optimization to embodied AI. Ban and her team are developing ReAgent; an end-to-end platform for applied RL, checkout open source version at https://reagent.ai/

    Linkedin
  • Naman Kohli

    Round Table Topic Leader: Causal Analysis

    Naman Kohli - Applied Scientist - Amazon

    Linkedin
  • 15:15

    Afternoon Networking Break

  • 15:45
    Frankie Cancino

    Ask Me Anything: Operationalizing MLOps

    Frankie Cancino - Data Scientist - Mercedes-Benz Research & Development

    Down arrow blue

    Frankie Cancino is a Data Scientist at Mercedes-Benz Research & Development, working on applied machine learning initiatives. Prior to joining Mercedes-Benz R&D, Frankie was a Senior AI Scientist at Target AI, focused on methods to improve demand forecasting and anomaly detection. He is also the organizer and founder of Data Science Minneapolis. Data Science Minneapolis is a community that brings together professionals, researchers, data scientists, and AI enthusiasts.

    Linkedin
  • 16:15
    Chris Van Pelt

    Strategic MLOps: Best Practices of Effective ML Teams

    Chris Van Pelt - Co-Founder - Weights & Biases

    Down arrow blue

    Strategic MLOps: Best Practices of Effective ML Teams

    What does it take to get the best model into production? We've seen industry-leading ML teams follow some of the same common workflows for dataset management, experimentation, and model management. I'll share case studies from customers across industries, outline best practices, and dive into tools and solutions for common pain points.

    Chris Van Pelt is a co-founder of Weights & Biases, a developer MLOps platform. In 2009, Chris founded Figure Eight/CrowdFlower. Over the past 10 years, Chris has dedicated his career optimizing ML workflows and teaching ML practitioners, making machine learning more accessible to all. Chris has worked as a studio artist, computer scientist, and web engineer. He studied both art and computer science at Hope College.

    Linkedin
  • 16:40
    David Liu

    Taming Signals, Features, and Training Datasets: How Data Management Shaped Pinterest ML

    David Liu - Head of ML Platform - Pinterest

    Down arrow blue

    Taming Signals, Features, and Training Datasets: How data management shaped Pinterest ML

    This talk examines how Pinterest evolved its handling of three types of data -- raw signals, ML features and ML training datasets -- and the effects on ML practitioners at Pinterest. Data management is the core complexity of production ML engineering, especially in Web-scale applications with billions of entities and training examples. Pinterest “signals,” or raw data about Pins, boards, and other entities, started as monolithic datasets that grew unmaintainable. We split them into individually owned datasets on a standardized “Signal Platform,” improving governance around lineage, ownership, and monitoring. We standardized ML features from highly custom formats to a flat “Unified Feature Representation,” enabling a shared feature store and model inference. Finally, we are transitioning ML training datasets from ad-hoc row-oriented datasets to standardized columnar table groups, enabling improved storage efficiency and shared training pipelines in the future.

    David is the Head of ML Platform at Pinterest, which comprises ML Data, ML Training, and ML Serving teams. These teams provide infrastructure for 200 engineers and data scientists for applications spanning ads, recommendations, search, and trust/safety, handling billions of events per day. Previously at Pinterest, David also started the Related Pins recommendations and visual search teams and built one of the first ML-based recommender systems at Pinterest. He holds a bachelor's and master's degree in computer science from Stanford.

    Linkedin
  • 17:10
    Korri Jones

    MLOps Closing Chair Remarks

    Korri Jones - Senior Lead Machine Learning Engineer - Chick-fil-A

    Down arrow blue

    Korri Jones is a Sr Lead Machine Learning Engineer and Innovation Coach at Chick-fil-A, Inc. in Atlanta, Georgia where he is focused on MLOps. Prior to his work at Chick-fil-A, he worked as a Business Analyst and Product Trainer for NavMD, Inc., was an adjunct professor at Roane State Community College, and instructor for the Project GRAD summer program at Pellissippi State Community College and the University of Tennessee, Knoxville. His accolades are just as diverse, and he was in the inaugural 40 under 40 for the University of Tennessee in 2021, Volunteer of the year with the Urban League of Greater Atlanta with over 1000 hours in a single calendar year and has received the “Looking to the Future” award within his department at Chick-fil-A among many others, including best speaker awards in business case competitions.

    Twitter Linkedin
  • 17:15

    Networking Reception

  • 18:15

    End of Day One

  • THIS EVENT STARTS AT 8:45

  • 08:45

    Coffee & Registration

  • 09:45
    Korri Jones

    MLOps Stage: Chair Welcome

    Korri Jones - Senior Lead Machine Learning Engineer - Chick-fil-A

    Down arrow blue

    Korri Jones is a Sr Lead Machine Learning Engineer and Innovation Coach at Chick-fil-A, Inc. in Atlanta, Georgia where he is focused on MLOps. Prior to his work at Chick-fil-A, he worked as a Business Analyst and Product Trainer for NavMD, Inc., was an adjunct professor at Roane State Community College, and instructor for the Project GRAD summer program at Pellissippi State Community College and the University of Tennessee, Knoxville. His accolades are just as diverse, and he was in the inaugural 40 under 40 for the University of Tennessee in 2021, Volunteer of the year with the Urban League of Greater Atlanta with over 1000 hours in a single calendar year and has received the “Looking to the Future” award within his department at Chick-fil-A among many others, including best speaker awards in business case competitions.

    Twitter Linkedin
  • 10:00
    Zachary Hanif

    Beyond MLOps: A Closer Look at Operational Machine Learning & Why it Matters

    Zachary Hanif - Vice President of Machine Learning - Capital One

    Down arrow blue

    Beyond MLOps: A Closer Look at Operational Machine Learning & Why it Matters

    While critical aspects of MLOps like infrastructure, CI/CD pipelines, logging and monitoring are much discussed in the machine learning world today, there has been less of a focus on the actual operations of machine learning. This talk will explore key topics and questions related to the operational considerations of developing and deploying machine learning models in real-world environments, including: how to respond when something goes awry; data-centric approaches to model development and how to course-correct for suboptimal training data; and what happens when novel external environments (e.g. a global pandemic) create outputs that run counter to developers’ expectations.

    Zachary Hanif is Capital One’s Vice President of Machine Learning, currently working to democratize and de-risk access to modeling capabilities across a data-driven enterprise. He has a career background focused on applications of machine learning and advanced modeling innovation, research, and product delivery across diverse problem domains in startups and large enterprises. Zachary is passionate about identifying novel uses of machine learning and large-scale distributed systems technologies that generate measurable value, pragmatic innovation, and ensuring that technical solutions, particularly those leveraging AI/ML, are responsible, ethically sound, and robust to challenge.

    Twitter Linkedin
  • 10:30
    Ville Tuulos

    Metaflow, A Unique ML Infrastructure Stack That is Human Centric

    Ville Tuulos - Co-founder & CEO - Outerbounds

    Down arrow blue

    Metaflow, A Unique ML Infrastructure Stack That is Human Centric

    Metaflow was born at Netflix, and grew into a widely-used open-source framework that helps data scientists and engineers develop and deliver real-life ML Projects. Metaflow is designed from the ground up to be accessible to data scientists while enabling observability, reproducibility, collaboration, and scalability. Outerbounds, the company that is shepherding the future of Metaflow, provides tools and resources that allow companies to be effective with Metaflow. In this talk, we will introduce the audience to Metaflow and discuss what makes it unique.

    Ville Tuulos has been developing infrastructure for machine learning for over two decades. He has worked as an ML researcher in academia and as a leader at a number of companies, including Netflix where he led the ML infrastructure team that created Metaflow, a popular open-source framework for data science infrastructure. He is the co-founder and CEO of Outerbounds, a company developing modern human-centric ML. He is also the author of an upcoming book, Effective Data Science Infrastructure, published by Manning.

    Linkedin
  • 11:00
    Lakshmi Ravi

    Selecting ML Algorithms and Validating

    Lakshmi Ravi - Applied Scientist - Amazon

    Down arrow blue

    Selecting ML Algorithms and Validating

    ML Practitioners often have a dilemma in identifying the right ML Model for the problem space. In this talk, I will be going over the common questions that will help in narrowing down the right next step. Developed model will have to meet certain validation metrics. The next common question is how the validation metrics proposed by scientists will have to be explained to business leaders and help them decide if the model is eligible to deployed. The next step is to find mechanisms to develop and study the online validation metrics. Often online metrics of an ML model launched will require studying the results in Treatment-Control fashion. In this talk, I will describe common development practices that helps in A/B testing of experiments.

    Lakshmi is an Applied Scientist with Amazon.She has been working with Amazon Machine Learning teams for the last 4.5 years. She had the chance to be part of Alexa's NLP team, Behavior Analytics (a causal Inference division in Amazon) and Amazon Music teams (improving the voice experience in Alexa).

    Linkedin
  • 11:30

    Break

  • 11:45
    Hiranmayi Ranganathan

    Data-Driven Modeling Approaches in Computational Drug Discovery

    Hiranmayi Ranganathan - Machine Learning Specialist - Accelerating Therapeutics for Opportunities in Medicine (ATOM)

    Down arrow blue

    Data-Driven Modeling Approaches in Computational Drug Discovery

    This session will introduce ATOM ( Accelerated Therapeutics for Opportunities in Medicine) and the work going on in the consortium. We will then talk about the issues around building predictive models for small molecule drug discovery, focusing on both target specific drug discovery approaches and structure based multi-target modeling.

    Some topics covered include: - Drug Discovery Cheminformatics Models - Introduction to Quantitative Structure Activity Relationships (QSAR) - ATOM Modeling Pipeline - Structural models

    Linkedin
  • 12:15
    Diego Klabjan

    MLOps for Deep Learning

    Diego Klabjan - Professor - Northwestern University

    Down arrow blue

    MLOps for Deep Learning

    In model serving, two important decisions are when to retrain the model and how to efficiently retrain it. Having one fixed model during the entire often life-long inference process is usually detrimental to model performance, as data distribution evolves over time, resulting in a lack of reliability of the model trained on historical data. It is important to detect drift and retrain the model in time. We present an ensemble drift detection technique utilizing three different signals to capture data and concept drifts. In a practical scenario, ground truth labels of samples are received after a lag in time, which we consider appropriate. Our framework automatically decides what data to use to retrain based on the signals. It also triggers a warning indicating a likelihood of drift.

    Model training in serving is not a one-time task but an incremental learning process. We address two challenges of life-long retraining: catastrophic forgetting and efficient retraining. To solve these two issues, we design a retraining model that can select important samples and important weights utilizing multi-armed bandits. To further address forgetting, we propose a new regularization term focusing on synapse and neuron importance.

    Only a significant minority of companies unlock the true potential of AI as trained models accumulate dust due to challenges in MLOps. Serving reliable AI predictions to customers involves cost, effort, and planning to set up a continuous deployment pipeline. MLOps for Deep Learning demands a carefully crafted deployment pipeline. We discuss our open-source project which is a robust continuous deployment pipeline by integrating our unique drift detection and model retrain algorithms for serving DL models. We show how to efficiently deploy, monitor, and maintain DL models in production using our solution which is a Kubernetes native POC solution.

    Diego Klabjan is a professor at Northwestern University, Department of Industrial Engineering and Management Sciences. He is also Founding Director, Master of Science in Analytics, and the Deep Learning Lab. His expertise is focused on data science and deep learning with a concentration in finance, insurance, and healthcare. Professor Klabjan has led projects with large companies such as The Chicago Mercantile Exchange Group, Intel, General Motors and many other, and he is also assisting numerous start-ups with their analytics needs. He is also a founder of Opex Analytics.

    Twitter Linkedin
  • 12:45
    Supreet Kaur

    Closing General Session: Striking the Right Balance: ML or No ML

    Supreet Kaur - Assistant Vice President - Morgan Stanley

    Down arrow blue

    MLOps/Trusted AI Summit: Closing General Session: Complexity vs Simplicity in ML and AI Projects

    Women in AI Reception: Pivoting into AI

    Supreet is an AVP at Morgan Stanley. Prior to Morgan Stanley, she was a management consultant at ZS Associates where she automated different workflows and built data driven solutions for fortune 500 clients. She is extremely passionate about technology and AI and hence started her own community called DataBuzz where she engages the audience by sharing the latest AI and Tech trends and also mentors people who want to pivot in this field.

  • 13:15

    Lunch

  • 14:15

    End of Summit

MLOps Summit San Francisco

MLOps Summit San Francisco

15 - 16 June 2022

Get your ticket
This website uses cookies to ensure you get the best experience. Learn more