O'Reilly logo
live online training icon Live Online training

Deploying machine learning models to production: A toolkit for real-world success

enter image description here

Armen Donigian

Companies are looking for ways to incorporate machine learning into their business to save money and increase revenue. In this hands-on practical training, Armen Donigian walks you through designing, developing, deploying, and monitoring machine learning models in production and shares common pitfalls and best practices to help you get started with your own generalizable machine learning platforms.

Materials will include hands on Jupyter notebooks using the Flights Cancellation Kaggle dataset to show you how to leverage open source tools such as XGBoost, TensorFlow, Clipper, TensorFlow Serving, DVC, vowpal-wabbit, MLFlow & SHAP.

This training will provide time for a personalized question & answer sessions, helping to ensure you can apply what you learn to problems you are working to solve.

What you'll learn-and how you can apply it

By the end of this live online course, you’ll understand:

  • Problem Framing technique for Machine Learning projects
  • An end-to-end implementation of a machine learning deployment pipeline
  • Most common mistakes & pitfalls during design, development, evaluation & deployment of models
  • Review data splitting & representation strategies for structured data
  • How version control differs for machine learning projects
  • How to engineer generalizable features, tips on model evaluation
  • How to use model interpretability to build more generalizable models
  • A hardware cost estimation model
  • Pros/Cons of offline and online model training
  • Pros/Cons of static and dynamic inference
  • How to manage package dependencies for experimental and critical path work

And you’ll be able to:

  • Design, build, evaluate, interpret, deploy & monitor predictive, generalizable machine learning models
  • Estimate the hardware capacity & cost needed for a given model
  • Optimize trained machine learning models to reduce training time & inference latency
  • Monitor performance of a machine learning model in production

This training course is for you because...

  • You're a data scientist, business analyst, or machine learning engineer who wants to build or deploy generalizable models in production.
  • You're a data engineer or software engineer who wants to deploy machine learning models in production.

Prerequisites

  • Intermediate software engineering or data science skills

Required materials and setup:

  • A machine with a modern browser (Chrome, Firefox, etc.) installed
  • Bookmark or download materials from course Github repo

Recommended preparation:

About your instructor

  • Armen Donigian has undergraduate and graduate degrees in Computer Science from UCLA and USC. He started his career building tracking & navigation algorithms at Orincon (later acquired by Lockheed Martin). Armen then joined the Global Differential GPS group at Jet Propulsion Laboratory (NASA), performing clock and orbit corrections using GPS/GLONASS satellites, which were also used for testing of Mars Science Laboratory Curiosity Rover.

    Bitten by the startup bug, Armen has helped several startups build data driven products and scale infrastructure as a Senior Data & Machine Learning Engineer. Armen has previously led the development of machine learning explainability methods & currently works as the Head of Personalization & Recommender Systems at Honey Science.

Schedule

The timeframes are only estimates and may vary according to how the class is progressing

Day 1:

An overview of the machine learning deployment pipeline (20 minutes)

  • Introduction to problem framing techniques for ML projects
  • Overview of an end-to-end machine learning process
  • Key questions to ask at each phase of the pipeline (experiment design, model build, evaluate, interpret)
  • Most common mistakes & pitfalls to avoid at each phase of the pipeline
  • A hands-on example of how to apply these principles to the Kaggle Flight Departure dataset
  • Knowledge check exercise
  • Break (10 minutes)

How machine learning projects differ from other software projects (95 minutes)

  • Data splitting & sampling strategies for structured data
  • How to create a repeatable and reproducible model training, test, and validation pipeline?
  • How to version your data, as you do with code?
  • Introduction to model interpretability methods (global/local feature importance, feature attribution etc...)
  • How to use model interpretability results to create more generalizable models?
  • Exploratory data analysis using open source tooling
  • A hands-on example of how to apply these principles to the Kaggle Flight Departure dataset
  • Knowledge check exercise
  • Break (10 minutes)

Package dependencies (10 minutes)

  • How and why to manage dependencies, including those by third parties?
  • A platform/operating system independent method to manage your dependencies
  • How to manage package dependencies if you’re using several languages (Python, R) ?
  • Knowledge check exercise

Cloud hosting cost estimation tools (35 minutes)

  • Review various hardware deployment configurations
  • How to price each cloud model serving offering?
  • How to choose which cloud provider is right for you?
  • Amazon AWS and Google Cloud platform resources
  • Knowledge check exercise

Day 2:

Pitfalls during model development (40 minutes)

  • The dos and don'ts of building a model for production
  • How to determine the size & quality of the dataset needed?
  • How to engineer generalizable features?
  • Most common mistakes to avoid
  • Best practices for how to test model generalizability offline
  • Tips for model evaluation & how to select the right evaluation metric
  • A hands-on example of how to apply these principles to the Kaggle Flight Departure dataset
  • Programming exercise: Detect common pitfalls such as overfitting, data leakage, and nonstationary data
  • Break (10 minutes)

Offline versus Online model training (50 minutes)

  • What is Offline vs Online model training & why do you need to know the difference?
  • When to use Offline vs Online training?
  • Pros/Cons of Offline vs Online training
  • How to transition from offline to online training? When should you not?
  • Hands-on implementations for both offline and online model training
  • Programming exercise: Implement an improved online learning model
  • Break (10 minutes)

Static versus Dynamic model inference (40 minutes)

  • What is Static vs Dynamic model inference & why do you need to know the difference?
  • When to use Static vs Dynamic model inference?
  • Pros/Cons of Static vs Dynamic model inference
  • How to transition from Static to Dynamic inference? When should you not?
  • Hands-on implementations for both Static and Dynamic model inference
  • Programming exercise: Implement an improved offline batch scoring model

Model monitoring in production (30 minutes)

  • Why and how to monitor a machine learning model in production?
  • How to define success & failure metrics?
  • Properties of good model metrics?
  • A hands-on example of how to use Clipper & Grafana to monitor features & model results
  • Programming exercise: Implement an input variable distribution monitor
  • Wrap-up, and Q&A