Deploying deep learning models in production could be challenging, as it's far beyond just training models with good perfromance. As you can see in the following figure, there are several components that need to be properly designed and developed in order to deploy a production level deep learning system:

This repo aims to serve as a an engineering guideline for building production-level deep learning systems to be deployed in real world applications.

(The material presented here is moslty borrowed from Full Stack Deep Learning Bootcamp (by Pieter Abbeel, Josh Tobin, and Sergey Karayev), TFX workshop by Robert Crowe, and Pipeline.ai's Advanced KubeFlow Meetup by Chris Fregly. )

The following figure represent a high level overview of different components in a production level deep learning system:

• Open source data (good to start with, not an advantage)
• Data augmentation
• Synthetic data

• Sources of labor for labeling:
  • Crowdsourcing
  • Service companies
    • FigureEight
  • Hiring annotators
• Labeling platforms:
  • Prodigy: An annotation tool powered by active learning (by developers of Spacy), text and image
  • HIVE: AI as a Service platform for computer vision
  • Supervisely: entire computer vision platform
  • Labelbox: computer vision
  • Scale AI data platform (computer vision & NLP)

• Data storage options:
  • Object store: Store binary data (images, sound files, compressed texts)
    • Aamzon S3
    • Ceph Object Store
  • Database: Store metadata (file paths, labels, user activity, etc).
    • Postgres is the right choice for most of applications, with the best-in-class SQL and great support for unstructured JSON.
  • Data Lake: to aggregate features which are not obtainable from database (e.g. logs)
    • Amazon Redshift
  • Feature Store: storage and access of machine learning features
    • FEAST (Google cloud, Open Source)
    • Michelangelo (Uber)
• At train time: copy data into a local or networked filesystem

• DVC: Open source version control system for ML projects
• Pachyderm: version control for data
• Dolt: versioning for SQL database

• Training data may come from different sources: Stored data in db and object stores, log processing, outputs of other classifiers.
• There are dependencies between tasks, one needs to kick off after its dependencies are finished.

• Workflows:
  • Airflow (most commonly used)

• Editors:
  • Vim
  • VS Code (Recommended by the author)
    • Built in git staging and diff, Lint code, open projects remotely through ssh
  • Jupyter Notebooks: Great as starting point of the projects, hard to scale
  • Streamlit: interactive data science tool with applets
• Compute recommendations ¹:
  • For solo/startup:
    • Development: a 4x Turing-architecture PC
    • Training/Evaluation: Use the same 4x GPU PC. When running many experiments, either buy shared servers or use cloud instances.
  • For larger companies:
    • Development: Buy a 4x Turing-architecture PC per ML scientist or let them use V100 instances
    • Training/Evaluation: Use cloud instances with proper provisioning and handling of failures

• Allocating free resources to programs
• Resource management options:
  • Old school cluster job scheduler ( e.g. Slurm workload manager )
  • Docker + Kubernetes
  • Kubeflow
  • Polyaxon (paid features)

• Unless having a good reason not to, use Tensorflow/Keras or PyTorch ¹

• Tensorboard
• Losswise (Monitoring for ML)
• Comet.ml
• Weights & Biases
• MLFlow tracking

• Hyperas
• SIGOPT
• Ray - Tune
• Weights & Biases

• Data parallelism: Use it when iteration time is too long (both tensorflow and PyTorch support)
• Model parallelism: when model does not fit on a single GPU
• Other solutions:
  • Ray
  • Horovod

Machine Learning production software requires a more diverse set of test suites than traditional software:

• Unit and Integration Testing:
  • Types of test:
    • Training system tests: testing training pipeline
    • Validation tests: testing prediction system on validation set
    • Functionality tests: testing prediction system on few important examples
• Continuous Integration: Running tests after each new code change pushed to the repo
• SaaS for continuous integration:
  • CircleCI, Travis
  • Jenkins, Buildkite

• Consists of a Prediction System and a Serving System
  • Prediction System: Process input data, make predictions
  • Serving System (Web server):
    • Serve prediction with scale in mind
    • Use REST API to serve prediction HTTP requests
    • Calls the prediction system to respond
• Serving options:
  • 1. Deploy to VMs, scale by adding instances
  • 2. Deploy as containers, scale via orchestration
    • Containers
      • Docker
    • Container Orchestration:
      • Kubernetes (the most popular now)
      • MESOS
      • Marathon
  • 3. Deploy code as a "serverless function"
  • 4. Deploy via a model serving solution
• Model serving:
  • Specialized web deployment for ML models
  • Batches request for GPU inference
  • Frameworks:
    • Tensorflow serving
    • MXNet Model server
    • Clipper (Berkeley)
    • SaaS solutions (Seldon, Algorithma)
• Decision making:
  • CPU inference:
    • CPU inference is preferable if it meets the requirements.
    • Scale by adding more servers, or going serverless.
  • GPU inference:
    • TF serving or Clipper
    • Adaptive batching is useful

• Purpose:
  • Alerts for downtime, errors, and distribution shifts
  • Catching service and data regressions
• Cloud providers solutions are decent

• Main challenge: memory footprint and compute constraints
• Solutions:
  • Quantization
  • Reduced model size
    • MobileNets
  • Knowledge Distillation
    • DistillBERT (for NLP)
• Embedded and Mobile Frameworks:
  • Tensorflow Lite
  • PyTorch Mobile
  • Core ML
  • ML Kit
  • FRITZ
• Model Conversion:
  • Open Neural Network Exchange (ONNX): open-source format for deep learning models

• Tensorflow Extended (TFX)
• Michelangelo (Uber)
• Google Cloud AI Platform
• Amazon SageMaker
• Neptune
• FLOYD
• Paperspace
• Determined AI
• Domino data lab

• Lessons learned from building practical deep learning systems

[1]: Full Stack Deep Learning Bootcamp

[2]: Advanced KubeFlow Workshop by Pipeline.ai

[3]: TFX: Real World Machine Learning in Production