Why to Monitor?
What is the goal of an ML system for a business use case? Do you think it is below? i.e Designing an ML system which has the best model accuracy
Not exactly. The goal looks more like below
True ROI from an ML system is only achieved by monitoring the system in production and tying the results to different aspects like data, model, system and business
What to monitor in this case?
source: https://www.evidentlyai.com/
- System: Eg. of this would be service health and system uptime
- Data: Eg. Data drifts, broken pipelines, schema changes, data outages, different segment performances
- Model: Eg. model accuracy, concept drift, Model bias,
- Business: Eg. increase in clickthrough rates, reduction in churn.
This post covers just this, we establish a high level of why to monitor and what components to monitor broad. We dive in further to understand why and how to monitor during later posts.
AI For Production 