Evaluating and Monitoring an AI System in Production
Evaluating and Monitoring an AI System is an advanced-level course of 5 lessons (about 1 h 40 min), with 15 exercises and 15 quiz questions. Prerequisites: Integrating an LLM into an Application. Exercises run in the browser, with nothing to install.
- Generative AI
- Advanced
- 5 lessons
- 1 h 40 min
- 15 exercises
- Exercises run in the browser
The course in brief
An AI system is not validated on a few tries. This course teaches you to build a test set, to use and calibrate a judge model, to measure quality, cost, and latency, to detect drift and use user feedback, then to bring it all together in a monitoring dashboard with alerts. The exercises are done in Python, with a locally simulated model and judge.
What you will be able to do
- Build a representative test set and use it as a regression test
- Evaluate free-form answers with a judge model and measure its agreement with humans
- Compute latency percentiles and a cost per request from the logs
- Choose a configuration based on quality, latency, and cost thresholds
- Detect input drift and a drop in satisfaction
- Build a monitoring dashboard with pandas and actionable alert rules
Syllabus: 5 lessons
- Build a representative test set for an AI system
- Write automatic checks: normalized matching and rules
- Use the test set as a regression test before every change
- 02
Evaluation by a Judge Model
- Design an evaluation rubric usable by a judge model
- Robustly read and validate a judge's output
- Measure the agreement between the judge and human annotations
- 03
Measuring Quality, Cost, and Latency
- Log the useful measurements of each model call
- Compute latency percentiles and a cost per request
- Choose a configuration based on quality, latency, and cost thresholds
Latency and percentilesCost per requestQuality, cost, latency trade-off
- 04
Drift and User Feedback
- Distinguish input drift from quality degradation
- Compute a stability index (PSI) between two periods
- Use user feedback to detect a drop in satisfaction
- 05
Building a Monitoring Dashboard
- Aggregate call logs into daily indicators with pandas
- Define alert rules with thresholds and severity levels
- Organize a dashboard that is useful to the team and to decision-makers
Final project
Evaluation File and Monitoring Dashboard for an Assistant
For a fictional customer service assistant (simulated model), build a test set of at least 30 cases distributed by category, evaluate two versions of the prompt with automatic checks and a simulated judge calibrated on human annotations, then generate a month of synthetic logs. From these logs, produce a daily monitoring dashboard, alert rules, and a drift analysis between the first and last week.
Deliverables
- The versioned test set (JSON or CSV) with categories and criteria
- The evaluation script comparing the two versions, global and per-category score
- The judge-human agreement calculation (agreement rate and kappa) and the judge's rubric
- The daily monitoring dashboard (pandas) and the alert rules with their justified thresholds
- A two-page note: recommended version, risks, indicators to watch
Assessment criteria
- The test set covers frequent cases, edge cases, out-of-scope requests, and manipulation
- Results are reproducible and the judge's invalid outputs are counted
- Latency is summarized by percentiles and costs use a dated grid in fictional units
- Drift is measured with an explained method and thresholds presented as conventions
- Alerts are few, graded, and each one is tied to an action
Frequently asked questions
Do you need access to a paid model for the exercises?
No. The evaluated model and the judge model are simulated by Python functions: the exercises are deterministic and run without a network or an API key.
Can a judge model replace human evaluation?
It complements it. The course shows how to measure its agreement with human annotations before trusting it, and how to correct its known biases.
Does the course give providers' prices?
No. Prices change often; the course uses fictional units and shows how to store a dated price grid to calculate the real costs of your system.
What prerequisites do you need?
Knowing how to integrate a language model into an application (course AI2) and knowing the basics of Python; some pandas notions are useful for the last lesson.
This course, told for your job
Say what you want to be able to do: “Evaluating and Monitoring an AI System” takes its place in your path, with examples from your field. The concepts stay those of the course.