AI monitoring after launch: test the complete alert path
After launch, monitor both the technical execution and the business result of your AI application. Then test the complete alert path: is the fault detected, does the notification reach the responsible person, and is the next action clear?
A dashboard does not answer those questions automatically. Missing measurements must also be visible so that a failed check cannot make the system look healthy.
The tables and examples below are engineering recommendations. The configuration and numerical examples are fictional; they have not been tested on a real system.
Measure availability and task correctness separately
A successful HTTP request does not establish that the application used the correct source or completed its task correctly. Google's SRE chapter on monitoring explicitly includes HTTP 200 responses with incorrect content among possible failures. It also distinguishes internal measurements from checks of externally visible behaviour.
| Check layer | Example | What this does not establish |
|---|---|---|
| Technical execution | Model calls and database access work. | The answer is correct. |
| Execution rules | Permitted tools and parameters were used. | The result solves the task. |
| Task result | The expected value and supporting source match. | All other requests are also correct. |
| Monitoring itself | Checks run and notifications arrive. | The checks cover every relevant failure. |
The AI agent evaluation guide explains how to assess individual execution paths.
Start with a fault someone can act on
Choose an important workflow and a specific failure condition. “Answer quality is falling” is too vague as an alert description. A more useful condition would be:
The reference question about the approved product classification returns an incorrect class or lacks the expected source.
You can then define the diagnostic evidence required and which function should be temporarily restricted.
- Immediate containment: A forbidden action was allowed or a critical safeguard failed.
- Prompt investigation: A repeatable task check fails.
- Scheduled review: A quality sample deteriorates or new failure types appear.
The required response depends on the process. Defined support hours may suit an internal drafting assistant. If a workflow needs immediate action outside those hours, responsibility and technical containment must be settled before operation. An alert does not replace preventive access control and cannot retract information already delivered or actions already completed.
Define a complete alert rule
The following is a fictional test configuration. Its intervals and thresholds are not general recommendations.
A RAG assistant answers a reference question about artificial demonstration record D-47. The expected result is test class B, supported by approved source Q-17, revision 3.
| Component | Definition in this example |
|---|---|
| Workflow | Internal product information in an isolated test area |
| Check | Compare the answer class and source revision with the test reference |
| Schedule | Every five minutes during agreed operating hours |
| Task-quality alert | Two directly consecutive scheduled checks are evaluable and return incorrect results |
| Monitoring gap | No completed, evaluable check for more than twelve minutes during operating hours: separately report “monitoring incomplete”. Repeated unevaluable checks do not reset this deadline. |
| Interrupted sequence | A missing or unevaluable check breaks the sequence; it counts as neither correct nor incorrect |
| Responsibility | A named technical owner and backup |
| First action | Restrict the affected information function and investigate using the run identifier |
| Recovery approval | Repeat the reference check and explicitly approve normal operation |
The gap check needs its own reliably executed monitoring. A stopped checking service cannot reliably report its own absence. Planned pauses must be accounted for.
A failed evaluator must not count as a passed quality check. A reference question also checks only a known path; it does not replace evaluation of changing user requests. RAG evaluation: metrics and golden datasets explains how to build a representative test set.
Test the alert path with controlled faults
Run the exercise in an isolated environment or an explicitly designated test area. Use artificial data and a test receiver. Do not trigger real payments, business messages or changes to production data.
| Controlled change | Expected evidence |
|---|---|
| The reference source deliberately returns an incorrect test class. | The quality check fails and triggers a notification under the defined rule. |
| A required data source becomes unavailable in the test. | Technical failure remains distinguishable from an unevaluable task result. |
| The evaluation service is stopped. | Missing evaluations appear as a gap rather than error-free results. |
| The regular test notification channel fails. | The agreed fallback is tested; an undelivered notification remains visible. |
| The test receiver does not acknowledge the alert. | The defined backup or escalation path activates in the test. |
| The underlying fault is fixed. | Rechecking and recovery approval are documented. |
An entry in an alert log is not sufficient completion evidence. Record whether the notification actually arrived and whether the recipient could infer the intended action. Separate three timestamps: fault detected, notification delivered, response started. This makes delays attributable.
Treat missing measurements as a separate state
“No errors measured” may mean everything works. It may also mean nothing is measuring anymore. Monitor:
- the last started and last completed check;
- expected and actually evaluated cases;
- pending or failed evaluations;
- the last confirmed test of the notification path.
A regular heartbeat establishes activity, but does not establish that a task-quality check completed.
Prometheus provides absent_over_time(), for example, to detect the absence of a time series over a window. A series that is still being emitted with a frozen application-level timestamp instead needs a check of that timestamp. “Metric present” and “check current” are different statements.
Show quality scores alongside coverage
A second fictional example: 200 requests receive an HTTP 200 response. Twenty answers are checked for correctness. Eighteen pass and two are wrong. The other 180 answers remain unevaluated.
| Metric | Result |
|---|---|
| HTTP 200 response share | 200 / 200 = 100% |
| Pass rate among evaluated answers | 18 / 20 = 90% |
| Correctness evaluation coverage | 20 / 200 = 10% |
The 90% describes the evaluated cases. It cannot simply be applied to every request without an appropriate sampling method and consideration of uncertainty. Add the period, case count, selection method and evaluation status to quality displays. Report important task classes separately so that failures in a rarely used function do not disappear in an overall average.
An LLM-based evaluator is itself a measurement component with limitations. The MOSS preprint, submitted on 16 October 2026, covers recurring quality evaluation in search and recommendation systems. It treats scores as aids for prioritisation and diagnosis, identifies calibration differences and possible false alerts, and does not replace human judgement. The work is under review and is not evidence of performance in other applications.
Keep enough evidence for the first investigation
A notification should identify the workflow and environment, timestamp and run ID, rule and version, observed condition and comparison basis, responsible owner and first action. Include the application, model and, where relevant, index version involved.
Full prompts and document content do not need to be copied into every notification by default. Use controlled references to the necessary evidence and restrict access.
Cost also belongs in operational monitoring. The complete task remains the relevant economic unit; the AI cost per successful task model explains this denominator.
Next step: exercise one alert from end to end
Choose a relevant workflow, document one alert rule and introduce the associated fault in a controlled test. Follow the path through to an acknowledged response. The result should answer:
- Which faults does the check detect, and which remain outside its coverage?
- How do missing evaluations become visible?
- Who acts, within which agreed hours?
- What establishes that normal operation may resume?
If you need to assess these operating and acceptance criteria for an existing PoC or AI application, the AI Production Readiness & Reliability Sprint is a suitable starting point. A bounded assessment can cover one workflow, its relevant failure states and its alert path.