How to build monitoring and AI summaries that preserve the difference between a quiet system and a failed observation.
Imagine a daily product summary that says: “Visits increased, but there were zero purchases.” It sounds like a useful diagnosis. Perhaps checkout is broken. Perhaps the new traffic is poorly targeted.
Now imagine that the purchase query actually timed out.
The system has turned an observation failure into a business result. If an AI assistant then explains the apparent drop, it can produce a persuasive analysis of something nobody measured.
This is the failure mode I want to design against. A zero is meaningful. It says the relevant measurement was made and nothing matching the definition was found. An unavailable measurement says something different.
The examples below are illustrative designs, not a report of a monitoring system I have shipped. The principle applies to dashboards, scheduled reports, and assistants that assemble answers from several services: uncertainty needs to survive every step between the source and the sentence.
An empty result needs a definition
An empty list can mean that a successful query found no matching records. It can also be the default value returned by a failed integration, a partial page, a restrictive filter, or an account that cannot see the expected records.
The list alone cannot tell you which situation occurred.
Before interpreting zero purchases, I would want to know that the query completed, used the intended account and time window, and covered the records relevant to the metric. If the source updates with a delay, I would also need to know whether that reporting window was complete enough for the decision.
Even database behavior contains useful distinctions. PostgreSQL documents that sum returns null when no rows are selected, while count returns the number of input rows. Replacing null with zero can be appropriate, but the application has to decide what that replacement means. The database does not know whether an upstream import silently stopped. PostgreSQL aggregate functions.
For a report, the right label might therefore be “zero matching purchases in the available records.” That is narrower than “nobody purchased,” and sometimes the narrower claim is all the evidence supports.
Preserve how each claim became known
I find four labels useful when reviewing a generated summary:
- Observed: directly inspected evidence, such as a response or event record from a named source
- Reported: a statement supplied by another person or system, such as a platform's summary total
- Derived: a value calculated from identified inputs, such as a rate or comparison
- Unknown: a question the available evidence cannot answer
These labels describe provenance, not a ranking of truth. An observed response can come from an incomplete dataset. A reported total may be the most authoritative available figure. A derived metric can be valid, provided its inputs and definition are valid.
Availability and freshness are separate dimensions. A derived result can be stale. A reported number can cover only part of the requested period. Labeling something “observed” does not make those limitations disappear.
For the opening example, the record might say that the analytics service reported visits for the day, the purchase read failed, and the conversion rate is unknown. That is enough for a useful summary. It simply does not support a conclusion about conversion performance.
Store the result of the check separately from the value
A monitoring record should carry more than a number. At minimum, I would keep the metric definition, source, requested window, check time, source timestamp, read outcome, and any known coverage limits.
Here is an illustrative record for a failed read:
1{2 "metric": "completed_purchases",3 "window": "previous_complete_day",4 "timezone": "UTC",5 "read_status": "failed",6 "value": null,7 "failure_reason": "timeout",8 "coverage": "unknown",9 "last_successful_observation": "stored_separately"10}The previous successful value belongs in a separate observation with its original window and timestamp. It should not quietly populate the current value.
I would use distinct outcomes for a successful value, a successful empty result, a partial result, a failed read, and a check that was not attempted. The exact names matter less than preserving the distinctions. “Not checked” is especially useful when a report deliberately omits an irrelevant source.
Code can enforce parts of this contract. A failed read must not contain a current numeric value. A partial read must carry a coverage note. A complete zero needs a successful read and a defensible definition of completeness.
Without those checks, a helper such as value ?? 0 can erase the evidence before the model ever sees it.
Freshness needs its own clock
A successful request at 09:00 does not establish that the returned data describes the world at 09:00. It may have retrieved a cached value or a report last updated overnight.
I would track when the check ran, when the source last updated, and which period the value describes. Those are different facts, even when a convenient dashboard collapses them into one “updated” label.
Prometheus provides a concrete example of explicit staleness handling: its documentation describes time series being marked stale and no longer returned in subsequent queries. The missing value is not automatically a measured zero. Prometheus querying and staleness.
A product report needs a freshness rule appropriate to its decision. Yesterday's final sales total can remain useful today. Yesterday's queue depth is poor evidence for whether a job is currently stuck.
The interface should make that distinction visible. “Last confirmed at 08:15” is useful context. A green “healthy” badge based on the same old reading can imply a current check that never happened.
Derived metrics inherit the limits of their inputs
Suppose the visit count is available and the purchase count is missing. There is no basis for displaying a zero-percent conversion rate.
Suppose both values are available, but one covers a local calendar day and the other covers a UTC calendar day. There is still work to do before dividing them. The same applies when visits include one set of channels and purchases include another.
A derived metric needs an explicit definition and compatible inputs. That includes time windows, timezone, filters, attribution rules where relevant, and the population being counted.
When the denominator is zero, the rate is undefined for that window. When an input is unavailable, the rate is unavailable. When the inputs are partial, any result needs to remain labeled partial rather than becoming an apparently final percentage.
Comparisons deserve the same care. If today's purchase records are still arriving and yesterday's are final, the apparent decline may describe the reporting pipeline. A summary should not turn that difference in completeness into a product recommendation.
This is straightforward arithmetic. The difficulty is keeping the metadata attached long enough for the arithmetic to mean something.
Separate a business alert from a visibility alert
A failed read may still require attention. It just requires a different message.
“No purchases were recorded in a complete reporting window” is a business observation. “Purchase data could not be checked” is a visibility problem. Either can matter, but the investigation starts in a different place.
Grafana makes a related distinction for Grafana-managed alert rules: a successful query with no data points can enter a No Data state, while a failed query evaluation can enter an Error state. Its handling is configurable, which is a reminder to make this behavior an explicit design choice. Grafana No Data and Error states.
A brief transient failure might justify a retry without a notification. A prolonged gap before an important decision might justify an alert that names the unavailable source and the last successful check. A confirmed failure in the product may justify a separate operational response.
Those policies should reflect the consequence of being wrong. I would be much more cautious about an automated action that changes spending or contacts customers than a read-only daily summary.
Unknown data should route the system toward verification, not silently satisfy a threshold intended for a measured value.
Give the assistant claims it is allowed to make
A language model should receive the evidence states, not a cleaned-up table that replaced every blank with zero. Otherwise, careful wording cannot repair the missing distinction.
For the opening example, an acceptable summary might be:
The analytics service reports higher visits for the completed day. Purchase data was unavailable because the read timed out, so I cannot assess conversion. The last successful purchase check covered the previous day.
That statement contains an observation, a limitation, and a useful next step for investigation. It does not need an invented explanation of customer behavior.
I would also validate the generated text. If a required source is unavailable, the output must not describe its metric as zero, improved, declined, healthy, or resolved. A global “some data may be missing” disclaimer is too weak when a specific conclusion depends on that data.
The limit belongs next to the claim it changes.
Test the quiet and broken paths
The most useful test cases are easy to overlook because they produce little data: a legitimate zero, a timeout, an expired permission, an empty first page with more pages available, and a successful response containing yesterday's snapshot.
I would add a partial import, incompatible reporting windows, a zero denominator, and a recovery where late records change an earlier total.
For each case, check the stored state, displayed value, generated language, and permitted action. A timeout should remain visible as a timeout all the way through. A legitimate zero should still be reported clearly rather than buried under generic caution.
Recovery matters too. When missing data arrives, recompute affected metrics and label a revised report where readers could otherwise act on the old one. Keep enough history to explain why the conclusion changed.
The goal is a monitoring system that can say something precise about both the product and its own ability to observe it. Sometimes the useful answer is a number. Sometimes it is a clear explanation of why there is no number yet.