Signals your production LLM stopped behaving like before

Production LLM drift rarely throws an error: slightly worse answers, slightly different formats, slightly off-spec behavior, enough to degrade the experience, not enough to trigger an alert. Spotting signals before users notice is the piece missing from most AI systems we see in production. We document it because it’s systematic, not occasional, the kind of problem that shows up often in the work we do at Snowinch.

Why drift is invisible by default

A traditional web app fails in binary ways: it responds or it doesn’t, data is there or it isn’t, tests pass or they don’t. A production LLM can degrade on a different axis, output quality, without any standard monitoring stack noticing.

Causes stack and overlap. The provider silently updates the underlying model. A prompt that worked on the previous version produces slightly different output on the new one. Real user inputs drift from what you had in mind when you wrote the prompt. Temperature chosen in staging behaves differently under real load. Someone “improved” the system prompt without a verification process.

The system still “works”, no exceptions, no timeouts, no 500s, but output no longer meets the original requirements. You find out when a user complains, or worse when nobody complains but conversions drop.

Signals worth monitoring

Output format distribution

If your LLM must return JSON with a defined shape, or answer in a fixed format, the first metric is how often responses match that contract. Not “JSON parses”, parsers catch that, but “expected fields are present, types are correct, values sit in the expected range.”

Even a small rise in responses that need fallback or repair is an early signal. Moving from ~1% to ~4% malformed responses in a week with no deploy on your side usually means something changed upstream, the numbers are illustrative, not universal thresholds.

// Minimal example: track every schema validation in production
const result = OutputSchema.safeParse(llmResponse);

await metrics.increment('llm.output.validation', {
  status: result.success ? 'valid' : 'invalid',
  // Do not log content, only where validation failed
  failure_path: result.success ? null : result.error.issues[0]?.path.join('.'),
});

Average response length

Output length is a crude but surprisingly reliable proxy. If your LLM produces summaries and average length drops ~30% in a week, something shifted, model, prompt, or input distribution. It won’t tell you what changed, but it tells you to look.

The reverse matters too: suddenly longer answers can mean unwanted verbosity or a looser output format.

Fallback activation rate

Almost every production LLM has fallback logic, retry, generic error copy, a default value. How often does it fire per day? If that count climbs, there’s an upstream issue worth investigating before users see it.

Latency per token

Higher latency doesn’t always mean quality regression, but it often precedes model behavior shifts. Providers rarely announce production model updates in advance, latency is an indirect way to notice infrastructure-side change.

Linked business signals

Technical monitoring alone isn’t enough. Business metrics tied to LLM output, flow completion rate, average interaction time, explicit user rejection rate, are often the first felt drift indicators. Noisier and slower, but they catch what technical tools miss: formally valid output that’s semantically useless.

The distinction that matters: model drift vs data drift

Not all drift comes from the model. Sometimes real user inputs move away from the cases you designed for. A classifier built for one text profile starts seeing slang, mixed languages, abbreviations, it degrades not because the model changed but because the domain widened.

The response differs. Model drift → revisit the prompt, consider provider or version change, refresh the golden set. Data drift → the system needs training cases or updated prompts covering new patterns. Treating both the same way leads to the wrong fix.

A practical split: periodically sample real inputs and check them manually against expectations. If inputs are still in the expected domain but outputs got worse, it’s model drift. If inputs changed, it’s data drift.

What doesn’t work as monitoring

Using one LLM to judge another’s output sounds obvious and creates more problems than it solves. The judge has its own bias, drift, and cost. It isn’t deterministic, two runs on the same output can disagree. It isn’t Git-trackable. It doesn’t tell you whether the fault is in the system under test or in the judge.

Deterministic monitoring, schema validation, length distributions, fallback rate, latency, is less flashy but more reliable and much cheaper. When you need qualitative judgment, human review on a sample is more honest than LLM-as-judge.

For a structured test harness with golden sets and deterministic matchers, see the LLM regression protocol article.

When to start monitoring

Before production, not after. Adding these metrics to a live system always costs more than baking them in upfront, you can’t recover historical baselines you never collected.

The practical minimum for a production LLM: logged schema validation rate, per-call latency, explicit fallback counters. Everything else can follow, but those three aren’t optional if you want to know what’s happening.

Operational summary

• Production LLM drift doesn’t throw errors, it degrades silently. Standard monitoring won’t catch it.
• Track: schema validation rate, output length distribution, fallback activation, latency per token, linked business signals.
• Separate model drift from data drift, different causes, different fixes.
• Avoid LLM-as-judge for monitoring: non-deterministic, not traceable, drifts on its own.
• Build the baseline before go-live. You can’t backfill history you never logged.

Tell us your context, constraints, and goals: we’ll say whether working together makes sense and how to set up a first step.

Email hello@snowinch.com