Skip to content

Six ObservabilityOps Symptoms and How to Diagnose Them

Published November 16, 2026 by Beena Yadav

A telemetry platform rarely fails with a bang. It fails as a slow drift: queries that took two seconds now take twenty, the monthly invoice creeps up, graphs show gaps on Mondays, and the on-call engineer mutes a noisy rule “just for tonight”. Each symptom has a cause, and each cause sits in the territory that ObservabilityOps covers. This article works through six common symptoms in problem-then-solution form, so you can diagnose before you buy anything.

For definitions we lean on the Atlas of Ops disciplines, in particular its ObservabilityOps entry, which reads the term as the operation of the observability platform itself. The Atlas states that the label is not an established standard, so treat the vocabulary as a working convention and the diagnostics as ordinary engineering.

Symptom 1: The metrics store is slow and memory-hungry

Likely cause: a surge in the number of distinct time series. Every unique mix of label values is stored separately, so one new label can multiply the total. The Atlas offers a lab to make this concrete: tagging one order counter with a customer ID that can take 50,000 values produces something like 1.8 million series for that single metric. Across a whole team, the exercise lands at 1,946,880 series against a budget of 200,000.

Solution: find the offenders, remove or bound the label, and set a budget per team. In the same lab, taking out the identifier brings the team to 146,916 series. Disk estimates move accordingly: 84.1 to 168.2 GB before, 6.3 to 12.7 GB after, for fifteen days of data. For sizing, multiply the retention period (counted in seconds) by the ingest rate, then by a byte or two for each sample. That product is worth keeping on a sticky note.

Symptom 2: A whole target vanishes from dashboards after a deploy

Likely cause: a scrape that crossed a configured limit. The Atlas points out that breaching a sample or label limit fails the entire scrape, not only the excess series. One verbose service can therefore make all its metrics disappear at once.

Solution: keep the limits, but alert on scrapes that fail for this reason and review per-service series counts before every release that touches instrumentation. A large edge-network operator described in the entry went a step further by patching its metrics software so that each server caps series instead of failing, and telling teams that a green CI run meant the capacity existed. Not every organisation can fork its tooling, so a pre-merge series estimate is the realistic version.

Symptom 3: The span-derived metrics have exploded

Likely cause: a per-instance identifier slipped into the labels when metrics were generated from spans. In the Atlas’s description, a collector instance UUID gets attached; with many replicas the series multiply, and since the UUID is regenerated on each restart, old series die and new ones appear continuously, which is churn.

Solution: aggregate the identifier away at query or recording time using a sum without that label. The Atlas is explicit that dropping the label at scrape time or raising a series limit is not the right fix. It is a good example of a problem a person with ownership notices in a week and a distributed team lives with for a year.

Symptom 4: The success-rate target is “met” on paper but users disagree

Likely cause: a measure that is computed in a flattering way. Here the entry uses telemetry export as its example. Out of ten million attempted exports, 9,950,000 got through, 40,000 failed and 10,000 were lost, which works out at 99.5 percent when the goal was 99.9, so the target is missed. The trap is counting retries as successes before retries are exhausted, which makes a poor number look better.

Solution: define the denominator and the point at which a failure is counted, and write both into the metric’s documentation. A measure that nobody can recompute by hand is a measure nobody can trust.

Symptom 5: Traces are chopped into fragments

Likely cause: context propagation broke somewhere along the path. Typical culprits include a message broker where no span links were configured, a reverse proxy which discards the traceparent field, and estates that blend the older B3 headers with the W3C ones. Tail sampling adds a second failure mode: each trace needs every one of its spans delivered to one collector instance, and the Atlas names routing keyed on the trace ID as the usual technique.

Solution: trace a synthetic transaction across the whole path on a schedule, and fail a check if the result has more than one root. Align on the W3C defaults, and begin a new trace at the public edge so outside callers cannot inject surprises. This also stops baggage from leaking to partners, a risk the Atlas calls out explicitly.

Symptom 6: Nobody trusts the pager

Likely cause: too many alerts with no clear action. The entry’s illustration: of 42 pages in a review window, only 15 required action, which is 35.7 percent. The remaining pages trained people to ignore the phone.

Solution: page on user-visible symptoms, route everything else to tickets or digests, and retire signals nobody reads. A Watchdog-style heartbeat catches the other failure, the silent pager, and a single black-box test of the alert path beats a dozen component checks.

How to run the diagnosis without a big project

Set aside two hours with the people who own the platform and walk through the symptoms in order. For each one, answer three questions: do we see it, how would we know, and who would fix it? The answers fall into three groups. Symptoms you see and can fix go straight to a backlog. Symptoms you could not detect become instrumentation tasks, because a blind spot is itself a finding. Symptoms with no obvious owner go to the manager, with a one-line explanation of the risk.

Keep the output small: one page, a ranked list, a date for the next review. Teams that produce long reports tend to file them. Teams that produce a short list tend to act on it, and the next review shows whether the numbers moved.

Pulling the diagnoses into a table of habits

  • Weekly: list the ten metrics with the most series and compare them to last week.
  • Monthly: compute unit cost, spend divided by request volume, and plot it next to traffic.
  • Quarterly: run an alert review and remove anything unused for ninety days.
  • Continuously: a canary that measures freshness at the 95th percentile.

A note on ownership and career consequences

The common thread in all six symptoms is the absence of a named owner. For engineers weighing their own next move, taking responsibility for the telemetry platform is a recognisable step on the ladder from implementer to platform thinker. A piece such as a walk through the typical steps from junior engineer to seniority frames the larger arc, and the roadmap in a career roadmap that covers skills, projects and certifications shows where such a project can sit in a portfolio.

A short worked diagnosis

To see the method in motion, imagine a generic platform team whose dashboards go blank every Thursday afternoon. They check Symptom 2 first and find that the Thursday release adds a new route label to a latency metric. The label has far more values than anyone expected, a scrape limit trips, and the entire target drops out until the next deploy. The fix has two parts: remove the route label in favour of a templated route name, and add an alert on failed scrapes so that the next breach is announced instead of discovered. The illustration is invented, but the pattern is common, and it shows why the symptom list is ordered the way it is.

What the evidence cannot tell you

A fair analysis admits limits. The Atlas is a single catalogue entry, and its worked examples are teaching exercises, not benchmarks from your estate. The public cases it cites, such as a very lean team running a high-traffic platform, are individual stories, and survivorship is likely. Treat the numbers as prompts for measurement, then measure your own environment and let those results drive decisions.

Start with one symptom you already have. Gather the figures, apply the matching fix, and record the before and after. Observability work earns its place the same way any engineering does: by changing a number you can show.