Skip to content

A Harbour Platform Team Quiets Its Pager: An AIOps Story

Published November 17, 2026 by Beena Yadav

What follows is fiction. It is stitched together from situations that operations teams commonly describe, and it refers to no real company, team or person. It follows a small platform group at a fictional logistics firm near a busy harbour as they try to tame their alerts, and it uses the Ops Atlas description of AIOps as the map they argue over. Names are made up and the numbers in the story are illustrative, except where the text says it is quoting the Atlas.

Monday: the night nobody slept

Femke, the team lead, arrived to find three colleagues grey-faced over coffee. A storage hiccup at 02:10 had produced a wave of messages: more than two hundred pages, all different in wording, all about the same degraded database. Whoever was on call had acknowledged them one by one, and by the time the real cause was found, the shift had lost two hours to reading the same complaint in new clothes.

“We keep buying more monitoring,” said Joost, the newest engineer, “and sleeping less.” Femke had heard about AIOps at a meetup and half expected a product demo to fix it. Instead she did something unglamorous: she read the reference page, the Atlas article on what AIOps is and is not, and printed its ordered list of eight practices. The first four, she noticed, needed no machine learning at all.

Tuesday: the argument about labels

The team’s first meeting was about tags. The Atlas asks that every alert carry a fixed set: service, team, severity, remediation, the cluster affected, a pointer to the runbook, plus which monitoring server fired it. Joost audited the live rules and came back with an uncomfortable table. Nearly half had no owner. A tag named “cluster” meant different things in different rule files. Two rules duplicated each other under different names.

They split the ambiguous tag into two and wrote a short linter check so that new rules without the full set could not be merged. In the Atlas, the first maturity stage ends once 95 percent of alerts comply; Femke wrote that number on the whiteboard, and by Friday they were at a little under three-quarters. It felt like the first real progress in months.

Wednesday: the fingerprint that never matched

On Wednesday Joost noticed something odd while testing deduplication. Alerts that were obviously the same were not collapsing. The reason took him an hour: a label holding the container’s generated name changed with every restart, so no two alerts ever shared a fingerprint. He moved that value out of the identifying labels and into the description, and the overnight pager volume in the test copy dropped sharply. He told the others the fix felt almost embarrassing because it was so small.

Thursday: the tempting shortcut

An enthusiastic colleague, Daan, suggested grouping anything that fired within five minutes. It would take an afternoon. Femke remembered a passage in the Atlas that describes the same experiment in its lab, where the time window reduced the alert count several-fold while also gluing unrelated incidents together. She asked a better question: how would they know whether the grouping was right?

That led to the part of the week that changed the team. They pulled out the last twenty-odd incidents, and Joost and Daan labelled by hand which alerts belonged together. Then they scored every grouping idea with precision and recall. The five-minute window looked wonderful on volume and mediocre on precision. Grouping by shared service label plus dependency scored better on both. For the first time the team was choosing by evidence rather than by the thrill of a smaller inbox.

The dependency map nobody owned

The better grouping needed a map of who calls whom. The map existed in a diagram last updated before two reorganisations. Femke assigned an owner, and Joost compared it against real traffic. Seven relationships were missing. She remembered that one Atlas test saw recall fall from 0.93 to 0.53 when a single edge went missing, so seven missing links were hardly cosmetic. They fixed the map, added a rule that every new service must declare its dependencies at launch, and rescored. Recall rose in step.

Along the way they noticed that nearly every serious incident in their records followed a deployment. Google’s SRE book, which the Atlas cites, ties most outages to changes made on live systems, so they began attaching the latest changes for each service to the alert as context. When the database degraded next time, the alert already said which release had touched the storage layer that afternoon.

Friday: guard rails before automation

Daan wanted to go further and let a script restart stuck workers. Femke agreed, with conditions copied from the Atlas pattern: only an approved list of actions, each reversible, with a rate limit per machine and per hour for the whole fleet, running as the application’s own user, verifying the result and paging a person on failure. The first automation was boring: restart one stuck consumer, at most a couple of times a day. It worked, which was the point.

They also read the security section and quietly fixed three things. The alert router’s API had been reachable from anywhere in the office network and now needed authentication. The webhook feeding the automation was moved onto a loopback address. And the group stopped letting any model read raw alert text unsupervised, since the Atlas warns that such text can be written by an attacker.

The weekly review that stuck

The habit that outlived the excitement was a Monday review of the ten most frequent alerts. Each got one of three verdicts: fix the rule, group it with its relatives, or delete it. Joost kept a running score of precision and recall for the grouping rules in a shared sheet, and anyone proposing a change had to post the new numbers beside the old ones. Arguments got shorter. When someone wanted a new detector for a sudden-traffic pattern, the first question became whether a calendar entry or a simple rule could handle it, and twice the answer was yes.

The team also agreed on a vocabulary. A rule was either proposed, in shadow, on canary, active, under review or retired, and the status lived in the rule’s own file. Nobody had to ask in chat whether something was live.

Six weeks later

The team did not buy a platform. They ended with a label contract at high compliance, a fingerprint that behaved, grouping rules scored against labelled incidents, a dependency map with an owner, and a calm weekly review in which the three noisiest alerts were fixed or deleted. Following the Atlas lifecycle, they ran their one experiment with a learned method strictly in shadow, scored but invisible to responders, and decided it was not yet good enough. Nobody felt cheated by that decision, because the evidence had been gathered openly.

Femke later told a visiting student that the story is less about intelligence than about discipline. For readers following the career side of this, our piece on how a DevOps engineer grows from junior to senior roles describes the same habits, and a longer career plan that mixes skills, hands-on projects and exams shows where this kind of work fits.