Skip to content

The Friday Night the Canal Street Crew Learned IncidentOps

Published November 15, 2026 by Beena Yadav

Pull up a chair for the tale of a team that never existed, though you may well know its cousins. Six engineers at an invented bike-sharing app, call them the Canal Street crew, had a Friday evening that went sideways. Their story is made up from ordinary ingredients, and I am telling it because it shows IncidentOps, the craft of handling unplanned outages as a team, better than any definition can.

Friday, 18:10: something is slow

Mila, a mid-level engineer, saw the rental screen taking ten seconds to load. She posted in the team channel: “anyone else seeing this?” Two people replied with a shrug. Everyone assumed it would pass. Forty minutes later the support inbox started filling.

The first lesson of the story is about the word “incident”. Nobody at Canal Street had ever decided what it meant, so nobody felt allowed to say it. Many teams are the same. The Ops Atlas, in its entry describing IncidentOps in detail, suggests written criteria that anyone can apply, such as: customers can see the problem, another team is needed, or it has resisted an hour of effort. The Canal Street crew had met two of those three conditions by seven o’clock, and still nobody declared.

Friday, 19:05: five people, one keyboard each

When the team lead, Joost, finally typed “this is an incident”, all six engineers dived in. Mila restarted a service. Pieter rolled a configuration change. Anouk was reading logs and, at the same time, restarting something else. Two of those actions cancelled each other out, and for twenty minutes nobody knew what state production was in.

Here the Atlas offers its sharpest point: the person coordinating is not meant to be a resolver. A small group changes production, one person keeps the map, and somebody else handles communication. It also gives a sensible guideline for group size, one supervisor per five people, so six engineers actually slightly exceeds a comfortable span.

Friday, 19:30: the quiet hat

Anouk, who had joined the team a year earlier, put her hand up in the channel. “Can I coordinate and stop touching things?” It felt odd. Everyone expected the most senior person to do it. But the published process the Atlas draws on gives the role to whoever outranks the others on the call, and it explicitly treats roles as hats rather than titles. Joost, relieved, became the person who changed production, and Mila started writing customer updates.

The change in the room was immediate. Anouk asked two questions on repeat: what do we know, and what is the cheapest thing that could make this stop? That second question mattered.

Friday, 19:50: stop the pain first

Somebody guessed that Thursday’s release was behind the slowness. Pieter wanted to dig into the database queries to prove it. Anouk asked whether they could simply go back to the previous version. They could, and ten minutes later the app loaded in under a second.

The Atlas names this move explicitly: mitigate before you explain. It lists rollback, draining traffic, switching off a new feature, failing over and shedding load as general-purpose remedies that need only to know where the fault lives. The team could have spent an hour proving the theory while riders stood at empty docks. Instead they got their service back, and the explanation could wait until Monday.

Saturday morning: the unsent message

Mila’s customer updates had gone out at a decent rhythm once she took the role, but the first one had been over an hour late. She felt bad, and that is where the Canal Street story gets interesting. In the review, the group did not blame her. They asked why a message could take an hour. The answer was that nobody owned it and nobody had a template. The Atlas suggests a first post within about five minutes, a follow-up soon after and regular refreshes through the first hours. The team wrote three short templates that morning, one per stage, using the four common wording stages: investigating, identified, monitoring and resolved.

Monday: the review

The meeting was run by Eveline from a neighbouring team, who had not been on the call. That choice came straight from the Atlas, which cites a well-known online marketplace’s habit of using a facilitator who was not directly involved. Joost had named himself the owner of the review before the Friday call ended. They agreed on three actions, each with one owner and one date: write declaration criteria, print role cards and move the public status notice to a host independent of the app itself.

When someone proposed adding “human error: bad config change” to the summary, Eveline gently asked what in the system had made the change easy to get wrong. They found the deployment tool accepted the setting without a confirmation step. That became the fourth action. The Atlas quotes a safety researcher on how looking back warps our verdicts about what people did, and advises ending the analysis at conditions in the system.

The numbers conversation nobody expected

A few weeks later the engineering manager asked for a chart showing how much faster the team had become. Eveline, again, pushed back. With only a handful of incidents in a quarter, she said, an average would jump around because of luck rather than ability. The Atlas backs her up: durations are lopsided, the abbreviation people use for repair time stands for at least four different measurements, and a single awful evening can overwhelm a small sample. They agreed to show the individual incidents on a simple timeline and to count how many follow-up actions were finished by their dates. The manager, to her credit, preferred it.

There was also a practical conversation about the rota. Six people covering evenings and weekends sounded like plenty until Joost did the sums and found each person would carry roughly one-third of the hours, which is above the quarter that the Atlas quotes as a sensible ceiling. They decided to shorten the scope of cover and borrow a colleague from another group.

What the crew did next

Over the next month they held a half-hour rehearsal with an invented failure. They rotated the coordinator role so that more people had practised it. At each shift change they used a spoken handover. Within a quarter, the team’s senior engineers were noticing something odd: Friday evenings had become quiet again.

What you can borrow

  • Agree three plain criteria for declaring an incident.
  • Pick one person to coordinate without touching the system.
  • Ask for the cheapest reversal before asking for the cause.
  • Give communication its own owner and templates.
  • Let a bystander facilitate the review, and keep the action list short.

If you are at an earlier stage and wondering how this fits into a career, the staircase described in our career-path article for junior-to-senior DevOps engineers shows where such skills sit, and the piece about moving across from sysadmin and support work will resonate with anyone who has been the person on call.

The Canal Street crew are invented, but the pattern is real: hesitation, crowding, and a rescue by structure. Next time something goes slow on a Friday evening, you can be Mila, asking the question aloud, or Anouk, offering to wear the quiet hat.