Skip to content

Two Engineers, One Outage: Acting on a Hunch vs Labelling It

Published November 20, 2026 by Beena Yadav

It was a quiet Thursday evening when the same alert reached two engineers in different teams of the same company. Both were competent and both were tired. What happened next is a useful comparison, because the outage was the same and the outcomes were not. The difference was a habit, and it is one of the habits that marks the move from a junior to a senior engineer.

The first engineer: the confident hunch

The first engineer read the alert, recalled that the last three incidents involving this service had been caused by a full disk, and went straight to the storage graph. It looked a bit high, though not alarming. That was enough. They cleared old logs, restarted the service and told the channel, “Fixed, it was the disk.”

For ten minutes the dashboard looked better. Then the alert came back. The engineer, now irritated, tried the next familiar cause, a stuck queue, and cleared that too. By the time the third idea was in motion the channel had four people in it, each offering a different story. Nobody could say what had been confirmed and what had merely been tried.

The second engineer: the labelled start

The second engineer opened a scratch note and made three short headings. Under “confirmed”, they wrote only what they had just looked at: the alert’s name, the time it first fired, the error rate on one endpoint and the fact that other endpoints were healthy. Under “assumed”, they wrote that the database was reachable and that the last release was unrelated. Under “guessing”, they wrote that it might be disk, or the queue, or a dependency.

Then they did something the first engineer had skipped: they chose the cheapest check that would separate the candidates. A single query showed that the failing endpoint was the only one calling a particular downstream service. That was a new confirmed fact, and it made the disk idea much less likely without any logs being touched.

A fair comparison

Both engineers acted quickly. The second spent roughly three minutes on the note, and in return avoided two interventions that would have changed the system without addressing the problem. The first engineer’s actions were not reckless, but each carried a small cost: clearing logs destroys evidence, restarting loses in-memory state, and every action muddies the timeline for the review afterwards.

Set side by side, the two styles compare like this:

  • Speed of the first action. The hunch wins by a few minutes. The labelled start wins on the speed of the correct action.
  • Evidence preserved. The labelled approach changes nothing until it has a reason; the hunch approach starts altering things at once.
  • Team coordination. A shared note lets a newcomer join in a minute. A chat full of competing stories takes far longer to read.
  • Review quality. The note becomes the timeline. The hunch approach has to reconstruct one from memory.

What this has to do with careers

Engineers are rarely promoted for being the fastest guesser. They are trusted with bigger systems when colleagues learn that their statements can be relied upon: “confirmed” from them means confirmed. That reliability is quiet and cumulative, and it appears in almost every account of how an engineer progresses from entry-level to senior DevOps work, even when it is not named directly. Likewise, talk of new career pathways for modern software developers usually assumes that people can reason calmly across the boundary between building and operating a system.

Why the hunch is so appealing

Pattern recognition is a real skill, and experienced people are often right. The risk is that being right several times builds a feeling of certainty that outruns the evidence. The first engineer’s belief about the disk was not foolish; it was a reasonable guess that was treated as a fact. The labelled approach keeps the pattern recognition but demotes its output to a hypothesis: “this resembles the old disk issue, which predicts that free space is low on the affected hosts.”

That is a gentler habit than it sounds. It does not ask anyone to ignore experience. It asks them to say aloud which part is experience and which part is observation.

What the team lead noticed

Reading both sets of notes the next morning, the team lead remarked on something small. The first engineer’s messages were full of verbs of action, such as cleared, restarted, retried. The second engineer’s were full of verbs of observation, such as checked, compared, confirmed. Neither style is wrong in itself, but only one leaves behind a trail that a stranger could follow six months later, and only one makes it obvious where the next question should go.

Building the habit deliberately

The simplest start is a template, kept somewhere easy to reach, with the three headings and a line for the next check. Use it on small, boring alerts first, so that by the time a serious one arrives you are not learning the technique under pressure.

Another way is to practise on old incidents. Take a past report and re-sort its statements: which were actually checked at the time, which were beliefs and which were hopes. A learning platform can supply material for that practice; DevOpsSchool is one, presenting itself as a learning platform for engineers with courses, certification tracks, a tools section and quizzes, which gives a place to practise testing a claim before trusting it.

The ending of the story

The second engineer found that a downstream certificate had expired, replaced it with the help of that team and closed the incident within the hour. The first engineer’s channel was still debating the disk when the news arrived. In the review the next day nobody blamed the first engineer. The team simply asked both of them to share their notes, and the contrast taught more than a slide deck could have.

The lesson is small and portable. When something breaks, a hunch is a perfectly good place to start and a poor place to stop. Label it, test it, and let the facts, not the loudest memory in the room, decide what to change.