Four engineers solved the same outage four times in one week

One team. One service. One alert that goes off four times in the same week. The alert is named after the database, so the dashboards they check all point there. The database is not the problem. Four people work that out separately, each in the middle of the night, each for the first time.

A handful of engineers taking turns with the pager, the on-duty phone that goes off when the service breaks. And a page on the company wiki they call the runbook, the how-to for when this goes wrong. Now put an AI agent on that rotation, taking the first look at each alert. By the end of this you will be able to show someone exactly why this team stopped restarting the database, and where that answer came from. Because here is the part nobody tells you. What one night's responder works out, the next one does not have.

And the written record ages while nobody watches.

Tuesday, three in the morning. Whoever answers the phone restarts the database. It holds for forty minutes. Thursday. Same alert, different person, same restart. Nobody knows Tuesday happened, because Tuesday happened to somebody else.

The runbook is a wiki page last edited eight months ago, whose first step opens a screen that no longer exists. Sunday, somebody answers the phone and asks the obvious question. Has this happened before. They get a shrug, a wiki page, and a link to a chat thread.

Run the same week again with one change. The team gets a shared notebook. The agent on that rotation reads it and writes to it, the people can too, and it outlives the night. The only moment anyone knows the answer is the moment nobody wants to type, so the thing that is already awake does the typing. Tuesday, three in the morning. The agent works the alert, and when it is over it writes down what went wrong, what it was doing, what it tried that did not work, and what it recorded as the fix.

That third one is what a tidy write-up loses, and the one the next person needs. Thursday, ten to three. Same alert. Before touching anything, the agent asks the notebook one question. Has this happened before, and what happened. Back come the notes that match, carrying whatever was filled in on the night.

So it skips the thing that looks right and goes to the thing that worked. The limit is real. The match runs on the name the trouble was filed under, and on how closely the description reads against the old ones. File it under a different name and Tuesday's note does not come back. A recorded fix is history, not a promise. The agent tries it and it holds.

The alert goes off twice more that week. By Sunday there are four notes saying much the same thing. So the notes get grouped, and the grouping hands back one suggestion, with a count and the dates of the first and last night. It does not write your runbook. A person reads it and the four nights underneath, then writes the procedure and approves it. The four notes stay where they are.

Here is the quiet part. The old wiki had forty runbooks and no way of knowing which three were any good. In this notebook, opening an entry is what counts in its favour, when the credit lands. Being offered does not count. Reading is the vote. So the pages the team uses come forward and the ones nobody opens go quiet.

Eight months ago somebody who has since left wrote one sentence worth more than the whole wiki. Do not restart the main database before the backup has finished taking over. On most teams it would sit in a document nobody opens. Here it sits with the team's own written rules, and underneath it, in plain words, what the risky move looks like. At the start of a shift, the agent asks for that list and reads it. The list does not stop anybody.

It gets put in front of whoever is about to act, rather than turning up at the meeting afterwards. The claim is not that it was prevented. It is that the sentence was still reachable, from somebody who left.

Monday morning. Somebody new answers the phone, and has not seen this alert before. They ask what Sunday's responder asked. Has this happened before. This time the answer is not a shrug. The procedure a person approved on Sunday comes back among the ranked answers.

Then the team's own software follows the four references it carries and fetches each night separately, with what went wrong, what was tried, what did not work, and when. Later that morning, at the incident review. Why did we stop restarting the database. The answer is on the record, and it does not depend on who is still on the team when the question gets asked. Be honest about what that proves. What was tried, what was written down, when.

Not that the fix was right. It is a claim a tired person made at three in the morning, not a verdict. Every runbook you have trusted was somebody's bad night. This one still has the night attached.

Nothing about this team got smarter. Thursday asked, Tuesday's note came back, the recorded fix held. That is what SmartMemory is. A memory your on-call agents share, that outlives the shift, the thread, and the goodbye. If you want to see what gets written at three in the morning, and what comes back on Thursday, that is Part 2. This was the week the person answering stopped starting from scratch.