Building an Organizational Failure Memory for CI/CD

An idea I've been turning over for a while, framed honestly: I don't have a finished product here. What I have is a recurring frustration from years of running CI/CD at scale in a large-scale financial services environment and again at an enterprise manufacturing company. The same pipeline failures keep happening. Not literally identical — the surface details differ — but the underlying pattern is something the team already debugged six months ago. The fix sits in a postmortem doc nobody reads at the moment of failure.

Postmortems don't prevent repeat incidents

This is uncomfortable to say out loud because postmortems are sacred in SRE culture. They should be. But the empirical reality, after watching dozens of them across multiple orgs: a postmortem is read carefully once, skimmed twice, and then it lives in a wiki nobody searches. The action items get tracked for about a quarter. The institutional knowledge — the why-this-happened, the subtle clue that gave it away — evaporates with the people who were on the call.

Six months later, a similar failure happens to a different team. They debug it from scratch. The postmortem they need is two clicks away, but they don't know to look for it because the failure looks different on the surface. Same root cause. New 40-minute outage.

What "failure memory" would have to do

To actually prevent repeats, the system has to do the work the human won't. Specifically:

• Match by signature, not by title. Postmortems are titled by symptom ("payments API 502s"). The recurring thing is the underlying cause ("DNS TTL expiry on the egress proxy during a deploy window"). Title search is useless for matching across symptoms.

• Surface at the moment of failure, not after. A link in a retro doc is too late. The relevant prior incident has to appear in the failing PR's check output, or in the on-call channel as the page is being acknowledged.

• Be wrong gracefully. If it surfaces three past incidents and only one is genuinely relevant, that's still a huge win. If it surfaces zero, it should say so plainly and not pretend.

[IMAGE: Diagram showing a new pipeline failure being matched against a database of past failure patterns]

A rough technical sketch

I'd build this in three pieces, none of which are novel on their own — the value is in the wiring:

1. Structured ingestion of past incidents. Postmortems get a lightweight schema: failing stage, error class, top-N log lines around the failure, root-cause tag, fix summary. Free text is fine but the structured fields are what makes retrieval tractable. Most orgs already have most of this; it's just not in a queryable shape.

2. Embedding store keyed by failure signature. For each incident, embed the concatenation of (failing stage + error class + canonical log snippet). Store the embedding in a vector DB (pgvector is enough for any org with fewer than a million past incidents). The embedding model matters less than people think — a general-purpose code-and-text embedding does fine here.

3. Query at failure time. When a pipeline fails, the CI runner extracts the same shape (stage, error, log snippet), embeds it, and does a top-K similarity search. The top hits get posted as a comment on the failing build with a one-line summary and a link to the postmortem.

[IMAGE: Simple flow showing postmortem data feeding into a searchable failure memory system]

Where this is harder than it looks

Three honest problems I haven't fully solved on paper:

• Log noise. CI logs are 90% irrelevant output. The "canonical log snippet" extraction has to be smarter than "last 50 lines" or the embeddings collapse into uselessness. Stage-aware extraction (only the lines after the failing step starts) helps a lot.

• Feedback drift. Past incidents from two years ago may describe a system that no longer exists. The store needs a freshness signal — either explicit ("this postmortem is for the legacy pipeline") or implicit (decay relevance scores over time).

• The "obvious match" trap. If the system fires on every flaky-test failure with "here are 47 past flakes," it becomes noise instantly. The match has to be selective — same-or-similar-cause selective, not same-error-message selective.

Where I disagree with the usual framing

The conventional pitch for "ML for incident response" is prediction — forecast that an incident is about to happen. I think that's the wrong place to start, for reasons I went into in Why Most AI for DevOps Tools Fail in Production. Recall is the easier, more honest problem: not "predict the future" but "remember the past." Memory is a much lower bar than prediction, and the value to a 3 a.m. on-call engineer is arguably higher.

This is the direction my own ongoing research is heading — adjacent to the pipeline-failure-prediction work I described in What I Learned Building an ML Model to Predict CI/CD Pipeline Failures. For background on similarity search at the scale this would need, the FAISS wiki is a good grounding read.

It's an idea, not a product. If someone has already built the version of this that handles the noise problem well, I want to hear about it.