Build Failure Triage — an Agent Skill over Jenkins

DRAFT — pending Harsh's verification and employer sign-off.

This page documents the build-failure triage capability on Memex — Auction.com's internal AI coding-agent platform, built on the pi-coding-agent framework and deployed on EKS. The capability is the work Harsh is proudest of in the agent-tooling space, and the one that most clearly shows the pattern this site is built to demonstrate: codifying an expert's implicit procedure into something a model can execute deterministically.

1. Two cooperating layers

Triage is built as two cooperating pieces, and the split is the whole point:

1. A read-only Jenkins connector — the tool layer. It wraps the Jenkins REST API as a set of narrow tools the agent can call: get build metadata, get the pipeline-stage breakdown, tail the last N log lines, regex-search the log with context, and list recent builds. No write actions — triage only. Counting these together with the skill layer's operations, the extension exposes 11 tools to the agent; the five named here are the core build-navigation tools detailed below.
2. A triage skill — the procedure layer. A markdown runbook that encodes the decision tree an experienced SRE follows: resolve the build → quick-exit if green or running → find the failed stage → categorize the failure against a signal table → gather targeted evidence → emit a structured report.

The tools give capability; the skill gives judgment. The model supplies only the reasoning glue between them — deciding which tool fires next, given what the previous one returned.

        ┌──────────────────────────────────────────┐
        │               Agent (LLM)                │
        │   reasoning = "which tool next, given    │
        │   the last result, per the skill"        │
        └───────────────┬──────────────┬──────────┘
        tools (capability)            skill (judgment)
            │                            │
            ▼                            │
   ┌─────────────────┐                   │
   │  Jenkins REST   │◀──── overlay ────┘
   │  API (read-only)│   (the skill tells the model
   └─────────────────┘    which tool to call, in what order)

The deliberate rejection here is the "one mega-tool" approach — a single triage_this_build call that hides every API interaction. Narrow tools keep the model's reasoning visible (every call is an auditable step) and let the skill compose them differently for different failure shapes. The model is explicitly forbidden from dropping to bash/curl directly; tools only, so behavior is consistent and replayable.

2. The tool layer: narrow, read-only, defensively coded

The extension exposes 11 tools to the agent across its two layers — the read-only Jenkins REST tools (the tool layer) plus the skill layer's operations. The five core build-navigation tools described in detail below are a subset of that full 11-tool surface; the remaining tools are not detailed in this page. The connector itself is narrow and read-only — zero write actions. Those five named tools are: build metadata, the pipeline-stage breakdown, log tail (last N, default 200), regex log search with ±N context lines, and list-recent-builds. Two implementation details matter because they recur across the whole platform:

• Job-path normalization as a pure helper. One function normalizes any Jenkins reference the user or model might type — a full URL, slash shorthand (service-pipeline/main), a flat name, with or without a trailing build number — into the API's /job/X/job/Y form, and extracts the build number if the last segment is numeric. Every tool shares it, so the model can pass whatever form it was given without a formatting step of its own.
• Defensive HTTP against the auth-HTML trap. When auth is misconfigured the API returns an HTML login page — not JSON, but HTTP 200. The helper captures the real status code out-of-band (-w __HTTP_STATUS__:%{http_code}), then hard-fails on any non-200 and on any non-JSON body, returning a truncated snippet instead. The model gets a clear "this was not JSON, here's what came back" error rather than trying to parse a login page into a failure category.

Reusable pattern: for agent tools over a remote API, normalize inputs in a shared helper and fail loudly on shape mismatches. A 200 with an HTML body is a real failure mode — surface it, don't let the model silently reason over a login screen.

3. Structured data before raw logs

The skill's first hard rule is: call the pipeline-stage breakdown first, and only fall back to raw console logs when stages aren't available. A stage list is a few hundred tokens and names the failure location directly; a full console log is tens of thousands of tokens of mostly-noise. The reasoning quality per token is dramatically higher on the structured view, and the failure location is given to the model rather than buried in a log it has to scan.

This is the same instinct as "look at the failing stage name before you scroll the log" — except encoded as a mandatory first step so every triage run pays the cheap cost first and the expensive cost only if the cheap one was inconclusive.

4. Bounded-log "tokenomics" — never dump a full log

The connector never offers "fetch the whole log." It exposes tail (last N, default 200 lines) and regex search with ±N context (default 5, with a max_matches cap). Truncation is explicit — the tool tells the model there is more log and how to get it (run a search), rather than silently clipping.

This was a deliberate rework. An "Improved tokenomics" commit restructured both the extension and the skill around bounded, targeted log access instead of bulk retrieval. The rejected alternative — bulk-log ingestion plus "let the model find the error" — loses on both cost and signal-to-noise: you pay for tens of thousands of tokens to surface a three-line stack trace.

Reusable pattern: for agent tools over large artifacts, expose tail + search + pagination — never "get all." Make truncation visible and tell the model the next move. The smallest useful slice first; the model asks for more only if it needs to.

5. The signal→action table — a runbook encoded as a lookup

Failure categorization is a table, not prose: category → signals → regex search pattern. The model matches observed log signals to a row, then runs that row's exact search pattern to gather evidence. The taxonomy covers twelve first-level categories:

1. Compilation / Build
2. Test Failure
3. Dependency / Artifact
4. Infrastructure (including OOM)
5. Deployment
6. Timeout
7. Permission / Credential
8. Approval Gate
9. Parameter Validation
10. Configuration
11. Quality Gate
12. Unknown

Beneath the table sit stack-specific known-signature diagnoses — short "if you see X, the fix is Y" entries, e.g. "Node heap limit exceeded → raise --max-old-space-size," or "coverage below threshold → check the skip flag or add tests." These are the fingertip diagnoses an experienced SRE reaches for without thinking; the table makes them addressable by row.

This is the core move of the whole design. An expert's triage procedure is mostly pattern-match then act — "I see this signal, so I go look at that." A signal→action table converts that implicit pattern-match into something a model executes deterministically, run after run, across model versions. The table is the part that survives a model swap; the model only supplies the glue.

Reusable pattern: a signal→action table converts an expert's pattern-match into something a model executes deterministically. This is the core "codify expert procedure as a skill" move — and it's the part worth keeping in version control, not in a prompt.

6. The report contract and the quick-exit ladder

Two things keep the output trustworthy and cheap:

• Quick-exit ladder. A green build yields a one-line summary and stops; an in-progress build reports progress and stops. Only failed, unstable, or aborted builds proceed to the full tree. Cheapest path first — most builds are green, and the agent shouldn't burn a full triage on them.
• Fixed report contract. Every report has the same shape: status + failed stage + category, duration + trigger, a root cause in one to three sentences written for an SRE audience, the evidence as log lines in a code block, one specific recommendation, and a deep link to the build. A fixed shape makes output skimmable and pasteable straight into a ticket — and it makes "the agent skipped the evidence block" an obvious, checkable failure.

7. What this codifies, and the honest gaps

Taken together, the design is an SRE runbook made executable: structured data first, bounded log access, a lookup-table taxonomy, and a fixed report. The skill is the imagination layer — the encoded judgment — and the tools are the capability layer. Swapping the model changes only the glue.

The triage system runs inside the broader CI/CD footprint Harsh operates: an in-house Jenkins + GitHub Actions toolchain (replacing a six-figure-per-year SaaS suite) running 10K+ monthly pipeline executions, on a cadence that moved from bi-weekly to daily releases with a <1% rollback rate.

One honest gap. No measured impact is recorded in the source — there is no before/after triage-time number, no adoption count, no median time-saved figure. A metrics-backed version of this story would need those pulled from real usage, and they are not in the material this page was distilled from.

Sources

This page was distilled from Harsh's commit history and session logs on the Memex platform — the Jenkins connector and the triage skill — cross-checked against the in-repo PRD and resume. No internal hostnames, credential identifiers, or proprietary source code appear; the design is described, not copied. Every architectural claim above traces to that material.