Every engineering team knows the ritual. A regression suite goes red on the QA pipeline, and someone has to stop what they’re doing, open the Jenkins build log, read through a wall of stack traces, cross-reference it against the source code, figure out whether it’s a real bug or a flaky test, and then hand-craft a JIRA ticket with all the right fields.
It takes 15–30 minutes per failure. On a bad build with five or ten reds, it swallows an entire morning — and the bugs that come out are often vaguely worded, misclassified, or missing the one detail that would actually help the developer fixing them.
So I built something to make that ritual disappear.
“60 seconds from failure to filed bug. 30× faster than manual triage. 108 hours saved per project per year.”
What AI Bug Analyzer Does
AI Bug Analyzer plugs into a Jenkins pipeline as a postStage hook. When an integration or regression stage fails, it springs into action — completely in the background, and without ever affecting the build result. Within about a minute it has:
- Collected the failures — pulls every failing test from the Jenkins Test Report API (JUnit / pytest results).
- Found the relevant source code — parses stack traces across Python, Java, JS/TS and Robot Framework, extracts keywords, and fetches the real application & test files from Bitbucket.
- Asked Claude to reason about it — sends the failure plus code context to
Claude Opus 4.6on Azure AI Foundry with a prompt that understands the bug-filing playbook. - Got a precise diagnosis — root cause pinpointed to
file:line, a suggested before/after code fix, severity, priority and a confidence score. - Filed the JIRA bug — creates a fully-structured ticket with every required field — or, if a matching bug already exists, comments on it instead of creating a duplicate.
The result: instead of “investigate the failing test,” the developer opens a ticket that already says exactly which line is wrong and what the fix looks like.
The Hard Part: Giving the AI the Right Context
An LLM is only as good as the context it gets. Handing Claude a raw stack trace produces a generic guess. The real engineering challenge was building a smart code-fetching pipeline that finds the handful of files that actually matter — out of a repository with thousands.
The system uses a tiered discovery strategy and a strict token budget so it never overwhelms the model (or the bill):
- Tier 1 — Stack trace resolution. Exact
file:linereferences, including resolving Java packages (com.jnj.its→com/jnj/its/…) against the live repo file tree. - Tier 2 — Keyword-matched application code. CamelCase class names, API path segments and test-name tokens are matched against
src/mainsource files. - Tier 3 — Test code. The test files themselves, so Claude can tell an application bug apart from a test-script defect.
Each tier gets its own character budget (roughly 60,000 characters total — about 12% of the 200K context window), with application code always prioritised over test code. That single design choice is what pushes the analysis from a “medium confidence” pattern-guess to a “high confidence” answer with real code evidence.
How It’s Built
The tech stack is deliberately lean:
- Azure Functions (Python 3.11) — serverless, zero cost when idle
- Azure AI Foundry + Claude Opus 4.6 — the reasoning engine
- Managed Identity — no API keys to leak, ever
- Bitbucket REST API — smart code fetching from the actual repo
- JIRA REST API — fully-structured ticket creation with duplicate detection
- Bicep IaC — entire infrastructure from a single template
- Application Insights — telemetry for every analysis run
- Jenkins (Groovy postStage) — the trigger point
Key Design Decisions
- Zero credential exposure — passwords, tokens and secrets in error messages are automatically redacted before anything reaches the AI.
- Zero pipeline impact — runs as a postStage hook. Every error is caught and logged as a warning — the build result is never touched.
- Language-agnostic — understands Python, Java, JavaScript, TypeScript and Robot Framework stack traces out of the box.
- Duplicate-aware — a JQL search catches existing bugs and comments on them instead of spamming new tickets.
Onboarding a Project Takes One Line
Because the whole thing is driven by an external manifest, adding it to any Jenkins project is a single entry:
# manifest.yaml — postStage section
postStage:
integrationTest:
- userScript:
name: ai_bug_analysis
externalRepoUrl: https://.../ai-bug-analyzer.gitThe Impact
Numbers tell the story best. Here’s the before-and-after for a single failing test:
- Time per bug: 15–30 minutes → ~60 seconds
- Root cause: varies by engineer → pinpointed to file:line
- Playbook compliance: inconsistent → 100% of fields populated
- Corrective action: often generic → specific fix + verification steps
- Duplicate detection: manual search → automatic via JQL
- Credential exposure: human error risk → zero (auto-redacted)
At a typical rate of five failures a week, a single project loses about 108 hours a year to manual bug filing. Across ten projects that’s over a thousand engineering hours — roughly $100K+ of time — handed back to the people who should be building, not triaging.
What I Took Away from Building It
The interesting lesson wasn’t the model — it was the plumbing around it. A capable LLM like Claude Opus 4.6 will happily reason its way to the exact broken line if you do the unglamorous work of feeding it the right code, keeping secrets out, and structuring the output so a human (and JIRA) can trust it.
The AI is the easy 20%; the context engineering, security, and CI/CD integration are the 80% that make it production-ready.
“60 seconds, not 30 minutes. file:line, not ‘investigate further’. 100% compliant, not ‘mostly there’.”