How Instacart's Blueberry AI Helps On-Call Engineers Fix Production Faster

When On-Call Starts with a Firehose

If you've ever been the on-call engineer for a large distributed system, you know the drill. An alert fires at 2 a.m. You open Slack, see a bunch of messages, and then the real work begins: figuring out which service is affected, who owns it, what changed recently, and where to look in the logs. That context gathering can eat up an hour or more before you even start diagnosing.

Instacart decided that had to change. The company built an AI assistant called Blueberry that sits in the same Slack channels where incidents are already being handled. It watches what's happening, pulls context from internal systems, and proposes root causes—all while the engineer stays in the conversation. The result: what used to take an hour of context collection now takes about three minutes for an initial assessment.

What Blueberry Actually Does

Blueberry is not a chat bot that gives generic advice. It's an AI agent system designed for production incident response. When an alert triggers, Blueberry spins up roughly ten sub-agents in parallel. These agents check service ownership data, review recent deployments, scan logs and metrics, hunt through documentation, and compare the current symptoms against more than fourteen years of incident history.

All that information gets synthesized into a root-cause hypothesis, which Blueberry posts directly into the Slack thread where the engineer is already working. The engineer doesn't have to leave the channel or switch to a different tool. The system handles the information gathering and hypothesis generation, but it doesn't make any changes to production. That's still the engineer's call.

Numbers That Make You Look Twice

Instacart shared some impressive metrics. In April alone, Blueberry ran about 25,000 diagnoses across more than 270 Slack channels. The diagnostic accuracy rate jumped from 60% to over 90%, and the company credits that improvement to the system's access to its long history of incident data as factual grounding.

The system also achieved a 99.9% workflow success rate, with over 58,000 MCP tool calls, and it adapted to the working styles of about 60 different teams. Those numbers suggest Blueberry isn't just a cool demo—it's being used every day by people who need it.

The Architecture Behind the Assistant

Instacart's engineering team built Blueberry on a few key layers. At the top, there's the agent itself, which coordinates the sub-agents and keeps track of what's been tried. Beneath that, the system integrates with internal resources through a tool-aware framework. It uses the Model Context Protocol (MCP) to connect to services like incident history, service ownership, logs, deployment pipelines, and other debugging signals.

One of the design decisions that stands out is persistence. Blueberry doesn't just fire off queries and forget. It maintains state about the incident as it progresses. That way, if the conversation moves or new information comes in, the system can update its context and keep the diagnosis on track.

There's also a feedback loop. After an incident is resolved, the system can learn from what happened, incorporating the new case into its knowledge base. That's how the accuracy improves over time—not by retraining the language model, but by expanding the operational memory it can draw on.

Why This Matters for Full-Stack Development

You might be thinking, "I'm a full-stack developer, not an SRE. Why should I care?" But Blueberry is a glimpse into how AI is changing the way we build and operate software. For full-stack teams, the boundary between development and operations is already blurry. Many of us are on call for the systems we build. Tools that reduce the cognitive load of incident response are directly relevant to our daily work.

More importantly, the principles behind Blueberry apply to any complex workflow. It's not just about throwing a large language model at a problem. It's about connecting that model to the specific context of your organization—your services, your logs, your history, your team's way of doing things. That's a lesson that translates to building internal tools, automating deployments, or even just making your own debugging more efficient.

What Full-Stack Developers Can Learn from Blueberry

For developers who want to build similar AI-assisted tools, there are a few takeaways from Instacart's approach. First, context is everything. A generic AI assistant is only as good as the data it can access. Blueberry's accuracy jumped from 60% to 90% when it was grounded in 14 years of incident history. That's a huge signal.

Second, integration matters more than raw model power. Blueberry works because it's embedded in the existing Slack workflow. Engineers don't have to learn a new interface or change their habits. The AI comes to them, not the other way around.

Third, keep the human in the loop. Blueberry provides hypotheses, but it doesn't take action. That's a smart design choice. AI can suggest, but humans should decide. This is especially important in production environments where mistakes are costly.

The Challenges of Building an AI Incident Responder

Of course, building something like this isn't easy. The biggest challenge is making sure the AI's suggestions are actually grounded in reliable facts. If the system hallucinates a root cause, it could send engineers down the wrong path, wasting valuable time during an outage.

Instacart addressed this by making Blueberry tool-aware. It can call APIs to retrieve real data, and it keeps track of what it's already checked. This reduces the chance of it making things up. But it also means the system is only as good as the tools it can access. If a service ownership database is out of date, Blueberry might point to the wrong team.

There's also the question of trust. Engineers need to feel comfortable relying on AI suggestions. That comes from seeing the system work correctly over time, and from knowing that the final decision is always theirs. Instacart's engineers apparently trust it enough to use it tens of thousands of times a month.

The Future of AI in Production Operations

Instacart's chief technology officer, Anirban Kundu, describes Blueberry as part of a broader exploration of AI agent systems. The company is not just building a one-off tool; it's thinking about how AI can augment its engineering workforce in many ways.

For the rest of us, the takeaway is clear. AI is becoming a practical tool for full-stack development and operations. It's not science fiction anymore. It's being used right now to help engineers respond to incidents faster, and the principles behind it—grounding AI in real data, integrating with existing workflows, and keeping humans in charge—are things we can all apply.

Whether you're building a full-stack app or running a massive e-commerce platform, the ability to diagnose problems quickly is a superpower. Blueberry shows that AI can help us get there, but only if we design it with the right context, the right tools, and the right boundaries.