Debug Log

From Vibe-Coded to Enterprise: Handing the Pager to Claude

April 03, 202618:24Debug Log

This episode explores Incident.io's new remote Model Context Protocol (MCP) server, which enables AI assistants like Claude to directly access and interact with live production incident data. Listeners will learn how this "USB-C for AI" standard aims to reduce "dashboard fatigue" and streamline incident response by providing consolidated information, while also considering the potential trade-offs regarding deep system understanding and the "vibe-coded" origin of the technology.

Key Takeaways

Detailed Report

AI Takes the Pager: Incident.io's Leap into Autonomous Incident Response

Imagine a 3 AM pager alert for a Sev-1 incident, but instead of frantically navigating multiple dashboards, you simply ask an AI assistant, "Hey Claude, what's going on with the payment gateway service and who's on call?" This scenario is now a reality thanks to Incident.io's new remote Model Context Protocol (MCP) server, a significant step towards integrating AI directly into live production incident management.

The Model Context Protocol: A Universal AI Adapter

Launched officially in March 2026, Incident.io's remote MCP server (`https://mcp.incident.io/mcp`) is being hailed as the "USB-C for AI." This analogy highlights its function as a universal standard, allowing various AI clients—like Claude Desktop, Claude Code, or Cursor—to directly access and interact with incident data. Prior to MCP, every AI assistant required bespoke integrations for each tool, creating a complex and inefficient ecosystem.

The MCP server translates natural language queries into the correct REST API calls, fetches data from Incident.io, and formats it for the AI to present. This abstraction is crucial for combating "dashboard fatigue," a common problem where engineers juggle numerous UIs (Datadog, PagerDuty, Jira, Slack, etc.) during high-stress incidents. By consolidating information and enabling direct querying, MCP promises psychological relief and faster root cause analysis.

From "Vibe-Coded" to Enterprise-Grade

The journey to this enterprise-ready solution has a unique origin. The first version, a local Golang MCP server, was released in August 2025. This prototype was not part of Incident.io's official product roadmap but was "vibe-coded" by a CMO and an engineer who simply wanted it to exist. "Vibe-coding" describes the rapid, intuitive development of software, often leveraging AI generation tools, outside traditional rigorous engineering practices.

Despite its "unsupported" and "vibe-coded" label, the prototype gained significant traction, proving market demand. However, its architecture—requiring local setup with GitHub repos, `.env` files, and raw API keys—was unscalable and unsuitable for large organizations. Customer feedback highlighted the need for a fully managed, remote solution, leading to the March 2026 release that offers zero local setup and deprecates the original Golang implementation.

This product pipeline demonstrates a "hyper-agile" approach: rapidly prototyping with AI, validating with the community, and then properly productizing if successful. While efficient for innovation, it raises questions about how initial "vibe-coded" tech debt is managed in the long run.

Security, Accountability, and the Human Proxy

A critical concern with giving LLMs read/write access to production systems is security. The initial prototype's reliance on raw API keys with potential admin privileges posed a massive risk. The March 2026 remote release addresses this with user-based OAuth authentication.

When a human connects via an AI assistant, they authenticate with their Incident.io account, and actions are attributed to their user account, inheriting their exact permissions. This means the AI isn't an autonomous agent but a direct proxy for the human. While solving the authorization problem, this creates a terrifying accountability dilemma: if an LLM hallucinates and takes an incorrect action (e.g., closing a Sev-1 incident), the audit log will show *you*, the human engineer, as responsible. This trade-off between convenience and the precision/safety of rigid UIs with confirmation dialogs is a significant challenge.

Shifting Paradigms in Incident Response

The advent of MCP servers signals a potential end to the era of Slack and ChatOps as the undisputed command center for incident response. While ChatOps required memorizing specific slash commands (e.g., `/incident declare`), AI tools leveraging MCP allow natural language interaction. Slack may evolve from a command center to more of a notification board, with the actual querying and action-taking moving into the LLM context window.

This shift also enables powerful IDE-driven operations, where engineers can create incidents or page teams directly from their code editor. Furthermore, integrations like "MCP Client by Zapier" allow chaining AI tools within workflows, enabling broader automation beyond single chat windows.

The Future: Automated SRE and Automation Complacency

The remote MCP server is foundational plumbing for fully automated Site Reliability Engineering (SRE). Incident.io is actively building an "AI SRE" agent, aiming to investigate incidents, diagnose issues, and eventually offer or execute fixes autonomously. Companies like Block are already deploying agents like "goose" for automated triage and root cause analysis.

However, this future also brings the risk of "automation complacency." If AI agents consistently provide neatly formatted summaries and suggest mitigation scripts, engineers might lose their operational "muscle memory" and the intuitive understanding of underlying systems. Blindly trusting an AI's conclusion without verifying raw logs could lead to catastrophic errors if the AI hallucinates, potentially skyrocketing time to resolution and incurring significant costs. The ultimate test will be how engineers balance the convenience of AI with the critical need for human verification during high-stakes incidents.

Show Notes

Works Referenced

  • Leverage Agentic AI for Autonomous Incident Response with AWS DevOps Agent: The foundational article discussing agentic AI for autonomous incident response.
  • Incident.io: A platform for incident management and response, central to the episode's discussion on AI integration.
  • Model Context Protocol (MCP): A universal standard developed by Anthropic that allows AI assistants to securely connect and interact with various operational tools and data sources.
  • Claude AI Assistants (Desktop, Code): AI assistants by Anthropic, capable of integrating with MCP servers to perform incident response actions.
  • Cursor IDE: An AI-native code editor that can integrate with incident management systems via MCP.
  • Datadog: A monitoring and analytics platform for cloud applications, often used during incident response.
  • PagerDuty: An on-call management and incident response platform for alerting and escalation.
  • Jira: Issue tracking and project management software, frequently used to manage incident tickets.
  • Retool Engineering Blog: A source of engineering insights, mentioned for its warning about the challenges of 'vibe-coded' applications.
  • Zapier: An automation platform that can connect AI tools and workflows using MCP servers.
  • Block (company): A technology company mentioned for developing 'goose,' an agent for automated triage and root cause analysis.
  • Notion: A workspace tool for notes, tasks, and documentation, useful for drafting incident post-mortems.

Glossary

  • Sev-1 / Sev-0: Short for 'Severity 1' or 'Severity 0,' indicating the highest priority and most critical incidents with significant business impact.
  • LLM: Large Language Model; an AI program trained on vast amounts of text data, capable of understanding and generating human-like text.
  • Model Context Protocol (MCP): A universal standard that allows AI assistants to securely connect and interact with various operational tools and data sources.
  • Vibe-coded: A term describing software built rapidly and intuitively, often with AI generation tools, without traditional rigorous engineering practices.
  • Dashboard Fatigue: The mental exhaustion experienced by engineers from constantly switching between multiple monitoring dashboards during an incident.
  • ChatOps: A collaborative approach to IT operations where tasks are executed using commands within a chat platform like Slack.
  • Hallucination (AI): When an AI model generates information that is plausible but factually incorrect or inconsistent with its training data or context.
  • Automation Complacency: A state where over-reliance on automated systems leads to a decrease in human vigilance, critical thinking, or skill retention.
  • OAuth: An open standard for access delegation, commonly used to allow users to grant websites or applications access to their information on other sites without giving them their password.
  • Agentic AI / AI Agent: An AI system designed to perceive its environment, make decisions, and take actions autonomously to achieve specific goals.

Sources / References

Full Transcript

HostOkay, so picture this: it’s 3 AM, your pager just went off, a Sev-1 is blowing up, and you're staring at five different dashboards, trying to figure out if it’s the database, the network, or that last-minute deploy from engineering. You’re bleary-eyed, fingers fumbling.
ExpertClassic. Been there, got the therapy bills to prove it.
HostExactly. Now imagine, instead of that, you just type into a chat window, "Hey Claude, what's going on with the payment gateway service and who's on call?" And it just… tells you. No clicking, no context switching. Just answers.
ExpertAnd it’s not just telling you, it’s actively *doing* things. Like, handing the actual pager to an LLM. That’s the wild part. I saw that announcement from Incident.io about their remote Model Context Protocol server and my jaw just about hit the floor.
HostYeah, "From Vibe-Coded to Enterprise: Handing the Pager to Claude" – that's the headline we're dissecting today. Because on March 31st, Incident.io officially rolled out something that feels like science fiction but is very much real: a remote server that lets AI assistants tap directly into your live production incident data.
ExpertAnd not just "tap in," but query incidents, analyze alerts, check schedules, manage escalations. Essentially, becoming your co-pilot, or maybe even your primary pilot, during an incident. It’s a huge shift.
HostIt is. This remote MCP server, hosted at `https://mcp.incident.io/mcp`, is essentially what they're calling the "USB-C for AI." Which, you know, makes sense as an analogy. Just plug in your AI client – Claude Desktop, Claude Code, Cursor – point it at this URL, and boom, your LLM now has access to all your incident data.
ExpertThe "USB-C for AI" is actually a fantastic analogy, because what Anthropic did with the Model Context Protocol back in late 2024 was create a universal standard. Before this, every AI assistant needed its own bespoke integration for every tool. It was a nightmare. MCP acts like a universal adapter, making it plug-and-play.
HostSo, when you ask Claude, "Show me all critical incidents from the last week," the MCP server translates that natural language into the correct REST API calls, fetches the data from Incident.io, and formats it for Claude to present back. It's not magic, it’s just incredibly clever abstraction.
ExpertAnd that abstraction is the key to killing what they call "dashboard fatigue." We’ve all been there, right? Sev-0, Slack's blowing up, you've got Datadog, PagerDuty, Jira, your internal wiki, and Incident.io all open, and you're just frantically flipping between tabs, trying to piece together a coherent picture.
HostOh, the psychological relief of not having to manually click through ten different UIs at 3:00 AM to figure out who's on call for the payments gateway. That sounds like a dream. No more "Wait, is this the right dashboard? Did I refresh?" Just ask Claude directly.
ExpertAbsolutely. Imagine asking, "How many P1 incidents did we have this quarter, and which teams were most affected?" or "What's the trend in overnight responder workload by alert source?" and getting an immediate, synthesized answer. That's a game-changer for incident post-mortems alone, let alone active incident response.
HostBut here's the thing that gnaws at me a little. If the AI is giving you this beautifully summarized, consolidated view, and you're not clicking through the dashboards yourself, are you losing that intuitive understanding of how the underlying systems are wired together? That muscle memory of where to look, what each metric means, how the pieces fit?
ExpertThat's a really good question. It's the classic trade-off, isn't it? Convenience versus deep understanding. I mean, do you really understand how your car works when you press the "start" button? Probably not as well as someone who had to crank it by hand. But it gets you where you need to go faster. The worry is when it *doesn't* get you where you need to go, and you don't have the fundamental knowledge to debug it yourself.
HostRight. And this entire journey, this enterprise-grade remote server we're talking about, has a fascinating origin story. It didn't just spring fully formed from a strategic planning session.
ExpertOh, the "vibe-coded" part of the title. That's one of my favorite bits. The first version, a local Golang MCP server, was released in August 2025. And it wasn't even on Incident.io's official product roadmap!
HostNo, it was built by their CMO, Tom, and an engineer, Chris Evans, because they "wanted it to exist, not because we had a roadmap requirement or huge customer demand." They just… built it. And they candidly slapped a warning on it: "Fair warning: this is largely vibe-coded and unsupported."
Expert"Vibe-coded." I love that term. It perfectly captures that trend of building software intuitively and rapidly, leaning heavily on AI generation tools like Claude Code or Cursor, rather than going through the traditional, rigorous software engineering practices.
HostIncident.io themselves are apparently all-in on this. They even admitted to vibe-coding a custom game called "PagerTron" for Anthropic using Claude Code. They said, "To think that you can build a simple game like that with a database of high scores without understanding very much about how it works is insane." Which, you know, is a *bit* terrifying when you think about production systems.
ExpertIt is. It reminds me of the early days of scripting, where you could cobble something together that worked, but it was held together with duct tape and good intentions. This is like that, but with a super-powered AI generating the duct tape. And while it's fun for prototyping, there's a dark side. A Retool engineering blog post from last year warned that vibe-coded apps are "fun to build, painful to run."
HostYeah, that's been my experience with some of the early AI-generated code I've seen. It’s notorious for adding tech debt, being verbose instead of refactoring, and potentially introducing security vulnerabilities like secret leakage and injection flaws. Because the AI isn't *understanding* security best practices, it's just generating code that *looks* like it works.
ExpertExactly. But here's the kicker: that vibe-coded prototype proved the market demand. Customers actually used it, a lot. But the architecture was fundamentally unscalable for enterprise teams. Running a GitHub repo, setting up `.env` files with raw API keys, `go run` or `docker-compose` locally? Forget about it for a large organization.
HostSo the customers told them, "We love the idea, but we can't adopt something that runs locally and isn't officially supported." Which brings us back to the March 2026 release: a fully managed, remote server with zero local setup. No repo to clone, no API keys to fuss with locally. Up and running in under a minute. They officially deprecated the old Golang implementation.
ExpertIt’s a fascinating product pipeline, though, isn't it? Use AI to rapidly prototype a feature over a weekend, get community validation, and if it sticks, *then* rewrite it properly and productize it. If it fails, you just drop it. It feels like a supercharged version of agile, maybe "hyper-agile"?
HostIt's like extreme lean startup methodology applied to engineering. Instead of months of product management and design docs, you just… build a working thing with AI in a weekend. It's almost bypassing the traditional 6-month product cycle entirely. The question is, does the initial "vibe-coding" tech debt ever truly get paid down? Or are we just building a house of cards faster?
ExpertWell, that leads us directly to the elephant in the room when you give an LLM "read/write" access to your production incident management system: security. The prototype's authentication was a raw `INCIDENT_IO_API_KEY` in environment variables. If that key had admin privileges, the LLM effectively had admin privileges. That's a massive risk.
HostYeah, a classic security anti-pattern. If you accidentally commit that `dotenv` file, or your local machine is compromised, you've just handed the keys to the kingdom to... well, anyone. The March 2026 remote release, thankfully, completely overhauls this with user-based OAuth.
ExpertWhich is a huge relief. When a human connects via an AI assistant like Claude, they authenticate with their Incident.io account using OAuth. The MCP client handles the flow, and the user approves access in their browser. This is the critical detail: "Actions are attributed to your user account, and you see the same data you'd see in the dashboard."
HostSo, Claude isn't some autonomous agent with its own service account. It's acting as a direct proxy for *you*. It inherits *your* permissions. If you can only read incidents, Claude can only read incidents. If you can close them, Claude can close them.
ExpertWhich solves the authorization problem, but creates a terrifying accountability problem. Imagine this scenario: it's 4 AM, you're exhausted, and you ask Claude to "Update the status of the database incident." Claude, being an LLM, hallucinates. It misinterprets the prompt, and instead of updating, it closes a Sev-1 incident that's actively costing the company $10,000 a minute.
HostOh no.
ExpertAnd because of that OAuth attribution model, the audit log will show that *you*, the human engineer, closed the incident. Not Claude. The LLM essentially committed an action under your name.
HostWait, really? So, if Claude goes rogue, I'm the one on the hook? That's… that's terrifying. Even if I approved the initial OAuth connection, the day-to-day commands happen instantly in the chat interface. There’s no "Are you sure you want to close this Sev-1, initiated by Claude?" pop-up.
ExpertExactly. So, are we comfortable taking the blame for our AI assistant's hallucinations? Does the convenience of natural language outweigh the precision and safety of a rigid UI with confirmation dialogs? Because right now, it feels like we're trading precision for speed, and hoping the AI doesn't screw up.
HostThat's a tough question. Because on one hand, the natural language interaction is so compelling. But on the other, the stakes during a Sev-1 are incredibly high. "Oh, my AI hallucinated and cost us half a million dollars" isn't going to fly with the CFO.
ExpertIt certainly isn't. And this shift away from rigid UIs and towards natural language isn't just about Incident.io. It signals a potential end to an era. For the last decade, Slack and ChatOps have been the undisputed command center for incident response.
HostAh, yes, the ubiquitous `/incident declare`, `/incident update status=investigating`, `/pd trigger` commands. You had to memorize all that syntax. It was like a mini-CLI within Slack.
ExpertRight. But with MCP servers, AI tools are starting to replace Slack as the primary interface for actual *action*. Instead of memorizing slash commands, you just say, "Draft my on-call handover notes to post in #handovers," or "Declare a new high-severity incident called 'Database performance degradation'," or "Who's on call for the platform team right now?"
HostSo Slack is moving from command center to more of a notification board. The actual thinking, querying, and action-taking are moving into the LLM context window. Which makes sense for developers who don't want to leave their IDE during an outage.
ExpertExactly. Think about tools like Cursor integrating MCP. An engineer can be looking at a failing block of code, highlight it, and tell the IDE, "Create an incident for this, page the platform team, and attach this stack trace." It's an IDE-driven operation. That's powerful.
HostAnd it's not just within proprietary tools. Zapier recently launched an "MCP Client by Zapier," which means you can use *any* remote MCP server within Zapier workflows. So you can chain AI tools together without writing any glue code.
ExpertLike a Zapier workflow that triggers on a "New Incident," automatically uses the MCP server to find who's on call, then executes subsequent actions. This is where it gets really interesting for broader automation beyond just a single chat window.
HostSo, is Slack dead for DevOps? The Incident.io changelog even noted you can use the MCP server to draft handover notes *to post in Slack*. It's basically saying Slack is just the broadcast medium, not the brain.
ExpertI don't think it's dead, but its role is definitely evolving. ChatOps forced you to learn a specific CLI-like syntax, which was a barrier, but also a guardrail. You couldn't accidentally declare an incident with a typo in the team name. MCP lets you talk like a human, but does that ambiguity of natural language become dangerous when managing a Sev-0?
HostThat’s the core tension, isn't it? The rigidity of ChatOps, while sometimes frustrating, might have actually prevented mistakes during those high-stress, sleep-deprived situations. Now, we're relying on the LLM to interpret our intent perfectly.
ExpertAnd that brings us to the future implications. This remote MCP server isn't just about making Claude smarter; it's the foundational plumbing for fully automated Site Reliability Engineering.
HostAutomated SRE. That's a term that gets my Spidey-sense tingling. Incident.io has explicitly stated they're building an "AI SRE," currently in private beta. Their vision is an agent that "will investigate the incident with you, show you what's wrong, and why it's wrong. Eventually, we'll show you how to fix it or offer to fix it on your behalf."
ExpertThat's the dream, isn't it? An agent that can detect an anomaly in Datadog, declare an incident via Incident.io's MCP, page the right team, and then draft the preliminary post-mortem in Notion—all without human intervention. That's not science fiction, that's what companies like Block are doing with their "goose" agent.
HostYeah, "goose" was described as a production tool that "automates triage and root cause analysis." So, the pieces are already in place. And to inject a little levity into this slightly terrifying future, I have to mention Incident.io's joke integration.
ExpertOh, the "Mind Control Protocol."
HostYes! Right there on their integrations page, alongside their real integrations, they list the "Mind Control Protocol": "The original MCP. Beam the incident summary directly into the head of your on-call engineer. No Slack, no PagerDuty, no excuses." And next to that, the "Ouija Board Webhook" with "Latency indeterminate, strongly discouraged for production workloads." They clearly have a sense of humor about the future.
ExpertThey do. But it also highlights the ultimate trade-off we're making here. If we remove all friction from incident management, if an LLM automatically reads the logs, summarizes the impact, and pages the team, do engineers lose their operational "muscle memory"?
HostThat's "automation complacency," right? When the pager goes off, and Claude hands you a neatly formatted summary, are you going to double-check the raw logs, or are you just going to blindly run the mitigation script Claude suggests?
ExpertExactly. Because if the AI SRE confidently states, "The database is out of memory due to the new deployment," and the human engineer anchors on that conclusion, but the AI is hallucinating and the real issue is a network partition, your time to resolution is going to skyrocket. Because you trusted the machine's summary over the raw telemetry.
HostIt's the ultimate test of trust. Do you trust the black box that summarizes, or do you trust your own ability to piece together the raw data? And how much time do you have to do both during a critical incident? It feels like we're trading one type of toil for another, where the new toil is constantly verifying the AI.
ExpertThis whole thing is a fascinating evolution. From a "vibe-coded" weekend project to a full-blown enterprise solution that's reshaping how we think about incident management. It's clear that AI is going to deeply integrate into our operational workflows.
HostIt is. The key takeaway for me is that the Model Context Protocol is truly a game-changer, acting as that universal translator for AI to interact with all our operational tools. It finally makes AI agents truly actionable in the incident response context.
ExpertAnd Incident.io's move to a remote, managed MCP server is what pushed this from a cool experiment to something enterprise teams can actually adopt. It shows how quickly AI-driven prototypes can evolve into critical infrastructure.
HostBut we have to be acutely aware of the new risks. The "human proxy" model with OAuth means you own the AI's mistakes. That's a massive shift in accountability that we need to bake into our processes and our culture, not just our tech.
ExpertAnd finally, this is going to fundamentally change how we interact with our systems during an incident. The days of rigid ChatOps might be waning, replaced by natural language interfaces that are more convenient, but also carry the inherent risk of ambiguity and automation complacency.
HostSo, as listeners think about this, what happens when the AI is wrong? When Claude hallucinates during a Sev-1? Who ultimately pays the price – the engineer who asked the question, or the company that deployed the AI?
ExpertAnd are we preparing our engineers for a world where their primary interface during an incident is an LLM, and not the raw dashboards? What skills will they need to maintain when so much of the traditional legwork is automated away?