Debug Log

30,000 Merges a Day: Why Anthropic Abandoned Git for 'Scion'

October 20, 202616:55Debug Log

This episode explores how Git, the industry-standard version control system, reaches its breaking point when confronted with extreme scale, such as Anthropic's 30,000 merges a day in a massive monorepo. Listeners will learn that Git's distributed architecture struggles with the demands of centralized, high-velocity development, leading to agonizingly slow operations and complex merge conflicts. This friction severely impacts developer productivity and makes version control a significant bottleneck.

Key Takeaways

Detailed Report

Anthropic, an AI safety and research company, faced an unprecedented challenge in its development workflow: processing 30,000 code merges per day. This extreme volume pushed Git, the industry-standard version control system, past its breaking point, leading the company to develop its own bespoke solution named 'Scion.'

Git's Breaking Point at Extreme Scale

For most development teams, 30,000 merges a day is an unimaginable figure. At Anthropic, this scale, combined with a massive monorepo containing millions of files, exposed fundamental limitations in Git's architecture. Operations that are typically instantaneous, such as `git status`, `checkout`, `diff`, or `log`, began taking up to ten minutes to complete. This slowdown wasn't just an inconvenience; it represented a significant barrier to developer productivity.

The core issue stemmed from a clash between Git's design and Anthropic's operational demands. Git excels at tracking individual changes and their lineage across distributed repositories, using a graph-based model. However, when thousands of developers contribute continuously to a single, colossal codebase (a monorepo), Git's efficiency plummets. The sheer volume of files, coupled with an incredibly dense and complex commit graph generated by daily merges, made basic file system traversals and history indexing computationally expensive.

The Toll on Developer Productivity

The impact on developer experience (DX) was profound. Waiting minutes for basic version control commands created significant cognitive load and context-switching overhead. More critically, the continuous torrent of changes led to a constant battle against merge conflicts. Git's default merge strategy, which often requires manual intervention, became a major bottleneck. Developers spent an inordinate amount of time resolving complex, multi-way conflicts, repeatedly pulling, resolving, and pushing changes. This 'death by a thousand paper cuts' shifted engineers' focus from building features to untangling version control issues, turning the system meant to facilitate collaboration into its primary hindrance.

The Clash with Trunk-Based Development

Anthropic, like many large, fast-moving organizations, favors a trunk-based development model. This approach involves integrating small, incremental changes very frequently into a single main branch, aiming to keep it always releasable and minimize divergence. Git, with its robust capabilities for distributed branching and managing complex divergent histories, is over-engineered for this specific workflow at Anthropic's scale. Its strengths in handling distributed development became liabilities when applied to a highly centralized, high-velocity trunk-based model, where the system needed to optimize for seamless, atomic changes directly to `main`.

Scion: A Purpose-Built Solution

To overcome these challenges, Anthropic built Scion, a custom version control system designed specifically for its hyper-scale, high-velocity monorepo environment. Scion fundamentally rethinks how changes are integrated, moving away from Git's traditional three-way merge that frequently demands manual intervention. Instead, Scion aims for automatic, conflict-free merges.

While specific technical details are proprietary, Scion likely incorporates several key architectural shifts:

  • Semantic Merges: The system might understand code structure (e.g., Abstract Syntax Trees) rather than just text lines, allowing for more intelligent reordering and merging of non-conflicting changes.
  • Optimized Data Model: Moving from Git's directed acyclic graph of commits, Scion might employ a snapshot-based approach, optimized for rapidly generating and comparing repository states, which speeds up operations like `status` and `checkout`.
  • Monorepo-First Design: Scion's internal data structures, indexing, and communication protocols are built from the ground up for millions of files and high change throughput, centralizing the 'truth' of the repository and orchestrating merges intelligently.

Scion is not merely a faster Git; it represents a fundamentally different approach tailored to shift the burden of conflict resolution from the developer to the system, restoring focus to actual coding.

Part of a Larger Trend

Anthropic's decision to build Scion is not unique. Other tech giants facing similar extreme scale challenges have also developed their own bespoke version control systems, including Google with Piper, Meta with Sapling, and Microsoft with GVFS. This trend highlights that while Git is incredibly powerful and suitable for most use cases, its architectural assumptions do not align with the unique operational realities of organizations operating at unprecedented scale and velocity.

Building a custom version control system is a monumental engineering undertaking. However, for these companies, the immense investment is justified by the critical importance of developer productivity. Even small percentage improvements in efficiency, multiplied across thousands of engineers, translate into significant cost savings and faster innovation cycles, making the return on investment substantial.

The Future of Version Control

Anthropic's experience with Scion underscores that foundational tools have limits. As software development continues to scale and monorepos become more prevalent, the pain points currently experienced by hyper-giants will likely become more widespread. This necessitates a re-evaluation of what constitutes an effective version control system, emphasizing the need for more intelligent, automated merging and highly performant operations on massive codebases.

The choice of version control is not a one-size-fits-all decision; it must align with a team's specific scale, workflow, and performance requirements. While Git remains dominant, the frontier of extreme scale demands purpose-built infrastructure, demonstrating that sometimes, building a custom solution is the only way to maintain development velocity and innovation.

Show Notes

Works Referenced

  • 30,000 Merges a Day: Why Anthropic Abandoned Git for 'Scion': The original article detailing Anthropic's challenges with Git and their custom version control system, Scion.
  • Anthropic: An AI safety and research company that developed Scion to manage its large-scale codebase.
  • Git: The widely used distributed version control system that Anthropic found insufficient for their extreme scale.
  • Scion: Anthropic's custom-built version control system, discussed in the source article, designed to handle 30,000 merges a day and extreme monorepo scale.
  • Google Piper: Google's custom version control system designed for its massive monorepo, handling billions of lines of code and hundreds of thousands of daily commits.
  • Meta Sapling: Meta's custom source control system, built to manage its large monorepo and improve developer experience at scale.
  • Microsoft GVFS (Git Virtual File System): Microsoft's solution to scale Git for large repositories, enabling efficient operations on massive codebases (now evolved into Scalar).

Glossary

  • Git: A widely used distributed version control system that tracks changes in source code during software development, allowing multiple developers to collaborate.
  • Monorepo: A software development strategy where all code for many projects is stored in a single repository, rather than in separate ones.
  • Version Control System (VCS): A system that records changes to a file or set of files over time so that you can recall specific versions later, essential for collaborative software development.
  • Trunk-based development: A software development practice where developers merge small, frequent updates to a single main branch (the 'trunk'), keeping it always releasable and minimizing long-lived feature branches.
  • Merge Conflict: A situation that occurs when two or more developers make conflicting changes to the same part of a file, requiring manual intervention to resolve before the changes can be integrated.
  • Developer Experience (DX): The overall quality of a developer's interaction with tools, systems, and environments, impacting their productivity, satisfaction, and ability to build software effectively.
  • Abstract Syntax Tree (AST): A tree representation of the syntactic structure of source code, used by compilers and code analysis tools to understand the program's structure rather than just its text.

Sources / References

Full Transcript

HostThirty thousand merges a day. That's a number that just… doesn't compute for most development teams. It sounds like a problem of a completely different order.
ExpertIt is. For Anthropic, that volume pushed Git, the industry's de facto standard for version control, past its breaking point. They describe "git status" taking ten minutes to run.
HostTen minutes for a `git status`? That's not just slow; that's actively hostile to developer productivity. It implies a level of friction that would bring a normal team to a standstill.
ExpertExactly. It indicates a fundamental mismatch between the tooling and the demands of their development process, which led them to build their own solution: 'Scion'.
HostSo, the core of this issue seems to be a collision between extreme scale and Git's architectural assumptions. Can you unpack what exactly breaks down when you're attempting 30,000 merges in a single day?
ExpertIt boils down to Git's design for a distributed, graph-based model clashing with a monorepo strategy at an almost unimaginable velocity. Git excels at tracking individual changes and their lineage across a network of repositories. But when you centralize thousands of developers on a single, massive codebase – a monorepo – and ask them to integrate changes continuously, Git's efficiency starts to plummet. Operations like `git status`, `checkout`, `diff`, or `log`, which are usually instantaneous, become agonizingly slow because they have to traverse an immense history and index an enormous working directory.
HostSo, it's not just the number of merges, but the size of the repository itself that's the culprit, exacerbating the performance issues. Is it the sheer volume of files, the depth of the history, or both?
ExpertBoth, but primarily the volume of files and the complexity of the change graph. Think of Git as a distributed ledger. Each commit is a snapshot, and merges integrate these snapshots. When you have a monorepo with millions of files, even a simple `status` operation requires Git to compare the state of your working directory against the index, and potentially against the head of the repository. This involves a vast amount of file system traversal and hashing. Then, when you add 30,000 merges a day, the commit graph becomes incredibly dense and complex. Operations that involve traversing this graph, like `git log` or even complex `rebase` scenarios, become computationally expensive. The system simply wasn't designed to efficiently operate on a codebase that large, with that many concurrent contributions.
HostIt sounds like trying to navigate a dense urban grid on foot when you really need a high-speed rail network. The tools are right, but for the wrong scale of problem.
ExpertA fair analogy. The issue isn't that Git is inherently bad, but rather that its strengths—its decentralized nature, its robust history, its object model—become liabilities when confronted with a workflow demanding extreme centralization and continuous integration on a colossal scale. The mental model and the underlying data structures simply don't scale efficiently to Anthropic's operational parameters.
HostAnd beyond the raw performance hits, what about the impact on the developers themselves? Ten minutes for a `git status` is just the tip of the iceberg, I imagine. What does that kind of friction do to a team?
ExpertThe developer experience, or DX, takes a massive hit. Imagine every time you want to see what files you've changed, or switch branches, or pull in the latest updates, you're waiting minutes. That cognitive load and context switching alone are killers for productivity. But the real productivity drain comes from merge conflicts. With 30,000 merges a day, you're talking about a continuous torrent of changes to the codebase. Even small, seemingly isolated changes have a high probability of colliding with others.
HostSo, it's not just that `git status` is slow, but that the act of integrating work itself becomes a constant battle against conflicting edits.
ExpertExactly. Git's default merge strategy, while effective for typical project sizes, relies heavily on developers manually resolving conflicts when changes overlap. When you have hundreds or thousands of changes landing simultaneously, the probability of complex, multi-way merge conflicts skyrockets. Developers spend an inordinate amount of time pulling changes, resolving conflicts, pushing, only to find another conflict, and repeating the cycle. It's a death by a thousand paper cuts. This leads to frustration, burnout, and crucially, a significant slowdown in the actual development work. It shifts developer focus from building features to untangling version control issues.
HostIt almost sounds like the version control system itself becomes the primary bottleneck, rather than the complexity of the code or the business logic.
ExpertIt does. The tooling meant to facilitate collaboration starts actively hindering it. The paper implies that engineers were spending a significant portion of their day not coding, but wrestling with Git. This is a crucial distinction from a typical project where merge conflicts are occasional annoyances. At Anthropic's scale, they become a systemic, daily, productivity-destroying feature of the development process. The system essentially forced a fragmented, often painful, integration cycle, which is the antithesis of a smooth, trunk-based development model.
HostYou mentioned trunk-based development. This seems like a critical distinction here. Git's flexibility with branching and distributed development is often touted as a feature. How does that clash with Anthropic's preferred workflow?
ExpertGit's design is inherently distributed. Every developer has a full copy of the repository and can work independently, creating branches and merging them back. This is fantastic for open-source projects or teams with distinct, long-lived feature branches. However, large, fast-moving organizations like Anthropic often prefer a trunk-based development model.
HostFor clarity, trunk-based development means everyone works directly off or integrates very frequently into a single main branch, right?
ExpertPrecisely. The idea is to keep the main branch—the "trunk"—always releasable and to integrate small, incremental changes very frequently. This minimizes divergence and the complexity of merges. But trying to force a highly distributed tool like Git, with its robust branching and merging capabilities, into a strict, high-velocity trunk-based workflow at Anthropic's scale presents significant challenges. Git's graph model is excellent for tracking complex branch histories, but less optimized for a scenario where 99% of activity is pushing small, atomic changes directly to `main` and expecting them to integrate seamlessly. The underlying data structures and algorithms optimized for distributed branch management don't perform well when applied to a single, monolithic, rapidly changing trunk.
HostSo, Git's strength—its ability to handle divergent histories and complex branching strategies—becomes a weakness because it's over-engineered for a simpler, more centralized workflow they want to achieve?
ExpertIn a way, yes. They needed the *outcome* of Git's robustness—reliable history, atomicity—but not necessarily its specific *mechanism* for achieving it, especially when that mechanism introduced performance penalties and merge hell. Scion, their custom solution, appears to be designed specifically for this high-velocity, trunk-based, monorepo paradigm. It shifts from Git's distributed, peer-to-peer merge model to something that facilitates more automatic, centralized, and conflict-free integration. It’s about creating a system where the "main" branch is always considered the authoritative source, and changes are integrated into it with minimal friction, almost as if they're queued up and applied in an intelligent, conflict-resolving manner.
HostThat brings us to Scion itself. If Git was breaking down, what did Anthropic build differently? How does Scion specifically address these performance and conflict issues?
ExpertThe source implies Scion fundamentally rethinks how changes are integrated. Instead of relying on Git's traditional three-way merge, which frequently demands manual intervention, Scion aims for automatic, conflict-free merges. While the precise technical details aren't exhaustively laid out, this suggests a few potential architectural shifts. One is likely a more sophisticated understanding of semantic merges, where the system might understand code structure rather than just lines of text, allowing for intelligent reordering or merging of non-conflicting changes.
HostSo, it's not just looking at raw text differences, but perhaps at the Abstract Syntax Tree, or the actual code components?
ExpertThat's a strong possibility, or at least a more granular understanding of change units. Another key aspect is likely its underlying data model. Git is essentially a directed acyclic graph of commits. Scion might employ a more snapshot-based approach, optimized for rapidly generating and comparing repository states, rather than tracing individual commit lineages through a complex graph. This would make operations like `status` and `checkout` much faster, as they wouldn't involve as much recursive traversal or re-indexing.
HostLike taking a photograph of the entire system at precise intervals, and then quickly comparing two photographs, rather than tracing the journey of every individual pixel.
ExpertA good analogy. Furthermore, Scion is likely built with an awareness of the monorepo scale from the ground up. This means its internal data structures, indexing, and communication protocols are optimized for operating on millions of files and a high throughput of changes. It's a system designed for a centralized workflow, even if developers still interact with it from their local machines. It centralizes the "truth" of the repository and orchestrates merges, rather than leaving it to distributed clients to figure out. This centralized intelligence is key to achieving those "automatic, conflict-free merges."
HostSo, Scion isn't just a faster Git; it's a fundamentally different approach to version control, tailored for this specific problem space of hyper-scale, high-velocity monorepos.
ExpertPrecisely. It's about shifting the burden of conflict resolution from the developer to the system, and optimizing for the specific access patterns and scale requirements of a single, enormous, rapidly evolving codebase. It prioritizes the continuous integration model over Git's more flexible, distributed branching model. The goal is to make the act of integrating code as seamless and invisible as possible, restoring developer focus to actual coding.
HostThis isn't an isolated incident, though. Other tech giants have also built their own bespoke version control systems – Google with Piper, Meta with Sapling, Microsoft with GVFS. Is Anthropic's Scion simply another data point in a larger trend, or does it represent something unique?
ExpertIt absolutely fits into that larger trend. What these companies have in common is operating at scales where off-the-shelf solutions, even robust ones like Git, cease to be effective. Google's Piper, for instance, handles a repository that effectively encompasses the vast majority of their codebase, with tens of millions of files and hundreds of thousands of daily commits. Meta's Sapling addresses similar challenges, particularly around monorepo performance and developer experience.
HostSo, it's less about "Git is bad" and more about "Git is not designed for this specific, extreme use case."
ExpertExactly. It highlights the principle that no single tool is a silver bullet for all problems. Git is incredibly powerful and widely adopted because it solves a wide range of version control problems very well. But when you push the boundaries of scale, concurrency, and specific workflow paradigms, you expose the limitations of any generalized solution. These companies aren't abandoning Git because they find it technically inferior in general, but because its architectural assumptions don't align with their unique operational realities. They're essentially demonstrating that at a certain scale, the cost of adapting existing tools or enduring their limitations outweighs the cost of building a custom solution.
HostAnd that cost of building a custom solution is immense. This isn't a small team hacking something together over a weekend. These are significant engineering investments.
ExpertThey are. Building a version control system from scratch is a monumental undertaking, requiring deep expertise in distributed systems, data structures, and performance optimization. The fact that multiple tech giants have chosen this path underscores the severity of the problem and the critical importance of developer productivity at their scale. A small percentage improvement in developer efficiency, when multiplied across thousands of engineers, translates into immense cost savings and faster innovation cycles. For companies like Anthropic, where rapid iteration on models and code is central to their mission, the ROI on a custom VCS like Scion is likely substantial. It's an investment in their core velocity.
HostSo, the takeaway here seems to be that while Git remains the dominant force for most software development, there's a frontier where its design hits a wall, and that frontier is marked by extreme scale, high velocity, and a preference for centralized, trunk-based monorepos.
ExpertThat's a precise summary. The "30,000 merges a day" figure isn't just a number; it's an indicator of a development environment operating at an intensity that demands purpose-built infrastructure, even for something as fundamental as version control. It forces a re-evaluation of what constitutes an effective version control system when traditional models break down.
HostSo, what does this indicate about the future of version control, or at least the challenges that might be seen more broadly?
ExpertIt underscores that as software development continues to scale, and as monorepos become more common even outside of these hyper-giants, the pain points currently experienced by companies like Anthropic will eventually trickle down. The need for more intelligent, automated merging, and highly performant operations on massive codebases, will only grow. It also highlights the trade-offs: the flexibility and decentralization of Git versus the raw performance and streamlined workflow of a specialized, potentially more centralized system.
HostIt raises the question of whether the next generation of general-purpose VCS tools will need to incorporate some of Scion's, or Piper's, or Sapling's underlying principles to cope with future scale.
ExpertExactly. The question isn't if, but when, these concepts find their way into more accessible, generalized tools. For now, Scion serves as a powerful testament to the fact that sometimes, the only way forward is to build your own road when the existing infrastructure simply can't handle the traffic.
HostFor listeners, the practical implication is understanding that even foundational tools have limits, and identifying those limits before they impact your team's velocity is critical.
ExpertAnd that the choice of version control isn't a one-size-fits-all decision; it needs to align with your specific scale, workflow, and performance requirements.
HostWhat becomes apparent is that Git, despite its dominance, has a specific operational envelope. When you exceed that envelope, the costs, both in terms of performance and developer morale, become prohibitive.
ExpertIndeed. The critical insight is recognizing when the cost of adapting to a tool's limitations exceeds the cost of innovation, even if that innovation means building a bespoke solution for a problem as fundamental as source control.
HostSo, the lesson here isn't to ditch Git tomorrow, but to be acutely aware of your project's scaling trajectory and whether your current tooling is equipped for the long haul.
ExpertAnd to constantly question whether the tools are serving the team, or if the team is serving the tools.
HostUltimately, this raises a fascinating question for any development lead: at what point does the perceived cost of maintaining a general-purpose, off-the-shelf system outweigh the very real, but often hidden, costs of developer friction and lost productivity?
ExpertAnd conversely, what level of development velocity and complexity warrants the extraordinary investment in building something entirely new?