Rachel Potvin, Josh Levenberg
2016 · Communications of the ACM
Google's monolithic repository provides a common source of truth for tens of thousands of developers around the world.
Metadata: Crossref. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
5 selected excerpts · 5 total mentions
I think it's also worth mentioning Piper, Critique, and the infamous monorepo. [1] Piper: https://en.wikipedia.org/wiki/Piper_(source_control_system) [2] Crituque: https://books.google.com/books?id=V3TTDwAAQBAJ&pg=PA399#v=on... [3] Monorepo: https://dl.acm.org/doi/10.1145/2854146
Yes. This is part of the no long lived branches[0] culture. Fwiw, I have found it superior to having to do huge merges. [0] https://dl.acm.org/doi/10.1145/2854146
> Is it possible to integrate Turborepo with general-purpose monorepo build tools? Bazel, in particular? It's definitely possible, but I think the practical limitations would make it too complex to reason around and maintain. You'd end up creating two separate and overlapping systems to declare dependency graphs and input sources and manage caching and execution. I haven't yet seen a case where the two are actually interleaved. Currently at Databricks, we use Bazel to provide the correctness guarantees and interop needed for CI, and we use JS-specific tooling (non-Bazel) locally to meet our performance needs, where the usage profile is different and where we're willing to…
make correctness tradeoffs. > (Is Bazel designed in a way that make it impossible to do JS monorepos well?) There are limitations in Bazel that don't play nicely with modern JS conventions. For example, Bazel's standard sandbox is based on symlink farms, and Node.js and the ecosystem by default follow symlinks[1] to their real locations, effectively breaking sandboxing. A FUSE or custom filesystem (Google's version of Bazel takes advantage of one[2]) would be better but is not as portable. As another example, Bazel's action cache tries to watch or otherwise verify the integrity of every input file to an action, and when node_modules is 100k+ files, this gets expensive and is prone to non-determinism. Bazel does this for correctness, which is noble but results in practical performance problems. You need to do extra work to "trick" Bazel into not reading these 100k+ files each time. The problems feel solvable to me, but not easily without adding yet more configuration options to Bazel. The influx of new JS-specific tooling is a reset to this, building the minimum viable set of functionality that the JS ecosystem specifically needs, without the burdens of being a general purpose build system. [1] https://nodejs.org/api/cli.html#--preserve-symlinks [2] https://dl.acm.org/doi/10.1145/2854146
Disclaimer: I haven't worked at Google, but you can easily read their papers like https://dl.acm.org/doi/pdf/10.1145/2854146 and understand what's going on. Refactoring tooling is integrated with the revision control system because in a giant codebase with thousands of contributors, you need to split a refactoring patch into smaller chunks and ideally use semantic rather than text-based diff tooling, or the changes will never get reviewed or merged before they conflict with more changes. As for the actual renaming, yes, definitely beyond what sed does but probably around what the fanciest IDE does. Imagine a big global symbol dependency graph produced by the entire build toolchain all at once and cached, I think? Also, this isn't the halting problem unless your codebase allows for fully dynamic invocation :)
The biggest feature I miss from Google is the constellation of citc and the distributed build system that revolves around it. Most is described or implied by "Why Google stores billions of lines of code in a single repository" (2016) [1]. The key idea is that any human-written change only ever touches ~1-100 files, and there's never any need to store or maintain the entire source tree in your local persistence store. By only ever working on deltas, you can have highly efficient distributed and cached builds. This architecture imposes significant constraints: you must express your build graph in a modular way [2], you collocate your distributed FUSE backend and build…
system for speed, and all development is over the network. But it comes with many benefits at scale: near-perfect caching, instant "branch switching", fast builds even with deep and wide dep graphs, and excellent affected target determination so flakes/outages in one part of the codebase don't affect the rest of it. [1] - https://dl.acm.org/doi/pdf/10.1145/2854146 [2] - https://bazel.build/
More from the same topics.
Dan Tong, David J. Farnham, Lei Duan +4
Abstract If future net-zero emissions energy systems rely heavily on solar and wind resources, spatial and temporal mismatches between resource availability and electricity demand may challenge system reliability.
Steven M. Paul, Daniel S. Mytelka, Christopher T. Dunwiddie +4