wtf( )unctionsystem design, drawn
← all problemsCloud PracticeHard

Every worker wanted a connection, and the database had ninety

A source-control and CI product, installed on a customer's own hardware because the data may not leave it. The interesting thing about it is that it has five different kinds of state and each one has a different failure story.

The install works on a laptop and falls over at the size the customer actually is. The application runs hundreds of worker processes and each one wants a connection to a database that accepts ninety. Repositories sit on local disks. Artifacts and backups are measured in terabytes.

  1. R1The application runs hundreds of worker processes and each wants a database connection. The database accepts far fewer and that limit is not negotiable. Whatever sits between them must let the database see a small, steady number of connections no matter how many workers exist — handing one out for the length of a statement rather than for the life of a worker.
  2. R2Repositories live on the local disks of the machines that serve them. Losing a disk must not lose a repository, and the business has ruled out a shared network filesystem — so the copies have to be made between the machines themselves rather than by all of them pointing at one thing.
  3. R3Artifacts, uploads and backups are large, binary, and measured in terabytes. They must not live on the machines that serve requests, and they must not be kept by anything whose job is to answer quickly.
Compose the parts that do not survive being scaled naively. Tier 1 is the database's connections and the repositories, tier 2 is everything large.
Components — tap one, then tap a slot on the diagram
?Nothing here is misconfigured. Every component is doing the right thing at the size it was tested at.

Boundaries, outermost first: THE CUSTOMER'S OWN HARDWARE: Web and job nodes (hundreds of workers), Product database (ninety connections), Repository router (the only way in), an empty slot for the what the workers actually connect to, tier 1, an empty slot for the where repositories survive a dead disk, tier 1, an empty slot for the where the terabytes go, tier 2 Connections: Web and job nodes calls what the workers actually connect to — hundreds of workers what the workers actually connect to calls Product database — a few connections Repository router calls where repositories survive a dead disk — reads and writes repos Web and job nodes calls Repository router — git over HTTP Web and job nodes calls where the terabytes go — artifacts, uploads

Web and job nodeshundreds of workers
Product databaseninety connections
Repository routerthe only way in