Segment: from 140 microservices back to one service

At a glance

Organization	Segment (now part of Twilio)
Period	About 2015 to 2018
Problem	One service and queue per destination multiplied operational work
Architecture	One service, one shared worker pool, one repository, with recorded tests
Patterns	Microservices, modular monolith, messaging

The problem

Segment receives hundreds of thousands of events per second from its customers and forwards them to more than 100 partner tools, which it calls destinations. Failed deliveries are retried.

At first, all destinations shared one queue. When one destination slowed down, its retries filled the queue and delayed every other destination: head-of-line blocking. To isolate them, Segment gave each destination its own service and queue, with a router copying each event to the destinations a customer had chosen. Later, because one failing test blocked everyone’s deploys, each destination also got its own repository.

That fixed isolation but created new problems as the count passed 140 services:

• Shared libraries diverged. Testing and deploying a library change across dozens of services was risky, so each service ended up on different versions.
• Scaling was unpredictable. Load ranged from a few events a day to thousands per second per destination. Quiet services needed manual scaling for spikes.
• Operations grew with every destination. Segment added about three destinations a month, and three full-time engineers spent most of their time keeping the system running.
• Tests were slow and flaky, because they called real partner endpoints.

What they did

• Built Centrifuge, a system that replaced the per-destination queues and handles delivery and retries for all destinations without head-of-line blocking.
• Merged the 140+ services into one service running on a large shared worker pool.
• Moved all destination code into one repository, with a single version of each of its 120 dependencies.
• Built Traffic Recorder, which records partner HTTP responses on the first test run and replays them afterwards. Test runs for all destinations went from as long as an hour to milliseconds.

flowchart LR
    subgraph before [Before]
        r[Router] --> q1[(Queue A)] --> s1[Service A]
        r --> q2[(Queue B)] --> s2[Service B]
        r --> q3[(Queue ...)] --> s3[Service ...]
    end
    subgraph after [After]
        c[Centrifuge] --> w[One service, shared worker pool]
        w --> d1[Destination A] & d2[Destination B] & d3[Destination ...]
    end

Results

• Improvements to shared libraries rose from 32 in the year before to 46 in the year after.
• One engineer can deploy the service in minutes, instead of deploying 140+ services.
• Traffic spikes for small destinations no longer page anyone, because the shared pool absorbs them.

Trade-offs

Segment listed its costs openly:

• Less fault isolation. A bug that crashes the service affects every destination.
• Less effective in-memory caching, because cached data is spread across thousands of processes.
• Shared dependency upgrades can break several destinations at once.

Lessons you can reuse

• Solve the specific problem you have. Head-of-line blocking needed isolated queues, not separate services and repositories.
• Before splitting by unit (one service per customer, partner or tenant), estimate the operational cost per unit and multiply by how many you expect.
• Keep dependencies consistent across services, or you will maintain many slightly different systems.
• Make tests independent of external services by recording and replaying their responses.

Related on ArchitectureCompass

• Microservices vs modular monolith
• Messaging
• Prime Video’s monitoring service, another move toward fewer, larger units

Sources

• Alexandra Noonan, “Goodbye Microservices: From 100s of problem children to 1 superstar” (opens in a new tab), Segment, July 2018