Airbnb: moving off the Monorail
At a glance
Section titled “At a glance”| Organization | Airbnb |
| Period | Described in 2018 |
| Problem | A Rails monolith (“Monorail”) slowed deployments as engineering grew from 200 to 1,000 people |
| Architecture | Services that own their data, behind an API gateway, with change events published to Kafka (opens in a new tab) |
| Patterns | Microservices, strangler fig migration, shadow traffic |
The problem
Section titled “The problem”Airbnb began as a Ruby on Rails application, nicknamed the Monorail. As engineering grew from about 200 engineers in 2015 to about 1,000, deploying it became a bottleneck: Airbnb reported that reverts and rollbacks delayed engineers’ deployments by an average of 15 hours a week.
What they did
Section titled “What they did”Agree the rules first. Before migrating, Airbnb set design tenets:
- A service owns reads and writes to its data; everyone else uses its API.
- Each service addresses one concern, sized to avoid both a new monolith and a tangled “distributed monolith”.
- Shared functionality goes in shared services or libraries, not copies.
- Every change to data publishes a standard event.
Build the platform. Client requests pass through an API gateway, and services are built, deployed and scaled independently, first on EC2 and later on Kubernetes (opens in a new tab). Apache Thrift (opens in a new tab) generated boilerplate across Ruby and Java (opens in a new tab). An open-source tool Airbnb built, SpinalTap, captured database changes and published them to Kafka.
Keep product engineers productive. A custom database adapter let existing Rails code call the new services without changing how product engineers worked.
Compare before switching. Each new service was validated against the monolith on live traffic:
- Reads: 1% of traffic went to the new service, results were compared with the monolith, and traffic was increased until the comparison was clean at 100%.
- Writes: the new service wrote to a shadow database, and results in the production and shadow databases were compared.
- A traffic-replay tool ran production requests against staging to catch regressions before release.
flowchart LR
req[Request] --> mono[Monorail: serves the response]
req -. copy of traffic .-> svc[New service]
mono --> cmp{Compare results}
svc --> cmp
cmp -- clean at 100% --> switch[Switch traffic to the service]
Results
Section titled “Results”Airbnb reported less downtime from rollbacks and page loads up to 10x faster.
Trade-offs
Section titled “Trade-offs”- Years of parallel running. The monolith and new services coexisted, and comparison infrastructure had to be built and maintained.
- Write comparison is harder than read comparison, needing shadow databases.
- A distributed system. Airbnb took on gateways, code generation and Kubernetes to make services workable at its scale.
Lessons you can reuse
Section titled “Lessons you can reuse”- Write down service-design rules before the first service, especially data ownership.
- Migrate behind the existing interface so product teams aren’t blocked.
- Prove each extracted service is equivalent by comparing it with the old path on real traffic, raising the share gradually.
Related on ArchitectureCompass
Section titled “Related on ArchitectureCompass”Sources
Section titled “Sources”- Reda Hmeid, “Airbnb’s Migration from Monolith to Services” (opens in a new tab), InfoQ, February 2019, reporting Jessica Tai’s QCon San Francisco 2018 talk
- Jessica Tai, “Airbnb’s Great Migration: From Monolith to Service-Oriented” (opens in a new tab), slides, QCon San Francisco 2018
Written and edited by Abhinav Taduka, enterprise software architect.