Netflix: from a single database to cloud-native services

At a glance


Organization	Netflix
Period	2008 to January 2016
Problem	A vertically scaled database was a single point of failure
Architecture	Hundreds of microservices on AWS (opens in a new tab), NoSQL data stores, continuous delivery, deliberate failure testing
Patterns	Microservices, well-architected cloud, chaos engineering

The problem

In August 2008 a major database corruption stopped Netflix from shipping DVDs for three days. The lesson Netflix drew was that large, vertically scaled components such as a single relational database in its own data center were single points of failure. It wanted horizontally scalable, distributed systems instead, and chose to build them on Amazon Web Services.

What they did

Netflix did not lift and shift its existing systems. It rebuilt nearly all of its technology for the cloud:

The monolithic application was split into hundreds of microservices.
The data model was denormalized and moved to NoSQL databases.
Releases moved from centrally coordinated, multi-week hardware provisioning to continuous delivery and self-service operations by each team.

flowchart LR
    subgraph dc [2008: own data center]
        app[Monolithic application] --> db[(Single relational database)]
    end
    subgraph cloud [2016: AWS]
        edge[API edge] --> s1[Service]
        edge --> s2[Service]
        edge --> s3[Service]
        s1 --> n1[(NoSQL store)]
        s2 --> n2[(NoSQL store)]
        s3 --> n1
        chaos[Chaos Monkey] -. terminates instances .-> s2
    end

Because cloud instances can fail at any time, Netflix also tested failure deliberately. In 2011 it described Chaos Monkey, a tool that randomly disables production instances during business hours while engineers watch, so weaknesses show up when people are ready to fix them. Related tools injected latency or simulated losing a whole availability zone.

Customer-facing services were in the cloud before 2015. Billing and customer and employee data took longer. Netflix shut down the last data center components of its streaming service in January 2016, about seven years after starting.

Results

Netflix reported that by 2016 it had eight times as many streaming members as in 2008, and overall viewing had grown by three orders of magnitude. Availability improved steadily despite some early cloud outages, and the company said it was nearing its goal of four nines. Cost was not the main reason for the move, but Netflix said its cloud cost per streaming start was a fraction of its data center cost.

Trade-offs

It took seven years. A full rebuild is a multi-year commitment, and Netflix kept both worlds running until the end.
Distributed systems fail in new ways. Netflix had to invest heavily in resilience tooling, observability and failure testing.
Denormalized NoSQL data gives scale and availability but moves consistency work into application code.

Lessons you can reuse

Migrate by rebuilding for the target platform when the old design is the problem; lift and shift only moves the problem.
Design every service to survive the loss of its dependencies, then prove it by injecting failures in a controlled way.
Move the least critical, most self-contained workloads first, and save regulated data such as billing for last.

Sources

Yury Izrailevsky, “Completing the Netflix Cloud Migration” (opens in a new tab), Netflix, 2016
Yury Izrailevsky and Ariel Tseitlin, “The Netflix Simian Army” (opens in a new tab), Netflix Tech Blog, July 2011

Written and edited by Abhinav Taduka, enterprise software architect.