Prime Video: when a distributed pipeline became one process

At a glance


Organization	Amazon Prime Video, Video Quality Analysis team
Period	Reported in 2023
Problem	A serverless pipeline hit scaling limits at about 5% of expected load and cost too much
Architecture	One application process per stream on containers, passing data in memory
Patterns	Serverless, modular monolith

The problem

The team’s tool watches video streams for quality problems, such as corrupted frames or audio out of sync, and triggers fixes. It splits each stream into video frames and audio buffers and runs machine-learning detectors over them.

The first version used serverless building blocks. AWS (opens in a new tab) Step Functions (opens in a new tab) orchestrated a set of Lambda functions, every frame and intermediate result was written to and read from S3, and results went out through SNS (opens in a new tab). It was quick to build, but it only handled about 5% of the load the team expected. Orchestration used several state transitions for every second of video, so the team hit its account limit on transitions. The many S3 reads and writes, and the state transitions, also made it expensive.

What they did

The team moved all the components into one application process running on Amazon ECS (opens in a new tab) on EC2. Frames now pass between steps in memory instead of through S3, and there is no orchestration service between steps. To scale, they run several copies of the process and split detectors across separate tasks, so adding detectors doesn’t hit the limits of a single machine.

flowchart TB
    subgraph v1 [First version: distributed]
        sf[Step Functions] --> l1[Lambda: split stream]
        sf --> l2[Lambda: detectors]
        l1 --> s3[(S3: frames and intermediate data)]
        s3 --> l2
        l2 --> sns[SNS: results]
    end
    subgraph v2 [Second version: one process]
        p[Container on ECS: split, detect, report in memory] --> out[Results]
    end

Results

The team reported a 90% cut in infrastructure cost, and the new design removed the scaling limit that had stopped it monitoring all streams.

What this does and doesn’t show

The write-up was widely shared as “Amazon abandons microservices”. That is not what it says:

It concerns one service with a very specific workload: high-volume data passed between tightly coupled steps.
The pipeline’s steps always ran together and shared large amounts of data. Splitting them across a network and storage added cost without adding independence.
The team still uses managed AWS services, just at a coarser grain.

Trade-offs

Less fine-grained scaling. The whole pipeline scales as one unit, which suited this workload.
One failure domain per process. A crash affects everything in that process, so the team runs many copies.
Serverless was still useful for getting the first version working quickly.

Lessons you can reuse

Count the bytes and calls that cross the network between your components. If steps always run together and exchange large data, they probably belong in one process.
Check per-request charges and service quotas (state transitions, storage operations) against your real volumes before choosing a fine-grained design.
Treat a design’s first version as a hypothesis. Measure it at expected load early.

Sources

Rafał Gancarz, “Prime Video Switched from Serverless to EC2 and ECS to Save Costs” (opens in a new tab), InfoQ, May 2023, summarizing the Prime Video Tech post by Marcin Kolny

Written and edited by Abhinav Taduka, enterprise software architect.