Discord: storing trillions of messages

At a glance

Organization	Discord
Period	2017 to 2022
Problem	A large Cassandra (opens in a new tab) cluster with unpredictable latency and heavy on-call load
Architecture	ScyllaDB (opens in a new tab) behind Rust (opens in a new tab) data services that coalesce requests and route by channel
Patterns	Request coalescing, consistent hashing, data migration with validation

The problem

Discord stores messages in a wide-column database, with messages for a channel grouped together by time. In 2017 it ran 12 Cassandra nodes holding billions of messages. By early 2022 the message cluster had 177 nodes holding trillions, and it was hard to run:

• Hot partitions. When a popular channel got a burst of traffic, the node serving it slowed down. Because reads and writes needed a quorum of nodes, one slow node raised latency across the cluster.
• Compaction backlogs made reads slower and more expensive, and engineers needed manual workarounds.
• Garbage-collection pauses in the JVM caused latency spikes, and some needed an operator to restart a node.

What they did

A different database engine. Discord had already moved its other databases to ScyllaDB, which is compatible with Cassandra but written in C++ without a garbage collector. The messages cluster was last, waiting on an improvement to reverse-order queries that ScyllaDB’s team delivered.

A layer in front of the database. Discord added data services written in Rust between its API and the database. They contain no business logic, just roughly one endpoint per query, and do two important things:

• Request coalescing: when many users request the same rows at once, such as everyone opening a busy channel after an announcement, the service sends one database query and shares the result.
• Consistent-hash routing: requests for the same channel always go to the same data-service instance, which makes coalescing far more effective.

flowchart LR
    api[API] --> router{{Route by channel ID}}
    router --> ds1[Rust data service 1]
    router --> ds2[Rust data service 2]
    ds1 -- one query for many identical requests --> db[(ScyllaDB)]
    ds2 --> db

A fast, verified migration. The first plan, using an existing Spark (opens in a new tab)-based migrator, was estimated at three months. The team wrote its own migrator in Rust on top of the data-service library instead; it was estimated at nine days and reached up to 3.2 million messages per second. A sample of reads was compared between the two databases to check the result, and ScyllaDB became the primary store in May 2022.

Results

	Before (Cassandra)	After (ScyllaDB)
Nodes	177	72
Disk per node	About 4 TB	About 9 TB
p99 latency, reading history	40 to 125 ms	About 15 ms
p99 latency, inserting a message	5 to 70 ms	About 5 ms

Trade-offs

• A new layer to own. The data services are another tier to build, deploy and monitor.
• Routing by channel concentrates load for the busiest channels on one instance, which coalescing then has to absorb.
• Migration surprises. The copy stalled at 99.9999% because of large ranges of deleted data in Cassandra, which had to be compacted first.

Lessons you can reuse

• When many clients ask for the same thing at the same moment, coalesce the requests before they reach the database.
• Route requests for the same key to the same instance to make caching and coalescing work.
• Validate a data migration by comparing live reads from old and new stores before switching.
• A runtime without garbage-collection pauses can matter more than raw speed for tail latency.

Related on ArchitectureCompass

• Slack moves to Vitess and Figma shards Postgres, two other ways to scale a database

Sources

• Bo Ingram, “How Discord Stores Trillions of Messages” (opens in a new tab), Discord, March 2023