Slack: re-sharding MySQL with Vitess

At a glance

Organization	Slack
Period	2017 to 2020
Problem	Each workspace lived on a single MySQL (opens in a new tab) shard, so big customers outgrew shards and one shard failure hit everyone on it
Architecture	Vitess (opens in a new tab) on top of MySQL, sharding by keys such as channel ID
Patterns	Sharding, double writes with read comparison

The problem

Slack used MySQL from the start. All data for a workspace (messages, channels, direct messages) lived on one shard, with a separate cluster mapping each workspace to its shard. Each shard had at least two MySQL servers in different data centers.

This design hit several limits:

• The largest customers outgrew the biggest available shard hardware.
• Some shards ran hot while many others sat underused, so Slack over-provisioned.
• When a shard failed, core features failed for every workspace on it.
• Newer products that cross workspaces, such as Enterprise Grid and Slack Connect, didn’t fit a one-workspace-per-shard model.

Options they weighed

• Sharding logic in the application: the code was too tightly tied to the storage layout, and it wouldn’t fix the operational problems.
• NoSQL or NewSQL databases: Slack wanted to keep MySQL and the operational knowledge it had built up.
• Vitess, an open-source sharding layer for MySQL: MySQL stays the storage engine, and Vitess adds horizontal sharding, automated failover and backups.

What they did

They proved Vitess on one small feature first, writing to both the old and new stores and comparing reads. Then, from 2017, they moved table after table, contributing fixes to Vitess along the way. Moving to Vitess also let them choose a better sharding key per table, such as channel ID instead of workspace.

flowchart LR
    subgraph before [Before]
        app1[Application] --> map[(Workspace-to-shard map)]
        app1 --> sh1[(Shard: workspaces A, B)]
        app1 --> sh2[(Shard: workspace C)]
    end
    subgraph after [After]
        app2[Application] --> vt[Vitess query routing]
        vt --> k1[(MySQL shard, by channel)]
        vt --> k2[(MySQL shard, by channel)]
        vt --> k3[(MySQL shard, by channel)]
    end

Results

By late 2020 Vitess served 99% of Slack’s database queries, peaking at 2.3 million queries per second (about 2 million reads and 300,000 writes), with a median latency of 2 ms and 11 ms at the 99th percentile. When usage jumped 50% in a single week in March 2020, the team scaled a busy keyspace by splitting it through Vitess. New products, including data residency in six regions, were built on it.

Trade-offs

• A long migration, running from 2017 until 2020 to move nearly all query load.
• Another distributed system to operate, with its own query routing and cluster metadata.
• Slack had to improve Vitess itself, contributing fixes for topology scalability, MySQL query compatibility and migration tooling.

Lessons you can reuse

• Choose sharding keys that spread load evenly. A customer-level key concentrates your biggest customers onto single machines.
• Prove a new data layer on a small, low-risk feature with double writes and read comparison before moving critical data.
• Keeping the storage engine your team knows can lower risk more than switching to something newer.

Related on ArchitectureCompass

• Figma shards Postgres, which rejected Vitess for its own reasons
• Discord’s message storage

Sources

• Arka Ganguli, Guido Iaquinti, Maggie Zhou and Rafael Chacón, “Scaling Datastores at Slack with Vitess” (opens in a new tab), Slack Engineering, December 2020