Dropbox: moving file storage off the public cloud
At a glance
Section titled “At a glance”| Organization | Dropbox |
| Period | 2013 to 2015 |
| Problem | File storage at hundreds of petabytes was core to the product and its costs |
| Architecture | An in-house storage system, Magic Pocket, in Dropbox’s own data centers, with AWS (opens in a new tab) kept for some workloads |
| Patterns | Hybrid cloud, build versus buy, dark launch |
The problem
Section titled “The problem”Dropbox kept file metadata on its own systems but stored the files themselves in Amazon S3 (opens in a new tab). Storage was growing fast: Dropbox reported 12x growth since 2012 and, by 2015, about 500 petabytes for 500 million users. For a company whose product is storage, performance and unit cost were core to the business. Dropbox also said it found nothing in open source reliable enough at the scale it needed.
What they did
Section titled “What they did”Dropbox designed Magic Pocket, its own storage software and hardware, with targets of encryption at rest, annual durability of twelve nines (99.9999999999%) and availability above 99.99%.
The rollout was gradual and reversible:
- Summer 2013: engineering work begins.
- September 2014: dark launch. Magic Pocket stores copies of data in two regions while S3 still serves users, proving the system under real load with no user impact.
- February 2015: Dropbox starts serving users from its own infrastructure.
- October 2015: 90% of user data is served exclusively from Dropbox’s own data centers.
flowchart LR
client[Dropbox clients] --> meta[Metadata service]
meta --> mp[(Magic Pocket: own data centers, about 90% of data)]
meta --> s3[(AWS: remaining workloads, some regions)]
Dropbox kept the remaining workloads on AWS where it made sense, for example to store data in Germany for European business customers.
Results
Section titled “Results”Dropbox reached the 90% milestone about two and a half years after starting, at roughly 500 PB. It cited better performance through end-to-end control of the stack, and better unit economics, as the main benefits.
Trade-offs
Section titled “Trade-offs”- A large, permanent investment. Dropbox now designs, buys, runs and replaces storage hardware and software itself.
- Less elasticity. Capacity must be planned and bought ahead of demand.
- It only made sense at this scale. Dropbox described the move in terms of the few exabyte-scale storage systems in the world. For most organizations, the public cloud remains cheaper and safer.
Lessons you can reuse
Section titled “Lessons you can reuse”- Consider owning infrastructure only for a capability that is both core to your product and large enough that cloud margins become a major cost.
- Keep a hybrid option: move what benefits, and leave the rest where it is.
- Dark-launch infrastructure changes. Mirror real traffic to the new system before any user depends on it.
Related on ArchitectureCompass
Section titled “Related on ArchitectureCompass”Sources
Section titled “Sources”- Akhil Gupta, “Scaling to exabytes and beyond” (opens in a new tab), Dropbox Tech Blog, March 2016
- “Dropbox Develops Magic Pocket, Moves Away From AWS” (opens in a new tab), InfoQ, March 2016
Written and edited by Abhinav Taduka, enterprise software architect.