LinkedIn: building Kafka to connect every system

At a glance

Organization	LinkedIn
Period	About 2008 to 2013
Problem	Every new data system needed custom pipelines to and from every other system
Architecture	A central, append-only log that every system writes to and reads from
Patterns	Event-driven architecture, messaging, change data capture

The problem

Around 2008 LinkedIn was outgrowing a single, central database. It added specialized systems one after another: search, the social graph, key-value and document stores, a recommendation engine, an OLAP engine, Hadoop (opens in a new tab), a data warehouse and monitoring. By 2013 its engineers listed nine major data systems.

Each system needed data from the others. The first attempt to load data into Hadoop was planned for a few weeks and became a long slog: every source needed custom configuration, pipelines broke often, and when one broke, Hadoop’s output was wrong. Connecting every system directly to every other one would need a pipeline in each direction for each pair, which grows with the square of the number of systems. The team concluded that approach could never be operated.

What they did

They built one central pipeline instead: a log, an append-only sequence of records in a fixed order, each with its own sequence number. Producers append to it; each consumer reads at its own pace and remembers its position. Adding a new system means connecting it to the log once, not to every other system.

flowchart LR
    subgraph before [Before: point-to-point]
        A1[Database] --> B1[Hadoop]
        A1 --> C1[Search]
        D1[Activity logs] --> B1
        D1 --> C1
        D1 --> E1[Monitoring]
        A1 --> E1
    end
    subgraph after [After: one shared log]
        A2[Database] --> K{{Kafka log}}
        D2[Activity logs] --> K
        K --> B2[Hadoop]
        K --> C2[Search]
        K --> E2[Monitoring]
    end

That system was Kafka (opens in a new tab). It was built first to carry user-activity events, then used for database changes, monitoring data and loading results back out of Hadoop. Kafka keeps records in order within each partition, not across the whole topic, which lets it spread load over many machines. LinkedIn described the design in a 2011 paper and released it as open source; it became Apache Kafka.

Results

In 2013 LinkedIn reported more than 60 billion unique message writes through Kafka per day, several hundred billion counting copies mirrored between data centers, and more than 75 TB stored per data center.

Trade-offs

• Ordering is per partition. Consumers that need a global order must design for it, usually by partitioning on the key whose order matters.
• Consumers own their progress. That makes replay easy, but every consumer must handle duplicates and its own failures.
• A central log becomes critical infrastructure. It needs the same care as a primary database.

Lessons you can reuse

• When the number of integrations grows faster than the number of systems, introduce a shared log or bus so each system connects once.
• Treat events as a durable, replayable record, not just notifications. Replay is what lets you add new consumers and rebuild broken ones.
• Partition on the business key whose order you care about, such as a member or order ID.

Related on ArchitectureCompass

• Event-driven architecture
• Messaging
• Lakehouse, which uses change data capture through Kafka

Sources

• Jay Kreps, “The Log: What every software engineer should know about real-time data’s unifying abstraction” (opens in a new tab), LinkedIn Engineering, December 2013. LinkedIn has retired its old engineering blog, so this links to the Internet Archive’s copy.
• Jay Kreps, Neha Narkhede and Jun Rao, “Kafka: a Distributed Messaging System for Log Processing” (opens in a new tab), NetDB workshop, 2011