Diskless Kafka removes local broker disks entirely and writes data straight to object storage. It's how modern teams cut Kafka costs by 10x while getting rid of operational overhead.
The old way: Traditional Kafka writes data to local SSDs on each broker, then replicates cross-zone to followers. In the cloud, 70-90% of a Kafka cluster's cost is just moving bytes between availability zones.
The new way: Diskless Kafka writes directly to cloud object storage (S3, GCS, Azure Blob). The compute layer is fully stateless. No disks, no rebalancing, no babysitting. The object store handles durability.
Stateless agents replace traditional Kafka brokers. Each agent is a lightweight Go binary that speaks the Apache Kafka protocol but holds no data on disk.
On the read side, consumers pull data from object storage through agents. WarpStream's service discovery ensures consumers always talk to an agent in the same availability zone — zero cross-zone costs on reads too.
Kafka was designed for LinkedIn's data centers in 2011. Cloud flips every one of those assumptions.
This isn't a marginal improvement. It's a structural cost difference.
At 1 GiB/s sustained write throughput, 7 days retention:
| Cost Component | Self-Hosted Kafka | WarpStream (Diskless) |
|---|---|---|
| Inter-AZ networking | ~$136,000/mo | $0 |
| Storage | ~$30,000/mo (3x replicated EBS) | ~$12,600/mo (S3, single copy) |
| Compute | ~$15,000/mo (dedicated brokers) | ~$5,000/mo (stateless, auto-scaled) |
| S3 API costs | $0 | ~$3,500/mo |
| Total | ~$181,000/mo | ~$21,100/mo |
| Savings | ~8.6x cheaper |
Tiered storage (KIP-405) keeps hot data on local broker disks and offloads cold data to S3. Reduces long-retention costs, but brokers are still stateful and you still pay for cross-zone replication.
Diskless puts everything in object storage from the start. No local disks at all. No inter-zone replication. No partition leaders. A ground-up rethink, not a bolt-on optimization.
| Diskless (WarpStream) | Tiered Storage (KIP-405) | Traditional Kafka | |
|---|---|---|---|
| Hot data location | Object storage | Local disks | Local disks |
| Inter-AZ replication | None | Still required for hot data | All data cross-zone |
| Broker state | Stateless | Stateful | Stateful |
| Partition rebalancing | Not needed | Still required | Required |
| Auto-scaling | Trivial (add containers) | Complex (data migration) | Complex (data migration) |
| Disk management | None | Reduced, not eliminated | Extensive |
| Cost at scale | 5-10x cheaper | ~2x cheaper | Baseline |
Writing to object storage takes longer than local SSD. In WarpStream's default config, produce P99 is ~400ms vs single-digit ms for Kafka on NVMe.
Two breakthroughs closed the gap: S3 Express One Zone (105ms median) and Lightning Topics (decoupled durability from sequencing).
Combined result:
KIP-1150 proposes adding diskless topic support directly to Apache Kafka — strong validation. But there are key differences:
WarpStream speaks the Apache Kafka protocol. No rewrites. No proprietary SDKs. Change the bootstrap URL and you're streaming.