What Is Diskless Apache Kafka®?

Definition

Diskless Kafka

Diskless Kafka is an Apache Kafka®-compatible architecture where brokers don't store data on local disks. Every write goes directly to cloud object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage, which handles durability and replication. That removes inter-zone replication traffic, makes compute stateless, and lets clusters scale without moving data.

Disk-based Kafka

Three copies on block storage, replicated by brokers across zones

AZ-A

AZ-B

AZ-C

Producer

Follower + disk

Leader + disk

Follower + disk

billed per GB

Diskless Kafka

Control plane: metadata

One copy in object storage, replicated by the object store

AZ-A

AZ-B

AZ-C

Producer

Agent stateless

Agent

Object storage (S3 / GCS / Azure Blob), regional

Same Kafka protocol. No cross-zone replication, and one copy in object storage instead of three on disks.

How Does Diskless Kafka Work?

Diskless Kafka splits Kafka into three parts that classic brokers bundle together: compute (stateless agents), storage (object storage), and metadata (a sequencer that assigns offsets and tracks which file holds which records). Because no compute node owns data, any node can serve any partition.

The write path

1. Buffer

The agent collects records from producers in its own zone for a short batching window.

2. Write

One file goes to object storage, holding records from many topics and partitions. Nothing touches a local disk.

3. Sequence

The metadata layer assigns offsets and records where the file lives.

4. Acknowledge

The producer gets the same ack it would from Kafka. Data is as durable as the object store.

The read path

Consumers connect to an agent in their own zone. The agent looks up which files hold the requested offsets, fetches them from object storage, and caches them so other consumers reading the same data don't trigger repeat downloads. Reads never cross a zone to reach a "leader."

Full request lifecycle: WarpStream architecture docs.

Why stateless compute changes operations

Scaling is adding containers

New agents serve traffic immediately. No partition reassignment or replica catch-up.

No hot brokers

Load spreads across agents regardless of which partitions are busiest.

Failures are non-events

If an agent dies, clients reconnect to another. No leader election, no data to recover.

Autoscale on CPU or network

Size for today's traffic, not next year's peak.

Why Are Disks the Problem for Kafka in the Cloud?

Kafka was designed for data centers, where traffic between racks is free and disks are the cheapest durable storage. In the public cloud, both assumptions flip: traffic between availability zones is billed per gigabyte, and object storage is far cheaper per gigabyte than replicated block storage.

The replication tax

With replication factor 3 across three zones, every byte crosses zone boundaries multiple times: producer to leader, leader to each follower, and leader to consumers in other zones. Each crossing is billed at roughly $0.02 per GB on the major clouds.

~80%

of a typical cloud Kafka bill is inter-AZ networking

Producer → leader when they're in different zones

Leader → follower 1 in a second zone

Leader → follower 2 in a third zone

Leader → consumers in other zones

The storage multiplier

Replicated block storage multiplies cost twice: three copies, plus headroom so disks never fill. Starting from EBS gp3 at $0.08 per GiB-month, that adds up to $0.48 per GiB-month. A single copy in S3 costs about $0.02, already replicated by the object store, with no headroom to provision.

24x

cheaper per GiB-month with object storage

Cost per GiB-month for a 3x replicated storage system on AWS. Source: Cloud Disks Are (Really!) Expensive.

The operations burden

Stateful brokers couple compute to data. Adding a broker means moving partitions onto it; replacing a failed disk means re-replicating it. Teams over-provision because scaling is slow and risky, and in practice many clusters only ever grow.

Root cause: Kafka ties compute and storage to the same machine. Diskless architectures separate them, and most of the cloud cost and operational pain goes away with that separation.

The Diskless Spectrum: Not Everything Called “Diskless” Is the Same

Nearly every Kafka platform uses object storage now. The useful question is: where does an acknowledged write live, and what still depends on a local disk?

	Classic Kafka	Tiered storage (KIP-405)	Disk-lite / WAL hybrid	Diskless topics in a classic cluster	Diskless cluster
Where recent data lives	Classic KafkaBroker disks	Tiered storage (KIP-405)Broker disks	Disk-lite / WAL hybridLocal or block WAL, then object storage	Diskless topics in a classic clusterObject storage (diskless topics only)	Diskless clusterObject storage
Cross-AZ replication for data	Classic KafkaYes (RF3)	Tiered storage (KIP-405)Yes, for recent data	Disk-lite / WAL hybridReduced or eliminated (varies)	Diskless topics in a classic clusterEliminated for diskless topics only	Diskless clusterEliminated
Brokers hold state	Classic KafkaYes	Tiered storage (KIP-405)Yes	Disk-lite / WAL hybridYes (WAL or log)	Diskless topics in a classic clusterYes (classic topics, KRaft, internal topics)	Diskless clusterNo
Scaling moves data	Classic KafkaYes	Tiered storage (KIP-405)Yes, less of it	Disk-lite / WAL hybridVaries	Diskless topics in a classic clusterFor classic topics	Diskless clusterNo
Examples	Classic KafkaApache Kafka, Amazon MSK	Tiered storage (KIP-405)Apache Kafka 3.9+, most managed Kafka	Disk-lite / WAL hybridAutoMQ, Redpanda Cloud Topics	Diskless topics in a classic clusterKIP-1150, Aiven for Apache Kafka	Diskless clusterWarpStream

None of these designs is wrong, but they produce different systems. A WAL hybrid can deliver lower produce latency, but reintroduces stateful storage to provision, monitor, and recover. Diskless topics bring object storage into a classic cluster without changing what the cluster is. A diskless cluster removes local state everywhere.

Comparing vendors? See WarpStream vs. Redpanda and WarpStream vs. Amazon MSK.

Diskless Topics vs. a Diskless Cluster

Since KIP-1150 was accepted, "diskless" increasingly means a topic type inside an otherwise classic cluster. That's a real improvement for the topics that use it, but a cluster with diskless topics is not a diskless cluster. The difference shows up the moment a workload needs a feature diskless topics don't support.

What stays on disk in a "diskless topics" deployment

Using Aiven for Apache Kafka's diskless topics as the reference:

Internal topics stay classic. Consumer offsets, Kafka Connect, MirrorMaker 2, and Schema Registry topics are service-managed; you can't change their storage type. (source)

Cluster metadata still runs through KRaft, shared across classic and diskless topics. (source)

Diskless metadata needs its own database: a Batch Coordinator backed, in BYOC, by a PostgreSQL service created in your project. (source)

Unsupported features force classic topics, with their replication and cross-AZ costs. (source)

Feature support

Capability	Diskless topics (Aiven for Apache Kafka)	WarpStream
Transactions / exactly-once	Diskless topics (Aiven for Apache Kafka)Not supported; use classic topics	WarpStreamSupported
Compacted topics	Diskless topics (Aiven for Apache Kafka)Not supported; use classic topics	WarpStreamSupported
Kafka Streams state stores	Diskless topics (Aiven for Apache Kafka)Not supported as write target	WarpStream[TBD]
Topic conversion	Diskless topics (Aiven for Apache Kafka)Classic → diskless only; can't convert back	WarpStreamN/A: every topic is diskless
Metadata layer	Diskless topics (Aiven for Apache Kafka)Batch Coordinator (PostgreSQL in your BYOC project)	WarpStreamWarpStream control plane; nothing to run or manage in your account
Capacity model	Diskless topics (Aiven for Apache Kafka)Service plans	WarpStreamAgents autoscale on CPU / network

Sources: Aiven docs (Limitations, Diskless vs. classic topics, Architecture). WarpStream Protocol and Feature Support.

What about WarpStream's own trade-offs?

WarpStream also has a topic type with restrictions: Lightning Topics, its lowest-latency mode, doesn't support idempotent producers or transactions. The difference is what you fall back to.

In a diskless-topics cluster

The fallback is classic Kafka, along with its disks, replication, and cross-AZ costs.

On WarpStream

Both topic types are diskless. Choosing "transactional" over "fastest" never puts data back on a broker disk.

Diskless Kafka Latency: What You Actually Need

Writing to object storage is slower than writing to a local SSD, so diskless Kafka has higher produce latency by default. On WarpStream it's tunable from hundreds of milliseconds down to roughly 33ms median. The more important question is how much latency your workload can actually tolerate.

Three latency tiers, one architecture

Produce latency, ms

Median / p99

DefaultStandard object storage

[TBD] / ~400

S3 Express One Zone+ lower batch timeout

~105 / ~170

Lightning Topics+ S3 Express One Zone

~33 / ~50

Medianp99

Sources: The Art of Being Lazy(log) and Lightning Topics docs.

No added cost from Lightning Topics. If you already use S3 Express One Zone, there's no extra cost. S3 Express One Zone is priced differently from standard S3, so factor that in for the lowest tier.

Two restrictions. Idempotent producers and transactions must be disabled for Lightning Topics. Durability guarantees are identical to standard topics.

Produce latency, not end-to-end. Lightning Topics shorten the produce path; records become readable once sequenced.

Be honest about your workload

Most Kafka clusters were tuned for single-digit-millisecond latency because that was the default, not because the workloads needed it. Before paying to replicate every byte across zones for latency headroom, check what your consumers actually do with the data.

Workload	Typical tolerance	Diskless fit
Logging, metrics, traces, observability	Typical toleranceSeconds are fine	Diskless fitDefault tier
Lakehouse ingestion, Iceberg, ETL, CDC into analytics	Typical toleranceSeconds to minutes	Diskless fitDefault tier
ML feature and training pipelines, telemetry, clickstream	Typical toleranceHundreds of ms	Diskless fitDefault or S3 Express
Event-driven microservices, async user workflows	Typical toleranceTens to low hundreds of ms	Diskless fitS3 Express or Lightning
Payments, order matching, HFT, real-time bidding	Typical toleranceSingle-digit ms	Diskless fitKeep on disk-based Kafka

You don't have to put everything in one system. Eternal (Zomato, Blinkit) runs latency-insensitive logging and benchmarking on WarpStream, saving 60% and 80% respectively, and keeps latency-sensitive workloads on Confluent Cloud.

How Much Does Diskless Kafka Cost Compared to Traditional Kafka?

Usually much less in the cloud, and the savings are structural: no inter-AZ replication, one copy in object storage instead of three on block storage, and no over-provisioning. The trade-off is a new, smaller cost: object storage API requests.

One workload, priced four ways

1 GiB/s write (uncompressed) · 3 AZs · 7-day retention · 8,192 partitions · WarpStream Fundamentals tier

Self-hosted Apache Kafka

~$226,739/mo

Amazon MSK Standardfetch-from-follower on

~$205,502/mo

Amazon MSK Expressfetch-from-follower on

~$60,791/mo

WarpStream (diskless)

~$25,966/mo

88% less than Kafka

From the WarpStream pricing calculator. WarpStream total includes the WarpStream license fee. Model your workload.

Savings are smaller if you've negotiated large inter-AZ discounts with your cloud provider. The storage and operational savings still apply.

What production teams report

90%cost reduction at 100 PiB and hundreds of thousands of partitions Read the case study

83%savings vs. Amazon MSK Read the case study

60%savings vs. multi-AZ Kafka Read the case study

60%savings on logging, scaling to 25+ GiB/s Read the case study

45%savings vs. open-source Kafka for logging Read the case study

Pricing is consumption-based with no per-agent or per-core fees; write throughput starts at $0.01/GiB. See pricing.

What Does the Control Plane Do, and Where Does My Data Live?

Agents run in your VPC and write to object storage buckets in your cloud account, so raw data never leaves your environment. WarpStream's control plane stores only metadata, such as which files hold which offsets. There's nothing to run or manage in your account, and no cross-account IAM access is required.

WarpStream Cloud receives metadata only; zero access to your VPC

Why separate metadata at all?

Every leaderless diskless design needs a single source of truth for ordering. KIP-1150 uses a Batch Coordinator; Aiven runs one per cluster; WarpStream runs it as a managed, replicated control plane. The real question is who operates it and what happens if it's unavailable.

What happens if the control plane is unavailable?

The control plane is a replicated state machine operated 24/7 by WarpStream, and Enterprise customers can deploy a dedicated control plane cell. As a safety net, Ripcord Mode lets Agents keep accepting writes during an outage, in a degraded mode where idempotent producers, transactions, and some consumer operations are unavailable until it recovers.

KIP-1150: Diskless Topics in Apache Kafka

KIP-1150 was accepted by the Apache Kafka community on March 2, 2026, with 9 binding votes. It makes diskless topics an official part of Kafka's roadmap, but it's a design mandate, not shipped code. Implementation lives in follow-up KIPs still in progress, and diskless topics aren't in a stable Kafka release yet.

KIP	What it covers	Status
KIP-1150	What it coversDiskless Topics (umbrella)	StatusAccepted Mar 2, 2026
KIP-1163	What it coversDiskless Core: produce and consume paths	StatusIn discussion
KIP-1164	What it coversBatch Coordinator	StatusIn discussion

Opt-in per topic. Classic and diskless topics coexist in the same cluster.

Leaderless. Any broker can accept writes for any partition.

Some disk remains. The KIP itself acknowledges diskless topics still need some local disk.

Should you wait for upstream diskless Kafka?

If you want diskless from the Apache project itself, KIP-1150 is the path. Expect years, not months: tiered storage (KIP-405) took more than two years from acceptance to a stable release. Even then, it will be a topic type in a self-managed, broker-based cluster with its own launch limitations. WarpStream launched in 2023 and has run diskless Kafka in production at multi-GiB/s scale, with transactions, compacted topics, and schema registry already supported.

2023

WarpStream launches diskless Kafka on object storage

Sept 2024

Confluent, founded by Kafka's original creators, acquires WarpStream

Apr 2025

KIP-1150 first published

Mar 2, 2026

KIP-1150 accepted by the Apache Kafka community

FAQs

Don't see an answer to your question? Check our docs, or contact us directly.

Get Started

Contact

Ask Our AI

What is diskless Kafka?

Diskless Kafka is a Kafka-compatible architecture where brokers don't store data on local disks. Writes go straight to cloud object storage (S3, GCS, or Azure Blob), which provides durability and replication. That eliminates inter-zone replication traffic, makes compute stateless, and lets clusters scale without moving data.

Is diskless Kafka production-ready?

Yes. WarpStream has run diskless Kafka in production since 2023 at companies including Robinhood, Grafana Labs, Character.AI, Cursor, and Goldsky at multi-GiB/s scale. The Apache Kafka community accepted KIP-1150 in March 2026, but upstream diskless topics aren't yet in a stable release.

Is diskless Kafka slower than regular Kafka?

By default, yes: writing to object storage takes longer than writing to a local SSD. WarpStream's latency is tunable, from about 400ms p99 by default, to about 105ms median on S3 Express One Zone, to about 33ms median and 50ms p99 produce latency with Lightning Topics.

Do I really need low-latency Kafka?

Often not. Observability, logging, ETL, CDC into analytics, and lakehouse ingestion typically tolerate hundreds of milliseconds or more. Payments, order matching, and high-frequency trading genuinely need single-digit milliseconds and are better served by disk-based Kafka. Many teams run both.

What is the difference between diskless Kafka and tiered storage?

Tiered storage (KIP-405) keeps recent data on broker disks and moves older segments to object storage, so brokers stay stateful and cross-zone replication continues for recent data. Diskless Kafka writes everything to object storage from the start.

What is the difference between diskless topics and a diskless cluster?

Diskless topics (KIP-1150, Aiven for Apache Kafka) are an opt-in topic type inside a classic cluster; brokers, internal topics, and topics needing unsupported features stay disk-based. A diskless cluster, like WarpStream, has no local data anywhere, including transactional and compacted topics.

Does diskless Kafka support transactions and exactly-once semantics?

It depends on the implementation. WarpStream supports transactions and exactly-once on standard topics; Lightning Topics don't. Aiven's diskless topics don't currently support transactions or compacted topics, so those workloads use classic, disk-based topics.

Do I need to change my Kafka clients?

No. WarpStream implements the Apache Kafka protocol and works with standard clients, including the Java client, librdkafka-based clients, and franz-go. You change the bootstrap URL.

How much cheaper is diskless Kafka?

At 1 GiB/s with 7-day retention, WarpStream's calculator shows about 88% less than self-hosted Kafka and 87% less than Amazon MSK Standard. Production customers report 45–90% savings.

Does my data leave my cloud account?

No. Agents run in your VPC and write to buckets in your account. The control plane stores only metadata, never message data. No cross-account IAM access is required, and there's nothing to run or manage in your account.

What happens if WarpStream's control plane goes down?

The control plane is replicated and operated 24/7, with a dedicated cell available on Enterprise. If it's unavailable, Ripcord Mode lets Agents keep accepting writes in a degraded mode until it recovers.

How is WarpStream different from Confluent Cloud Freight clusters?

Freight clusters are a Confluent Cloud cluster type for high-throughput, relaxed-latency workloads, fully managed in Confluent's cloud. WarpStream is BYOC: Agents and data stay in your own cloud account.

Is WarpStream part of Confluent or IBM?

Confluent acquired WarpStream in September 2024, and IBM completed its acquisition of Confluent in March 2026. WarpStream operates as its own product with its own website, console, pricing, and roadmap, separate from Confluent Cloud.

Which clouds does diskless Kafka run on?

WarpStream runs on AWS (S3 and S3 Express One Zone), Google Cloud (GCS), and Microsoft Azure (Blob Storage), as well as S3-compatible object stores.

What Is Diskless Apache Kafka®?

Diskless Kafka

How Does Diskless Kafka Work?

The write path

The read path

Why stateless compute changes operations

Why Are Disks the Problem for Kafka in the Cloud?

The replication tax

The storage multiplier

The operations burden

The Diskless Spectrum: Not Everything Called “Diskless” Is the Same

Diskless Topics vs. a Diskless Cluster

What stays on disk in a "diskless topics" deployment

Feature support

What about WarpStream's own trade-offs?

Diskless Kafka Latency: What You Actually Need

Three latency tiers, one architecture

Be honest about your workload

How Much Does Diskless Kafka Cost Compared to Traditional Kafka?

What production teams report

What Does the Control Plane Do, and Where Does My Data Live?

Why separate metadata at all?

What happens if the control plane is unavailable?

KIP-1150: Diskless Topics in Apache Kafka

Should you wait for upstream diskless Kafka?

Who Runs Diskless Kafka in Production?

How to Move to Diskless Kafka

Diskless Kafka, in your own cloud account

Explore WarpStream

FAQs