WarpStream Executive Summary

Zero Interzone Networking

One of the most expensive parts of running data streaming systems like Kafka in cloud environments is the cost of interzone networking.

Consider a typical Kafka cluster deployed across three availability zones for high availability. In a perfectly-ideal scenario, where fetch from follower is enabled, every GiB of data transmitted through the system will cost a minimum of $0.053. At scale, this adds up very quickly, especially considering that for the same cost, this data could be stored in object storage for more than two months.

For high-volume workloads, interzone networking fees quickly become the largest line item on the bill. In the pricing calculator, they’re 60% of the total cost of running a 1 GiB/s Kafka cluster with 1 day of retention even with fetch from follower enabled, and 72% with Kafka’s default of consuming from the leader. WarpStream eliminates these interzone networking fees entirely by streaming data to and from object storage directly.²

Writing and reading from object storage isn’t free, but WarpStream’s storage engine was designed from the ground up around the unit economics of object storage. As a result, the object storage API fees incurred by WarpStream cost less than 1/10th of what you’d otherwise spend on interzone networking with a traditional Kafka cluster (3% at 1 GiB/s).

Note that WarpStream isn’t magic, we do make a small trade-off for latency. In WarpStream’s default configuration, Produce requests take approximately 500ms to complete at the P99, and end-to-end latency is just under one second at the P99. This is completely configurable. The more latency you can trade, the more savings you capture.

In our experience, this latency trade-off is acceptable for the vast majority of workloads, especially given WarpStream’s other benefits.

With Lightning Topics, that trade-off shrinks to nearly zero. Lightning Topics are a topic type where Agents skip committing metadata to the control plane in the critical path of a Produce request, instead journaling the write to object storage and committing metadata asynchronously.

Combined with S3 Express One Zone, Lightning Topics achieve 33ms median / 50ms P99 produce latency, a 70% reduction versus WarpStream’s previous best-in-class low-latency configuration, with the same durability guarantees as regular topics (any acknowledged write is durable and eventually consumable).

Lightning Topics aren’t a fit for every workload, e.g., transactions, exactly-once semantics, and idempotent producers still require standard topics, but for the majority of latency-sensitive use cases, low latency and low cost are no longer in tension.

Note that latency is the only trade-off that WarpStream’s architecture makes. WarpStream always provides the same durability and correctness guarantees as Apache Kafka running with the most strict configuration (acks=all). Produce requests are never acknowledged until the data has been durably persisted in object storage, so any Produce requests that succeed are guaranteed to be durable and available.

Zero Disks

Zero Ops Auto-Scaling

Kafka and its alternatives are infamous for being difficult to manage. WarpStream’s diskless architecture makes WarpStream trivial to manage because the Agents are completely stateless. WarpStream is so easy to manage that it can be auto-scaled on CPU usage like a stateless web server with no special tooling or expertise.

WarpStream’s ability to auto-scale doesn’t just free up valuable engineering time, it also dramatically reduces infrastructure costs. Auto-scaling Kafka and alternative systems is almost impossible (because the brokers are stateful), so the vast majority of Kafka clusters are overprovisioned by at least 2–4x to handle expected peak throughput.

On the other hand, auto-scaling WarpStream clusters are always perfectly provisioned for your current workload, so you only pay for what you use. Teams can skip the weekly burden of partition rebalancing, scaling headaches, volume management, capacity planning, and more.

Zero Access

WarpStream was designed from the ground up with a data plane / control plane split such that all of the data in the cluster is processed by your own compute (VMs, Kubernetes, bare metal, etc.) and stored in your object storage buckets. Raw data never leaves your own environment, whether that’s a cloud VPC, on-prem datacenter, or hybrid setup. The only data that ever leaves your own environment is metadata required to provide the abstraction of Kafka.

This architecture makes WarpStream secure by default, and much more secure than alternative BYOC providers that require invasive cross-account privileges and complex networking setups like VPC peering.³

The ability to escalate privileges in an emergency is a requirement for all competitor BYOC solutions because their products are incredibly stateful and it’s impossible to manage stateful services in someone else’s cloud account without the ability to escalate privileges. The ability to escalate privileges in an emergency means that our competitors’ “Zero Trust” BYOC deployments provide a false sense of security.

Effectively, these BYOC solutions often have a break glass provision that allows them to access customer data.

The key differentiating factor with WarpStream is that its diskless architecture makes the WarpStream Agents completely stateless, and thus trivial for the customer to manage themselves. WarpStream doesn’t need any ability to escalate privileges, even for emergencies. This makes WarpStream’s BYOC product the most secure offering on the market.

Agent Groups are distinct sets of Agents that all belong to the same logical cluster that use a shared object storage bucket as the storage and network layer.

Groups enable a single logical cluster to be split into many different “groups” that are isolated at the network / service discovery layer.

You can leverage Agent Groups to:

• Isolate analytical workloads from transactional ones.
• Flex a single, logical cluster across multiple VPCs, regions, or cloud providers without the need for complex VPC peering.
• Split workloads at the hardware level, while still sharing data sets.

Watch the engineering lightboard video on Agent Groups

WarpStream Managed Data Pipelines combine the power of WarpStream’s control plane with Bento, a powerful, yet simplified stream processing and integration tool that offers much of the functionality of Kafka Connect, as well as additional stream processing functionality like single message transforms, aggregations, multiplexing, enrichments, and more. It also has native support for WebAssembly (WASM) for more advanced processing.

Like WarpStream, Bento is written in Go, so we embedded it natively into the WarpStream Agents to make it possible to perform ETL and stream processing tasks with a single, stateless binary and no additional infrastructure beyond the WarpStream Agents.

By embedding Bento inside the WarpStream Agents, data pipelines can be written, edited, and fully controlled from WarpStream’s managed control plane.

Managed data pipelines run in your own environment, on your VMs, and process data in your buckets. WarpStream has no access to any of the data processed by your managed pipelines. Instead, WarpStream’s control plane enhances Bento by adding:

• A helpful UI or our API / Terraform for creating and editing the pipelines.
• The ability to pause and resume pipelines dynamically.
• A full version control system so pipeline configuration is versioned and can easily be rolled forward or backward.

Completing a data engineering task like multiplexing, where you combine multiple data streams into a single, consolidated stream, is a straightforward YAML file:

input:
  kafka:
    addresses: [ TODO ]
    topics: [ foo, bar ]
    consumer_group: foogroup

output:
  switch:
    cases:
      - check: doc.tags.contains("AWS")
        output:
          aws_sqs:
            url: https://sqs.us-west-2.amazonaws.com/TODO/TODO
            max_in_flight: 20

      - output:
          redis_pubsub:
            url: tcp://TODO:6379
            channel: baz
            max_in_flight: 20

Because Bento is embedded in WarpStream Agents, you can leverage Pipeline Groups to isolate pipelines to particular groups of Agents. This allows you to isolate the larger pipelines onto a dedicated group of Agents so that they can’t interfere with the smaller pipelines.

Orbit allows you to automatically replicate topics (including record offsets), consumer groups, offset gaps, ACLs, and cluster configurations, without an additional component to manage unlike Replicator or MirrorMaker2. It works with any source system that is Apache Kafka® protocol compatible and can replicate from more than one source cluster into WarpStream, making migrating from sources like Amazon MSK to WarpStream a breeze.

In addition to copying your data 1:1 to WarpStream, you can use Orbit for:

• Disaster recovery. Replicate your primary Kafka cluster to a secondary WarpStream cluster and instantly flip over to the read replica if you have a hardware failure or networking issues to minimize downtime and data loss.
• Cost-effective read replicas. Offload analytical or batch jobs to isolated hardware. Scale read throughput infinitely and on demand in seconds. Set up dedicated clone clusters per team without impacting production workloads.
• Performant tiered storage. Transform any existing Kafka cluster into tiered storage that scales to massive read throughput with consistent performance. Reduce your storage costs by 30x.
• Low-latency geo replication. Geographically distribute your data to read from the nearest read-only replica to reduce latency.

WarpStream’s BYOC Schema Registry serves as a central repository to store and retrieve schemas used to serialize / deserialize and validate your Kafka records. The Schema Registry exposes a REST server that is API-compatible with Confluent’s Schema Registry.

All schemas are stored in your object store while metadata that requires consensus (e.g., schema ID, versioning, etc.) is offloaded to WarpStream’s control plane. WarpStream supports schema formats of Avro, JSON, and Protobuf.

WarpStream’s Schema Registry is embedded natively into the WarpStream Agents. This makes deploying schema registries as easy as deploying the stateless WarpStream Agents.

Unlike traditional Kafka schema registries in which only the leader node is capable of performing writes to the underlying Kafka log, any WarpStream Agent is capable of both writing to and reading from the registry. Scaling the number of Schema Registry Agents during traffic spikes is trivial due to the stateless nature of WarpStream Agents.

With WarpStream Schema Validation, you can configure the WarpStream Agent to not only validate that the record contains a valid schema ID, but that the record conforms to the corresponding schema. The Agent can then reject or emit metrics when it receives invalid records.

The process of how producers, consumers, the WarpStream Agent, and your preferred Schema Registry operate is illustrated in this diagram:

WarpStream Schema Validation can be used with WarpStream BYOC Schema Registry or any Kafka-compatible Schema Registry or AWS Glue.

WarpStream Schema Linking can migrate schemas from Confluent-compatible schema registries to WarpStream’s BYOC Schema Registry. During migration, your schemas will never leave your own environment and object storage buckets.

In addition to migrating schemas, WarpStream Schema Linking also preserves schema IDs, subjects, subject versions, compatibility rules, etc. It even preserves whether subject versions are soft deleted. This means that after the migration, the destination schema registry should behave identically to the source schema registry from an API level.

WarpStream also has additional Data Governance features like:

• Access Control Lists (ACLs). Dictate what an authenticated user or service is allowed to do, such as reading, writing, or deleting, on specific cluster resources like topics or consumer groups.
• ACL Shadowing. Evaluate Kafka ACLs against live traffic without enforcing authorization decisions. Validate ACL behavior before enabling full ACL enforcement.
• Audit Logs. Get a complete, structured record of every authentication action, authorization decision, and platform operation across your WarpStream clusters.

WarpStream Tableflow is an Iceberg-native database that materializes tables from any Kafka topic, automating ingestion, compaction, and table maintenance. It’s a complete, real-time data lake solution.

Tableflow is not a connector or a “zero-copy” version of Kafka tiered storage. It’s a stateless, auto-scaling, single-binary database that runs in your own environment, connects to your Kafka cluster(s), and produces Iceberg tables to a declarative YAML specification, handling ingestion, compaction, retention enforcement, and table maintenance automatically.

Tableflow follows the same zero-access BYOC design as the rest of WarpStream: raw data is processed on your compute and stored in your object storage buckets, and is never accessible to WarpStream.

Tableflow is vendor-agnostic and works with any Kafka-compatible source (open-source Kafka, Amazon MSK, Confluent Cloud, Redpanda, or WarpStream itself), including ingesting from multiple clusters simultaneously into a single lake.

Catalogs like AWS Glue, Databricks Unity Catalog, and Iceberg REST, and query engines like Athena, BigQuery, DuckDB, ClickHouse, Trino, Snowflake, and Databricks are supported.

Kafka can technically achieve zero data loss across regions, but only with heavy manual configuration, careful leader-election tuning, and expensive cross-region networking, and even then, only if everything goes right.

WarpStream Multi-Region Clusters guarantee zero data loss (RPO=0) out of the box, with zero additional operational overhead. Both the data plane and control plane are made RPO=0 independently: Agents write to multiple object storage buckets across more than one region (guaranteeing at least one durable copy survives a regional outage), while WarpStream manages a multi-region control plane for you.

Unlike Apache Kafka’s primary/secondary model, where each topic-partition has a single leader living in one region, and a regional failure requires a new leader election, complex networking, and careful KRaft configuration, WarpStream handles multi-region consensus and automatic failover natively, with no tuning and no trade-offs.

Multi-Region Clusters are available on all Enterprise-tier WarpStream clusters, and backed by WarpStream’s 99.999% SLA.

Multi-Region Clusters protect you from a regional outage. Ripcord Mode protects you from a control plane outage.

By design, WarpStream’s control plane handles metadata and coordination while your data plane (the Agents) does all the heavy lifting of moving data to and from object storage in your own environment. But what happens if the control plane itself becomes unavailable?

With Ripcord Mode enabled, Agents continue processing Produce requests during a control plane outage by operating in a degraded-but-functional state, rather than halting your pipeline entirely.

Ripcord Mode is a resilience backstop, not a normal operating mode: idempotent producers, transactions, and some consumer operations aren’t supported while it’s active, and Agents running in Ripcord mode effectively treat all topics as Lightning Topics for the duration of the outage. WarpStream’s control plane already carries strong availability guarantees on its own. Ripcord Mode exists as an extra layer of protection for teams that can’t tolerate a pipeline stall under any circumstances.⁴

Open Source Apache Kafka (OSS Kafka)

Everything described in the WarpStream Zero sections applies to Apache Kafka. Apache Kafka is fundamentally disk-based: every broker relies on local storage and multi-broker replication for durability. WarpStream is significantly more cost-effective, scalable, and easier to operate than self-hosting disk-based OSS Kafka.

To make this concrete, consider the cost difference between running a 1 GiB/s workload with 3x consumer fan out:

WarpStream is 67% cheaper than running OSS Kafka yourself (including WarpStream’s licensing fee and not including engineering hours spent maintaining OSS Kafka) in the best-case scenario where fetch from follower is properly configured.

Amazon MSK (Managed Streaming for Apache Kafka) is essentially just managed EC2 instances running open-source, disk-based Kafka. However, MSK has a few advantages over running Kafka yourself:

• AWS takes care of version upgrades and provides a few additional utilities for scaling up clusters.
• Integration with IAM.
• Simplified networking options.
• Interzone networking for replication between brokers is free, although you still have to pay for interzone networking between your producers/consumers and the brokers.

On the flip side, MSK still inherits almost all of the disadvantages of running open-source Kafka yourself:

• Operational burden is still heavy. MSK doesn’t rebalance partitions automatically or auto-scale. In practice this means that the vast majority of MSK users run their clusters heavily overprovisioned which makes it extremely expensive.
• Scaling up can take hours or even days due to the fact that the brokers are stateful and large amounts of data must be copied around when changing the topology of the cluster.
• Triply-replicated EBS storage is extremely expensive.
• Scheduled patching does rolling broker restarts.

In addition, MSK has some additional disadvantages not present when running open source Kafka yourself:

• AWS charges a ~2–2.5x premium on EC2 instances.
• MSK doesn’t let you tune the brokers as much as you can with open source Kafka, making it an awkward in-between where you give up a lot of control, but it still doesn’t really present itself as a managed service.
• No multi-cloud capabilities or on-prem option.

The sample pricing below shows that TCO (total cost of ownership) with WarpStream is 48% cheaper than Amazon MSK at 1 GiB/s of write throughput with 1 day of retention and 87% cheaper than Amazon MSK at 7 days of retention, in best-case scenarios where fetch from follower is properly configured.

Additionally, MSK has a SLA of 99.9% whereas WarpStream’s SLA on its Fundamentals tier starts at 99.9% and goes as high as 99.999% on its Enterprise tier (if using Multi-Region Clusters).

Amazon MSK Serverless alleviates some of the operational issues with regular Amazon MSK, making it an attractive choice for smaller use cases and dev / staging / pre-prod environments.

However, MSK Serverless has several downsides compared to Amazon MSK:

• It’s significantly more expensive for high-throughput workloads: at 1 GiB/s with 1 day of retention, MSK Serverless costs 4.7x as much as a provisioned Amazon MSK cluster, and 9.2x as much as WarpStream.
• Maximum write throughput per partition is 5 MiB/s.
• Maximum read throughput per partition is 10 MiB/s.
• Maximum total cluster ingress of 200 MiB/s and maximum total cluster egress of 400 MiB/s (2x fan out).
• Maximum number of client connections is 3,000.
• Maximum number of leader partitions is 2,400 per cluster.
• Maximum number of clusters per account is 10.
• IAM authentication only. No SASL/SCRAM or mTLS.

A maximum number of 3,000 client connections is particularly problematic because every Kafka client needs to connect to every MSK Serverless broker, but you don’t control the number of brokers with the MSK Serverless product, so this limit can be exhausted very quickly.

The sample pricing below shows that TCO (total cost of ownership) with WarpStream is 89% cheaper than Amazon MSK Serverless at 1 GiB/s of write throughput with 1 day of retention.

Redpanda is basically Kafka++. It’s a rewrite of Kafka in C++, but it retains local disks and almost the same exact architecture as Apache Kafka, and therefore inherits the exact same prohibitively-expensive cost structure and operational issues.

In fact, Redpanda is actually more expensive to use than just running Kafka yourself because Redpanda has almost the exact same cost structure as Apache Kafka in terms of interzone networking and expensive cloud disks, but you’ll also have to pay significant licensing and support fees.

We don’t provide a direct cost comparison of WarpStream against Redpanda because Redpanda doesn’t have public pricing for their flagship products. However, Redpanda has almost the exact same cost structure as Apache Kafka, and the total cost of ownership for WarpStream BYOC (cloud infrastructure and WarpStream licensing fees) using our list prices is 82% cheaper than just the infrastructure costs for running OSS Kafka at the pricing calculator’s default workload (100 MiB/s with 7 days of retention). As a result of this, it’s simply not possible for Redpanda to offer their product at a total cost of ownership that is remotely competitive with WarpStream. Their architecture simply doesn’t allow it.

For Redpanda Serverless, there are these downsides:

• Your data lives in Redpanda’s cloud, not your account.
• 100 MBps write, 300 MBps read, and 5,000 partitions limits per cluster.
• While it auto-scales, the above limits apply.

Finally, as noted in our BYOC section, Redpanda’s BYOC product is significantly different than WarpStream and requires a very high number of cross-account privileges and the ability to escalate to root in emergencies, whereas WarpStream’s BYOC model requires zero cross-account privileges and WarpStream engineers have no ability to escalate to root access ever.

In addition to the security issue flagged above, Redpanda BYOC has limitations of:

• Tiered storage, not diskless. Fresh writes land on broker-local disks and are replicated across zones; only older segments offload to object storage. You pay for both tiers, plus interzone replication.
• Fixed usage tiers, no on-demand autoscaling. You select one of nine tiers, from 20 MBps to 2,000 MBps ingress. Outgrowing your tier means moving to a larger one.
• Published ceilings: 2,000 MBps (2 GB/s) ingress, 4,000 MBps (4 GB/s) egress, and 112,500 partitions at the top tier, with lower partition limits at higher replication factors. Azure supports tiers 1–5 only (400 MBps ingress max).

AutoMQ

AutoMQ is a fork of Apache Kafka that replaces the storage layer with object storage while retaining Kafka’s original broker model and JVM codebase. The distinction from WarpStream is architectural, not incremental: WarpStream replaced the entire broker model with stateless Agents, while AutoMQ kept brokers and moved the storage underneath them.

That difference has real operational consequences. AutoMQ brokers still own a write-ahead log (WAL), so when a broker fails, its unflushed WAL data must be recovered before its partitions are fully back, which can cause latency spikes and temporary unavailability. A WarpStream Agent failure, by contrast, is a non-event: the Agent is simply replaced, with zero data recovery required.

AutoMQ also inherits Kafka’s JVM operational burden (GC tuning, heap sizing, pause-latency spikes, etc.) and keeps Kafka’s KRaft metadata layer. WarpStream’s metadata is fully managed and never runs in your account.

On latency, AutoMQ’s sub-10ms figures require a separate low-latency WAL (EBS, Regional EBS, or NFS/FSx), which means provisioning and paying for another stateful storage layer on top of compute and object storage. With S3 as the WAL, AutoMQ’s latency is closer to 500ms. WarpStream Lightning Topics achieve 33ms median produce latency on S3 Express One Zone, with no additional storage layer required.

AutoMQ is offered as BYOC or as self-managed Software. With AutoMQ BYOC, both the control plane and brokers run inside your cloud account, and AutoMQ performs maintenance only after you authorize it, along with service-account access to a bucket for health monitoring. With AutoMQ Software, you operate everything yourself. WarpStream BYOC runs only stateless Agents in your account; its control plane is hosted and operated by WarpStream, and it needs zero access to your account or data.

AutoMQ also prices in AKUs, capacity units of 30 MiB/s write or 60 MiB/s read, while WarpStream charges per GiB written and stored, with public unit prices. There’s no WarpStream SDK; clients connect with standard bootstrap servers.

Aiven for Apache Kafka

Aiven for Apache Kafka offers Diskless Topics (originally launched as “Inkless”) as a deployment option that runs in your own cloud account. It’s currently in limited availability, by request, for BYOC projects on AWS. Diskless topics store data in object storage, but they sit inside an otherwise classic, broker-based Kafka cluster: diskless and classic topics coexist side by side, and Kafka’s internal topics (consumer offsets, Kafka Connect, MirrorMaker 2, and Schema Registry) always remain classic, disk-backed topics.

This is a meaningfully different model from WarpStream, where the entire cluster is diskless, internal topics included. With WarpStream, there’s no topic-type decision to make and no classic fallback mode to reason about. Even WarpStream’s low-latency Lightning Topics are diskless, so choosing transactional over fastest never puts data back on a broker disk.

Aiven’s hybrid model also means brokers and capacity planning don’t go away. Cluster metadata still runs through KRaft, broker capacity is set by your service plan, and classic topics carry all the operational overhead of traditional Kafka. WarpStream’s stateless Agents autoscale on CPU, with nothing to size in advance.

Architecturally, Aiven’s diskless topics are coordinated by a single-leader Batch Coordinator that tracks offsets and batch locations. In BYOC, it’s backed by an Aiven-managed PostgreSQL service in your cloud account, a second metadata system alongside KRaft. Aiven upgrades that database’s plan as load grows, and the upgraded plan shows up on your bill. WarpStream’s metadata lives in a managed control plane, with nothing to run in your account.

Diskless topics on Aiven also don’t support transactions, compacted topics, or Kafka Streams state stores. Those workloads must use classic topics, and existing classic or tiered topics can’t be converted to diskless, so teams end up running both models side by side, and migrating existing topics means creating new ones.