WarpStream is compatible with the Apache Kafka® protocol, but we are not running Kafka. Instead, we run a stateless Agent that has no local disks, and writes directly to object storage, which avoids 100% of cross-AZ replication charges. These stateless Agents are also much easier to operate than Kafka brokers, so you can drastically reduce the amount of time that you spend managing your streaming infrastructure by switching to WarpStream.
WarpStream effectively uses object storage as both the storage layer and the network layer, which avoids much of the cost associated with running Kafka in cloud environments. By writing directly to object storage, WarpStream avoids replicating data between Agent nodes. Instead, data is durably persisted to object storage before WarpStream provides an acknowledgement to the producer. Once data is written to object storage, replication is handled by the object store. If you use Amazon S3 as the object store, for example, this means that the data that you write to WarpStream has an eleven nines (99.999999999%) durability guarantee. And because your data is not replicated between Agents before reaching object storage, you don't pay anything extra for this durability.
In addition, the WarpStream Service Discovery system ensures that your clients are 100% zone-aligned with Agents for both Produce() and Fetch() requests, which completely avoids cross-AZ traffic in both the write path and read path by default. Kafka's Fetch from Follower feature helps reduce cross-AZ traffic in the read path, but because Kafka clients must always write to a leader partition, cross-AZ traffic in the write path is unavoidable. WarpStream Agents do not have the concept of leader partitions, so any producer can produce to any Agent.
Finally, because the compute layer is stateless, you can autoscale the Agents, which means you don't need to overprovision compute for your cluster to be able to handle peak load. This stateless model also enables WarpStream to run on smaller instance types with lower memory requirements for lower-throughput workloads. For example, whereas Kafka is recommended to run on at least d2.xlarge or r4.xlarge EC2 instances in AWS, WarpStream can use much smaller instances in less expensive instance families, and smaller instance sizes as well.