Tiered storage is a hot topic in the world of data streaming systems, and for good reason. Cloud disks are (really) expensive, object storage is cheap, and in most cases, live consumers are just reading the most recently written data.

Cloud disks are expensive. Really expensive. Most engineers intuitively understand this, but the magnitudes are worth considering.

‍The troubled past of open source cloud infrastructure, and why we raised $20M to try and fix it.

How we leverage Antithesis to deterministically simulate our entire SaaS platform and verify its correctness, all the way from signup to running entire Kafka workloads.

A huge part of building a drop-in replacement for Apache Kafka® was implementing support for compacted topics. The primary difference between a “regular” topic in Kafka and a “compacted” topic is that Kafka will asynchronously delete records from compacted topics that are not the latest record for a specific key within a given partition.

Serverless products and usage based billing models go hand in hand, almost by definition. A product that is truly serverless effectively has to have usage based pricing, otherwise it’s not really serverless!

The future of modern data infrastructure is object storage.

How we separated data from metadata to build support for idempotent producers in our Apache Kafka protocol layer.

We first introduced WarpStream in our blog post: "Kafka is Dead, Long Live Kafka", but to summarize: WarpStream is a Kafka protocol compatible data streaming system built directly on top of object storage.

How we built stateless load balancing into a protocol that was never designed for it.

The problems with Kafka, and how we created WarpStream to solve them.