Engineering

WarpStream Blog

Multiple Regions, Single Pane of Glass

Jun 20, 2024
Emmanuel Pot
How we built support for running WarpStream's control plane and Metadata Store in multiple regions, while still presenting our platform as a single pane of glass.

Secure by default: How WarpStream’s BYOC deployment model secures the most sensitive workloads

Jun 10, 2024
Caleb Grillo
WarpStream's Zero Disk Architecture enables a BYOC deployment model that is secure by default and does not require any external access to the customer's environment.

Announcing Bento, the open source fork of the project formerly known as Benthos

May 31, 2024
Richard Artoul
Announcing Bento, the open source fork of the project formerly known as Benthos.

Zero Disks is Better (for Kafka)

May 23, 2024
Richard Artoul
Follow up to "Tiered Storage Won't Fix Kafka", this post covers all the different advantages that WarpStream's Zero Disk Architecture provides over Apache Kafka.

Tiered Storage Won’t Fix Kafka

Apr 28, 2024
Richard Artoul
Tiered storage is a hot topic in the world of data streaming systems, and for good reason. Cloud disks are (really) expensive, object storage is cheap, and in most cases, live consumers are just reading the most recently written data.

Cloud Disks are (Really!) Expensive

Apr 20, 2024
Richard Artoul
Cloud disks are expensive. Really expensive. Most engineers intuitively understand this, but the magnitudes are worth considering.

The Original Sin of Cloud Infrastructure

Mar 14, 2024
Richard Artoul
‍The troubled past of open source cloud infrastructure, and why we raised $20M to try and fix it.

Deterministic Simulation Testing for Our Entire SaaS

Mar 12, 2024
Richard Artoul
How we leverage Antithesis to deterministically simulate our entire SaaS platform and verify its correctness, all the way from signup to running entire Kafka workloads.

Kafka as a KV Store: deduplicating millions of keys with just 128 MiB of RAM

Mar 4, 2024
Manu Cupcic
A huge part of building a drop-in replacement for Apache Kafka® was implementing support for compacted topics. The primary difference between a “regular” topic in Kafka and a “compacted” topic is that Kafka will asynchronously delete records from compacted topics that are not the latest record for a specific key within a given partition.

Anatomy of a serverless usage based billing system

Feb 8, 2024
Richard Artoul
Serverless products and usage based billing models go hand in hand, almost by definition. A product that is truly serverless effectively has to have usage based pricing, otherwise it’s not really serverless!

S3 Express is All You Need

Nov 28, 2023
Richard Artoul
The future of modern data infrastructure is object storage.

Unlocking Idempotency with Retroactive Tombstones

Nov 18, 2023
Richard Artoul
How we separated data from metadata to build support for idempotent producers in our Apache Kafka protocol layer.

Minimizing S3 API Costs with Distributed mmap

Oct 9, 2023
Richard Artoul
We first introduced WarpStream in our blog post: "Kafka is Dead, Long Live Kafka", but to summarize: WarpStream is a Kafka protocol compatible data streaming system built directly on top of object storage.

Hacking the Kafka PRoTocOL

Sep 18, 2023
Richard Artoul
How we built stateless load balancing into a protocol that was never designed for it.

Kafka is dead, long live Kafka

Jul 25, 2023
Richard Artoul
The problems with Kafka, and how we created WarpStream to solve them.