All Posts

WarpStream Blog

Tiered Storage Won’t Fix Kafka

Tiered storage is a hot topic in the world of data streaming systems, and for good reason. Cloud disks are (really) expensive, object storage is cheap, and in most cases, live consumers are just reading the most recently written data. Paying for expensive cloud disks to store historical data isn’t cost-effective, so historical data should be moved (tiered) to object storage. On paper, it makes all the sense in the world.

The Original Sin of Cloud Infrastructure

Many of today's most highly adopted open source “big data” infrastructure projects – like Cassandra, Kafka, Hadoop, etc. – follow a common story. A large company, startup or otherwise, faces a unique, high scale infrastructure challenge that's poorly supported by existing tools. They create an internal solution for their specific needs, and then later (kindly) open source it for the greater community to use. Now, even smaller startups can benefit from the work and expertise of these seasoned engineering teams. Great, right?

Kafka is dead, long live Kafka

Chances are you probably had a strong reaction to the title of this post. In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it.
Latest Posts

Announcing WarpStream Schema Validation

Jul 18, 2024
Brian Shih
WarpStream now has the capability to connect to external schema registries, and verify that records actually conform to the provided schema.

The Kafka Metric You're Not Using: Stop Counting Messages, Start Measuring Time

Jul 16, 2024
Aratz Manterola Lasa
Traditional offset-based monitoring can be misleading due to varying message sizes and consumption rates. To address this, you can introduce a time-based metric for a more accurate assessment of consumer group lag.

Multiple Regions, Single Pane of Glass

Jun 20, 2024
Emmanuel Pot
How we built support for running WarpStream's control plane and Metadata Store in multiple regions, while still presenting our platform as a single pane of glass.

Secure by default: How WarpStream’s BYOC deployment model secures the most sensitive workloads

Jun 10, 2024
Caleb Grillo
WarpStream's Zero Disk Architecture enables a BYOC deployment model that is secure by default and does not require any external access to the customer's environment.

Announcing Bento, the open source fork of the project formerly known as Benthos

May 31, 2024
Richard Artoul
Announcing Bento, the open source fork of the project formerly known as Benthos.

Zero Disks is Better (for Kafka)

May 23, 2024
Richard Artoul
Follow up to "Tiered Storage Won't Fix Kafka", this post covers all the different advantages that WarpStream's Zero Disk Architecture provides over Apache Kafka.