All Posts

WarpStream Blog

WarpStream is Dead, Long Live WarpStream

WarpStream is joining Confluent to bring data streaming to customer's cloud accounts with WarpStream BYOC.

Tiered Storage Won’t Fix Kafka

Tiered storage is a hot topic in the world of data streaming systems, and for good reason. Cloud disks are (really) expensive, object storage is cheap, and in most cases, live consumers are just reading the most recently written data. Paying for expensive cloud disks to store historical data isn’t cost-effective, so historical data should be moved (tiered) to object storage. On paper, it makes all the sense in the world.

Kafka is dead, long live Kafka

Chances are you probably had a strong reaction to the title of this post. In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it.
Latest Posts

Announcing Schema Validation with AWS Glue Schema Registry

Sep 25, 2024
Brian Shih
WarpStream now supports AWS Glue Schema Registries, in addition to the Kafka-compatible schema registries. The WarpStream Agent can use schemas stored in the user’s AWS Glue Schema Registries to validate records.

WarpStream is Dead, Long Live WarpStream

Sep 9, 2024
Richard Artoul
WarpStream is joining Confluent to bring data streaming to customer's cloud accounts with WarpStream BYOC.

Dealing with rejection (in distributed systems)

Aug 13, 2024
Richard Artoul
Backpressure is a really simple concept. When the system is nearing overload, it should start “saying no” by slowing down or rejecting requests. Of course, the big question is: How do we know when we should reject a request?

Announcing WarpStream Schema Validation

Jul 18, 2024
Brian Shih
WarpStream now has the capability to connect to external schema registries, and verify that records actually conform to the provided schema.

The Kafka Metric You're Not Using: Stop Counting Messages, Start Measuring Time

Jul 16, 2024
Aratz Manterola Lasa
Traditional offset-based monitoring can be misleading due to varying message sizes and consumption rates. To address this, you can introduce a time-based metric for a more accurate assessment of consumer group lag.

Multiple Regions, Single Pane of Glass

Jun 20, 2024
Emmanuel Pot
How we built support for running WarpStream's control plane and Metadata Store in multiple regions, while still presenting our platform as a single pane of glass.

Secure by default: How WarpStream’s BYOC deployment model secures the most sensitive workloads

Jun 10, 2024
Caleb Grillo
WarpStream's Zero Disk Architecture enables a BYOC deployment model that is secure by default and does not require any external access to the customer's environment.

Announcing Bento, the open source fork of the project formerly known as Benthos

May 31, 2024
Richard Artoul
Announcing Bento, the open source fork of the project formerly known as Benthos.

Zero Disks is Better (for Kafka)

May 23, 2024
Richard Artoul
Follow up to "Tiered Storage Won't Fix Kafka", this post covers all the different advantages that WarpStream's Zero Disk Architecture provides over Apache Kafka.

Introducing WarpStream Managed Data Pipelines for BYOC clusters

May 14, 2024
Richard Artoul
Managed Data Pipelines provide a fully-managed SaaS user experience for Bento, without sacrificing any of the cost benefits, data sovereignty, or deployment flexibility of the BYOC deployment model.

Tiered Storage Won’t Fix Kafka

Apr 28, 2024
Richard Artoul
Tiered storage is a hot topic in the world of data streaming systems, and for good reason. Cloud disks are (really) expensive, object storage is cheap, and in most cases, live consumers are just reading the most recently written data. Paying for expensive cloud disks to store historical data isn’t cost-effective, so historical data should be moved (tiered) to object storage. On paper, it makes all the sense in the world.

Cloud Disks are (Really!) Expensive

Apr 20, 2024
Richard Artoul
Cloud disks are expensive. Really expensive. Most engineers intuitively understand this, but the magnitudes are worth considering.

Real-Time Website Security Monitoring with WarpStream, RisingWave, and Grafana

Apr 9, 2024
Fahad Shah
This blog is guest authored by Fahad Shah from RisingWave, and cross-posted from RisingWave's blog. In this blog, we have presented the development of a real-time security threat monitoring system that integrates RisingWave, WarpStream, and Grafana. The setup process for the entire system is quite straightforward. To monitor each metric, you only need to create a single materialized view in RisingWave and visualize it in Grafana.

Fancy Stream Processing Made (even more) Operationally Mundane

Mar 26, 2024
Richard Artoul
We’re excited to announce that WarpStream now natively embeds Bento, a stateless stream processing framework that connects to many data sources and sinks. Bento offers much of the functionality of Kafka Connect, as well as additional lightweight stream processing functions.

The Original Sin of Cloud Infrastructure

Mar 14, 2024
Richard Artoul
Many of today's most highly adopted open source “big data” infrastructure projects – like Cassandra, Kafka, Hadoop, etc. – follow a common story. A large company, startup or otherwise, faces a unique, high scale infrastructure challenge that's poorly supported by existing tools. They create an internal solution for their specific needs, and then later (kindly) open source it for the greater community to use. Now, even smaller startups can benefit from the work and expertise of these seasoned engineering teams. Great, right?

Deterministic Simulation Testing for Our Entire SaaS

Mar 12, 2024
Richard Artoul
How we leverage Antithesis to deterministically simulate our entire SaaS platform and verify its correctness, all the way from signup to running entire Kafka workloads.

Public Benchmarks and TCO Analysis

Mar 5, 2024
Richard Artoul
Benchmarking databases – and maintaining fairness and integrity while doing so – is a notoriously difficult task to get right, especially in the data streaming space. Vendors want their systems to produce mouth watering results, and so unnatural configurations divorced from customer realities (AKA “vanity” benchmarks) get tested, and it's ultimately the end-user that is left holding the bag when they realize that their actual TCO is a lot higher than they were led to believe.

Kafka as a KV Store: deduplicating millions of keys with just 128 MiB of RAM

Mar 4, 2024
Manu Cupcic
A huge part of building a drop-in replacement for Apache Kafka® was implementing support for compacted topics. The primary difference between a “regular” topic in Kafka and a “compacted” topic is that Kafka will asynchronously delete records from compacted topics that are not the latest record for a specific key within a given partition.

Anatomy of a serverless usage based billing system

Feb 8, 2024
Richard Artoul
Serverless products and usage based billing models go hand in hand, almost by definition. A product that is truly serverless effectively has to have usage based pricing, otherwise it’s not really serverless!

Middleware saves 85% by replacing Apache Kafka with WarpStream to power AI observability platform

Feb 1, 2024
Richard Artoul
Middleware is a cloud observability company that uses AI to help customers identify, understand, and fix issues across their infrastructure. It’s probably not a surprise to you that their entire business relies on scalable streaming infrastructure: they’re ingesting and processing tens of TiBs of telemetry every day, and need to process it very efficiently.

S3 Express is All You Need

Nov 28, 2023
Richard Artoul
The future of modern data infrastructure is object storage.

Unlocking Idempotency with Retroactive Tombstones

Nov 18, 2023
Richard Artoul
How we separated data from metadata to build support for idempotent producers in our Apache Kafka protocol layer.

Minimizing S3 API Costs with Distributed mmap

Oct 9, 2023
Richard Artoul
We first introduced WarpStream in our blog post: "Kafka is Dead, Long Live Kafka", but to summarize: WarpStream is a Kafka protocol compatible data streaming system built directly on top of object storage.

WarpStream + Materialize: Simpler Streaming for Operational Data Products

Oct 6, 2023
Richard Artoul, Andy Hattemer
If you’re on a Data or Data Platforms team, you’ve probably already seen the productivity boost that comes from pulling business logic out of various ETL pipelines, queries, and scripts and centralizing it in SQL in a clean, version-controlled git repo managed by dbt. The engine that unlocked this approach is the analytical data warehouse: typically Snowflake or BigQuery.

Hacking the Kafka PRoTocOL

Sep 18, 2023
Richard Artoul
How we built stateless load balancing into a protocol that was never designed for it.

Kafka is dead, long live Kafka

Jul 25, 2023
Richard Artoul
Chances are you probably had a strong reaction to the title of this post. In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it.
No results found.
Clear Search