All Posts

WarpStream Blog

RSS
Subscribe
Featured Posts
ShareChat + Jason Lauritzen
June 16, 2025

Cost-Effective Logging at Scale: ShareChat’s Journey to WarpStream

By switching from open-source Apache Kafka to WarpStream, ShareChat was able to implement zero-ops auto-scaling and saved 60% vs. multi-AZ Kafka. They also shared some best practices for optimizing WarpStream.
Richard Artoul
June 15, 2025

The Hitchhiker's Guide to Disaster Recovery and Multi-Region Kafka

Disaster recovery and data sharing between regions are intertwined. We explain how to handle them on Kafka and WarpStream, as well as talk about RPO=0 Active-Active Multi-Region clusters, a new product that ensures you don't lose a single byte if an entire region goes down.
WarpStream Team
May 27, 2025

Your WarpStream Questions, Answered

The WarpStream team receives lots of questions about our architecture, pricing, unique features, and other aspects of WarpStream. We created this page to serve as an up-to-date repository or list of frequently asked questions. 
Richard Artoul
April 18, 2025

WarpStream S3 Express One Zone Benchmark and Total Cost of Ownership

Excited about the recent 85% drop in S3 Express One Zone (S3EOZ) prices? We've supported S3EOZ since December 2024. This latest pricing update means our latency is 3x better at only a 15% higher TCO.
Richard Artoul
April 10, 2025

Taking out the Trash: Garbage Collection of Object Storage at Massive Scale

Distributed systems built on object storage all have one common problem: removing files that have been logically deleted either due to data expiry or compaction. We review the pros and cons of five ways to solve this problem.
Ella Chao
April 3, 2025

A Trip Down Memory Lane: How We Resolved a Memory Leak When pprof Failed Us

Pprof is an amazing tool for debugging memory leaks, but what about when it's not enough? Read about how we used gcore and viewcore to hunt a particularly nasty memory leak in a large distributed system.
Brian Shih
March 25, 2025

Zero Ops Schema Migration: WarpStream Schema Linking

Today, we're excited to announce WarpStream Schema Linking, a tool to continuously migrate any Confluent-compatible schema registry into a WarpStream BYOC Schema Registry. WarpStream now has a comprehensive Data Governance suite to handle schema needs.
Aratz Manterola Lasa
March 18, 2025

WarpStream Diagnostics: Keep Your Data Stream Clean and Cost-Effective

We’ve released Diagnostics, a new feature for WarpStream clusters! Diagnostics continuously analyzes your clusters to identify potential problems, cost inefficiencies, and ways to make things better. It looks at the health and cost of your cluster and gives detailed explanations on how to fix and improve them.
February 24, 2025

How WarpStream Powers Grafana Labs’ Redesigned Architecture

"...we needed to ensure WarpStream could also support our scalability needs before settling on it. Given the size of Grafana Cloud Metrics, we knew it needed to handle read and write rates of tens of gigabytes per second."
Character.AI
February 20, 2025

Character.AI's Transition to WarpStream

Character.AI operates at scale, supporting over 20 million monthly active users across our services. Despite being a relatively small company, Character.AI services have significant and complex data storage needs.
Manu Cupcic
January 13, 2025

Kafka Transactions Explained (Twice!)

In this blog post we'll explain how transactions work in Kafka by comparing and contrasting the implementations of transactions in two different Kafka implementations: the official Apache Kafka project, and WarpStream.
Aratz Manterola Lasa
December 3, 2024

Getting Rid of (Kafka) Noisy Neighbors Without Having to Buy a Mansion

In this post, we’ll look at what noisy neighbors are, the current ways to handle them (cluster quotas and mirroring clusters), and how WarpStream’s solution compares in terms of elasticity, operational simplicity, and cost efficiency.
Brian Shih
November 25, 2024

Introducing WarpStream BYOC Schema Registry

WarpStream BYOC reimplements the Kafka protocol with a stateless, zero-disk cloud-native architecture, replacing Kafka brokers with WarpStream Agents to simplify operations. But data streaming extends beyond Kafka clusters alone.
Richard Artoul
November 19, 2024

The Case for Shared Storage

In this post, I’ll start off with a brief overview of “shared nothing” vs. “shared storage” architectures in general. This discussion will be a bit abstract and high-level, but the goal is to share with you some of the guiding philosophy that ultimately led to WarpStream’s architecture.
Arjun Nair
November 13, 2024

Kafka Replication Without the (Offset) Gaps

Orbit is a tool which creates identical, inexpensive, scaleable, and secure continuous replicas of Kafka clusters. It is built into WarpStream and works without any user intervention to create WarpStream replicas of any Apache Kafka-compatible source cluster.
Brian Shih
September 25, 2024

Announcing Schema Validation with AWS Glue Schema Registry

WarpStream now supports AWS Glue Schema Registries, in addition to the Kafka-compatible schema registries. The WarpStream Agent can use schemas stored in the user’s AWS Glue Schema Registries to validate records.
Richard Artoul
September 9, 2024

WarpStream is Dead, Long Live WarpStream

WarpStream is joining Confluent to bring data streaming to customer's cloud accounts with WarpStream BYOC.
Richard Artoul
August 13, 2024

Dealing with rejection (in distributed systems)

Backpressure is a really simple concept. When the system is nearing overload, it should start “saying no” by slowing down or rejecting requests. Of course, the big question is: How do we know when we should reject a request?
Brian Shih
July 18, 2024

Announcing WarpStream Schema Validation

WarpStream now has the capability to connect to external schema registries, and verify that records actually conform to the provided schema.
Aratz Manterola Lasa
July 16, 2024

The Kafka Metric You're Not Using: Stop Counting Messages, Start Measuring Time

Traditional offset-based monitoring can be misleading due to varying message sizes and consumption rates. To address this, you can introduce a time-based metric for a more accurate assessment of consumer group lag.
Emmanuel Pot
June 20, 2024

Multiple Regions, Single Pane of Glass

How we built support for running WarpStream's control plane and Metadata Store in multiple regions, while still presenting our platform as a single pane of glass.
Caleb Grillo
June 10, 2024

Secure by default: How WarpStream’s BYOC deployment model secures the most sensitive workloads

WarpStream's Zero Disk / Diskless Architecture enables a BYOC deployment model that is secure by default and does not require any external access to the customer's environment.
Richard Artoul
May 31, 2024

Announcing Bento, the open source fork of the project formerly known as Benthos

Announcing Bento, the open source fork of the project formerly known as Benthos.
Richard Artoul
May 23, 2024

Zero Disks is Better (for Kafka) - Diskless Kafka

Follow up to "Tiered Storage Won't Fix Kafka", this post covers all the different advantages that WarpStream's Zero Disk / Diskless Architecture provides over Apache Kafka.
Caleb Grillo
May 20, 2024

Pixel Federation Powers Mobile Analytics Platform with WarpStream, saves 83% over MSK

‍Pixel Federation is the developer of nearly a dozen highly popular mobile games with players from all over the world. They have millions of monthly active users, and those millions of users generate lots of events. In fact, Pixel Federation uses an event-driven architecture for almost everything: logging, events, billing, tracking game state, etc. Find out how they saved 83% using WarpStream over MSK.
Richard Artoul
May 14, 2024

Introducing WarpStream Managed Data Pipelines for BYOC clusters

Managed Data Pipelines provide a fully-managed SaaS user experience for Bento, without sacrificing any of the cost benefits, data sovereignty, or deployment flexibility of the BYOC deployment model.
Richard Artoul
April 28, 2024

Tiered Storage Won’t Fix Kafka

Tiered storage is a hot topic in the world of data streaming systems, and for good reason. Cloud disks are (really) expensive, object storage is cheap, and in most cases, live consumers are just reading the most recently written data. Paying for expensive cloud disks to store historical data isn’t cost-effective, so historical data should be moved (tiered) to object storage. On paper, it makes all the sense in the world.
Richard Artoul
April 20, 2024

Cloud Disks are (Really!) Expensive

Cloud disks are expensive. Really expensive. Most engineers intuitively understand this, but the magnitudes are worth considering.
Fahad Shah
April 9, 2024

Real-Time Website Security Monitoring with WarpStream, RisingWave, and Grafana

This blog is guest authored by Fahad Shah from RisingWave, and cross-posted from RisingWave's blog. In this blog, we have presented the development of a real-time security threat monitoring system that integrates RisingWave, WarpStream, and Grafana. The setup process for the entire system is quite straightforward. To monitor each metric, you only need to create a single materialized view in RisingWave and visualize it in Grafana.
Richard Artoul
March 26, 2024

Fancy Stream Processing Made (even more) Operationally Mundane

We’re excited to announce that WarpStream now natively embeds Bento, a stateless stream processing framework that connects to many data sources and sinks. Bento offers much of the functionality of Kafka Connect, as well as additional lightweight stream processing functions.
Richard Artoul
March 14, 2024

The Original Sin of Cloud Infrastructure

Many of today's most highly adopted open source “big data” infrastructure projects – like Cassandra, Kafka, Hadoop, etc. – follow a common story. A large company, startup or otherwise, faces a unique, high scale infrastructure challenge that's poorly supported by existing tools. They create an internal solution for their specific needs, and then later (kindly) open source it for the greater community to use. Now, even smaller startups can benefit from the work and expertise of these seasoned engineering teams. Great, right?
Richard Artoul
March 12, 2024

Deterministic Simulation Testing for Our Entire SaaS

How we leverage Antithesis to deterministically simulate our entire SaaS platform and verify its correctness, all the way from signup to running entire Kafka workloads.
Richard Artoul
March 5, 2024

Public Benchmarks TCO Analysis

Benchmarking databases – and maintaining fairness and integrity while doing so – is a notoriously difficult task to get right, especially in the data streaming space. Vendors want their systems to produce mouth watering results, and so unnatural configurations divorced from customer realities (AKA “vanity” benchmarks) get tested, and it's ultimately the end-user that is left holding the bag when they realize that their actual TCO is a lot higher than they were led to believe.
Manu Cupcic
March 4, 2024

Kafka as a KV Store: deduplicating millions of keys with just 128 MiB of RAM

A huge part of building a drop-in replacement for Apache Kafka® was implementing support for compacted topics. The primary difference between a “regular” topic in Kafka and a “compacted” topic is that Kafka will asynchronously delete records from compacted topics that are not the latest record for a specific key within a given partition.
Richard Artoul
February 8, 2024

Anatomy of a serverless usage based billing system

Serverless products and usage based billing models go hand in hand, almost by definition. A product that is truly serverless effectively has to have usage based pricing, otherwise it’s not really serverless!
Richard Artoul
November 28, 2023

S3 Express is All You Need

The future of modern data infrastructure is object storage.
Richard Artoul
November 18, 2023

Unlocking Idempotency with Retroactive Tombstones

How we separated data from metadata to build support for idempotent producers in our Apache Kafka protocol layer.
Richard Artoul
October 9, 2023

Minimizing S3 API Costs with Distributed mmap

We first introduced WarpStream in our blog post: "Kafka is Dead, Long Live Kafka", but to summarize: WarpStream is a Kafka protocol compatible data streaming system built directly on top of object storage.
Richard Artoul
September 18, 2023

Hacking the Kafka PRoTocOL

How we built stateless load balancing into a protocol that was never designed for it.
Richard Artoul
July 25, 2023

Kafka is dead, long live Kafka

Chances are you probably had a strong reaction to the title of this post. In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it.