All Posts

WarpStream Blog

Real-Time Website Security Monitoring with WarpStream, RisingWave, and Grafana

Apr 9, 2024
Fahad Shah, Developer Advocate - RisingWave
This blog is guest authored by Fahad Shah from RisingWave, and cross-posted from RisingWave's blog. In this blog, we have presented the development of a real-time security threat monitoring system that integrates RisingWave, WarpStream, and Grafana. The setup process for the entire system is quite straightforward. To monitor each metric, you only need to create a single materialized view in RisingWave and visualize it in Grafana.

The Original Sin of Cloud Infrastructure

Mar 14, 2024
Richard Artoul
‍The troubled past of open source cloud infrastructure, and why we raised $20M to try and fix it.

Deterministic Simulation Testing for Our Entire SaaS

Mar 12, 2024
Richard Artoul
How we leverage Antithesis to deterministically simulate our entire SaaS platform and verify its correctness, all the way from signup to running entire Kafka workloads.

Public Benchmarks and TCO Analysis

Mar 5, 2024
Richard Artoul
Benchmarking databases – and maintaining fairness and integrity while doing so – is a notoriously difficult task to get right, especially in the data streaming space. Vendors want their systems to produce mouth watering results, and so unnatural configurations divorced from customer realities (AKA “vanity” benchmarks) get tested, and it's ultimately the end-user that is left holding the bag when they realize that their actual TCO is a lot higher than they were led to believe.

Kafka as a KV Store: deduplicating millions of keys with just 128 MiB of RAM

Mar 4, 2024
Manu Cupcic
A huge part of building a drop-in replacement for Apache Kafka® was implementing support for compacted topics. The primary difference between a “regular” topic in Kafka and a “compacted” topic is that Kafka will asynchronously delete records from compacted topics that are not the latest record for a specific key within a given partition.

Anatomy of a serverless usage based billing system

Feb 8, 2024
Richard Artoul
Serverless products and usage based billing models go hand in hand, almost by definition. A product that is truly serverless effectively has to have usage based pricing, otherwise it’s not really serverless!

Middleware saves 85% by replacing Apache Kafka with WarpStream to power AI observability platform

Feb 1, 2024
Richard Artoul
Middleware is a cloud observability company that uses AI to help customers identify, understand, and fix issues across their infrastructure. It’s probably not a surprise to you that their entire business relies on scalable streaming infrastructure: they’re ingesting and processing tens of TiBs of telemetry every day, and need to process it very efficiently.

S3 Express is All You Need

Nov 28, 2023
Richard Artoul
The future of modern data infrastructure is object storage.

Unlocking Idempotency with Retroactive Tombstones

Nov 18, 2023
Richard Artoul
How we separated data from metadata to build support for idempotent producers in our Apache Kafka protocol layer.

Minimizing S3 API Costs with Distributed mmap

Oct 9, 2023
Richard Artoul
We first introduced WarpStream in our blog post: "Kafka is Dead, Long Live Kafka", but to summarize: WarpStream is a Kafka protocol compatible data streaming system built directly on top of object storage.

WarpStream + Materialize: Simpler Streaming for Operational Data Products

Oct 6, 2023
Richard Artoul, Andy Hattemer
If you’re on a Data or Data Platforms team, you’ve probably already seen the productivity boost that comes from pulling business logic out of various ETL pipelines, queries, and scripts and centralizing it in SQL in a clean, version-controlled git repo managed by dbt. The engine that unlocked this approach is the analytical data warehouse: typically Snowflake or BigQuery.

Hacking the Kafka PRoTocOL

Sep 18, 2023
Richard Artoul
How we built stateless load balancing into a protocol that was never designed for it.

Kafka is dead, long live Kafka

Jul 25, 2023
Richard Artoul
The problems with Kafka, and how we created WarpStream to solve them.