.svg)
WarpStream is joining Confluent to bring data streaming to customer's cloud accounts with WarpStream BYOC.
Many announcements like this proceed to lament that the product is shutting down or radically changing, but we’re doing quite the opposite. WarpStream is about to get better – a lot better – with the resources and backing of the leader in streaming. Logistically, the product will continue on in the same form as it does today, all existing customers will continue to be served, and we’re still actively onboarding new customers. We’re adjusting our roadmap to be even more ambitious, and our mission remains as steadfast as it has always been.
If you were considering WarpStream, but were hesitant to bet on a small startup, you can now do so knowing that the full weight of Confluent is behind us!
Thirteen months ago we launched the company with our “Kafka is Dead, Long Live Kafka” blog post. Since then, we grew the team from three to thirteen, GA’d the product, launched new products like Managed Data Pipelines, added new cloud regions, got our SOC2 Type II certification, and most importantly on-boarded incredible customers including Grafana Labs, Zomato, ShareChat, Character.ai, PostHog, Goldsky, and more!
In our original launch post I talked about how Jay Kreps’s The Log blog post is one of my favorite pieces of writing of all time. So when I got an email from Jay himself asking to talk, I was excited.
We met at a diner in New York, barely able to hear each other over the blaring music, and discussed the future of both our companies. Surprisingly, we actually agreed about a lot of things, including the fact that BYOC was going to continue to grow in popularity as a packaging model for infrastructure software, and that Zero Disk Architectures would transform the data streaming space.
Over the next few months we met with more people at Confluent to determine if there was a path forward for WarpStream to exist under the Confluent umbrella, and today we’re happy to announce that we’ve found one.
The WarpStream team, product, and brand will fold into Confluent. In addition, we’re doubling down on our flagship product: WarpStream BYOC. We’ll continue serving all of our existing customers, as well as onboarding new ones.
Our existing (and new!) customers will benefit from what we’ve already built, as well as:
If you’re reading this and you’re still skeptical, I understand. But I’m so confident about how much better WarpStream is going to get at Confluent that today we’re going to do something unusual: share our internal roadmap. Over the next few months, you can expect some amazing stuff as we work toward a more complete platform than either WarpStream or Confluent could build on their own.
We know. This is a table-stakes feature and we’re sorry it’s taken us so long, but we’re almost done and we swear it’s coming soon. Kafka transactions are often used for companies’ most critical use-cases so we want to be absolutely sure our implementation is correct and battle-hardened before releasing it!
One of the most painful aspects of managing self hosted Apache Kafka clusters is dealing with multi-tenant environments with producers and consumers written and maintained by different teams. Often a single errant or misconfigured application can overwhelm an entire multi-tenant Kafka cluster and disrupt every other workload. The solution in open source Kafka is to use a feature called quotas, but many companies forgo using this feature because it operates at a broker-level which makes it extremely difficult to tune. Soon we’ll bring cluster-level quotas to WarpStream that will enable operators of WarpStream clusters to safely implement multi-tenancy and protect their clusters from misbehaving applications.
Schema registries are a critical part of many companies' governance story, but existing WarpStream users have to choose between either using an external cloud-managed schema registry, or running their own. With the launch of WarpStream’s BYOC schema registry, users will get the benefit of WarpStream’s data plane / control plane split for their schema registries as well. All schemas will be stored in the customer’s environment and object storage buckets, with WarpStream responsible for scaling the schema registry metadata (concurrency control, versioning, etc). Your schemas will never leave your cloud account!
Orbit is a new WarpStream product that will enable WarpStream to perfectly mirror any Kafka-compatible source (Open Source Kafka, Confluent Cloud, Amazon MSK, etc). It’s similar to Confluent Cluster Linking in that it performs offset preserving replication, unlike MirrorMaker. This will enable customers to adopt WarpStream incrementally as an enhancement to their existing Kafka clusters that provides: disaster recovery, a local cache, tiered storage, and an additional set of arbitrary read replicas for high fanout workloads.
WarpStream Agents run in any cloud environment, and even on-prem if an S3-compatible object store is available. But WarpStream currently only offers its managed control plane in AWS. That means WarpStream customers running in GCP and Azure have to connect their Agents to a nearby AWS region. Soon we’ll launch our first GCP and Azure control plane regions so customers can keep all their WarpStream metadata traffic local to their region and CSP.
More on this coming soon, but active-active multi region clusters will allow customers to deploy WarpStream clusters that can read and write from multiple cloud regions and will continue functioning even if an entire cloud region goes down. Depending on the configuration, these clusters will even be able to tolerate the failure of an entire cloud provider.
Server side partitioning, partition count auto-scaling, native integration with Apache Iceberg (Similar to Confluent Cloud Tableflow), and much more, so stay tuned! We’ll be announcing more about the future of WarpStream in a few weeks at Current 2024, the data streaming event.
In the meantime, it’s business as usual for us, and there’s never been a better time to adopt WarpStream. Click here to get started with $400 in free credits!
.png)
