Knock is an agent-native customer engagement platform. Knock helps teams power their customer messaging for transactional, lifecycle, and marketing in a single tool across in and out-of-app channels like email, push, SMS, and more.
In mid 2026, the platform engineering team at Knock undertook a project to tackle infrastructure spend and optimize margins by reducing COGS (Cost of Goods Sold). As part of this project, we decided to tackle a long-standing source of headaches for us: our "data streams" built on Amazon Kinesis.
By replacing our Kinesis infrastructure with WarpStream, we expect to reduce annual data streaming costs by 77%, from about $112,000 to $25,000. The migration took one engineer about a month and moved five topics and roughly 20 consumers without service disruption.
As a messaging service, Knock produces and processes a lot of data — observability logs, engagement events, and product events that drive messaging flows. And because we handle customer data, teams need to trust Knock with their privacy, which brings another set of challenges.
Knock’s backend services are primarily written in Elixir. At the very beginning of the company's development, we picked Kinesis as our "message hub" for a variety of events broadcast to the rest of the system – events related to the lifecycle of messages, state updates from our in-house workflow engine, and more.
Kinesis is a reliable technology, but it proved to be a costly operational burden for us. Kinesis is designed to be used by KCL (Kinesis Client Library), the Java driver built by AWS itself. That driver is complex. Being on Elixir, we had to instead rely on a community driver that caused us many issues over the years.
The lack of support for enhanced fan-out in our driver meant we had to concoct a "manifold" abstraction for fanning events out to multiple consumers. A manifold was made of a parent Kinesis stream consumed by a "fan-out worker", whose job would be to republish each message to a set of children Kinesis streams. This worked just fine for a few years, but it caused occasional inconsistencies and was a major source of wasted infrastructure spend – we'd pay for the bytes we put on the parent plus each child stream.
.png)
A few months ago we started looking around for solutions. We researched message broker solutions (like RabbitMQ), native AWS solutions (like SNS topics fanning out to SQS queues), newer technologies like Apache Pulsar®, and event logs like Apache Kafka®.
Our problem space is event-log shaped: we push too much data on the wire for solutions like RabbitMQ (copy on each queue) or SNS/SQS, which turned out to be prohibitively expensive. Kafka has good support in the Erlang / Elixir ecosystem via the brod driver and the broadway_kafka integration with Broadway, so we pretty quickly landed on it. However, we were concerned by the cost of AWS MSK and self-hosting was not an option.
WarpStream to the rescue. WarpStream provided Knock with all the things we were looking for. Its higher latency – in the standard, cost-effective configuration – was not a problem for our use cases. We were able to replace "manifolds" and their children with Kafka-native topics and consumer groups. We now operate topics that range from two to eight different consumers each, where adding a consumer doesn't require a new stream (or significant additional costs).
WarpStream (by way of Kafka semantics) also unlocked fan-in for us: we can now easily have consumer groups that consume from multiple topics, something we had to hack together on Kinesis.
WarpStream’s BYOC model also simplified our privacy and compliance review. Unlike a managed broker that stores customer data in the provider’s cloud account, WarpStream keeps our message data in S3 inside our own AWS accounts.
The whole migration went really smoothly. We de-risked the cutover by running the same consumer code against both Kinesis and WarpStream at the same time. Then, we transitioned published events from one to the other via extensive feature flagging. We publish plenty of metrics, so we were able to observe traffic shifting over to WarpStream topics.
The whole migration (five topics and around 20 consumers in total) took us about a month with most of the work carried out by one engineer.
WarpStream turned out to be exactly what we were looking for. We've been operating it for a few months now, and it met our cost, reliability, and operational goals. We're able to run our main application's needs off of three <span class="codeinline">m6in.xlarge</span> EC2 boxes, which sit at a comfortable ~20% CPU usage. By reserving those instances as well, we're paying an extremely cost-effective ~$400/mo for all those instances.
We haven't had the need for horizontal autoscaling yet, but we're feeling great about the fact we could just throw more nodes at our main Agent group if our usage ramped up significantly. We also run two additional, lower-cost agents for ClickPipes (see below).
We were also pleasantly surprised by the latency behavior we've observed. None of our use cases needed sub-second end-to-end latency; we observed a consistent sub-2s latency – less than we expected and good enough for our use cases, which often are centered around eventually-consistent logs.
As an aside, we were extremely happy with the number and quality of metrics exposed by WarpStream. Our main observability platform is Datadog, where WarpStream is a supported integration: this allowed us to enable high-cardinality metrics (per topic and per partition!) without incurring any additional custom-metric-related costs.
Knock is an avid ClickHouse user. In fact, all our Kinesis manifolds had one consumer each dedicated to persisting data from the stream to ClickHouse. Just a couple of months before the WarpStream migration, we had started making heavy use of ClickPipes, ClickHouse Cloud's solution for natively ingesting data into ClickHouse.
The fact that WarpStream is an officially-supported source for ClickPipes was an important factor in our decision to use WarpStream. We now run a ClickPipe attached to each WarpStream topic. We spun up a dedicated Agent group for ClickPipes to isolate their traffic patterns. We loved being able to constrain that group to only serving <span class="codeinline">Fetch</span> requests (meaning cheap nodes).
Our production traffic on WarpStream sits at around 10-15k messages per second, or around 15 MiB/s of uncompressed data published to our topics.
We expect to reduce our yearly spend significantly. Kinesis was costing us approximately $112k/year between shard-hours and DynamoDB tables for checkpointing. We're (painfully!) aware that our custom manifold design made Kinesis more expensive than a more-idiomatic enhanced fan-out setup. However, our estimates tell us that even an optimal Kinesis setup would cost us more than the roughly $25k/year we're forecasting with WarpStream – a whopping ~77% decrease from today. This figure includes:
This doesn't count ClickPipes costs as we were paying those exact same costs with Kinesis.
For us, an important improvement here is the agility we gained. Adding a consumer to an existing topic costs us very little (only the additional <span class="codeinline">Fetch</span> requests); that's something we do consistently. Adding topics, or rather adding data throughput, costs money but it's cheaper than it was on Kinesis.
The main result of the migration is undoubtedly COGS savings. However, we ended up with quite a few additional positive outcomes:
The end result was not only a cheaper system, but also a simpler system overall. We removed infrastructure that caused repeated problems for us, turned adding consumer groups to topics into a non-event, and gave the team a solid Kafka-compatible foundation with great Elixir support.
.svg)
