Getting started with WarpStream on Tigris

WarpStream is a Kafka®-compatible streaming platform that stores data directly in object storage instead of relying on local disks. This architecture unlocks virtually unlimited retention and simplifies operations. Because WarpStream is zone-aligned, it avoids costly inter-AZ network traffic by ensuring that agents and object storage communicate within the same availability zone. However, in multi-cloud architectures where clients and agents communicate across clouds, egress fees from cross-cloud traffic can quickly dominate your cloud bill as you scale.

Tigris is a globally distributed, multi-cloud object storage platform with native S3 API support and zero egress fees. It dynamically places data in the region where it’s being accessed—eliminating cross-cloud data transfer costs without sacrificing on performance.

By combining WarpStream with Tigris, you get a bottomless, durable, and globally aware message queue. Your data is stored efficiently, close to where it’s consumed, without incurring hidden transfer fees or needing to plan for regional bucket placement.

Before we get started, let’s cover the moving parts:

The Moving Parts

Apache Kafka is a durable, distributed messaging system. In Kafka, Producers write records into Topics hosted by Brokers that are read by Consumers. Used by 80% of Fortune 500 companies, Kafka connects applications and data stores using a real-time publish/subscribe model with strong durability guarantees. One of the main differences between Kafka and a traditional queueing system is persistence. Data written to Kafka is retained, even after it is read by a Consumer, which enables interesting patterns such as multiple consumers reading from the same topic. The main downside is that Kafka relies on local storage, meaning that your Kafka Brokers require fast disks and intra-cluster replication to ensure durability and availability.

WarpStream is fully compatible with the Kafka protocol, but it fundamentally reimagines the architecture. Instead of using local disks, it writes all data directly to object storage, completely decoupling compute and storage. While Apache Kafka 3.6.0 introduced Tiered Storage to enable more flexible scaling of compute and storage, it stops short of full decoupling—brokers still require local disk, retain state, manage replication, and serve all client traffic.

With WarpStream, the compute layer is completely stateless. The data does not need to be replicated between nodes since durability is handled entirely by the object store. This architecture is much simpler to operate: you don’t need to set up and manage all of Kafka’s dependencies (Zookeeper, the JVM, etc). WarpStream also ships an easy to use command line utility that helps you administrate your message queue and test functionality.

In addition, we’ll use Docker, the universal package format for the Internet. Docker lets you put your application and all its dependencies into a container image so that it can’t conflict with anything else on the system.

Today we’re going to deploy a WarpStream cluster backed by Tigris into a Docker container so you can create your own bottomless durable message queue. This example will use Docker compose, but it will help you understand how to create your own broker so you can deploy it anywhere.

Prerequisites

Clone the warpstream-tigris demo repo to your laptop and open it in your favourite editor, such as VS Code.

Make sure you have the following installed on your computer:

• Docker Desktop or another similar app like Podman Desktop.
• The AWS CLI.
• WarpStream’s CLI.

You will need the following accounts:

• A Tigris account from storage.new.
• A WarpStream account from console.warpstream.com

Building a compose file

First, clone tigrisdata-community/warpstream-tigris to your laptop and open it in your favourite text editor. If you use development containers, tell your editor to open this repository in a development container to get up and running in a snap!

Take a look at the docker-compose.yaml file in the root of the repository:

services:
  warp:
    # Grab the latest copy of the warpstream agent for your computer
    image: public.ecr.aws/warpstream-labs/warpstream_agent:latest
    # Run warpstream in "playground" mode for testing
    command:
      - playground
      - -advertiseHostnameStrategy
      - custom
      - -advertiseHostnameCustom
      - warp
    environment:
      # this is a no-op as it will default on the custom advertised hostname defined above, but you can change this if you want to use a different hostname with Kafka
      - WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE=warp
    healthcheck:
      # Wait for the Agent to finish setting up the demo before marking it as healthy
      # to delay the diagnose-connection command from running for a few seconds.
      test: ["CMD", "sh", "-c", "sleep 10"]
      interval: 5s
      timeout: 15s
      retries: 5

‍

Open a new terminal in your development container and make sure WarpStream is up and running:

warpstream kcmd --bootstrap-host warp --type diagnose-connection

‍

This should return output like the following:

running diagnose-connection sub-command with bootstrap-host: warp and bootstrap-port: 9092

Broker Details
---------------
  warp:9092 (NodeID: 1547451680) [playground]
    ACCESSIBLE ✅

GroupCoordinator: warp:9092 (NodeID: 1547451680)
    ACCESSIBLE ✅

‍

Excellent! Create a new topic with <span class="codeinline">warpstream kcmd</span>:

warpstream kcmd --bootstrap-host warp --type create-topic --topic hello

‍

This should return output like the following:

running create-topic sub-command with bootstrap-host: warp and bootstrap-port: 9092

created topic "hello" successfully, topic ID: MQAAAAAAAAAAAAAAAAAAAA==

‍

Perfect! Now let’s make it work with Tigris. Create a <span class="codeinline">.env</span> file in the root of the repository:

cp .env.example .env
code .env

‍

Create a new bucket at storage.new in the Standard access tier. Copy its name down into your notes. Create a new access key with Editor permissions for that bucket. Copy the environment details into your <span class="codeinline">.env</span> file:

## Tigris credentials
AWS_ACCESS_KEY_ID=tid_access_key_id
AWS_SECRET_ACCESS_KEY=tsec_secret_access_key
AWS_ENDPOINT_URL_S3=https://t3.storage.dev
AWS_ENDPOINT_URL_IAM=https://iam.storage.dev
AWS_REGION=auto

‍

Then fill in your WarpStream secrets from the console, you need the following:

• Cluster ID from the virtual clusters list (begins with <span class="codeinline">vci_</span>)
• Bucket URL (explained below)
• Agent key from the agent keys page for that virtual cluster (begins with <span class="codeinline">aks_</span>)
• Cluster region from the admin panel (such as <span class="codeinline">us-east-1</span>)

If your bucket is named <span class="codeinline">xe-warpstream-demo</span>, your bucket URL should look like this:

s3://xe-warpstream-demo?region=auto&endpoint=https://t3.storage.dev

‍

Altogether, put these credentials in your <span class="codeinline">.env</span> file:

## Warpstream credentials
WARPSTREAM_AGENT_KEY=aks_agent_key
WARPSTREAM_BUCKET_URL='s3://xe-warpstream-demo?region=auto&endpoint=https://t3.storage.dev'
WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID=vci_cluster_id
WARPSTREAM_REGION=us-east-1

‍

Edit your <span class="codeinline">docker-compose.yaml</span> file to load the <span class="codeinline">.env</span> file and start warpstream in agent mode:

# docker-compose.yaml
services:
  warp:
    image: public.ecr.aws/warpstream-labs/warpstream_agent:latest
    command:
      - agent
    environment:
      WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE: warp
      WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE: 9092
      WARPSTREAM_REQUIRE_AUTHENTICATION: "false"
    env_file:
      - .env

‍

Then restart your development container with control/command shift-p “Dev Containers: Rebuild Container”. Test the health of your Broker:

warpstream kcmd --bootstrap-host warp --type diagnose-connection

‍

You should get output like this:

running diagnose-connection sub-command with bootstrap-host: warp and bootstrap-port: 9092

Broker Details
---------------
  warp:9092 (NodeID: 1415344910) [warpstream-unset-az]
    ACCESSIBLE ✅

GroupCoordinator: warp:9092 (NodeID: 1415344910)
    ACCESSIBLE ✅

‍

It’s working! Create a topic and publish some messages:

warpstream kcmd --bootstrap-host warp --type create-topic --topic hello
warpstream kcmd --bootstrap-host warp --type produce --topic hello --records "world,,world"

‍

This should create the topic <span class="codeinline">hello</span> and two messages with <span class="codeinline">world</span> in them. You should get output like this:

result: partition:0 offset:0 value:"world"
result: partition:0 offset:1 value:"world"

‍

Now let’s read them back:

warpstream kcmd --bootstrap-host warp --type fetch --topic hello --offset 0

‍

You should get output like this:

consuming topic:"hello" partition:0 offset:0
result: partition:0 offset:0 key:"hello" value:"world"
result: partition:0 offset:1 key:"hello" value:"world"

‍

It works! You’ve successfully put data into a queue and fetched it back from the queue. From here you can connect to your broker on host <span class="codeinline">warp</span> and port <span class="codeinline">9092</span>. All your data is securely backed by Tigris and you can access it from anywhere in the world.