A few months ago, we announced WarpStream Tableflow as the latest addition to our platform. At the time, the product was functional, but fairly limited in terms of features. Since then, we have been working with our Early Access program design partners to add features and functionality. WarpStream Tableflow has become a fully-fledged automated tool for materializing Kafka topics as Apache Iceberg tables, and we’re now ready to announce that WarpStream Tableflow is Generally Available!
WarpStream Tableflow can read from any Kafka topic, build Iceberg tables in your object storage bucket, and incrementally maintain those tables over time. It automatically handles data expiration, cleans up orphan files, and compacts files in the background to ensure data locality. It deploys as a single, stateless binary, and automatically scales to meet the demand of a given workload using standard auto-scaling technologies such as the Kubernetes HPA.
When you deploy a WarpStream Tableflow cluster, you’re actually deploying the same WarpStream Agent binary as if you deployed WarpStream Agents for a WarpStream Kafka cluster. This means that your Tableflow Agents behave similarly to WarpStream Agents that you might already be familiar with. It’s the same software! You can use the standard WarpStream Helm chart, and Tableflow is fully supported in our Terraform provider. To learn more about how it works, check out our previous blog post.
During the Early Access program, we added support for JSON, Avro, and Protobuf formats. Tableflow translates Kafka records into Iceberg records, applying the schema defined in your Tableflow configuration file. Types are defined in this configuration, and nested types are supported. You can also define the behavior if Tableflow encounters a record that is incompatible with the provided schema, and either skip the record or stop and block future processing.
Tableflow also supports stateless transformations of records at ingest using Bento. This eliminates the need for intermediary processing upstream of Tableflow. You can use Tableflow transformations to restructure records from one schema to another, rename fields, delete fields, add fields, convert types, or conditionally drop entire records.
Tables can be unpartitioned, or partitioned by hour, day, month, year, or any arbitrary field. Time-based partitioning uses the record timestamp as the partition key. The partitioning key is specified for each table, so different tables processed as part of the same WarpStream Tableflow cluster can have different partitioning schemes.
WarpStream Tableflow integrates directly with Snowflake, Databricks Unity Catalog, BigQuery, and AWS Glue. Iceberg-aware query engines such as Clickhouse and DuckDB can also query the Iceberg tables created by WarpStream Tableflow, and Amazon Athena integrates via the AWS Glue catalog integration. You can learn more about these integrations in our docs.
WarpStream Tableflow is the simplest, easiest, and most cost-effective way of building data lakes out of Kafka topics. It does all of the tedious background jobs so that you don’t have to build and manage them yourself. Best of all, it plugs into whatever existing Kafka-compatible cluster that you already have, so there’s no upstream migration required to get up and running. It’s a complete data lake engine for Apache Iceberg. We like to think of it as the “bottom half” of the database, with your query engine of choice forming the top half.
To learn more about WarpStream Tableflow, check out the docs. If you’re ready to get started, sign up for WarpStream.
Try our demo in 30 seconds! <span class="codeinline" style="white-space: nowrap;">curl https://console.warpstream.com/install.sh | sh</span>
