Going All in on Protobuf With Schema Registry and Tableflow

Maud Gautier

Software Engineer

December 3, 2025

going-all-in-on-protobuf-with-schema-registry-and-tableflow

HN Disclosure: WarpStream sells a drop-in replacement for Apache Kafka built directly on-top of object storage.

Protocol Buffers (Protobuf) have become one of the most widely-adopted data serialization formats, used by countless organizations to exchange structured data in APIs and internal services at scale.

At WarpStream, we originally launched our BYOC Schema Registry product with full support for Avro schemas. However, one missing piece was Protobuf support.

Today, we’re excited to share that we have closed that gap: our Schema Registry now supports Protobuf schemas, with complete compatibility with Confluent’s Schema Registry.

A Refresher of WarpStream’s BYOC Schema Registry Architecture

Like all schemas in WarpStream’s BYOC Schema Registry, your Protobuf schemas are stored directly in your own object store. Behind the scenes, the WarpStream Agent runs inside your own cloud environment and handles validation and compatibility checks locally, while the control plane only manages lightweight coordination and metadata.

This ensures your data never leaves your environment and requires no separate registry infrastructure to manage. As a result, WarpStream’s Schema Registry requires zero operational overhead or inter-zone networking fees, and provides instant scalability by increasing the number of stateless Agents (for more details, see our previous blog post for a deep-dive on the architecture).

Compatibility Rules in the Schema Registry

In many cases, implementing a new feature via an application code change also requires a change to be made in a schema (to add a new field, for example). Oftentimes, new versions of the code are deployed to one node at a time via rolling upgrades. This means that both old and new versions of the code may coexist with old and new data formats at the same time.

Two terms are usually employed to characterize those evolutions:

Backward compatibility, i.e., new code can read old data. In the context of a Schema Registry, that means that if consumers are upgraded first, they should still be able to read the data written by old producers.
Forward compatibility, i.e., old code can read new data. In the context of a Schema Registry, that means that if producers are upgraded first, the data they write should still be readable by the old consumers.

This is why compatibility rules are a critical component of any Schema Registry: they determine whether a new schema version can safely coexist with existing ones.

Like Confluent’s Schema Registry, WarpStream’s BYOC Schema Registry offers seven compatibility types: BACKWARD, FORWARD, FULL (i.e., both BACKWARD and FORWARD), NONE (i.e., all checks are disabled), BACKWARD_TRANSITIVE (i.e., BACKWARD but checked against all previous versions), FORWARD_TRANSITIVE (i.e., FORWARD but checked against all previous versions) and FULL_TRANSITIVE. (i.e., BACKWARD and FORWARD but checked against all previous versions).

Getting these rules right is essential: if an incompatible change slips through, producers and consumers may interpret the same bytes on the wire differently, thus leading to deserialization errors or even data loss.

Wire-Level Compatibility: Relying on Protobuf’s Wire Encoding

Whether two schemas are compatible ultimately comes down to the following question: will the exact same sequence of bytes on the wire using one schema still be interpreted correctly using the other schema? If yes, the change is compatible. If not, the change is incompatible.

In Protobuf, this depends heavily on how each type is encoded. For example, both int32 and bool types are serialized as a variable-length integer, or “varint”. Essentially, varints are an efficient way to transmit integers on the wire, as they minimize the number of bytes used: small numbers (0 to 128) use a single byte, moderately large number (129 to 16384) use 2 bytes, etc.

Because both types share the same encoding, turning an int32 into a bool is a wire-compatible change. The reader interprets a 0 as false and any non-zero value as true, but the bytes remain meaningful to both types.

However, a change from an int32 into a sint32 (signed integer) is not wire-compatible, because sint32 uses a different encoding: the “ZigZag” encoding. Essentially, this encoding remaps numbers by literally zigzagging between positive and negative numbers: -1 is encoded as 1, 1 as 2, -3 as 3, 2 as 4, etc. This gives negative integers the ability to be encoded efficiently as varints, since they have been remapped to small numbers requiring very few bytes to be transmitted. (Comparatively, a negative int32 is encoded as a two’s complement and always requires a full 10 bytes).

Because of the difference in encoding, the same sequence of bytes would be interpreted differently. For example, the bytes 0x01 would decode to 1 when read as an int32 but as -1 when read as a sint32 after ZigZag decoding. Therefore, converting an int32 to a sint32 (and vice-versa) is incompatible.

Note that since compatibility rules are so fundamentally tied to the underlying wire encoding, they also differ across serialization formats: while int32 -> bool is compatible in Protobuf as discussed above, the analogous change int -> boolean is incompatible in Avro (because booleans are encoded as a single bit in Avro, and not as a varint).

The Testing Framework We Used To Guarantee Full Compatibility

These examples are only two among dozens of compatibility rules required to properly implement a Protobuf Schema Registry that behaves exactly like Confluent’s. The full set is extensive, and manually writing test cases for all of them would have been unrealistic.

Instead, we built a random Protobuf schema generator and a mutation engine to produce tens of thousands of different schema pairs (see Figure 1). We submit each pair to both Confluent’s Schema Registry and WarpStream BYOC Schema Registry and then compare the compatibility results (see Figure 2). Any discrepancy reveals a missing rule, a subtle edge case, or an interaction between rules that we failed to consider. This testing approach is similar in spirit to CockroachDB’s metamorphic testing: in our case, the input space is explored via the generator and mutator combo, while the two Schema Registry implementations serve as alternative configurations whose outputs must match.

Our random generator covers every Protobuf feature: all scalar types, nested messages (up to three levels deep), oneof blocks, maps, enums with or without aliases, reserved fields, gRPC services, imports, repeated and optional fields, comments, field options, etc. Essentially, any feature listed in the Protobuf docs.

Our mutation engine then applies random schema evolutions on each generated schema. We created a pool of more than 20 different mutation types corresponding to real evolutions of a schema, such as: adding or removing a message, changing a field type, moving a field into or out of a oneof block, converting a map to a repeated message, changing a field’s cardinality (e.g., switching between optional, required, and repeated), etc…

For each test case, the engine picks one to five of those mutations randomly from that pool to generate the final mutated schema. We repeat this operation hundreds of times to generate hundreds of pairs of schemas that may or may not be compatible.

Figure 1: Exploring the input space with randomness: the random generator generates an initial schema and the mutation engine picks 1-5 mutations randomly from a pool to generate the final schema. This is repeated N times so we can generate N distinct pairs of schemas that may or may not be compatible.

Each pair of writer/reader schemas is then submitted to both Confluent’s and WarpStream’s Schema Registries. For each run, we compare the two responses: we’re aiming for them to be identical for any random pair of schemas.

Figure 2: Comparing the responses of Confluent’s and WarpStream’s Schema Registry implementations with every pair of writer-reader schemas. An identical response (left) indicates the two implementations are aligned but a different response (right) indicates a missing compatibility rule or an overlooked corner case that needs to be looked into.

This framework allowed us to improve our implementation until it perfectly matched Confluent’s. In particular, the fact that the mutation engine selects not one, but multiple mutations atomically allowed us to uncover a few rare interactions between schema changes that would not have appeared had we tested each mutation in isolation. This was notably the case for changes around oneof fields, whose compatibility rules are a bit subtle.

For example, removing a field from a oneof block is a backward-incompatible change. Let’s take the following schema versions for the writer and reader:

// Writer schema
message User {
  oneof ContactMethod {
    string email = 1;
    int32 phone = 2;
    int32 fax = 3;
  }
}

// Reader schema
message User {
  oneof ContactMethod {
    string email = 1;
    int32 phone = 2;
  }
}

‍

As you can see, the writer’s schema allows for three contact methods (email, phone, fax) whereas the reader’s schema allows for only the first two. In this case, the reader may receive data where the field fax was set (encoded with the writer’s schema) and incorrectly assume no contact method exists. This results in information loss as there was a contact method when the record was written. Hence, removing a oneof field is backward-incompatible.

However, if the oneof block gets renamed to ModernContactMethod on top of the removal of the fax field from the oneof block:

// Reader schema
message User {
  oneof ModernContactMethod {
    string email = 1;
    int32 phone = 2;
  }
}

‍

Then the semantics change: the reader no longer claims that “these are the possible contact methods” but instead “these are the possible modern contact methods”. Now, reading a record where the fax field was set results in no data loss: the truth is preserved that no modern contact method was set at the time the record was written.

This kind of subtle interaction where the compatibility of one change depends on another was uncovered by our testing framework, thanks to the mutation engine’s ability to combine multiple schemas at once.

All in all, combining a comprehensive schema generator with a mutation engine and consistently getting the same response from Confluent’s and WarpStream’s Schema Registries over hundreds of thousands of tests gave us exceptional confidence in the correctness of our Protobuf Schema Registry.

So what about WarpStream Tableflow? While that product is still in early access, we've had exceptional demand for Protobuf support there as well, so that's what we're working on next. We expect that Tableflow will have full Protobuf support by the end of this year.

If you are looking for a place to store your Protobuf schemas with minimal operational and storage costs, and guaranteed compatibility, the search is over. Check out our docs to get started or reach out to our team to learn more.

Get started with WarpStream today and get $400 in credits that never expire. No credit card is required to start.

Get Started

Explore WarpStream

Bring Your Own Cloud

Deploy in any cloud or self-host, using your own compute and object storage.

Managed Data Pipelines

Managed integrations and connectors. Powerful, scalable stream processing.

Orbit

Offset-preserving replication from any source Apache Kafka cluster.

Data Governance

Schema registry, linking, and validation in your own cloud account.

Multi-Region Clusters