One Weird Trick to Make Rapid Storage 40x Faster

Make It Work, Make It Correct, Then Make It Fast

When WarpStream was first announced, we framed it as a significantly cheaper and easier to use implementation of the Apache Kafka® protocol if your workload could tolerate higher latency. The product was well received, but that was a very big if!

Since then, we’ve iterated relentlessly to improve the latency profile of the product to make it suitable for a wider variety of use cases. We made the Agents faster, reduced the latency of our control plane, built Lightning Topics, and added support for S3 Express One Zone. With these improvements, a WarpStream cluster tuned for low latency can achieve a P99 Produce latency of less than 50ms with a P50 of less than 35ms.

These latencies represent an order of magnitude improvement in latency since we launched V1 of the product and make WarpStream suitable for 99% of the world’s Kafka workloads without compromising on cost or ease of operations.

However, there was still one big gap in our low-latency story: GCP. In order for WarpStream to achieve a low-latency profile, it needs a low-latency storage medium. In AWS, this became available with the launch of S3 Express One Zone in November of 2023. In Azure, the “premium” (low-latency) tier of blob storage has been available since March of 2019.

GCP Rapid Buckets

But what about GCP? Well, for a long time there wasn’t a low-latency object storage product available in GCP, so we had no way of offering a low-latency GCP version of WarpStream. That changed in May of 2026 when GCP GA’d their new “Rapid” Buckets. GCP Rapid Buckets expose a similar API as a “traditional” regional GCS bucket, but they have a few differences also.

First, Rapid Buckets offer much lower latency than GCP traditional buckets, hence the name. This was exactly what we needed.

Second, Rapid Buckets only store data in a single availability zone. While that may sound like a show-stopper for a product like WarpStream that prides itself on delivering 11 9s of durability, in practice it’s a non-issue because WarpStream already supports storing data in a quorum of zonal buckets (a feature we developed specifically for S3 Express One Zone when it was first launched).

Third, Rapid Buckets charge significantly more money for every GiB of data stored. This is especially problematic because to achieve high durability, systems like WarpStream need to store the data in at least two different Rapid Buckets as discussed in the previous paragraph. This makes the effective storage cost of Rapid Buckets more than 10x higher than a traditional regional GCS bucket.

Again, this sounds like it would be a deal-breaker for a product like WarpStream that prides itself on minimizing cloud infrastructure costs, but in practice it’s also a non-issue because WarpStream is a distributed log-structured merge tree (LSM) that performs background compactions. As a result, WarpStream can be configured to land newly ingested data in a quorum of GCP Rapid Buckets to minimize latency, but then asynchronously compact those files into a regional GCS bucket a few minutes later. As a result of this architecture, our GCP customers get the low-latency benefit of Rapid Buckets without the correspondingly high storage costs.

There was just one problem: in our testing, GCP Rapid Buckets didn’t actually seem any faster than traditional GCS buckets. Specifically, when we benchmarked the latency of creating a new file, writing a few hundred KiBs of data to it, and then closing the file, we found that Rapid Buckets and traditional buckets had about the same latency of 80-100ms.

When we investigated more closely, it became obvious that the intended usage of Rapid Buckets was to use long-lived files that were regularly appended. Once a file was created / opened, appending to it was very fast, on the order of 1ms, but it appeared that the latency of creating a new file and “finalizing” (marking as sealed / immutable) an existing file had not been optimized at all. This was quite different from S3 Express One Zone, where creating a new file, writing some data to it, and then closing it was extremely fast.

There was an obvious solution to this problem: we could rearchitect WarpStream to have a mode where it used long-lived files and appended to them instead of constantly creating and flushing new files, but this would have been a large, invasive change that we hadn't budgeted for, so we shelved the project.

A Few Weird Tricks

A few months went by and the request for low-latency WarpStream clusters kept piling up. We decided to talk to some engineers at GCP to see if there was anything they or we could do that wouldn’t require rewriting half of WarpStream. When we met with the engineers at GCP, they confirmed our suspicion that the reason we weren’t seeing any meaningful latency improvements was because our workload had two critical-path metadata operations: (1) opening the file and (2) closing the file, and those operations weren’t any faster with Rapid Buckets than they were with GCS buckets.

There was disappointment on both sides. We really wanted to take advantage of Rapid Buckets’ low-latency capabilities without a big rewrite, and the GCP engineers clearly felt like this was an important use case they wanted to address.

“You could just not close the files,” one GCP engineer mused.

“Wait what?” I said, sitting up straighter. “Can we do that?”

— “Yeah, definitely. Unfinalized files are readable and you can always read your writes. You’d just need to make sure that all your writes had been flushed successfully, but flushes are super fast, on the order of 1ms of latency.”

— “OK, so an unfinalized file will just look like any other file? It will be visible to GET requests and LIST requests? It won’t show up as a failed multi-part upload or something weird like that?”

— “Correct, the only difference between a finalized and unfinalized file is that a finalized file is immutable and won’t accept further appends if you’re using our gRPC interface. If you’re using the JSON/XML interfaces and don’t close the file, then it may take some time for the CRC/Size of the file to be updated.”

— “OK, that’s perfect then. I’ll have to make sure we never try to append to a file that we’re done with, but our system already never does that, I can just add a few extra safeguards to be sure. So, if I do this, how fast will it get?”

— “Well, that will eliminate one of the two critical-path metadata operations and since the latency of metadata operations dominates your workload, it should be about half the latency of regular GCS.” 

— “OK, that’s not bad, but it would be really nice to get rid of the first metadata operation also.” 

We both stared at each other, clearly formulating the same idea at the same time. “What if I just maintained a pool of pre-warmed files, would that work?” I asked tentatively.

“Yes, that would one hundred percent work,” the GCP engineer replied immediately and enthusiastically.

We had a plan, and it might just work! We agreed that I would go test our hypothesis and report back. To confirm the performance characteristics worked the way we expected, I wrote a short test program that looked roughly like the following:

client, _ := storage.NewGRPCClient(ctx, experimental.WithZonalBucketAPIs())
bucket := client.Bucket(bucketName)
payload := []byte("0123456789")

// Phase 1: open N writers (create-if-not-exists), flush to establish each handle
var writers []*storage.Writer
for i := 0; i < fileCount; i++ {
	w := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).
		If(storage.Conditions{DoesNotExist: true}).
		NewWriter(ctx)
	w.FinalizeOnClose = false
	writers = append(writers, w)
}

// Phase 2: write + close each open writer
for _, w := range writers {
	start := time.Now()
	w.Write(payload)
	w.Close() // Doesn't finalize file because w.FinalizeOnClose = false
	fmt.Println("write+close:", time.Since(start))
}

// Phase 3: read them back
for i := 0; i < fileCount; i++ {
	r, _ := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).NewReader(ctx)
	contents, _ := io.ReadAll(r)
	r.Close()
	fmt.Printf("%q\n", contents)
}  

‍

Unfortunately, when I ran this program the reported latency for write + close was on the order of 80-100ms, and we really needed it to be sub-10ms for this to work. A few minutes later I realized that I’d forgotten one key detail that the GCP engineers had told me: the GCP clients are lazy and so my Phase 1 for-loop was not actually pre-warming the files, it was just creating an array of in-memory objects and my program would pay the full latency penalty of opening the file on the first call to <span class="codeinline">Write()</span>.

I updated my program with one additional line, changing the pre-warming code from:

w := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).
    If(storage.Conditions{DoesNotExist: true}).
    NewWriter(ctx)
w.FinalizeOnClose = false
writers = append(writers, w)

‍

And voila, the reported latency was now about 9ms, exactly where I needed it to be. I reported this back to the GCP engineer and he notified me that while reproducing my workload, they’d found a few inefficiencies with how the first write to a new file was handled and were making some improvements on the backend. When they notified me they’d rolled out the improvements, I ran my test program again and this time it reported latency of about 6-7ms. Nice!

Finally, I sent my entire test program over to the GCP engineer to confirm it was correct and he pointed out that it would be faster if I replaced my terminal <span class="codeinline">w.Close()</span> with a <span class="codeinline">w.Flush()</span> because <span class="codeinline">Close()</span> performs some extra work that is not required for my workload like accelerating how fast the size/CRC updates will be made visible to JSON/XML clients, as well as performing some client-side and server-side cleanup. That said, it was still important that the process called <span class="codeinline">Close()</span> eventually to ensure client-side resources were cleaned up, which was simple to implement with an asynchronous goroutine.

In the end, the final program ended up looking like this:

client, _ := storage.NewGRPCClient(ctx, experimental.WithZonalBucketAPIs())
bucket := client.Bucket(bucketName)
payload := []byte("0123456789")

// Phase 1: open N writers (create-if-not-exists), flush to establish each handle
var writers []*storage.Writer
for i := 0; i < fileCount; i++ {
    w := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).
        If(storage.Conditions{DoesNotExist: true}).
        NewWriter(ctx)
    w.FinalizeOnClose = false
    w.Flush()
    writers = append(writers, w)
}

// Phase 2: write + close each open writer
for _, w := range writers {
    start := time.Now()
    w.Write(payload)
    w.Flush()
    go func() {
        // Doesn't matter if this fails, just cleaning up some client
        // resources.
        w.Close()
    }()
    fmt.Println("write+close:", time.Since(start))
}

// Phase 3: read them back
for i := 0; i < fileCount; i++ {
    r, _ := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).NewReader(ctx)
    contents, _ := io.ReadAll(r)
    r.Close()
    fmt.Printf("%q\n", contents)
}

‍

With all of these tweaks and improvements the program was now reporting a latency of 2-3ms. Almost 40x better than the 80ms we started with! So I set about translating my test program into concrete changes in WarpStream to take advantage of these tips and tricks.

Summarizing the tricks (I know I lied in the title when I said it was just one):

1. Don’t call <span class="codeinline">NewWriter()</span> just-in-time when we’re ready to flush a new file. Instead, maintain a background pool of pre-created writers that can be retrieved on-demand.
2. The pool should also call <span class="codeinline">Flush()</span> on the writers before considering them “ready” to ensure that the file is actually created and the metadata operation completed so that the first real write to the object doesn’t incur that penalty.
3. Objects should not be finalized and we can ensure that by setting <span class="codeinline">w.FinalizeOnClose = false</span>.
4. “Close” the file by calling <span class="codeinline">Flush()</span>, not <span class="codeinline">Close()</span>.

The last three tricks were trivial, a few lines each at most. But trick 1 was … tricky. Specifically, we had to juggle a number of different requirements:

1. The object pool always had to have spare files, otherwise a call to retrieve a file from the pool may block on creating a new file on the fly and cause a latency spike.
2. Files that had been open for too long should be thrown away / discarded. This is important for two reasons:
   
   1. To prevent a latency spike from trying to use a stale connection.
   2. But also because WarpStream has a background reconciliation loop that scans the object store for files that aren’t tracked by the control plane and deletes them. To prevent this loop from deleting files that are “in-flight” we look at the creation time of the file and apply a grace window. Therefore, if a file had been in the pool for a period that was approaching the grace window, we couldn’t use it without risk of it being deleted in-between us writing it to the bucket and it being committed to the control plane (this would result in data loss).
3. The pool should refill itself at a rate roughly equal to the rate at which it was drained.
4. The pool should not create and throw away an excessive number of files as this would be expensive from both a performance and cost perspective.

Luckily we already had an abstraction in the codebase called a <span class="codeinline">LoadingQueue</span> that was almost exactly what we needed:

// LoadingQueue is a FIFO queue abstraction that abstracts away
// the process of fetching items to load into the queue.
// 
// It allows the user to provide a load function and then just
// call Get() in a loop and the queue will automatically handle
// refilling itself in the background.
type LoadingQueue[T any] interface {
   Get(ctx context.Context) (T, error)
}

‍

The implementation of the <span class="codeinline">LoadingQueue</span> itself is quite simple. You can think of it as a light abstraction over a buffered channel in Golang where a Goroutine calls a user-provided <span class="codeinline">LoadFunction</span> in a loop and inserts the result into the channel. When the channel is full, backpressure is applied naturally to the goroutine running the <span class="codeinline">LoadFunction</span>.

We made just three simple changes to the configuration of the <span class="codeinline">LoadingQueue</span> struct to support our new use-case:

1. We added an optional <span class="codeinline">DiscardFunction</span> parameter which is a user-supplied function that is called on any object that ends up being discarded (I.E not returned in a call to <span class="codeinline">Get()</span>). In the case of the Rapid Buckets implementation, this function just calls <span class="codeinline">Close()</span> on the file.
2. We added a <span class="codeinline">MaxItemAge</span> parameter that limits the age of an object that will be returned by the loading queue. Any object older than this parameter is automatically discarded using the <span class="codeinline">DiscardFunction</span>.
3. We added a <span class="codeinline">LoadConcurrency</span> parameter that controls the maximum number of Goroutines that will run the <span class="codeinline">LoadFunction</span>concurrently. This was necessary because the latency of pre-warming files was on the order of 80ms, which meant that we needed more than one Goroutine doing it continuously to keep Agent VMs with many cores satisfied.

The Final Result

Once that was done, we could finally measure the actual impact on latency in one of our test clusters:

With these changes in place, the P99 latency to flush a 1MiB file with Rapid storage was about 33ms and the P50 was 20ms, about 2-3x faster than with a traditional GCS bucket. The end result didn’t end up being 40x faster like in the previous example. This is because the gap between Rapid and traditional GCS buckets drops when the amount of data written increases, and also because this measurement includes a few non-bucket related operations like compression, which are performed streaming and whose latency can’t be disentangled from the latency of writing the file. Still, a 3x reduction in P99 is nothing to sneeze at!

‍Note: The Rapid Buckets have an important limitation: they are not compatible with WarpStream’s Lightning Topics feature. This is a temporary limitation due to uninteresting details of how lightning topics are implemented. This limitation will be fixed in the next few months.

If you want to get started with WarpStream and GCP Rapid Buckets, check out our documentation.

‍