chore: use otter as the primary cache implementation and get rid of alternative implementations by tstirrat15 · Pull Request #3112 · authzed/spicedb

tstirrat15 · 2026-05-12T18:45:44Z

Description

We've had three alternative implementations for a while now, with ristretto being the one that is available by default, with two other implementations behind flags. We've load tested this internally and it appears that all three of the implementations are interchangeable from a performance perspective. otter is the most modern of the three and it supports generics, which means that we don't have to muck around with calculating our own keys.

This PR makes otter the primary cache, removes the other implementations, and simplifies where possible.

Changes

Will annotate.

Testing

Review. See that benchmarks are the same.

codecov · 2026-05-12T18:49:07Z

Codecov Report

❌ Patch coverage is 84.70588% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.69%. Comparing base (f7fda69) to head (3a395e3).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/cmd/server/cacheconfig.go	50.00%	4 Missing and 1 partial ⚠️
internal/dispatch/keys/keys.go	82.36%	3 Missing ⚠️
internal/dispatch/keys/computed.go	89.48%	2 Missing ⚠️
internal/dispatch/keys/hasher.go	92.86%	1 Missing and 1 partial ⚠️
pkg/cache/standard.go	66.67%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3112      +/-   ##
==========================================
+ Coverage   75.66%   75.69%   +0.03%     
==========================================
  Files         505      502       -3     
  Lines       62111    61989     -122     
==========================================
- Hits        46989    46914      -75     
+ Misses      11701    11660      -41     
+ Partials     3421     3415       -6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite	Current: `3a395e3`	Previous: `f7fda69`	Ratio
`BenchmarkDatastoreDriver/cockroachdb-overlap-static/TestTuple/SnapshotReverseRead (github.com/authzed/spicedb/internal/datastore/benchmark)`	`34826186` ns/op 175010 B/op 20202 allocs/op	`6959651` ns/op 172710 B/op 20195 allocs/op	`5.00`
`BenchmarkDatastoreDriver/cockroachdb-overlap-static/TestTuple/SnapshotReverseRead (github.com/authzed/spicedb/internal/datastore/benchmark) - ns/op`	`34826186` ns/op	`6959651` ns/op	`5.00`
`BenchmarkDatastoreDriver/cockroachdb-overlap-insecure/TestTuple/SnapshotReverseRead (github.com/authzed/spicedb/internal/datastore/benchmark)`	`32581099` ns/op 174731 B/op 20200 allocs/op	`7535757` ns/op 172724 B/op 20195 allocs/op	`4.32`
`BenchmarkDatastoreDriver/cockroachdb-overlap-insecure/TestTuple/SnapshotReverseRead (github.com/authzed/spicedb/internal/datastore/benchmark) - ns/op`	`32581099` ns/op	`7535757` ns/op	`4.32`

This comment was automatically generated by workflow using github-action-benchmark.

tstirrat15

See comments

tstirrat15 · 2026-05-12T18:46:16Z

 // DatastoreProxyTestCache returns a cache used for testing.
 func DatastoreProxyTestCache(t testing.TB) cache.Cache[cache.StringKey, CacheEntry] {
 	cache, err := cache.NewStandardCache[cache.StringKey, CacheEntry](&cache.Config{
-		NumCounters: 1000,


This configuration value isn't used by otter, so I deprecated it and removed places where it's set.

tstirrat15 · 2026-05-12T18:47:36Z

 	}
 }

-func TestComputeOnlyStableHash(t *testing.T) {


This is related to removing ristretto - there was a process-specific hash that was a part of its key computation, but that isn't used by the other cache implementations, so I got rid of its computation and got rid of logic and tests around that.

tstirrat15 · 2026-05-12T18:48:00Z

-
 func (dck DispatchCacheKey) KeyString() string {
 	firstBytes := binary.AppendUvarint(make([]byte, 0, 8), dck.stableSum)
-	secondBytes := binary.AppendUvarint(make([]byte, 0, 8), dck.processSpecificSum)


This value is always 0 unless it's been computed in the ristretto logic.

tstirrat15 · 2026-05-12T18:48:12Z

-// AsUInt64s returns the cache key in the form of two uint64's. This method returns uint64s created
-// from two distinct hashing algorithms, which should make the risk of key overlap incredibly
-// unlikely.
-func (dck DispatchCacheKey) AsUInt64s() (uint64, uint64) {


This was only used in the ristretto implementation.

tstirrat15 · 2026-05-12T18:53:11Z

-	if h.computeOption == computeBothHashes {
-		h.processSpecificSum = runMemHash(h.processSpecificSum, []byte(value))
-	}
-}
-
-// From: https://github.com/outcaste-io/ristretto/blob/master/z/rtutil.go
-type stringStruct struct {
-	str unsafe.Pointer
-	len int
-}
-
-//go:noescape
-//go:linkname memhash runtime.memhash
-func memhash(p unsafe.Pointer, h, s uintptr) uintptr
-
-func runMemHash(seed uint64, data []byte) uint64 {
-	ss := (*stringStruct)(unsafe.Pointer(&data))
-	return uint64(memhash(ss.str, uintptr(seed), uintptr(ss.len))) //nolint:gosec
-}


This is the part that I removed - this is the only place where computeOption is actually referenced, and the processSpecificSum is only referenced in the key logic in the ristretto implementation.

tstirrat15 · 2026-05-12T19:01:20Z

These changes are just flattening the test and testing Otter directly.

tstirrat15 · 2026-05-12T19:01:36Z

+	// TODO: check name here
+	return NewOtterCache[K, V]("", config)


Should there be a name here? I wasn't sure.

func mustRegisterCache(name string, c withMetrics) { if _, loaded := caches.LoadOrStore(name, c); loaded { panic("two caches with the same name: " + name) } } func unregisterCache(name string) { caches.Delete(name) }

how does that work if the key is empty?
why do we have _?
should we not panic if _ is actually false? so many questions..

tstirrat15 · 2026-05-12T19:02:00Z


 // CompleteCache translates the CLI cache config into a cache config.
 func CompleteCache[K cache.KeyString, V any](cc *CacheConfig) (cache.Cache[K, V], error) {
-	if !cc.Enabled || cc.MaxCost == "" || cc.MaxCost == "0%" || cc.NumCounters == 0 {


NumCounters is deprecated, so we don't switch on it anymore.

tstirrat15 · 2026-05-12T19:02:15Z

 		return nil, errors.New("could not cast max cost to int64")
 	}

-	if cc.CacheKindForTesting != "" {


Otter is now the standard, so we get rid of this switching.

tstirrat15 · 2026-05-12T19:02:29Z

 	flags.BoolVar(&config.Enabled, flagPrefix+"-enabled", defaults.Enabled, "enable caching of "+flagDescription)

-	// Hidden flags.
-	flags.StringVar(&config.CacheKindForTesting, flagPrefix+"-kind-for-testing", defaults.CacheKindForTesting, "choose a different kind of cache, for testing")


This flag was already hidden, so I'm comfortable with deleting it.

tstirrat15 · 2026-05-12T19:10:10Z

Hmm... seems Otter is still incompatible with wasm. I'll reinstate the wasm build flags.

tstirrat15 · 2026-05-14T15:10:17Z

Okay, this is ready for review.

miparnisari · 2026-05-14T17:46:22Z

-		}
-	}
-
 	if cc.Metrics {


why do we have this if here? the config that gets sent in the constructor is exactly the same, and so is the underlying code.

Looking at the implementation, it looks like we're always emmitting metrics 🤔 😬 (which is fine by me, i don't know why we expose --cache-metrics as a flag?)

tstirrat15 · 2026-05-14T18:11:48Z

An LLM note about why otter isn't compatible with WASM:

Here's the chain:

xruntime.CacheLineSize = unsafe.Sizeof(cpu.CacheLinePad{}) — defined in otter's internal package

cpu.CacheLinePad is struct{ _ [cacheLineSize]byte } from golang.org/x/sys/cpu

In cpu_wasm.go (the WASM-specific file), cacheLineSize = 0 with the comment: "Make CacheLinePad an empty struct and hope that the usual struct alignment rules are good enough."

So unsafe.Sizeof(cpu.CacheLinePad{}) → size of an empty struct → 0.

This is why otter can't be built for WASM — any code that uses CacheLineSize as an array size or divisor (e.g [CacheLineSize]byte or modulo operations) would either produce a zero-size array or a division by zero at runtime.

tstirrat15 requested a review from a team as a code owner May 12, 2026 18:45

github-actions Bot added area/cli Affects the command line area/datastore Affects the storage system area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools) area/dispatch Affects dispatching of requests labels May 12, 2026

github-actions Bot reviewed May 12, 2026

View reviewed changes

tstirrat15 force-pushed the tstirrat/coalesce-cache-on-otter branch from 39dad92 to d8c6070 Compare May 12, 2026 18:56

tstirrat15 commented May 12, 2026

View reviewed changes

tstirrat15 force-pushed the tstirrat/coalesce-cache-on-otter branch 2 times, most recently from 5b87fe4 to 2691223 Compare May 12, 2026 19:07

tstirrat15 marked this pull request as draft May 12, 2026 19:10

chore: use otter instead of ristretto, get rid of theine

0d53a6e

tstirrat15 force-pushed the tstirrat/coalesce-cache-on-otter branch from 2691223 to 0a785e8 Compare May 14, 2026 13:42

chore: simplify things around otter use

5a2bf18

tstirrat15 force-pushed the tstirrat/coalesce-cache-on-otter branch from 0a785e8 to 5a2bf18 Compare May 14, 2026 14:11

tstirrat15 marked this pull request as ready for review May 14, 2026 14:34

miparnisari reviewed May 14, 2026

View reviewed changes

Comment thread CHANGELOG.md Outdated

miparnisari reviewed May 14, 2026

View reviewed changes

Comment thread internal/dispatch/keys/dispatchcheckkey.go Outdated

miparnisari reviewed May 14, 2026

View reviewed changes

Comment thread pkg/cache/cache_wasm.go

miparnisari reviewed May 14, 2026

View reviewed changes

miparnisari changed the title ~~Use otter as the primary cache implementation and get rid of alternatives~~ chore: use otter as the primary cache implementation and get rid of alternative implementations May 14, 2026

chore: address feedback

3a395e3

tstirrat15 force-pushed the tstirrat/coalesce-cache-on-otter branch from c04ccb9 to 3a395e3 Compare May 14, 2026 21:44

miparnisari approved these changes May 14, 2026

View reviewed changes

github-actions Bot added the area/dependencies Affects dependencies label May 14, 2026

tstirrat15 enabled auto-merge (squash) May 14, 2026 21:55

tstirrat15 merged commit 8a01e69 into main May 14, 2026
44 of 45 checks passed

tstirrat15 deleted the tstirrat/coalesce-cache-on-otter branch May 14, 2026 22:04

github-actions Bot locked and limited conversation to collaborators May 14, 2026

		// TODO: check name here
		return NewOtterCache[K, V]("", config)

Conversation

tstirrat15 commented May 12, 2026

Description

Changes

Testing

Uh oh!

codecov Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

⚠️ Performance Alert ⚠️

Uh oh!

tstirrat15 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miparnisari May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tstirrat15 commented May 12, 2026

Uh oh!

tstirrat15 commented May 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tstirrat15 commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented May 12, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

miparnisari May 14, 2026 •

edited

Loading

tstirrat15 commented May 14, 2026 •

edited

Loading