Add TTL values for client caches of key locations and auto remove expired entries #12514

jzhou77 · 2025-10-24T19:29:33Z

Currently, client side cache only remove entries when the cache size limit is reached. However, if a storage server has changed IP, but the client is not accessing the server, the old entry in the cache will never be removed. Thus the client will keep try connecting to the old IP addresses (maybe until some timeout value?).

The change here is to add TTL for cached entries and remove old entries if they were never renewed/used recently.

20251024-210832-jzhou-96ffca1323c55610 compressed=True data_size=37340840 duration=4572667 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=2:24:43 sanity=False started=100000 stopped=20251024-233315 submitted=20251024-210832 timeout=5400 username=jzhou

20251120-192454-jzhou-1c85ca93ab32246b compressed=True data_size=37340794 duration=5168185 ended=100000 fail=5 fail_fast=10 max_runs=100000 pass=99995 priority=100 remaining=0 runtime=2:03:10 sanity=False started=100000 stopped=20251120-212804 submitted=20251120-192454 timeout=5400 username=jzhou

Only saw 1 failure in the log, which is TestUnexpectedlyNotFinished but reproduction passed, likely due to ssd-sharded-rocksdb being slow.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

So the client may automatically remove stale entries, e.g., storage server has since changed IP addresses.

foundationdb-ci · 2025-10-24T19:40:23Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: bc933ea
Duration 0:10:35
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T19:40:55Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: bc933ea
Duration 0:11:09
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T19:41:21Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: bc933ea
Duration 0:11:33
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T19:43:39Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: bc933ea
Duration 0:13:56
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T19:44:09Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: bc933ea
Duration 0:14:24
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-24T19:45:14Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: bc933ea
Duration 0:15:27
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T19:48:11Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: bc933ea
Duration 0:18:26
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T21:18:49Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 2967d31
Duration 0:11:34
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T21:32:37Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 2967d31
Duration 0:25:22
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T21:51:30Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 2967d31
Duration 0:44:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T21:54:15Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 2967d31
Duration 0:46:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T21:59:22Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 2967d31
Duration 0:52:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-24T22:16:55Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 2967d31
Duration 1:09:42
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

Doesn't seem to be effective in simulation runs. will go back to this.

saintstack

LGTM. One question in below.

fdbclient/DatabaseContext.actor.cpp

johscheuer · 2025-11-19T16:22:29Z

fdbclient/DatabaseContext.actor.cpp


 	clientDBInfoMonitor = monitorClientDBInfoChange(this, clientInfo, &proxiesChangeTrigger);
 	tssMismatchHandler = handleTssMismatches(this);
+	locationCacheCleanup = cleanupLocationCache(this);


Should we add a cancel() call for this actor in the destructor (e.g. https://github.com/apple/foundationdb/blob/main/fdbclient/DatabaseContext.actor.cpp#L1573)?

We don't need to explicitly call cancel(), because locationCacheCleanup destructor will automatically cancel the actor. Usually explictly calling cancel is to proactively clean the state to avoid destruction order problems. To be on the safe side, I'll add the cancel call in the destructor.

johscheuer · 2025-11-19T17:08:06Z

fdbclient/include/fdbclient/DatabaseContext.h

 	using Locations = MultiInterface<ReferencedInterface<StorageServerInterface>>;
 	explicit LocationInfo(const std::vector<Reference<ReferencedInterface<StorageServerInterface>>>& v)
-	  : Locations(v) {}
+	  : Locations(v),


Please correct me if I'm wrong, but do I understand the code correctly that the LocationInfo holds information of one or more StorageServerInterface and the expire time is set for the LocationInfo and not the information of the individual StorageServerInterface in this LocationInfo?

that's correct. Each shard is stored on multiple storage servers, so LocationInfo points to all replicas.

This seems to be a field added for StorageCache feature, which was removed.

foundationdb-ci · 2025-11-20T19:43:40Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 78ca1a2
Duration 0:25:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-20T19:56:33Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 78ca1a2
Duration 0:38:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-20T20:02:16Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 78ca1a2
Duration 0:43:51
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-20T20:16:49Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 78ca1a2
Duration 0:58:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-20T20:17:34Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 78ca1a2
Duration 0:59:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-20T20:28:35Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 78ca1a2
Duration 1:10:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

jzhou77 added 2 commits October 24, 2025 12:24

Add TTL values for client caches of key locations

cf3480c

So the client may automatically remove stale entries, e.g., storage server has since changed IP addresses.

Add the actor to clean up expired location cache entries

bc933ea

Add code probe for location cache cleanups

2967d31

jzhou77 marked this pull request as ready for review October 24, 2025 22:12

Tweak knobs

18b1b4a

Doesn't seem to be effective in simulation runs. will go back to this.

saintstack approved these changes Nov 11, 2025

View reviewed changes

fdbclient/DatabaseContext.actor.cpp Show resolved Hide resolved

saintstack approved these changes Nov 12, 2025

View reviewed changes

johscheuer reviewed Nov 19, 2025

View reviewed changes

Remove hasCaches from LocationInfo

78ca1a2

This seems to be a field added for StorageCache feature, which was removed.

johscheuer mentioned this pull request Nov 21, 2025

[RELEASE-7.3] Fix client locality cache #12572

Open

5 tasks

Add TTL values for client caches of key locations and auto remove expired entries #12514

Are you sure you want to change the base?

Add TTL values for client caches of key locations and auto remove expired entries #12514

Conversation

jzhou77 commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 24, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

saintstack left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

johscheuer Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jzhou77 Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

johscheuer Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jzhou77 Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Nov 20, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 20, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Nov 20, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 20, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 20, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 20, 2025

jzhou77 commented Oct 24, 2025 •

edited

Loading