[Enhancement] Add a cleaner for BrpcStubCache to cleanup unused connections

[duanyyyyyyy]

[Enhancement] Add a cleaner for BrpcStubCache to cleanup unused connections

Why I'm doing:

The BRPC Stub cache will never be evicted and will always stored in the cache.
Therefore there will be some bug here.
In the case that when users are using K8s to deploy the StarRocks backend, after sometime running there will be many log like this.

After check the BRPC code in https://github.com/apache/brpc/blob/1.8.0/src/brpc/socket.cpp#L1340
And from the grafana monitor

This cluster have 148 BE nodes, normally it will be about 148 BRPC stub but after some times running, some pods restarted and the IP changed, some of the node BRPC stub creased to over 200.
And as we all known the endpoint in BRPC are made of ip and host, that means if the IP is changed the endpoint will still in the cache.

What I'm doing:

Here I introduce a BrpcStubManager to periodically check and cleanup expired BRPC stub for BrpcStubCache, HttpBrpcStubCache and LakeServiceBrpcStubCache

1. Record the last access time when get the stub for given endpoint
2. Loop all the endpoint in _stub_map or _last_access_time_map check if now - last_access_time is larger than expire time or not
3. If less, will not clean up the endpoint
4. If larger or equal, will try to clean up the endpoint

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 4.0
  • 3.5
  • 3.4
  • 3.3

[duanyyyyyyy]

Hi @kevincai master, are there any suggestion for this case? Or any comment for this PR?

[kevincai]

Hi @kevincai master, are there any suggestion for this case? Or any comment for this PR?

will take a look today.

[xhumanoid]

@duanyyyyyyy gentle ping =)

[duanyyyyyyy]

@duanyyyyyyy gentle ping =)

sry for forget.

[duanyyyyyyy]

@duanyyyyyyy gentle ping =)

Seems some unrelated test failed, let me retry for it

[duanyyyyyyy]

@kevincai Master, any other comments?

[Copilot]

Pull Request Overview

This PR adds a cleanup mechanism for BRPC stub caches to prevent accumulation of unused connections in Kubernetes environments where pod IPs change frequently. The implementation periodically cleans up expired BRPC stubs from three cache types.

Key Changes:

• Introduces timer-based cleanup tasks for BRPC stub expiration
• Adds configurable expiration time (brpc_stub_expire_s) with 60-minute default
• Refactors cache data structures to support scheduled cleanup

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
be/src/util/brpc_stub_cache.h	Adds cleanup task classes and timer support to cache headers
be/src/util/brpc_stub_cache.cpp	Implements timer-based cleanup logic for all three BRPC cache types
be/src/common/config.h	Adds configurable expiration time parameter
be/src/runtime/exec_env.cpp	Updates BrpcStubCache constructor to accept ExecEnv parameter
be/test/util/brpc_stub_cache_test.cpp	Adds comprehensive tests for cleanup functionality
docs/en/administration/management/BE_configuration.md	Documents new configuration parameter
be/test/http/stream_load_test.cpp	Updates test to use new constructor signature
be/test/http/transaction_stream_load_test.cpp	Updates test to use new constructor signature

[kevincai]

overall looks good to me.

[github-actions]

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 95 / 101 (94.06%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/util/brpc_stub_cache.cpp	92	98	93.88%	[66, 182, 183, 194, 259, 270]
🔵	be/src/runtime/exec_env.cpp	1	1	100.00%	[]
🔵	be/src/util/brpc_stub_cache.h	2	2	100.00%	[]

[duanyyyyyyy]

@stdpain Master, any comments?