feat(compression-coordinator): Introduce max_concurrent_tasks config to limit job-handler concurrency. - #2435
Conversation
WalkthroughThe change adds a validated concurrency limit to compression coordinator configuration. The coordinator uses a semaphore and FIFO queue for new-job scheduling, while startup recovery can launch all recovered jobs. The package template documents coordinator and Spider connection settings. ChangesCompression coordinator concurrency
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CompressionCoordinator
participant JobDatabase
participant JobHandler
CompressionCoordinator->>JobDatabase: Fetch pending jobs when capacity exists
CompressionCoordinator->>CompressionCoordinator: Add fetched jobs to the FIFO queue
CompressionCoordinator->>JobHandler: Spawn handlers after acquiring permits
JobHandler->>CompressionCoordinator: Retain the permit until completion
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bill-hbrhbr
changed the title
max_concurrent_tasks config to limit job-handler task concurrency.
Bill-hbrhbr
changed the title
max_concurrent_tasks config to limit job-handler task concurrency.max_concurrent_tasks config to limit job-handler concurrency.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/clp-py-utils/clp_py_utils/clp_config.py`:
- Line 798: Add a shared Tokio semaphore upper-bound validation for
max_concurrent_tasks in components/clp-py-utils/clp_py_utils/clp_config.py:798
and components/clp-rust-utils/src/clp_config/package/config.rs:488, preserving
the existing positive-value validation. In
components/compression-coordinator/src/coordination.rs:137-138, validate the
configured value before constructing job_handler_sem and return the established
configuration error when it exceeds the supported limit.
In `@components/compression-coordinator/src/coordination.rs`:
- Around line 49-52: Update the recovery admission logic around the
coordinator’s recovered-handler tracking so every recovered handler contributes
to the active count, including handlers that do not hold a permit. Prevent new
jobs from being admitted until the total active recovered and newly started
handlers is below max_concurrent_tasks, and revise the recovery documentation
near the constructor to describe this limit-enforced behavior.
- Around line 270-275: Update the pending-job refill flow around
fetch_new_job_rows so it retrieves eligible jobs in bounded pages rather than
loading and retaining the entire backlog in pending_job_queue. Add durable
paging state, such as a cursor that advances only after successful processing or
equivalent recovery-safe state, and preserve first-fetch recovery semantics so
retries resume without skipping jobs or causing unbounded memory growth.
In `@components/package-template/src/etc/clp-config.template.json.yaml`:
- Line 72: Update the commented max_concurrent_tasks configuration entry in the
template to explicitly state that its value must be greater than zero,
distinguishing it from compression_scheduler.max_concurrent_tasks_per_job where
zero disables the limit.
- Line 67: Update the compression coordinator configuration note near the
“Compression coordinator config” comment to document every runtime prerequisite:
`spider`, `logs_input.type: s3`, and `archive_output.storage.type: s3`; clarify
that the related template examples must use these required S3 settings rather
than `fs` when enabling the configuration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 8fd3f43d-d433-437d-ab55-235ab39cfb5f
📒 Files selected for processing (4)
components/clp-py-utils/clp_py_utils/clp_config.pycomponents/clp-rust-utils/src/clp_config/package/config.rscomponents/compression-coordinator/src/coordination.rscomponents/package-template/src/etc/clp-config.template.json.yaml
| if self.pending_job_queue.is_empty() && self.job_handler_sem.available_permits() > 0 { | ||
| let new_job_rows = self.fetch_new_job_rows().await.inspect_err(|e| { | ||
| tracing::error!(error = % e, "Failed to fetch new jobs from database."); | ||
| })?; | ||
| self.pending_job_queue.extend(new_job_rows); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Bound the pending-job queue.
fetch_new_job_rows returns every eligible database row, and this code retains every row and serialized CLP I/O configuration in pending_job_queue. A large backlog can still cause unbounded memory growth.
Fetch jobs in bounded pages. Preserve the first-fetch recovery semantics with a durable cursor or equivalent paging state.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@components/compression-coordinator/src/coordination.rs` around lines 270 -
275, Update the pending-job refill flow around fetch_new_job_rows so it
retrieves eligible jobs in bounded pages rather than loading and retaining the
entire backlog in pending_job_queue. Add durable paging state, such as a cursor
that advances only after successful processing or equivalent recovery-safe
state, and preserve first-fetch recovery semantics so retries resume without
skipping jobs or causing unbounded memory growth.
There was a problem hiding this comment.
We currently only fetch new jobs when the pending queue has been exhausted. so it's not like we will accumulate a backlog or get any duplicated pending jobs.
also this PR is about the limit for tokio spawns.
if you want to do paged fetches that's another story.
There was a problem hiding this comment.
@Bill-hbrhbr, agreed that this change correctly limits concurrent Tokio job-handler tasks.
The queue does not accumulate across refills. However, fetch_new_job_rows still loads all eligible rows during one refill, so one large backlog can make pending_job_queue large.
Paged fetching is separate from this PR’s scope. Would you like me to create a follow-up GitHub issue for bounded job fetching?
🐇💭
You are interacting with an AI system.
There was a problem hiding this comment.
@LinZhihao-723 do we need bounded job fetching
There was a problem hiding this comment.
Yeah, I thought it's a part of the plan.
There was a problem hiding this comment.
Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!
| # port: 6000 | ||
| # logging_level: "INFO" | ||
| # | ||
| ## Compression coordinator config. When set, the `spider` config below must also be set. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Document all required runtime prerequisites.
components/compression-coordinator/src/bin/compression_coordinator.rs requires logs_input.type and archive_output.storage.type to be s3, in addition to requiring spider. The template examples use fs at Lines [29-34] and [150-155]. The new note names only spider, so enabling this configuration can still produce a startup error. Update the note at Line [67].
Proposed documentation update
-## Compression coordinator config. When set, the `spider` config below must also be set.
+## Compression coordinator config. When enabled, set the `spider` config below and use
+## S3 for `logs_input.type` and `archive_output.storage.type`.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ## Compression coordinator config. When set, the `spider` config below must also be set. | |
| ## Compression coordinator config. When enabled, set the `spider` config below and use | |
| ## S3 for `logs_input.type` and `archive_output.storage.type`. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@components/package-template/src/etc/clp-config.template.json.yaml` at line
67, Update the compression coordinator configuration note near the “Compression
coordinator config” comment to document every runtime prerequisite: `spider`,
`logs_input.type: s3`, and `archive_output.storage.type: s3`; clarify that the
related template examples must use these required S3 settings rather than `fs`
when enabling the configuration.
There was a problem hiding this comment.
we will implement fs output in the near future. check docstring for other wording updates.
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
entrypoint=$(fd -t f '^compression_coordinator\.rs$' components)
if [ -n "$entrypoint" ]; then
ast-grep outline "$entrypoint" --items all
rg -n -C 5 --glob '*.rs' \
'(logs_input|archive_output|storage\.type|compression_coordinator|Spider|spider|S3|s3)' \
"$(dirname "$entrypoint")"
fi
rg -n -C 4 --glob '*.{rs,md,rst,yaml,yml,json}' \
'(requires? .*s3|requires? .*S3|archive_output.*s3|logs_input.*s3|compression coordinator config)' \
componentsLength of output: 13354
@Bill-hbrhbr, I checked the runtime validation. The coordinator currently rejects non-S3 logs_input and non-S3 archive_output.storage. It also requires spider.
The template should describe the current restriction and allow a future filesystem-output change:
## Compression coordinator config. Currently requires `logs_input.type` and
## `archive_output.storage.type` to be "s3", and `spider` to be set.I will keep the finding open because the current wording omits the archive-output requirement.
✏️ Learnings added
Learnt from: Bill-hbrhbr
Repo: y-scope/clp PR: 2435
File: components/package-template/src/etc/clp-config.template.json.yaml:0-0
Timestamp: 2026-08-01T09:17:11.203Z
Learning: The Rust compression coordinator currently supports only S3 log inputs and S3 archive outputs. Filesystem archive output is planned for a future implementation, so configuration documentation should state the S3 restriction as a current limitation.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
You are interacting with an AI system.
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
components/compression-coordinator/src/coordination.rs (1)
294-300: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winRecord a job as dispatched only after handler creation succeeds.
dispatched_job_ids.push(job_id)runs before configuration deserialisation andcreate_job_handle. If either operation fails,runstill passes the ID tomark_jobs_dispatched. AnError::UnsupportedInputConfigjob therefore receivesdispatch_timewithout a coordinator handler, which removes it from the subsequent pending-job query and can interfere with the documented legacy-scheduler hand-off.Move the push after
create_job_handlereturnsOk:Proposed fix
let job_id = job_row.id; - dispatched_job_ids.push(job_id); let clp_io_config: ClpIoConfig = match BrotliMsgpack::deserialize(&job_row.serialized_clp_io_config) { Ok(clp_io_config) => clp_io_config, Err(e) => { ... } }; let Ok(job_handle) = self.create_job_handle(job_id, clp_io_config).await else { continue; }; +dispatched_job_ids.push(job_id);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/compression-coordinator/src/coordination.rs` around lines 294 - 300, Move dispatched_job_ids.push(job_id) in the run flow to after configuration deserialization and successful create_job_handle completion, so only jobs with a coordinator handler are marked dispatched; preserve failure handling without passing failed job IDs to mark_jobs_dispatched.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/package-template/src/etc/clp-config.template.json.yaml`:
- Line 67: Update the compression coordinator configuration note near the
existing logs_input.type and spider requirements to also state that
archive_output.storage.type must be "s3". Keep the documentation aligned with
the coordinator prerequisites and the archive output setting shown in the
template.
---
Outside diff comments:
In `@components/compression-coordinator/src/coordination.rs`:
- Around line 294-300: Move dispatched_job_ids.push(job_id) in the run flow to
after configuration deserialization and successful create_job_handle completion,
so only jobs with a coordinator handler are marked dispatched; preserve failure
handling without passing failed job IDs to mark_jobs_dispatched.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: b74354c7-b2d8-4659-a5ac-82902ed1fabb
📒 Files selected for processing (3)
components/compression-coordinator/src/coordination.rscomponents/compression-coordinator/src/error.rscomponents/package-template/src/etc/clp-config.template.json.yaml
| # port: 6000 | ||
| # logging_level: "INFO" | ||
| # | ||
| ## Compression coordinator config. Requires `logs_input.type` to be "s3" and `spider` to be set. |
There was a problem hiding this comment.
Document the archive output prerequisite.
Line [67] documents the logs_input.type: "s3" and spider requirements, but the coordinator also requires archive_output.storage.type: "s3". The template still shows "fs" for archive output at Lines [150-155]. If users enable this configuration with the template values, the coordinator fails during startup. Update this note to include the archive output requirement.
Proposed documentation update
-## Compression coordinator config. Requires `logs_input.type` to be "s3" and `spider` to be set.
+## Compression coordinator config. Requires `logs_input.type` and `archive_output.storage.type` to be "s3", and `spider` to be set.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@components/package-template/src/etc/clp-config.template.json.yaml` at line
67, Update the compression coordinator configuration note near the existing
logs_input.type and spider requirements to also state that
archive_output.storage.type must be "s3". Keep the documentation aligned with
the coordinator prerequisites and the archive output setting shown in the
template.
| # port: 6000 | ||
| # logging_level: "INFO" | ||
| # | ||
| ## Compression coordinator config. Requires `logs_input.type` to be "s3" and `spider` to be set. |
There was a problem hiding this comment.
We are removing Spider config from this file, as it is only used for docker compose.
| while !self.pending_job_queue.is_empty() { | ||
| let Ok(permit) = self.job_handler_sem.clone().try_acquire_owned() else { | ||
| break; | ||
| }; |
There was a problem hiding this comment.
Getting a semaphore in a busy loop is not a good design.
There was a problem hiding this comment.
As discussed offline, I think it should be fine: the performance overhead isn't that large though especially in this use case.
There was a problem hiding this comment.
I don't think we should update this config. The current compression coordinator only supports helm (k8s) deployment which doesn't use this template. We should update it when we have docker compose support ready,
| job_polling_interval: Duration, | ||
| cancellation_token: CancellationToken, | ||
| job_handler_sem: Arc<Semaphore>, | ||
| pending_job_queue: VecDeque<PendingJobRowProjection>, |
There was a problem hiding this comment.
I'm not sure if we need this. Can you explain why we can't do the following instead:
- On the main loop, fetch compression jobs with a limit and ID ordering (sth like
SELECT * FROM t ORDER BY id ASC LIMIT 100;) - Get a permit before spawning the coroutine. The permit automatially drops itself if the coroutine exits, aborts, or crashed.
- Let the semaphore to block coroutine creation implicitly. The main loop may be blocked, which is fine because it shouldn't push more jobs into Spider.
This should lead to the behavior we expect iiuc.
3 participants
Description
Limit the number of compression jobs processed concurrently to prevent the coordinator from creating an unbounded number of job-handler tasks and consuming excessive memory when a large backlog of jobs accumulates.
On restart, the coordinator first resumes tracking all jobs that were previously submitted to Spider before accepting new work. This recovery step bypasses the concurrency limit, but counts resumed jobs against it, preventing new jobs from being submitted until capacity becomes available.
This preserves the existing recovery behavior while ensuring that newly admitted jobs respect the configured processing capacity.
The concurrency limit is configured through a new
max_concurrent_tasksfield in ClpConfig and is populated from the configuration file.Checklist
breaking change.
Validation performed
Summary by CodeRabbit