feat(log-ingestor): Make database connection pool size configurable (fixes #2039). - #2413
Open
junhaoliao wants to merge 12 commits into
Open
feat(log-ingestor): Make database connection pool size configurable (fixes #2039).#2413junhaoliao wants to merge 12 commits into
junhaoliao wants to merge 12 commits into
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe log-ingestor MySQL pool size is now configurable through Python and Rust configuration, validated as positive, exposed through deployment templates, and passed to pool creation. Missing configuration returns an error, with accompanying documentation and test updates. ChangesLog-ingestor pool configuration
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant LogIngestorConfig
participant ClpDbIngestionConnector
participant MySQLPool
LogIngestorConfig->>ClpDbIngestionConnector: provide configured pool size
ClpDbIngestionConnector->>MySQLPool: create pool with configured size
ClpDbIngestionConnector-->>LogIngestorConfig: return contextual error if configuration is missing
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
junhaoliao
marked this pull request as draft
July 22, 2026 01:48
…ction-pool-size' into feat/log-ingestor-database-connection-pool-size
junhaoliao
marked this pull request as ready for review
July 22, 2026 02:21
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/clp-py-utils/clp_py_utils/clp_config.py`:
- Line 100: Add or confirm Python-side validation tests for the
DatabaseConnectionPoolSize alias covering 0, 2**32, and 2**32 - 1; assert the
exclusive lower and upper bounds reject 0 and 2**32 while accepting 2**32 - 1,
preserving alignment with Rust’s NonZeroU32 contract.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 9e7a8fb2-fb79-46dc-b1b2-6e1b57248347
📒 Files selected for processing (5)
components/clp-py-utils/clp_py_utils/clp_config.pycomponents/clp-rust-utils/src/clp_config/package/config.rscomponents/log-ingestor/src/ingestion_job_manager.rscomponents/log-ingestor/src/ingestion_job_manager/clp_ingestion.rscomponents/package-template/src/etc/clp-config.template.json.yaml
💤 Files with no reviewable changes (1)
- components/log-ingestor/src/ingestion_job_manager.rs
LinZhihao-723
requested changes
Jul 22, 2026
Comment on lines
+232
to
+233
| const DEFAULT_LOG_INGESTOR_DATABASE_CONNECTION_POOL_SIZE: NonZeroU32 = | ||
| NonZeroU32::new(100).unwrap(); |
Member
There was a problem hiding this comment.
- How about moving it into
default? - If not, we should move it after all public symbols to follow the symbol ordering guideline here.
- Please don't use
unwrap: useexpect("Readable reason").
…abase-connection-pool-size
…abase-connection-pool-size
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #2039.
The log-ingestor previously constructed its MySQL connection pool with a hardcoded maximum of 100 connections. This PR adds
log_ingestor.database_connection_pool_sizeso operators can tune that limit while preserving 100 as the default, and passes the configured value to the pool constructor:clp/components/log-ingestor/src/ingestion_job_manager/clp_ingestion.rs
Lines 204 to 215 in 956f27c
The Python configuration model validates the setting as a positive value that fits in a Rust
u32, while the Rust mirror usesNonZeroU32to reject zero during deserialization. The configured value is passed tocreate_clp_db_mysql_pool, and a missing log-ingestor configuration is reported as a startup error instead of introducing a new panic path.The packaged
clp-config.yamltemplate documents the new setting, and focused Rust tests cover the default, a custom value, and rejection of zero. The tests use the project'sanyhow::Resultconvention, and adjacent Rust error, panic, andexpectmessages follow the lowercase, no-period style. The Helm chart exposes the same setting throughclpConfig.log_ingestor, renders it into the CLP ConfigMap, and bumps the chart version to0.4.1-dev.2.Checklist
Validation performed
Rust lint
Rust tests
Configuration and generated schema
Helm chart
Full build
Package smoke test
The package smoke test could not reach compression because the local Docker Compose v5.3.1 installation hit the zero-replica dependency regression tracked in #2374. The partially started package was stopped successfully.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation