Skip to content

Conversation

DevSeongmin
Copy link

@DevSeongmin DevSeongmin commented Aug 6, 2025

Summary

This PR introduces a batchSize parameter to RedisItemReader and its builder to solve the classic N+1 issue faced during batch reads from Redis.
Currently, RedisItemReader executes individual GET commands per Redis key resulting in a significant number of network round-trips and performance bottlenecks.

By enabling batch reading with Redis MGET when batchSize > 1, this enhancement drastically reduces network calls and improves throughput while maintaining full backward compatibility when batchSize is not set (default = 1).

Key Changes

  • Added batchSize property with default value 1 in RedisItemReader and RedisItemReaderBuilder.
  • When batchSize > 1, the reader fetches multiple keys in batch using Redis multiGet() and internally buffers results.
  • Existing behavior preserved for batchSize = 1, ensuring no breaking changes.
  • Added thorough JavaDoc explaining usage, limitations, and batch size considerations.
  • Updated unit tests to cover both single and batch modes, verifying correct data retrieval and network call optimization.

Benefits

  • Significant performance improvements through reduced network round-trips during batch processing.
  • Configurable batching allowing tuning based on memory and performance requirements.
  • Maintains Spring Batch's principle of backward compatibility and progressive enhancement.
  • Handles Redis SCAN cursor nature without breaking existing semantics.

Technical Considerations

  • Batch size validated to be greater than 0; recommending values between 1 and 1000 to prevent excessive memory use and large network calls.
  • The internal queue buffers values for efficient retrieval during batch reads.
  • Users should be aware of Redis SCAN characteristics such as key duplication and lack of ordering; idempotent batch jobs recommended.
  • Restartability remains unsupported due to unordered nature of Redis SCAN results.

Related Issue

Fixes #4941


"Terminated by KILL-9 SQUAD 💀"

…+1 problem mitigation

- Introduced batchSize parameter to RedisItemReader and builder
- When batchSize > 1, read keys in batches using Redis MGET
- Backward compatible: batchSize = 1 retains existing behavior
- Added unit tests covering single and batch modes
- Updated Javadoc with usage and limitations

Signed-off-by: DevSeongmin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enhance RedisItemReader with batchSize option to optimize N+1 problem via MGET operations
1 participant