Skip to content

feat: stream large submission exports directly from db without memory… - #2123

Merged
RUKAYAT-CODER merged 4 commits into
EarnQuestOne:mainfrom
AbuJulaybeeb:feature/stream-submission-exports
Jul 30, 2026
Merged

feat: stream large submission exports directly from db without memory…#2123
RUKAYAT-CODER merged 4 commits into
EarnQuestOne:mainfrom
AbuJulaybeeb:feature/stream-submission-exports

Conversation

@AbuJulaybeeb

Copy link
Copy Markdown
Contributor

closes #1971.

This PR addresses memory spikes caused by generating large submission exports for quests with thousands of participants by transitioning entirely away from offset-based, chunked data queries and removing memory buffering in our streaming service.

Changes:

True DB Streaming Cursors: Refactored getQueryIterator in StreamExportService to utilize TypeORM's native .stream() method. Previously, getQueryIterator achieved "chunking" via .skip(offset).take(chunkSize), which executes repeated offset queries causing degraded SQL performance for deep records and buffering arrays in memory.
Implement Network Backpressure: Rewrote streamAsCSV, streamAsJSONLines, and streamAsJSON in StreamExportService to evaluate the boolean return value from response.write(). If the underlying TCP buffer backs up (returns false), the stream now pauses via await new Promise(resolve => response.once('drain', resolve)) until the buffer clears, capping concurrency safely.
Testing & Benchmarks: Updated stream-export.service.spec.ts mocks to handle the new once and backpressure assertions. A new benchmark script (BackEnd/scripts/benchmark-export.ts) was added to demonstrate stable memory (less than a few MBs of variance) regardless of how large the requested dataset is.

@drips-wave

drips-wave Bot commented Jul 28, 2026

Copy link
Copy Markdown

@AbuJulaybeeb Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@RUKAYAT-CODER

Copy link
Copy Markdown
Contributor

Kindly resolve conflict

@AbuJulaybeeb
AbuJulaybeeb marked this pull request as draft July 30, 2026 16:01
@AbuJulaybeeb
AbuJulaybeeb marked this pull request as ready for review July 30, 2026 16:14
@RUKAYAT-CODER

Copy link
Copy Markdown
Contributor

Thank you for contributing to the project.

@RUKAYAT-CODER
RUKAYAT-CODER merged commit abbe197 into EarnQuestOne:main Jul 30, 2026
4 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream large submission exports instead of buffering in memory

2 participants