feat(relay): add Accept-Encoding gzip to all https.request calls#1681
Open
feat(relay): add Accept-Encoding gzip to all https.request calls#1681
Conversation
Node.js https.request() does NOT send Accept-Encoding by default (unlike fetch()). Add gzip, deflate to 22 external API call sites in the AIS relay and use the existing _collectDecompressed() helper for transparent decompression. This reduces Railway egress bandwidth for all relay upstream calls (Yahoo Finance, Finnhub, GDELT, UCDP, CelesTrak, AviationStack, ICAO, World Bank, Polymarket, OpenSky auth, cyber threat feeds). 4 calls already had gzip support (YouTube, OpenSky raw). Redis/internal calls are unchanged.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
_collectDecompressed() now accepts optional maxBytes parameter that aborts the decompression stream mid-flight when the limit is exceeded, instead of buffering the full response first. CelesTrak fetch uses 2MB limit to prevent memory pressure in the long-running relay process.
…uard 10 tests covering the streaming size limit: - Source contract: maxBytes param exists, size check before push, stream+response destroyed on limit, descriptive error message, CelesTrak uses 2MB limit - Behavioral: under-limit resolves, unlimited resolves, over-limit rejects mid-stream, identity passthrough, corrupt gzip rejects
…nto test:sidecar - Extract _collectDecompressed from ais-relay.cjs into scripts/_relay-decompress.cjs so tests can import the real function without triggering relay startup/process.exit - Update ais-relay-gzip.test.cjs to require the real implementation instead of duplicating it (eliminates drift risk between test copy and production code) - Add scripts/ais-relay-gzip.test.cjs to test:sidecar in package.json so the maxBytes regression test runs in pre-push and CI automatically
…ent behavior proxyToCloud() strips the browser Origin then injects 'https://worldmonitor.app' so the cloud API key validator doesn't reject the request with 401. The test was written before this injection was added and still asserted null. Same pattern as #1329 which fixed the equivalent assertion for local handlers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Node.js
https.request()does NOT sendAccept-Encodingby default (unlikefetch()which does). The AIS relay had 22 external API call sites making uncompressed requests, wasting egress bandwidth.Accept-Encoding: gzip, deflateto all 22 externalhttps.get/requestcallslet body = ''; resp.on('data', ...)) with existing_collectDecompressed()helperAPIs affected
Yahoo Finance, Finnhub, GDELT, UCDP, CelesTrak TLE, AviationStack, ICAO NOTAM, World Bank, Polymarket, OpenSky auth, cyber threat feeds (OTX, URLhaus, Feodo, C2Intel, AbuseIPDB)
What was already gzip-enabled (unchanged)
Why not seed scripts?
Node.js
fetch()(used by seed scripts) already sendsAccept-Encoding: gzipby default and auto-decompresses. PR #1676 was closed after proving this empirically.Test plan
node -c scripts/ais-relay.cjssyntax valid