Problem / motivation
Some feed providers rate-limit aggressively per client IP. Reddit is the common case: it allows roughly 1 request/minute per IP for its .rss endpoints and returns 429 Too Many Requests for bursts. If you subscribe to a number of subreddit feeds, CommaFeed cannot keep them refreshed — the refresh engine fetches them back-to-back (often within the same second), the provider 429s most of them, and they never reliably update.
Current behavior
As far as I can tell from the refresh engine (FeedRefreshEngine / FeedDAO.findNextUpdatable / setDisabledUntil):
- feed-refresh.interval is global — there's no per-host or per-feed interval.
- There's no per-host rate limiter, so all feeds due at the same moment are fetched as fast as the worker threads allow.
- On error (e.g. 429), a feed is rescheduled to now + backoff-interval × errorCount (capped at max-interval).
This last point creates an unfortunate failure mode when several feeds from the same host are involved. Feeds that fail in the same pass share the same now, so they all reschedule to (nearly) the same instant and stay grouped. The group keeps fetching together, keeps mutually 429-ing, errorCount climbs, and they
all pile onto the max-interval cap as a single synchronized burst. Once this happens it doesn't self-heal — effectively only one feed from the host succeeds per cycle.
Why the obvious workarounds fall short
- Raising feed-refresh.interval is global, so it slows down every feed, not just the throttled host — and it still doesn't prevent same-host feeds from bursting together within a pass.
- Manually staggering each feed's next-refresh time works only until the first 429; the backoff rescheduling described above re-synchronizes them, and they collapse back into a burst.
Proposed feature
Either (or ideally both) of:
- Per-host request rate limiting. A configurable minimum spacing between requests to the same host (derived from the feed URL's host), e.g. feed-refresh.per-host-min-interval with optional per-host overrides. The worker would serialize/delay requests to a host so they never exceed the configured rate, regardless of how many feeds share that host. Honoring Retry-After on 429 to set the per-host cooldown would be a natural fit.
- Per-feed (or per-subscription) refresh interval override. Allow a feed/subscription to declare its own minimum interval, overriding the global one, so high-volume hosts can be polled slowly without slowing everything else.
A smaller, complementary improvement: when rescheduling failed feeds, add a small jitter to the backoff so feeds that fail together don't re-converge on an identical next-refresh timestamp.
Environment
CommaFeed v7.1.0 (Quarkus), H2 backend, single-node deployment. Happy to provide logs or test a patch.
Problem / motivation
Some feed providers rate-limit aggressively per client IP. Reddit is the common case: it allows roughly 1 request/minute per IP for its .rss endpoints and returns 429 Too Many Requests for bursts. If you subscribe to a number of subreddit feeds, CommaFeed cannot keep them refreshed — the refresh engine fetches them back-to-back (often within the same second), the provider 429s most of them, and they never reliably update.
Current behavior
As far as I can tell from the refresh engine (FeedRefreshEngine / FeedDAO.findNextUpdatable / setDisabledUntil):
This last point creates an unfortunate failure mode when several feeds from the same host are involved. Feeds that fail in the same pass share the same now, so they all reschedule to (nearly) the same instant and stay grouped. The group keeps fetching together, keeps mutually 429-ing, errorCount climbs, and they
all pile onto the max-interval cap as a single synchronized burst. Once this happens it doesn't self-heal — effectively only one feed from the host succeeds per cycle.
Why the obvious workarounds fall short
Proposed feature
Either (or ideally both) of:
A smaller, complementary improvement: when rescheduling failed feeds, add a small jitter to the backoff so feeds that fail together don't re-converge on an identical next-refresh timestamp.
Environment
CommaFeed v7.1.0 (Quarkus), H2 backend, single-node deployment. Happy to provide logs or test a patch.