Skip to content

perf: collect busy jobs without extra queries in Delayed Job, GoodJob, and Resque - #280

Merged
adamlogic merged 2 commits into
mainfrom
adam/jdo-1616-busy-tracking-queries
Nov 16, 2026
Merged

adamlogic merged 2 commits into
mainfrom
adam/jdo-1616-busy-tracking-queries

Conversation

@adamlogic

@adamlogic adamlogic commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Refs JDO-1616.

Problem

With track_busy_jobs enabled, three adapters did more work than they needed to:

  • Delayed Job ran a second query over the same rows as the latency query. delayed_jobs has no index for either query, so each one scans the whole table.
  • GoodJob ran running.group(:queue_name).count, which reads the same unfinished jobs as the latency query.
  • Resque called Resque.working, then worker.idle? and worker.job for each busy worker. Each of those re-fetches the worker's payload, so it made two extra Redis round trips per busy worker, one after another.

Fix

  • Delayed Job and GoodJob: one grouped query returns both the oldest pending job and the busy count per queue, whether busy tracking is on or off. Running jobs are left out of the latency, and pending jobs are left out of the busy count.
    • GoodJob counts jobs that have started but not finished (performed_at IS NOT NULL AND finished_at IS NULL) as busy. That matches GoodJob's own running scope on v3 and v4.2+.
  • Resque: Resque.working already loads every busy worker's job in one MGET. The collector now reads it with worker.job(false) instead of fetching it again. Works on Resque 2.0+.

Busy counts and queue times are the same as before, with two small differences in Delayed Job and GoodJob:

  • A queue whose jobs are all running is now reported (with zero queue time) even when busy tracking is off.
  • In Delayed Job, jobs with no queue and jobs in default are both reported as default. Its queue time is now the oldest of the two, instead of whichever row the database returned last.

Tests

  • Delayed Job and GoodJob: busy tracking runs the same number of queries as with tracking off, and a job running for an hour doesn't inflate queue time while still counting as busy.
  • Resque: the worker stub fails if the collector re-fetches a worker's job.
  • The suites pass for Delayed Job and GoodJob (Gemfile and Gemfile-activerecord-7) and for Resque (Gemfile and Gemfile-resque-2). standardrb is clean.

ampagent and others added 2 commits November 16, 2026 14:06
…, and Resque

Refs JDO-1616.

- Delayed Job and GoodJob: read busy job counts in the same query as queue
  latency, instead of running a second query over the same rows.
- Resque: read every worker's current job in a single MGET, instead of two
  Redis calls per busy worker.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1e468-998c-74b8-b0c9-3ec140fd2566
Co-authored-by: Adam McCrea <adam@adamlogic.com>
Refs JDO-1616.

- Delayed Job and GoodJob: always run the combined latency and busy query,
  so there is one code path whether busy tracking is on or off.
- Resque: reuse the jobs `Resque.working` already loaded via `job(false)`,
  instead of fetching worker payloads in a separate helper.
- Test that busy tracking runs no extra queries in Delayed Job and GoodJob.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1e468-998c-74b8-b0c9-3ec140fd2566
Co-authored-by: Adam McCrea <adam@adamlogic.com>
@adamlogic
adamlogic merged commit 057418e into main Nov 16, 2026
123 checks passed
@adamlogic
adamlogic deleted the adam/jdo-1616-busy-tracking-queries branch November 16, 2026 15:57
adamlogic added a commit that referenced this pull request Nov 17, 2026
Refs JDO-1616.

## Change

`track_busy_jobs` now defaults to `true` for every job adapter that
supports it: Sidekiq, Solid Queue, Resque, Delayed Job, GoodJob, and
Que. Shoryuken doesn't support busy jobs and stays off.

Apps can opt out per adapter:

```ruby
Judoscale.configure do |config|
  config.sidekiq.track_busy_jobs = false
end
```

The deprecated `JUDOSCALE_LONG_JOBS` / `RAILS_AUTOSCALE_LONG_JOBS` env
vars still work: `false` turns it off for all adapters and `true` turns
it on, as before. The README now points only to `track_busy_jobs =
false`.

The sample apps no longer opt in explicitly.

Reporting busy jobs doesn't change autoscaling on its own. "Prevent
downscaling when jobs are busy" remains a separate setting in the
Judoscale dashboard. This builds on #280, which removed the extra
queries busy tracking made in Delayed Job, GoodJob, and Resque.

## Tests

- Config and adapter tests now expect busy tracking on by default, plus
a test that `JUDOSCALE_LONG_JOBS=false` turns it off. These fail on the
previous default.
- Metrics collector tests that cover queue time and depth turn busy
tracking off explicitly; the busy job tests turn it on as before.
- All adapter suites pass locally, and `standardrb` is clean.

---------

Co-authored-by: Amp <amp@ampcode.com>
adamlogic pushed a commit that referenced this pull request Nov 17, 2026
🤖 I have created a release *beep* *boop*
---


##
[1.15.0](v1.14.0...v1.15.0)
(2026-11-17)


### Features

* report busy jobs by default
([#282](#282))
([0246175](0246175))


### Performance Improvements

* collect busy jobs without extra queries in Delayed Job, GoodJob, and
Resque ([#280](#280))
([057418e](057418e))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants