Skip to content

Add --synthetic_duration option for time-based synthetic workloads#342

Merged
zeke-emerson merged 2 commits intomainfrom
zeke-simplify-duration-run
Mar 2, 2026
Merged

Add --synthetic_duration option for time-based synthetic workloads#342
zeke-emerson merged 2 commits intomainfrom
zeke-simplify-duration-run

Conversation

@zeke-emerson
Copy link
Copy Markdown
Contributor

Problem

Running synthetic benchmarks for a specific wall-clock duration currently requires calculating the right --synthetic_requests value (desired_duration * requests_per_sec). This is unintuitive — users just want to say "run for 5 minutes at 10 QPS" without doing math.

Solution

Add a --synthetic_duration CLI option (in seconds) as an alternative to --synthetic_requests. When specified, query iterators yield indefinitely and RunUser checks a time-based deadline to end the Run phase. The progress bar shows elapsed/total time instead of request counts.

Key design decisions:

  • Mutually exclusive with --synthetic_requests — specifying both produces an error
  • Backward compatible — when neither is specified, defaults to --synthetic_requests=100
  • Scoped to synthetic and synthetic-proportional — not supported with synthetic-runbook (its multi-step structure makes duration splitting complex; can be added later)
  • Deadline check in RunUser — keeps timing logic in one place rather than threading it through every iterator

Files changed:

  • vsb/cmdline_args.py — new argument + validation
  • vsb/workloads/base.pysynthetic_duration() method on base class
  • vsb/workloads/synthetic_workload/synthetic_workload.py — duration mode for InMemoryWorkload, SyntheticWorkload, and SyntheticProportionalWorkload
  • vsb/users.py — deadline check in RunUser.do_run(), time-based progress bar in LoadShape

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update
  • Infrastructure change (CI configs, etc)
  • Non-code change (docs, etc)
  • None of the above: (explain here)

Test Plan

  • All 33 existing unit tests pass (poetry run pytest tests/unit)
  • Verified --synthetic_duration=30 --requests_per_sec=10 runs for ~30 seconds against Pinecone
  • Verified omitting both flags defaults to --synthetic_requests=100 (backward compat)
  • Verified specifying both --synthetic_duration and --synthetic_requests produces an error
  • Verified --synthetic_duration with synthetic-runbook produces an error

Copy link
Copy Markdown

@dmsmith25 dmsmith25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with and without the --synthetic_duration tag. Both tests succeeded.

@zeke-emerson zeke-emerson merged commit 7d860a6 into main Mar 2, 2026
4 checks passed
@zeke-emerson zeke-emerson deleted the zeke-simplify-duration-run branch March 2, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants