sts: add JMH benchmark suite by iamabhilaksh · Pull Request #588 · salesforce/multicloudj

iamabhilaksh · 2026-08-12T16:37:50Z

Summary

Adds a JMH benchmark suite for STS: AbstractStsBenchmarkTest (sts-client) plus thin AWS/GCP concretes. Gated by @EnabledIfSystemProperty(named="runBenchmarks", matches="true") — inert in CI (green without fixtures); the chameleon-example-service pipeline opts in explicitly.

Swept per provider — AWS 3, GCP 4: getCallerIdentity, getAssumeRoleCredentials, concurrentAssumeRole on both, plus getAccessToken on GCP. getAccessToken and getAssumeRoleWithWebIdentity are kept as @Benchmarks but gated out of the sweep when the provider/run can't exercise them (see Capability gating below); they re-enter automatically once runnable.

Testing proof

Run locally against live AWS + GCP, both JMH modes (Throughput + SampleTime), producing populated results with zero errors on both clouds. Per-provider swept counts and the capability gating are detailed under Capability gating below.

Invocation (creds via OS env only, never -D):

mvn test -pl sts/sts-aws -Dtest=AwsStsBenchmarkTest -DrunBenchmarks=true \
  -DSTS_BENCHMARK_AWS_REGION=us-west-2 \
  -DSTS_BENCHMARK_AWS_ROLE_ARN=<assumable-role-arn>
  # optional: -DSTS_BENCHMARK_AWS_ENDPOINT=<custom-endpoint>
mvn test -pl sts/sts-gcp -Dtest=GcpStsBenchmarkTest -DrunBenchmarks=true \
  -DSTS_BENCHMARK_GCP_SERVICE_ACCOUNT=<sa-email>

Rendered AWS-vs-GCP comparison pages are being published to the benchmark-visualizer (git.soma Pages) alongside the other suites.

Capability gating (`getAccessToken`, `getAssumeRoleWithWebIdentity`)

JMH has no per-method skip and ignores JUnit assumptions under Runner, so unexercisable benchmarks are excluded in runBenchmarks() based on per-provider capabilities rather than one blanket .exclude():

supportsGetAccessToken() — AWS false, GCP true
supportsWebIdentity(probe) — true iff STS_BENCHMARK_<CLOUD>_WEB_IDENTITY_TOKEN is set

This replaces the previous unconditional .exclude(getAccessToken), which also stripped GCP even though GCP implements the operation. It also removes the old in-method if (token == null) return no-op in getAssumeRoleWithWebIdentity — that shipped an empty method to JMH, which timed it as a ~1.5×10⁹ ops/s row indistinguishable from real data.

Verified live on 2026-08-13 — both clouds, both token states:

AWS getAccessToken still unrunnable: a direct GetSessionToken under the assumed-role creds returns AccessDenied: Cannot call GetSessionToken with session credentials (AWS maps getAccessToken → GetSessionToken). supportsGetAccessToken()=false excludes it; the gate keeps this AWS-only so GCP is no longer stripped.
AWS, no token: swept 3 (getCallerIdentity, getAssumeRoleCredentials, concurrentAssumeRole); getAccessToken and web-identity excluded. Run omitted STS_BENCHMARK_AWS_ENDPOINT, confirming the client derives the default regional endpoint.
AWS, token set: web-identity entered the sweep and made a live AssumeRoleWithWebIdentity call; AWS rejected the placeholder token server-side (InvalidIdentityTokenException, 400) — proof the call is live, not a no-op.
GCP, no token: swept 4 (adds getAccessToken); web-identity excluded. getAccessToken produced a real datapoint — coverage the old blanket exclude was denying GCP.
GCP, token set: web-identity entered the sweep and made a live STS token-exchange call; GCP rejected the placeholder token (400 invalid_request).

Gating is covered by a deterministic unit test (AbstractStsBenchmarkTestGatingTest, 5 cases, no creds) that runs in CI.

Run contract

sts-aws: STS_BENCHMARK_AWS_{REGION,ROLE_ARN} required; STS_BENCHMARK_AWS_ENDPOINT optional (default regional endpoint derived from region); STS_BENCHMARK_AWS_WEB_IDENTITY_TOKEN optional (enables the web-identity benchmark)
sts-gcp: STS_BENCHMARK_GCP_SERVICE_ACCOUNT required; STS_BENCHMARK_GCP_WEB_IDENTITY_TOKEN optional (enables the web-identity benchmark)

JMH config note

The class-level @Warmup/@Measurement/@Fork are the local-run baseline. The chameleon pipeline vendors these files and drives JMH via its own BenchmarkRunner, overriding warmup/measurement/forks via env vars — so those annotations are not the pipeline's config.

Downstream

Running this in the pipeline needs a separate vendor-sync PR into sfdc-bazel (new Bazel target + runtime_deps wiring + any infra/cron); it does not propagate automatically.

Merge order

Independent of the other suite PRs. Recommend the root-pom JMH fix (build: add jmh-generator-annprocess…) merges first so this is runnable on JDK 21+.

Adds an abstract AbstractStsBenchmarkTest in sts-client plus thin AWS/GCP concretes, gated by @EnabledIfSystemProperty(runBenchmarks=true) so CI stays green without fixtures. Swept: getCallerIdentity, getAssumeRoleCredentials, concurrentAssumeRole, getAssumeRoleWithWebIdentity. getAccessToken is kept but excluded from the sweep (AWS GetSessionToken rejects session creds).

codecov-commenter · 2026-08-12T16:49:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.61%. Comparing base (aea85ed) to head (a7fb99d).

Additional details and impacted files

@@            Coverage Diff            @@
##               main     #588   +/-   ##
=========================================
  Coverage     83.61%   83.61%           
  Complexity      674      674           
=========================================
  Files           215      215           
  Lines         15010    15010           
  Branches       2076     2076           
=========================================
  Hits          12550    12550           
  Misses         1636     1636           
  Partials        824      824

Flag	Coverage Δ
unittests	`83.61% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

LihaoLiuXs · 2026-08-12T22:53:18Z

+  @Benchmark
+  public void benchmarkGetAssumeRoleWithWebIdentity(Blackhole bh) {
+    String webIdentityToken = harness.getWebIdentityToken();
+    if (webIdentityToken == null) {


⚠️ This benchmark and the .exclude() at :184 are the same problem, and one hook fixes both

Both shipped harnesses return null from getWebIdentityToken() — AwsStsBenchmarkTest:38-41 and GcpStsBenchmarkTest:36-39 — so this guard fires on every provider and JMH times an empty method under the name of a federation round trip. Nothing excludes it; runBenchmarks() excludes only benchmarkGetAccessToken.

I measured what that actually publishes rather than guessing. Reproducing this guard shape under this suite's own class annotations with JMH 1.37: ~1.5 × 10⁹ ops/s (≈0.6 ns/op) in Throughput, against 17.4 ops/s for a 50 ms control — a factor of ~10⁸. The "reads ~0" intuition is true only of the SampleTime rows; in the JSON the Throughput row is the largest number in the file, and the record is schema-identical to a real measurement, so nothing downstream can tell it apart. The sharper consequence is trend continuity: the day someone wires a real token, this benchmark drops eight orders of magnitude and reads as a catastrophic regression when it is actually the benchmark starting to work.

The .exclude(".*benchmarkGetAccessToken.*") at :184 is the same problem from the other side. It's justified by an AWS credential constraint, but it's unconditional, so it also strips GCP — which implements the operation for real (sts-gcp/src/main/java/com/salesforce/multicloudj/sts/gcp/GcpSts.java:313-323).

You're right that JMH has no skip primitive and that Assumptions.assumeTrue is ignored under Runner — the javadoc at :119-121 says so and that's correct. But this repo already has the answer for this exact operation in this exact module: AbstractStsIT.Harness.supportsGetAccessToken() (sts-client/src/test/java/com/salesforce/multicloudj/sts/client/AbstractStsIT.java:48), decided per provider — AWS true, GCP true (GcpStsIT.java:64), Ali false. A boolean supportsX() on the benchmark Harness, with runBenchmarks() computing .exclude() from it, keeps the AWS reason in the AWS module, gives GCP back coverage it can support, and removes the empty-method datapoint — one mechanism for both halves. You already used exactly this shape for toPolicyMember in the sibling IAM PR, so it's not a new idea, just applied unevenly.

One correction while you're here: the javadoc at :146 says this mirrors multicloud-py's supports_get_access_token, but Python's gate is a per-harness predicate evaluated at test time — the base returns True (tests/sts/conformance/base_sts_conformance.py:136-143), AWS narrows it by cassette and credential type, GCP keeps it enabled. The two behave oppositely rather than alike, so that line is worth dropping or rewording.

Fixed — went with the capability-gate approach. Added supportsGetAccessToken() (AWS false, GCP true) and a token-driven supportsWebIdentity(); runBenchmarks() now computes .exclude() from those instead of the blanket line, so GCP gets getAccessToken back and the empty web-identity method is dropped rather than timed. I also deleted the in-method if (token == null) return no-op — the method now runs a real federation call whenever a token is supplied.

Verified both halves live on both clouds: with no token, web-identity is excluded on both and getAccessToken produces a real datapoint on the provider that supports it; with a token set, web-identity enters the sweep and makes a live call (the placeholder token gets rejected server-side, which is the proof it is no longer a no-op). Added AbstractStsBenchmarkTestGatingTest (5 cases, no creds) so the gating is covered in CI. And +1 on the sibling-SDK reference — dropped it.

Addressed in the capability-gate change above — thanks both.

Yaminik1996 · 2026-08-13T06:01:14Z

+
+  /**
+   * Short-lived access/session token issuance. Excluded from the default sweep (see
+   * runBenchmarks): AWS GetSessionToken rejects temporary/session credentials, so it cannot run


Provider-specific keywords (GetSessionToken, AWS) should ideally not appear in the cloud-agnostic sts-client module

Good catch — reworded the getAccessToken javadoc so the cloud-agnostic module no longer names a specific operation or provider. It now just says the benchmark is dropped on providers that report they cannot issue a token under the run credentials, with the provider-specific reason living in the concrete. Also genericized the new gating test so its method names describe capability shapes rather than provider names.

Yaminik1996 · 2026-08-13T06:01:45Z

+
+    @Override
+    public StsClient createStsClient() {
+      return StsClient.builder("aws").withRegion(region).withEndpoint(URI.create(endpoint)).build();


STS_BENCHMARK_AWS_ENDPOINT is a required env var and always applied via .withEndpoint(URI.create(endpoint))
Benchmarking real AWS STS normally uses the default regional endpoint, so forcing an override makes the common case awkward and diverges from the GCP harness, which requires only the service account.
Suggested fix: make the endpoint optional — apply withEndpoint only when the var is present.

Agreed, made it optional — withEndpoint is only applied when STS_BENCHMARK_AWS_ENDPOINT is set, so the common case just needs region and the SDK derives the default regional endpoint. Confirmed live: a run with the var omitted works fine against real STS. Matches the GCP harness now.

Yaminik1996 · 2026-08-13T06:02:18Z

+
+    @Override
+    public StsClient createStsClient() {
+      return StsClient.builder("aws").withRegion(region).withEndpoint(URI.create(endpoint)).build();


[Nit] Uses the literal "aws" instead of AwsConstants.PROVIDER_ID (= "aws"), which already exists. The GCP harness in this same PR uses GcpConstants.PROVIDER_ID, so this is internally inconsistent within the PR

Done — using AwsConstants.PROVIDER_ID for both getProviderId() and the builder, consistent with the GCP harness.

Yaminik1996 · 2026-08-13T06:03:26Z

+   * Short-lived access/session token issuance. Excluded from the default sweep (see
+   * runBenchmarks): AWS GetSessionToken rejects temporary/session credentials, so it cannot run
+   * under the assumed-role creds the pipeline uses. Kept for anyone running with long-term
+   * IAM-user creds. (Mirrors multicloud-py's supports_get_access_token capability gate.)


The "(Mirrors multicloud-py's supports_get_access_token capability gate.)" reference points at a sibling Python SDK. We should refrain from referencing other SDKs.

Removed — dropped the sibling-SDK reference from that javadoc.

- Add supportsGetAccessToken() (abstract) + supportsWebIdentity(probe); runBenchmarks() computes excludes from these instead of one static .exclude(getAccessToken) that also stripped GCP. - Drop the in-method no-op in benchmarkGetAssumeRoleWithWebIdentity so it makes a real federation call when a token is supplied. - Make STS_BENCHMARK_AWS_ENDPOINT optional (default regional endpoint). - Use AwsConstants.PROVIDER_ID instead of literal "aws". - Keep cloud-agnostic module provider-neutral (no provider keywords). - Add AbstractStsBenchmarkTestGatingTest (5 cases, no creds) for CI.

salesforce-cla Bot added the cla:signed label Aug 12, 2026

LihaoLiuXs mentioned this pull request Aug 12, 2026

build: add jmh-generator-annprocess to annotationProcessorPaths #587

Open

LihaoLiuXs reviewed Aug 12, 2026

View reviewed changes

This was referenced Aug 12, 2026

registry: add JMH benchmark suite #590

Open

dbbackuprestore: add JMH benchmark suite #591

Open

Yaminik1996 reviewed Aug 13, 2026

View reviewed changes

iamabhilaksh force-pushed the feat/sts-benchmark-suite branch from e06a997 to a7fb99d Compare August 13, 2026 12:22

Yaminik1996 approved these changes Aug 13, 2026

View reviewed changes

Navigation Menu

Uh oh!

sts: add JMH benchmark suite - #588

sts: add JMH benchmark suite#588
iamabhilaksh wants to merge 2 commits into
salesforce:mainfrom
iamabhilaksh:feat/sts-benchmark-suite

iamabhilaksh commented Aug 12, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 12, 2026 •

edited

Loading

Uh oh!

LihaoLiuXs Aug 12, 2026

Uh oh!

Yaminik1996 Aug 13, 2026

Uh oh!

iamabhilaksh Aug 13, 2026

Uh oh!

iamabhilaksh Aug 13, 2026

Uh oh!

Yaminik1996 Aug 13, 2026

Uh oh!

iamabhilaksh Aug 13, 2026

Uh oh!

Yaminik1996 Aug 13, 2026

Uh oh!

iamabhilaksh Aug 13, 2026

Uh oh!

Yaminik1996 Aug 13, 2026

Uh oh!

iamabhilaksh Aug 13, 2026

Uh oh!

Yaminik1996 Aug 13, 2026

Uh oh!

iamabhilaksh Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

iamabhilaksh commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing proof

Capability gating (getAccessToken, getAssumeRoleWithWebIdentity)

Run contract

JMH config note

Downstream

Merge order

Uh oh!

codecov-commenter commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iamabhilaksh commented Aug 12, 2026 •

edited

Loading

Capability gating (`getAccessToken`, `getAssumeRoleWithWebIdentity`)

codecov-commenter commented Aug 12, 2026 •

edited

Loading