Skip to content

test(openfeature): add STATIC and SPLIT reason assertions#5506

Draft
leoromanovsky wants to merge 4 commits intomasterfrom
leo.romanovsky/openfeature-reason-assertions
Draft

test(openfeature): add STATIC and SPLIT reason assertions#5506
leoromanovsky wants to merge 4 commits intomasterfrom
leo.romanovsky/openfeature-reason-assertions

Conversation

@leoromanovsky
Copy link
Copy Markdown
Contributor

Motivation

The Ruby evaluation_engine_spec.rb tests assert ERROR and MATCH reason paths, but do not verify STATIC or SPLIT reason values. The native evaluator (Rust FFI via libdatadog) already returns correct reason values, but without test assertions, regressions in the reason contract would go undetected.

This is part of a cross-SDK effort to ensure all OpenFeature reason values are covered by tests (see dd-trace-cross-repo planning).

Changes

  • Added STATIC reason test context: verifies the override path (no rules, 1 split, no shards) returns reason: 'STATIC'
  • Added SPLIT reason test context: verifies shard-based assignment returns reason: 'SPLIT'
  • Added MATCH reason assertion to the existing resolution details context (previously only asserted value, not reason)
  • All existing ERROR reason tests remain unchanged

Decisions

  • Mock-based approach: These tests stub NativeEvaluator#get_assignment to return ResolutionDetails with specific reason values, matching the existing test pattern. This verifies the engine correctly propagates reason values from the evaluator.
  • No lib/ changes: The native evaluator already computes correct reasons; only test coverage was missing.

…on engine spec

- Add STATIC reason test context (override path: no rules, 1 split, no shards)
- Add SPLIT reason test context (shard-based assignment)
- Add MATCH reason assertion to existing resolution details context
- All existing ERROR reason tests unchanged
@github-actions
Copy link
Copy Markdown

👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description.

If changes need to be present in CHANGELOG.md you can state it this way

**Change log entry**

Yes. A brief summary to be placed into the CHANGELOG.md

(possible answers Yes/Yep/Yeah)

Or you can opt out like that

**Change log entry**

None.

(possible answers No/Nope/None)

Visited at: 2026-03-26 01:40:24 UTC

@github-actions github-actions bot added the dev/testing Involves testing processes (e.g. RSpec) label Mar 26, 2026
@datadog-datadog-prod-us1-2
Copy link
Copy Markdown

datadog-datadog-prod-us1-2 bot commented Mar 26, 2026

⚠️ Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

❄️ 103 New flaky tests detected

OpenFeature evaluation fixtures test-case-kill-switch-flag.json case 0/alice from rspec   View in Datadog   (Fix with Cursor)
wrong number of arguments (given 1, expected 0)

Failure/Error: let(:configuration) { Datadog::Core::FeatureFlags::Configuration.new(config_json) }

ArgumentError:
  wrong number of arguments (given 1, expected 0)
./spec/datadog/open_feature/fixture_spec.rb:9:in 'BasicObject#initialize'
./spec/datadog/open_feature/fixture_spec.rb:9:in 'Class#new'
./spec/datadog/open_feature/fixture_spec.rb:9:in 'block (2 levels) in <top (required)>'
./spec/datadog/open_feature/fixture_spec.rb:53:in 'block (5 levels) in <top (required)>'
...
OpenFeature evaluation fixtures test-case-kill-switch-flag.json case 1/bob from rspec   View in Datadog   (Fix with Cursor)
wrong number of arguments (given 1, expected 0)

Failure/Error: let(:configuration) { Datadog::Core::FeatureFlags::Configuration.new(config_json) }

ArgumentError:
  wrong number of arguments (given 1, expected 0)
./spec/datadog/open_feature/fixture_spec.rb:9:in 'BasicObject#initialize'
./spec/datadog/open_feature/fixture_spec.rb:9:in 'Class#new'
./spec/datadog/open_feature/fixture_spec.rb:9:in 'block (2 levels) in <top (required)>'
./spec/datadog/open_feature/fixture_spec.rb:53:in 'block (5 levels) in <top (required)>'
...
OpenFeature evaluation fixtures test-case-kill-switch-flag.json case 10/6 from rspec   View in Datadog   (Fix with Cursor)
wrong number of arguments (given 1, expected 0)

Failure/Error: let(:configuration) { Datadog::Core::FeatureFlags::Configuration.new(config_json) }

ArgumentError:
  wrong number of arguments (given 1, expected 0)
./spec/datadog/open_feature/fixture_spec.rb:9:in 'BasicObject#initialize'
./spec/datadog/open_feature/fixture_spec.rb:9:in 'Class#new'
./spec/datadog/open_feature/fixture_spec.rb:9:in 'block (2 levels) in <top (required)>'
./spec/datadog/open_feature/fixture_spec.rb:53:in 'block (5 levels) in <top (required)>'
...
View all

🧪 1 Test failed

gem release process datadog.gemspec files includes all important files from rspec   View in Datadog   (Fix with Cursor)
Failure/Error:
  expect(gemspec.files)
    .to match_array(
      \`git ls-files -z\`
        .split("\x0")
        .reject { |f| f.match(directories_excluded) }
        .reject { |f| f.match(single_files_excluded) }
    )

  expected collection contained:  [".gitmodules", "CHANGELOG.md", "LICENSE", "LICENSE-3rdparty.csv", "LICENSE.Apache", "LICENSE.BSD3", ...ls.rb", "lib/datadog/tracing/workers.rb", "lib/datadog/tracing/writer.rb", "lib/datadog/version.rb"]
...
This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8eeaf9e | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

- Copy ufc-config.json and 24 evaluation-case JSON files
- Files are byte-identical to Go reference fixtures
…Go canonical fixtures

- Loop over all 24 evaluation-case JSON files and assert value + reason
- Handle known discrepancies: FLAG_NOT_FOUND (ERROR vs DEFAULT), startAt/endAt
  reason (TARGETING_MATCH vs STATIC), insignificant shard optimization (STATIC vs SPLIT)
- 220 fixture test cases passing, existing evaluation_engine_spec.rb untouched
- Add ffe-system-test-data as git submodule at spec/datadog/open_feature/
- Update fixture_spec.rb paths from testdata/ to ffe-system-test-data/
- Remove local testdata/ directory (now sourced from submodule)
- Add .github/dependabot.yml for git submodule auto-updates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev/testing Involves testing processes (e.g. RSpec)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant