Skip to content

Adversarial fixtures: citations that mention a topic but do not support the claim #3

Description

@alsoleg89

Problem

The current 12-case benchmark proves structural traceability and safe rendering. It deliberately does not prove that an excerpt entails the attached evidence statement.

We need inspectable fixtures for the next benchmark layer.

Fixture families

Contributions can add one or more minimal examples:

  • a citation that merely mentions the subject;
  • a source whose scope is narrower than the claim;
  • correlation presented as causation;
  • a stale source used for a current-state claim;
  • two sources that appear contradictory only because populations or time windows differ;
  • an excerpt that drops a decisive negation or qualifier.

Acceptance criteria

  • Each fixture includes the claim, bounded source excerpt, locator, and expected verdict.
  • The failure is explainable without a model-specific score.
  • Copyrighted excerpts stay short and link to the source.
  • At least one nearby valid control prevents an over-broad rule.
  • Fixtures contain no secrets, personal data, or unverifiable private sources.
  • Benchmark documentation states which checks are deterministic and which require a reviewer/model.

See benchmarks/README.md and skill/doubt/references/evidence-ladder.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    evidence-contractChanges to validation rules or the map schemahelp wantedExtra attention is neededneeds-evidenceClaim or fixture needs stronger source support

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions