Merge extracted data into existing Wikidata statements (augment-in-place)

[monneyboi]

Merge extracted data into existing Wikidata statements (augment-in-place)

Problem

When enrichment extracts dates for a position that already has a Wikidata statement (e.g. a dateless P39), the current flow is: user accepts the new extraction (creates a new statement) and deprecates the existing one. This causes needless statement churn: it destroys the statement ID, severs its references and other qualifiers, and uses deprecated rank for statements that were vague rather than false.

Instead, users should be able to merge an extraction into an existing statement: the extracted dates and source are added to the statement in place.

Semantics: union-only merge

Merging is strictly additive — it can never destroy information:

• Add an extracted qualifier (P580/P582) only if the live statement has no value for that property.
• Append the extraction's reference, deduped by URL.
• Never replace or remove existing qualifiers, references, value, or rank.

Because nothing can be overwritten, no conflict resolution or automatic classification is needed. The user decides statement identity ("is this extraction about the same tenure?"); the system only executes their choice. Value replacement (fixing wrong dates) is explicitly out of scope and remains a manual Wikidata edit.

Backend shape: edits as first-class domain objects

The unit PoliLoom operates on is an edit to Wikidata, made explicit instead of being derived at push time from (is_accepted, statement_id IS NULL).

Edit algebra (new module poliloom/wikidata/edits.py, all pure):

@dataclass
class CreateStatement:
    property_id: str          # P39
    value: dict
    qualifiers: list[dict]
    references: list[dict]

@dataclass
class AugmentStatement:       # union-only merge
    statement_id: str
    qualifiers: list[dict]
    references: list[dict]

@dataclass
class DeprecateStatement:
    statement_id: str

One pure resolver — the only place decision logic lives; also validates merge targets (same politician, same type + entity_id, target has statement_id):

def resolve_edit(prop, is_accepted, target) -> Edit | None:
    # extracted + accepted            -> CreateStatement
    # extracted + accepted + target   -> AugmentStatement
    # extracted + rejected            -> None (local soft delete)
    # statement + rejected            -> DeprecateStatement
    # statement + accepted            -> None (no-op)

Executors — thin effectful shell around pure builders, one per edit type, uniform shape fetch live → build operation → apply → reconcile local state from the API response:

• Create → POST statement; extracted row gains the returned statement_id.
• Augment → GET live statement → pure build_augment_patch(live, edit) (JSON Patch, REST statement format) → PATCH. Live-state drift degrades gracefully: the patch shrinks to whatever is still missing, down to a no-op. Reconcile the local target row from the PATCH response body, move PropertyReferences over, soft-delete the extracted row.
• Deprecate → PATCH rank (unchanged).

Cache principle: Wikidata is the source of truth; a local Property row with a statement_id is a read cache, a row without one is a proposal. Acceptance graduates a proposal into a cached statement; augment absorbs a proposal into an existing statement. Reconciliation always writes back what Wikidata returned.

Schema & API

• Migration: evaluations.target_property_id (nullable FK → properties), for audit.
• Evaluation create accepts optional target_property_id; validation delegated to resolve_edit.
• push_evaluation becomes resolve → execute → reconcile; record outcome (resulting statement id / patch summary / error) on the evaluation for audit and retry.

Frontend: drag-and-drop merge

• Users merge by dragging an extracted property card onto an existing statement card for the same entity. The target highlights as a drop zone; on drop the extracted card docks into it as a pending merge (chip showing what will be added: e.g. + P580: 2020-01-01, + 1 source), with undo.
• Pending merges participate in the normal session submit flow like any other evaluation.
• Since merges are union-only, dropping needs no confirm dialog.
• Must be keyboard-accessible and work on touch (dnd-kit sensors).
• Tutorial: new/changed steps explaining merging — when to merge into a statement vs. create a new one (different tenure), and that merging preserves the statement's history, references, and qualifiers.

Build order

1. poliloom/wikidata/edits.py (edit types, resolve_edit, build_augment_patch) + tests
2. Migration + target_property_id on evaluation create
3. Executor refactor of push_evaluation
4. GUI: drag-and-drop merge with pending state
5. Tutorial updates