Skip to content

feat: expand PII scrubber regression benchmarks - #874

Open
Oluwatobi843 wants to merge 1 commit into
Pulsefy:mainfrom
Oluwatobi843:feature/pii-scrubber-regression-benchmarks
Open

feat: expand PII scrubber regression benchmarks#874
Oluwatobi843 wants to merge 1 commit into
Pulsefy:mainfrom
Oluwatobi843:feature/pii-scrubber-regression-benchmarks

Conversation

@Oluwatobi843

Copy link
Copy Markdown

Summary

Resolves #767 — adds false-positive/false-negative regression fixtures, a reproducible benchmark script, scrubber bug fixes, and CI integration.


Problem

The original PII scrubber had two known bugs:

  • Phone regex too greedy: matched years (2024), dates (2024-07-28), prices ($4500), port numbers (8080)
  • ID regex too broad: any 4+ digit standalone number was redacted, including years and version numbers

The original fixture set (4 cases) had no false-positive or false-negative coverage, so these bugs were invisible.


Changes

scrubber.py

  • Phone regex now uses a broad capture + digit-count filter (≥ 7 digits required)
  • ISO 8601 dates (YYYY-MM-DD) are stashed before phone matching and restored after
  • Currency-prefixed numbers ($£€) and slash-separated numbers (date paths) are excluded
  • Numeric ID threshold raised from 4+ digits → 8+ digits
  • Email pass runs first so digit groups inside email tokens don't trigger phone/ID passes

tests/fixtures/pii_inputs.json + expected_outputs.json

Expanded from 4 to 22 test cases across four categories:

Category Count Purpose
Basic 4 Core PII types (email, phone, ID, safe text)
False-positive 7 Safe data that must NOT be redacted (years, dates, versions, prices, ports)
False-negative 6 PII variants that MUST be caught (subdomain email, local phone, national ID, mixed PII)
Edge 5 Tricky in-context cases

tests/benchmark_pii.py (new)

Reproducible benchmark script with:

  • Accuracy %, per-category pass/fail breakdown
  • Throughput measurement (calls/sec, avg latency ms)
  • --output FILE for JSON report (CI artifact upload)
  • --min-accuracy PCT flag — exits 1 if below threshold

.github/workflows/ci-python-tests.yml

  • Extended paths triggers to include scrubber.py and tests/**
  • New pii-scrubber job: runs pytest + benchmark with --min-accuracy 95
  • Benchmark JSON report uploaded as a 30-day retained CI artifact

Test results

22 cases — 100% accuracy
  ✓ basic           4/4
  ✓ false_positive  7/7
  ✓ false_negative  6/6
  ✓ edge            5/5
Throughput: ~51,000 calls/sec

Acceptance criteria

  • Fixture set includes hard false-positive and false-negative examples
  • Benchmark output is reproducible in CI or local runs
  • Results are easy to compare between revisions (JSON artifact + stdout table)

- Expand tests/fixtures/pii_inputs.json from 4 to 22 cases covering:
  - Basic PII (email, phone, numeric ID, safe text)
  - False-positive cases: years, ISO dates, software versions,
    currency prices, port numbers, plain sentences
  - False-negative cases: subdomain emails, plus-addressed emails,
    local phone numbers, phones with parentheses, national IDs,
    mixed PII in a single string
  - Edge cases: multiple emails, punctuation-adjacent email,
    safe short numbers, lone year, ID embedded in alphanumeric text

- Fix scrubber.py to eliminate false positives:
  - Phone regex now requires >= 7 digit characters (rejects years/ports)
  - ISO 8601 dates (YYYY-MM-DD) are explicitly preserved before phone pass
  - Currency-prefixed numbers ($£€) are excluded from phone matching
  - Slash-separated tokens (dates like 2024/07/28) are not matched
  - Numeric ID redaction raised to 8+ digits (from 4+)

- Add tests/benchmark_pii.py:
  - Reports accuracy %, per-category breakdown, and throughput
  - Supports --output FILE for JSON report (CI artifact upload)
  - Supports --min-accuracy PCT flag (exits 1 if below threshold)

- Update .github/workflows/ci-python-tests.yml:
  - Extend path triggers to include scrubber.py and tests/**
  - Add pii-scrubber job: runs pytest + benchmark with --min-accuracy 95
  - Uploads benchmark_report.json as a retained CI artifact

Closes Pulsefy#767
@drips-wave

drips-wave Bot commented Jul 28, 2026

Copy link
Copy Markdown

@Oluwatobi843 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@Cedarich Cedarich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix failed CI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand PII Scrubber Regression Benchmarks

2 participants