Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 23 additions & 5 deletions .github/workflows/config/.secrets.baseline

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Why is this diff here? Maybe an incorrect merge conflict resolution? Looks like possibly some other unexpected changes like pyproject.toml that are effectively reverting PRs from main.

If we do need to make changes to this secrets config, can we do it in a totally separate PR that just merges directly to main, and is not part of the feature branch changes?

Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
{
"name": "GitHubTokenDetector"
},
{
"name": "GitLabTokenDetector"
},
{
"name": "HexHighEntropyString",
"limit": 3.0
Expand All @@ -37,21 +40,26 @@
"name": "IbmCosHmacDetector"
},
{
"name": "JwtTokenDetector"
"name": "IPPublicDetector"
},
{
"name": "KeywordDetector",
"keyword_exclude": ""
"name": "JwtTokenDetector"
},
{
"name": "MailchimpDetector"
},
{
"name": "NpmDetector"
},
{
"name": "OpenAIDetector"
},
{
"name": "PrivateKeyDetector"
},
{
"name": "PypiTokenDetector"
},
{
"name": "SendGridDetector"
},
Expand All @@ -67,6 +75,9 @@
{
"name": "StripeDetector"
},
{
"name": "TelegramBotTokenDetector"
},
{
"name": "TwilioKeyDetector"
}
Expand All @@ -77,7 +88,7 @@
},
{
"path": "detect_secrets.filters.common.is_baseline_file",
"filename": ".secrets.baseline"
"filename": ".github/workflows/config/.secrets.baseline"
},
{
"path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies",
Expand Down Expand Up @@ -109,7 +120,14 @@
},
{
"path": "detect_secrets.filters.heuristic.is_templated_secret"
},
{
"path": "detect_secrets.filters.regex.should_exclude_file",
"pattern": [
"\\.github/workflows/config/\\.secrets\\.baseline"
]
Comment thread
nina-xu marked this conversation as resolved.
}
],
"results": {}
"results": {},
"generated_at": "2026-08-04T15:08:14Z"
}
3 changes: 2 additions & 1 deletion .markdownlint.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{
"MD013": false,
"MD034": false
"MD034": false,
"MD046": { "style": "fenced" }
}
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Source code lives in `src/nemo_safe_synthesizer/`:
| `generation/` | GeneratorBackend, VllmBackend, regex manager, batch gen |
| `holdout/` | Train/test splitting |
| `llm/` | Model loading, metadata, memory management |
| `pii_replacer/` | NER-based PII detection and replacement |
| `pii_replacer/` | Heuristic tabular PII discovery and persona replacement |
| `privacy/` | DP transformers (Opacus integration) |
| `sdk/` | SafeSynthesizer builder, library_builder |
| `training/` | TrainingBackend, HuggingFace backend, timeseries_preprocessing (`timeseries_preprocessing.py`) |
Expand Down
2 changes: 0 additions & 2 deletions STYLE_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,8 +154,6 @@ def is_json_object(value: object) -> TypeIs[dict[str, JsonValue]]:
)
```

Legacy modules (`pii_replacer/`) still use `Optional`/`List`/`Dict`. Only `pii_replacer/` is excluded from `ty` type-checking (see `[tool.ty.src] exclude` in `pyproject.toml`). The remaining legacy usages will be migrated when those modules come under the type checker.

### Control flow

- Prefer `match`/`case` for dispatch on types or tagged values. Not a blanket rule -- `if`/`elif` is fine for simple boolean predicates.
Expand Down
8 changes: 4 additions & 4 deletions design.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ graph TB
TrainConfig[TrainingHyperparams]
GenConfig[GenerateParameters]
EvalConfig[EvaluationParameters]
PIIConfig[PiiReplacerConfig]
PIIConfig[ReplacePiiConfig]
DPConfig[DifferentialPrivacyHyperparams]
end
end
Expand Down Expand Up @@ -207,7 +207,7 @@ Path: `src/nemo_safe_synthesizer/config/`
- `TrainingHyperparams`: Training settings (learning rate, epochs, batch size, etc.)
- `GenerateParameters`: Generation settings (temperature, top_p, num_records, etc.)
- `EvaluationParameters`: Evaluation component toggles and settings
- `PiiReplacerConfig`: PII detection and replacement settings
- `ReplacePiiConfig`: PII detection and replacement settings
- `DifferentialPrivacyHyperparams`: DP training parameters (epsilon, delta, clipping norm)

### 2. Data Processing Pipeline
Expand Down Expand Up @@ -415,7 +415,7 @@ results = synthesizer.results

### PII Protection

- Named Entity Recognition (NER) to detect PII
- Heuristic discovery to find PII columns (names, values, dtypes)
- Deterministic replacement with synthetic values
- Preserves column statistics and distributions

Expand Down Expand Up @@ -469,7 +469,7 @@ safe-synthesizer-artifacts/
2. Custom Generation Backend: Implement `GeneratorBackend` abstract class
3. Custom Evaluation Component: Extend `Component` base class
4. Custom Data Actions: Add to `data_processing/actions/`
5. Custom PII Detectors: Extend NER pipeline
5. Custom PII discovery: extend heuristic detectors in `pii_replacer/`

---

Expand Down
10 changes: 5 additions & 5 deletions docs/developer-guide/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ graph TB
TrainConfig["TrainingHyperparams"]
GenConfig["GenerateParameters"]
EvalConfig["EvaluationParameters"]
PIIConfig["PiiReplacerConfig"]
PIIConfig["ReplacePiiConfig"]
DPConfig["DifferentialPrivacyHyperparams"]
end
end
Expand Down Expand Up @@ -188,7 +188,7 @@ flowchart TB
training_p["TrainingHyperparams"]
gen_p["GenerateParameters"]
eval_p["EvaluationParameters"]
pii_p["PiiReplacerConfig | None"]
pii_p["ReplacePiiConfig | None"]
dp_p["DifferentialPrivacyHyperparams | None"]
ts_p["TimeSeriesParameters"]
end
Expand Down Expand Up @@ -233,7 +233,7 @@ Exactly what avenues of configuration are available, and thus how precedence is
- CLI: CLI flags > dataset registry overrides > YAML config file > defaults
- SDK: Python SDK builder calls > YAML config file > defaults

Nullable sub-configs (`PiiReplacerConfig | None`, `DifferentialPrivacyHyperparams | None`)
Nullable sub-configs (`ReplacePiiConfig | None`, `DifferentialPrivacyHyperparams | None`)
use `None` as the sole disabled signal. The `@pydantic_options` decorator auto-generates
`--no_<field>` is-flags for these fields; `parse_overrides` translates them into
`{field: None}` in the overrides dict.
Expand Down Expand Up @@ -297,7 +297,7 @@ Path: `src/nemo_safe_synthesizer/config/`
- TrainingHyperparams: training settings (learning rate, epochs, batch size, etc.)
- GenerateParameters: generation settings (temperature, top_p, num_records, etc.)
- EvaluationParameters: evaluation component toggles and settings
- PiiReplacerConfig: PII detection and replacement settings
- ReplacePiiConfig: PII detection and replacement settings
- DifferentialPrivacyHyperparams: DP training parameters (epsilon, delta, clipping norm)

### 2. Data Processing Pipeline
Expand Down Expand Up @@ -395,4 +395,4 @@ The execution follows a clear pipeline: Data --> PII Replacement --> Training --
2. Custom Generation Backend: implement `GeneratorBackend` abstract class
3. Custom Evaluation Component: extend `Component` base class
4. Custom Data Actions: add to `data_processing/actions/`
5. Custom PII Detectors: extend NER pipeline
5. Custom PII discovery: extend heuristic detectors in `pii_replacer/`
2 changes: 1 addition & 1 deletion docs/developer-guide/configuration_management.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ The generated Click options use `Field(description=...)` as CLI help text.
Optional sub-configs use `None` as the disabled signal. Do not add a second
boolean flag for the same feature.

For a field such as `replace_pii: PiiReplacerConfig | None`,
For a field such as `replace_pii: ReplacePiiConfig | None`,
`pydantic_options()` emits:

```bash
Expand Down
Loading
Loading