Flag process-based requirements (testing, evaluation, approval, documentation) across C01–C13

[jmanico]

Flag process-based requirements (not verifiable against a live system or by a deterministic pipeline artifact)

A requirement is process-based when it is satisfied by performing an activity like external security testing, scanning, evaluation, simulation, audit, assessment, or human approval, or by maintaining documentation, inventories, plans, or formal proofs.

It does not establish a property that can be verified by exercising the live system or by inspecting a deterministic technical artifact (signing, hashing, lineage, version control, access controls, enforcement gates, runtime monitoring).

Automated security scanning DOES belongs here when it runs in a training pipeline because it's a verifiable security property in a live system, not an external review process.

Requirements in this category across C01–C13:

• C01 — Training Data Integrity: 1.1.1 (source inventory), 1.1.3 (change-approval workflow), 1.2.1 (retirement impact assessment), 1.4.2 (poisoning-detection testing), 1.4.5 (pre-deployment bias evaluation), 1.4.6 (documented poisoning-defense risk assessment)
• C03 — Model Lifecycle: 3.1.1 (deployed-model inventory), 3.2.1 (input-validation testing), 3.2.2 (output-sanitization testing), 3.2.3 (pre-deployment safety evaluation), 3.2.5 (security testing harness), 3.2.7 (re-evaluation of compressed models), 3.4.2 (approved review before deployment), 3.5.1 (hosted-model inventory), 3.5.2 (re-evaluation on provider change), 3.6.2 (separation of duties), 3.6.3 (reward-hacking detection)
• C05 — Access Control: 5.2.3 (documented classification taxonomy)
• C06 — Supply Chain: 6.1.2 (malicious-layer scanning), 6.1.3 (quarantine pending human sign-off), 6.1.4 (behavioral acceptance testing), 6.1.5 (adversarial evaluation), 6.2.2 (key re-approval on rotation), 6.3.1 (pre-training content scrubbing), 6.3.2 (dataset poisoning risk assessment), 6.4.1 (incident-response playbooks)
• C08 — Memory & Embeddings: 8.4.1 (embedding-leakage targets & regression gating)
• C09 — Orchestration: 9.1.3 (runaway-loop security testing), 9.2.4 (tested compensating actions)
• C11 — Adversarial Robustness: 11.1.2 (alignment test suite), 11.1.4 (documented/reproducible alignment training), 11.1.5 (evaluation-awareness assessment), 11.2.1 (adversarial evaluation), 11.2.4 (robustness-metric tracking), 11.2.5 (documented/reproducible hardening), 11.2.6 (adaptive-attack evaluation), 11.2.7 (formal robustness verification), 11.2.8 (post-transform re-certification), 11.2.9 (post-training integrity verification), 11.3.3 (membership-inference simulations), 11.6.2 (threshold tuning on validation sets), 11.6.4 (false-positive-rate measurement), 11.10.1 (stratified robustness evaluation)
• C12 — Privacy: 12.1.2 (k-anonymity audits), 12.1.3 (feature-importance analysis), 12.1.4 (formal re-identification proofs), 12.2.2 (post-unlearning evaluation), 12.2.3 (certified unlearning), 12.3.1 (DP-budget tracking), 12.3.2 (empirical privacy audits), 12.3.3 (formal privacy proofs), 12.6.3 (canary privacy auditing), 12.6.4 (formal ε-budget proofs)
• C13 — Monitoring & Logging: 13.3.3 (documented baseline profiles), 13.5.1 (AI incident-response plans), 13.5.2 (IR tooling/expertise), 13.5.3 (post-incident analysis)

This list either should be candidates to be removed or clarified that they are live processes in a training pipeline or live system.

[ottosulin]

Agreed. At this point we should err on the side of considering requirements to be too much on the governance side and thus removed, rather than keeping too many requirements.

[jmanico]

I am not going to PR this, it's quite large. I just wanted to flag these during our next review meeting.

[csfreak92]

@jmanico, I was about to ask this when I looked at the AISVS document whether we have flags too for which requirements are testable and which can done only via security architecture and security design reviews. Do we have that somewhere? Or is there a plan to clarify that here in AISVS?

[jmanico]

They should all be directly testable. Please point out and process ones we can remove!

[jmanico]

cc @razashariff I know you were asking about this, I am happy to take PR's for obvious documentation requirements....

ones with (pure paperwork):

13.5.2 — "the incident-response team has the right expertise" (you can't test a team)
13.5.1 — an incident-response plan exists
6.4.1 — incident-response playbooks exist
5.2.3 — a data-classification taxonomy is documented
11.1.4 — safety-training procedures are documented
11.2.5 — hardening procedures are documented

[razashariff]

So Jim a couple of these are real AI controls, not just paperwork - 13.5.1 is AI incident containment, 6.4.1 is model rollback / signature revocation. The weak ones I'd tighten to be testable in 1.1 rather than cut. Let's run through them on the call tmmrw

Suggestion if you want them out of the requirements then we put them in an Annex. One to discuss.