feat(agents): align RAI planner with guide, remove scoring, improve UX#1287
feat(agents): align RAI planner with guide, remove scoring, improve UX#1287WilliamBerryiii wants to merge 13 commits intomainfrom
Conversation
- restructure Phase 2 into binary trigger assessment with T1/T2/T3 tiers
- replace likelihood-impact scoring with restricted-use gate framework
- adopt AI STRIDE extensions with eight AI element types in Phase 4
- unify threat IDs to T-RAI-{NNN} format across all phases
- add rai-sensitive-uses-triggers.instructions.md for Phase 2 depth
- update collection manifests, plugins, and documentation accuracy
- add Sign-RaiArtifacts.ps1 signing script with Pester tests
Closes #1281
🚀 - Generated by Copilot
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF ScorecardScorecard details
Scanned Files
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1287 +/- ##
==========================================
- Coverage 87.63% 86.88% -0.76%
==========================================
Files 61 63 +2
Lines 9328 9575 +247
==========================================
+ Hits 8175 8319 +144
- Misses 1153 1256 +103
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
raymond-nassar
left a comment
raymond-nassar
left a comment
There was a problem hiding this comment.
First impressions on this draft PR:
The three-part disclaimer formulation exceeds the baseline requirement: (1) covers legal, compliance, and ethics; (2) frames all outputs as suggestions; (3) establishes organizational policy precedence.
Replacing numerical scores with qualitative concern levels and maturity indicators strengthens the disclaimer posture. Qualitative assessments naturally invite human judgment.
Look forward to seeing this in action.
raymond-nassar
left a comment
raymond-nassar
left a comment
There was a problem hiding this comment.
Files affected: All files containing the current disclaimer blockquote:
.github/agents/rai-planning/rai-planner.agent.md
.github/instructions/rai-planning/rai-identity.instructions.md (Session Start Display, Exit Point Reminder, State Creation)
.github/prompts/rai-planning/rai-capture.prompt.md
.github/prompts/rai-planning/rai-plan-from-prd.prompt.md
.github/prompts/rai-planning/rai-plan-from-security-plan.prompt.md
Current text:
This tool provides structured prompts and frameworks to support responsible AI planning. It is not a substitute for professional legal, compliance, or ethics review. All outputs are suggestions for human evaluation. Organizational RAI policies and applicable regulations take precedence.
Required text:
This agent is an assistive tool only. It does not provide legal, regulatory, or compliance advice and does not replace Responsible AI review boards, ethics committees, legal counsel, compliance teams, or other qualified human reviewers. The output consists of suggested actions and considerations to support a user's own internal review and decision‑making. All RAI assessments, sensitive use screenings, security models, and mitigation recommendations generated by this tool must be independently reviewed and validated by appropriate legal and compliance reviewers before use. Outputs from this tool do not constitute legal approval, compliance certification, or regulatory sign‑off.
Rationale:
The current disclaimer is too general and does not adequately protect against misinterpretation. The replacement version:
- Enumerates specific professional roles the tool does not replace (RAI review boards, ethics committees, legal counsel, compliance teams)
- Names the specific output types requiring validation (RAI assessments, sensitive use screenings, security models, mitigation recommendations)
- Explicitly excludes certification and sign-off — preventing misinterpretation as approval
- States a mandatory validation requirement ("must be independently reviewed and validated") rather than a softer suggestion
- Addresses regulatory advice directly ("does not provide legal, regulatory, or compliance advice")
- This change should be applied everywhere the current disclaimer blockquote appears, including the Session Start Display and Exit Point Reminder sections in rai-identity.instructions.md which govern when the disclaimer is shown to users.
- update disclaimer in rai-planner agent, three RAI prompt files, and rai-identity instructions - regenerate plugin outputs and reformat doc tables ⚖️ - Generated by Copilot
This has been resolved. |
|
The disclaimer text was strengthened across all existing locations in Given that the new verbatim text explicitly states "Outputs from this tool do not constitute legal approval, compliance certification, or regulatory sign‑off", the handoff document is arguably the most critical place for this disclaimer to appear, since it's the artifact most likely to be shared outside the immediate user session with review boards, legal, or leadership. Suggested change: add between the handoff header metadata and the Work Item Summary. |
Agree and I am addressing this now; but I am still trying to figure out how to do this tastefully. My main area of consideration here is that the handoff artifacts, in their "intended use" are to be persisted locally (not saying they WILL be, but as HVE Core is currently designed, the "intended use" is user specific local storage). Local artifacts, just like GHCP generating code or documentation generation are intended for human review. As currently designed the agent receivers of these handoff artifacts (the backlog agents) must be operationalized by the user explicitly, the artifacts passed to them, and they are backed by 3 operational modalities:
All three operational modalities have multi-step human interaction requirements, and there is no masked workflows happening. The human is identifying and selecting a specific agent for execution, and any delegation (only available to subagents), only has the bounded context of the original agent's human request. I think at the end of the day, the thing we really care about here from a system/agent operations perspective is to ensure that a human has reviewed the artifacts that are intended for other humans to consume. If we can agree on that, then I think the most important thing about AI generation attribution is to provide, as part of the disclaim footer, a markdown checkbox that offers, "has" or "has not" been reviewed by a human, with a default selection of "has not". This sends a much stronger signal, of attestation of review ... which is really what we are trying to set the behavioral outcome for. Consuming agents can also gate on this field ... as an added benefit. |
…system - remove human review checkbox from agentic artifacts (Control Surface Catalog, Evidence Register) in impact assessment instructions - remove superseded qualifier prose from RAI Review Summary template in backlog handoff instructions - add Artifact Attribution and Review section to handoff pipeline docs with footer classification table - add footer classification notes to Phase 5 and Phase 6 outputs in phase reference docs - add conversational vs persisted disclaimer note to agent overview docs 📝 - Generated by Copilot
Artifact Attribution and Review — Summary of ChangesThank you for the thorough review, @raymond-nassar. We implemented a two-tier attribution system across the RAI planning artifact templates and added documentation to support the new conventions. Tier 1: AI-Content NoteAll persisted artifacts now include an AI-content note adapted from the Microsoft Learn per-article pattern:
Tier 2: Full DisclaimerThe handoff summary — the primary stakeholder-facing deliverable — includes the complete verbatim disclaimer after the AI-content note and human review checkbox. Human Review CheckboxHuman-facing artifacts include a review checkbox to track validation status:
Artifact Classification
Control Surface Catalog and Evidence Register are classified as agentic artifacts (consumed by later pipeline phases) and receive only the AI-content note. All other artifacts are human-facing and include the review checkbox. Files ChangedInstruction files:
Documentation:
|
- add centralized config files for disclaimers and footers with human review - add JSON schema for config file validation - add Validate-AIArtifacts.ps1 with scope-filtered artifact classification - add Pester tests (20 passing) for all validation functions - add ai-artifact-validation.yml reusable workflow gated in pr-validation - add lint:ai-artifacts and validate:ai-artifacts npm scripts 🔧 - Generated by Copilot
AI Artifact Validation Infrastructure — Update SummaryThis latest commit ( What was added
Expansion plansThe current scope filter targets
The goal is for each focus area to opt in by adding its scope patterns to the config files — keeping validation centralized while allowing incremental adoption across the codebase. 🔧 - Generated by Copilot |
Correction to previous commentThe scope in Updated scope: Updated expansion plan: When other focus areas (e.g., Validation results after fix: 2 files / 7 issues (down from 3 files / 9 issues), all within 🔧 - Generated by Copilot |
…iling newline Security instruction files were not modified in this PR. Remove .github/instructions/security/** scope patterns from footer-with-review.yml to avoid false positive validation warnings. Fix missing trailing newline in package.json. 🔧 - Generated by Copilot
- switch Find-ArtifactReferences from content-based to filename-based matching - update Pester tests for filename-based artifact matching logic - fix BOM encoding and empty catch blocks for PSScriptAnalyzer - update Docusaurus collection card counts for rai-planning - auto-fix markdown table formatting in handoff-pipeline 🔧 - Generated by Copilot
… consistency - prettify tables in 5 rai-planning instruction files using LF-normalized input - resolves CRLF/LF formatting divergence between Windows and Linux CI 🎨 - Generated by Copilot
|
Resolved merge conflicts with origin/main (�d98b00a). Conflicted files (3):
All conflicts resolved by combining both sides: HEAD's PR-specific changes (rai-planner guide alignment) alongside main's independent updates (security reviewer renaming, secure-by-design skill). |
WilliamBerryiii
force-pushed
the
feat/rai-planner-guide-alignment
branch
from
ad98b00 to
ad1bc61
Compare
| label: "AI-Content Note (Tier 1)" | ||
| text: >- | ||
| > **Note** — The author created this content with assistance from AI. | ||
| All outputs should be reviewed and validated before use. |
There was a problem hiding this comment.
From Jackie:
Line 16: All outputs should be reviewed and validated by a qualified human reviewer before use.
There was a problem hiding this comment.
Addressed. The footer-with-review.yml centralized configuration updated at both locations:
ai-content-note(L16): "All outputs should be reviewed and validated by a qualified human reviewer before use."human-review-checkbox(L21): "Reviewed and validated by a qualified human reviewer"
Both the centralized config and all 6 inline footer locations in rai-backlog-handoff.instructions.md now use identical "qualified human reviewer" language. No discrepancy remains.
| * Pass: composite >= 70. | ||
| * Conditional: composite 50-69 (proceed with mandatory remediation work items). | ||
| * Fail: composite < 50 (return to assessment phase). | ||
| * **Ready for stakeholder review** — All dimensions addressed with supporting evidence; no critical gaps identified. |
There was a problem hiding this comment.
I would suggest removing no critical gaps identified portion. I don't think we want to say if something is critical or not?
There was a problem hiding this comment.
Addressed. Replaced the "no critical gaps identified" language at line 35 with a 3-tier neutral assessment framework. The review quality checklist uses 6 dimensions (Standards Alignment, Threat Completeness, Control Effectiveness, Evidence Quality, Tradeoff Resolution, Risk Classification) with these status tiers:
- "Ready for stakeholder review" — All dimensions addressed with supporting evidence
- "Additional attention suggested" — Most dimensions addressed; one or more areas flagged
- "Significant areas need further consideration" — Multiple dimensions have limited coverage
Added the governing instruction at line 43: "Frame all assessments as suggested observations based on session findings."
The file no longer makes criticality judgments about gaps. The agent produces observations and suggested status tiers; the qualified human reviewer determines significance.
| ### Remediation Suggested: {Yes/No} | ||
| ### Work Items Generated: {count} | ||
|
|
||
| > **Note** — The author created this content with assistance from AI. All outputs should be reviewed and validated before use. |
There was a problem hiding this comment.
reviewed and validated by a qualified human before use.
There was a problem hiding this comment.
Addressed. All 6 inline footer locations in rai-backlog-handoff.instructions.md updated with "qualified human reviewer" language:
| # | Lines | Context |
|---|---|---|
| 1 | L98–99 | RAI Review Summary markdown blockquote + checkbox |
| 2 | L182–183 | ADO HTML <p> tag template |
| 3 | L245–246 | ADO Bug work item footer |
| 4 | L369–372 | GitHub issue body footer |
| 5 | L418–419 | GitHub Bug issue template |
| 6 | L437–438 | GitHub Research issue template |
Each footer follows the two-line pattern:
> **Note** — The author created this content with assistance from AI. All outputs should be reviewed and validated before use.
> - [ ] Reviewed and validated by a qualified human reviewer
Both the centralized footer-with-review.yml config and the 6 inline footers in rai-backlog-handoff.instructions.md now use identical "qualified human reviewer" language.
| <li>{criterion_2}</li> | ||
| </ul> | ||
| <blockquote> | ||
| <p><strong>Note</strong> — The author created this content with assistance from AI. All outputs should be reviewed and validated before use.</p> |
There was a problem hiding this comment.
All outputs should be reviewed and validated by a qualified human before use
There was a problem hiding this comment.
Addressed. The ADO HTML template <blockquote> at line 182 updated with the full footer pattern:
<p><strong>Note</strong> — The author created this content with assistance from AI.
All outputs should be reviewed and validated before use.</p>
<p>☐ Reviewed and validated by a qualified human reviewer</p>This is footer location #2 of 6 in rai-backlog-handoff.instructions.md. All 6 locations use identical reviewer language. See Comment #5 for the full location table.
| * User Story: Specific control or mitigation. | ||
| * Task: Implementation steps for a user story. | ||
| * Bug: Existing RAI violations requiring remediation. | ||
| * Bug: Existing RAI areas of concern requiring attention. |
There was a problem hiding this comment.
Suggested RAI area requiring attention.
There was a problem hiding this comment.
Addressed. Updated at line 196 in rai-backlog-handoff.instructions.md:
- Before:
* Bug: Existing RAI areas of concern requiring attention. - After:
* Bug: Suggested RAI area requiring attention.
Removes the "Existing" qualifier (implies pre-existing known issues rather than newly identified observations) and replaces "areas of concern" with neutral "requiring attention" phrasing. This was identified as a gap during deep review and amended into commit 15f9d1dc.
| * Impact: High, Medium, Low, or ❓ (unassessed). | ||
| * Risk: Calculated using the standard risk matrix (High x High = Critical, High x Medium or Medium x High = High, Medium x Medium = Medium, Low x any or any x Low = Low). | ||
| * Suggested Threat Origin: Where the threat originates (Data Pipeline, Model, Interface, Infrastructure, or Cross-cutting). | ||
| * Concern Level: Qualitative assessment of threat significance (Low Concern, Moderate Concern, or High Concern). |
There was a problem hiding this comment.
Do we say how this is calculated?
There was a problem hiding this comment.
It's over in the STRIDE model for security, I'll update to add a comment to point to it.
There was a problem hiding this comment.
Addressed per your earlier thread reply. Changes to rai-security-model.instructions.md:
-
Cross-reference added: Inserted "See Concern Level Assessment below for criteria." into the Concern Level column definition in the ML STRIDE output table. This links the column header to the assessment guidance section.
-
Concern Level Assessment defines three qualitative tiers (not numerical risk ratings):
Level Criteria Low Concern "Threat is theoretical or mitigated by existing controls; no immediate action suggested." Moderate Concern "Threat is plausible and partially mitigated; additional controls recommended." High Concern "Threat is likely or unmitigated; priority mitigation suggested." Includes the caveat: "The concern level is a suggested assessment for the team's consideration, not a definitive risk rating."
-
ML STRIDE matrix columns updated: 8×6 matrix (8 components × 6 STRIDE categories) now uses 7 NIST trustworthiness characteristic columns instead of the legacy 6 Microsoft principles. Each cell annotates
Applicability / NIST Characteristicfor traceability.
The cross-reference note was identified as a gap during deep review and amended into commit 15f9d1dc.
| applyTo: '**/.copilot-tracking/rai-plans/**' | ||
| --- | ||
|
|
||
| # Sensitive Uses Trigger Screening |
There was a problem hiding this comment.
Should this be included? This is internal to Microsoft. will this be internal only?
There was a problem hiding this comment.
Maybe you take the same approach as restricted use and allow a user to specify the framework to use to determine sensitive use.
There was a problem hiding this comment.
This is the 3 buckets, that are annotated in the public MSFT Impact Assessment and Guide documents. It's unaffiliated with any MSFT specific (internal) processes. This instruction set also allows for the importing of sensitive use triggers from other pubic specifications like those from Singapore, UK/AUS, and EU governing standards bodies.
There was a problem hiding this comment.
Addressed comprehensively. The entire file has been redesigned:
-
Renamed from
rai-sensitive-uses-triggers.instructions.md→rai-risk-classification.instructions.md(commite221ae0f) -
Replaced legacy binary T1/T2/T3 triggers with 3 NIST AI RMF 1.0-derived risk indicators:
Indicator NIST Source Method Domain safety_reliabilityMS-2.5, MS-2.6 Binary (present/absent) Physical harm, psychological injury, operational disruption rights_fairness_privacyMS-2.8, MS-2.10, MS-2.11 Categorical (None/Indirect/Direct/Primary) Discrimination, rights restriction, equitable access, personal data misuse, accountability gaps security_explainabilityMS-2.7, MS-2.9 Continuous (0.0–1.0) Adversarial attacks, data poisoning, model theft, inability to explain consequential decisions MS-2.8 (Accountable and Transparent) groups with fairness/privacy because "accountability gaps directly affect rights recourse mechanisms."
-
Added framework isolation: Custom framework documents can fully replace the default NIST indicators via the
replaceDefaultIndicators: trueflag inriskClassificationstate. Risk Indicator Extensions are always additive: they never replace defaults even whenreplaceDefaultIndicatorsis set. -
Added Prohibited Uses Gate: Safety-critical check before risk indicator assessment, modeled on EU AI Act Article 5 prohibited practices. Renamed from "Restricted Uses Gate" across all files.
-
Added code-of-conduct reference type: Agent prompts users for AI service provider codes of conduct during Phase 1 and cross-references them during risk indicator evaluation.
No platform specific trigger categories remain. Default indicators use universal NIST-aligned language; organizations can supply their own internal policies as custom framework documents without exposing them as hardcoded defaults.
|
|
||
| Custom categories do not replace the fixed T1/T2/T3 triggers. They extend the evaluation with additional criteria the user considers relevant to their organizational context. | ||
|
|
||
| ## Sensitive Use Trigger Screening |
There was a problem hiding this comment.
See above for comment on applying same logic for restricted use here. That way we can use our internal policies for SU without exposing to external users.
There was a problem hiding this comment.
Oh yes ... I actually have some cool stuff for customers to be able to override these with their own specific RAI systems. Let' me make that more clear in the documents how it works.
There was a problem hiding this comment.
Addressed via the complete Phase 2 redesign:
-
Default risk indicators now use universal NIST-aligned language mapped to NIST AI RMF subcategories (MS-2.5 through MS-2.11), not proprietary categories
-
Custom framework override via
replaceDefaultIndicators: truein theriskClassificationstate object enables organizations to load their own internal policies as custom framework documents without exposing them as hardcoded defaults. Risk Indicator Extensions are always additive on top of whichever base set is active — they never replace defaults even whenreplaceDefaultIndicatorsis set -
The cross-framework terminology mapping informed naming choices throughout:
Term Used Source Standard "Risk Classification" EU AI Act, ISO 42001 "risk indicators" NIST AI RMF Map-1.1 "risk characteristics" "prohibited uses" EU AI Act Article 5 "code-of-conduct" reference AI service provider compliance documents -
No platform-specific categories remain in the codebase. Organizations supply their own policies through the
referencesProcessedmechanism and the agent cross-references them during risk indicator evaluation
|
|
||
| Evaluate the AI system against three binary triggers derived from the Microsoft Responsible AI Impact Assessment Template §3.6. Each trigger is binary (triggered or not triggered). There is no partial activation or severity grading within triggers. | ||
|
|
||
| ### Trigger Table |
There was a problem hiding this comment.
Ack. All of this is from our internal policies correct? The EU act has it's own:
Biometrics
Critical infrastructure
Education access or evaluation
Employment decisions
Access to essential services
Law enforcement
Migration / border control
Justice systems
and OECD AI system classification framework has dimensions like:
Context of use
Potential societal impact
Data sensitivity
Level of autonomy
Transparency / explainability
Here's how they map:
Microsoft RAI EU AI Act OECD NIST
Sensitive Use High‑Risk AI High‑Impact Context MAP‑identified risk
I don't think we should specify our SU's in this if it will be used outside of Microsoft.
There was a problem hiding this comment.
This is ONLY from the public documents we provide through AKA.ms/RAI and the backing unauthenticated CDN hosting of the content. Also much of what you are seeing, language wise, is taken from the MSFT AI Code of Conduct and all the associated documents that hang off it from service agreements. That said, I'm going to fully remove all the content that references MSFT's content in this space and just refer to the public/government aligned standards and enhance the ability to customize.
|
|
||
| Phase 2 screens AI systems for sensitive use triggers using three binary assessments derived from the Microsoft Responsible AI Impact Assessment Template §3.6. For copyright and intellectual property notices applicable to the Microsoft Responsible AI Impact Assessment Guide, see the Attribution section in `rai-standards.instructions.md`. The restricted uses gate executes first as a safety-critical binary check. Trigger results determine the suggested assessment depth tier for subsequent phases. | ||
|
|
||
| ## Restricted Uses Gate |
There was a problem hiding this comment.
"Restricted" is a very Microsoft coined term. Could go with Prohibited to make this less Microsoft focused if the goal is to be used outside of Microsoft.
There was a problem hiding this comment.
Addressed. Renamed all occurrences across 3 instruction files:
restrictedUsesGate→prohibitedUsesGate(state field inrai-identity.instructions.md)restricted-use-framework→prohibited-use-framework(reference type)- "Restricted Uses Gate" → "Prohibited Uses Gate" (all prose references)
- Gate protocol renamed:
gateResults.restrictedUsesGate→gateResults.prohibitedUsesGatein state.json schema
The "Prohibited" terminology aligns with EU AI Act Article 5 "prohibited practices," which is the international standard term. ~45 occurrences updated across:
| File | Approximate Occurrences |
|---|---|
rai-risk-classification.instructions.md |
~20 (complete rewrite of file) |
rai-identity.instructions.md |
~15 (state schema, phase definitions, gate protocol) |
rai-impact-assessment.instructions.md |
~10 (cross-references, evidence register) |
There was a problem hiding this comment.
I've left several comments on the use of Microsoft terms and language around RAI like "sensitive use" and "restricted use" etc. There are other terms being used in the industry that might land a bit better and feel less like Microsoft pushing our way etc. Also I would remove the Sensitive Use as that is proprietary to Microsoft. I've added some examples from other frameworks. Perhaps the same approach that is taken for "restricted" could be applied for sensitive use.
There was a problem hiding this comment.
Addressed. This comment is the summary motivation for the framework-agnostic redesign:
- "Sensitive uses" → "Risk Classification" everywhere (universal term per EU AI Act, ISO 42001)
- "Restricted" → "Prohibited" everywhere (aligns with EU AI Act Article 5 "prohibited practices")
- All legacy binary triggers replaced with 3 NIST AI RMF 1.0-derived risk indicators (binary, categorical, and continuous assessment methods)
- Control surface taxonomy updated from 6×3 (6 legacy principles × Prevent/Detect/Respond) to 7×3 (7 NIST trustworthiness characteristics × Prevent/Detect/Respond)
- Evidence register abbreviations updated to NIST-aligned keys using the format
EV-{CHARACTERISTIC_ABBR}-{NNN}where abbreviations are: VR (Valid and Reliable), SAFE, SR (Secure and Resilient), AT (Accountable and Transparent), EI (Explainable and Interpretable), PRIV (Privacy-Enhanced), FAIR (Fair with Harmful Bias Managed) - RAI tags updated from
rai:{legacy}torai:{nist-key}format (e.g.,rai:valid-reliable,rai:secure-resilient) in both ADO and GitHub templates
| | Control Effectiveness | Coverage and effectiveness of controls for identified threats | | ||
| | Evidence Quality | Quality and availability of evidence supporting control effectiveness | | ||
| | Tradeoff Resolution | Clarity of principle tradeoff analysis and resolution rationale | | ||
| | Sensitive Uses | Whether sensitive use triggers were addressed with appropriate depth | |
There was a problem hiding this comment.
Eeek. This one worries me a bit. What does "appropriate depth" mean? and with SU we typically speak in terms of mitigation vs. addressed
There was a problem hiding this comment.
Addressed. Replaced the ambiguous "appropriate depth" phrasing with mitigation-focused language in both documentation files:
handoff-pipeline.md: Review dimension updated from "addressed with appropriate depth" to "evaluated with documented mitigations"phase-reference.mdL283: Same replacement — "Whether risk indicators were evaluated with documented mitigations"
The mitigation-focused language aligns with NIST's risk management approach: Phase 2 risk indicators should be evaluated (assessed against evidence) with documented mitigations (specific controls recorded), not vaguely "addressed." This was identified as a gap during deep review and amended into commit 15f9d1dc.
| ## Phase 2: RAI Standards Mapping | ||
| ## Phase 2: Sensitive Uses Assessment | ||
|
|
||
| > NIST AI RMF alignment: Govern |
There was a problem hiding this comment.
Is this NIST or is it Microsoft SU? i'm a bit confused.
There was a problem hiding this comment.
Addressed. The Phase 2 section in phase-reference.md has been fully rewritten to eliminate the NIST vs. Microsoft confusion:
- Section title: "Sensitive Uses Assessment" → "Risk Classification"
- Content: Removed all platform specific language. Now references NIST AI RMF 1.0 Map function contextual assessment with 3 risk indicators (binary, categorical, continuous methods)
- State fields: Updated all references from legacy to NIST-aligned naming:
sensitiveUsesTriggers→riskClassificationrestrictedUsesGate→prohibitedUsesGateprincipleTracker→characteristicTrackerwith 7 NIST keys:validReliable,safe,secureResilient,accountableTransparent,explainableInterpretable,privacyEnhanced,fairBiasManaged
- NIST subcategory annotations: Each characteristic now includes traceable references (MS-2.5 through MS-2.11) for audit purposes
- Review dimension: "addressed with appropriate depth" → "evaluated with documented mitigations"
|
|
||
| ### Purpose | ||
|
|
||
| Screen the AI system for sensitive use triggers to determine the suggested assessment depth tier for subsequent phases. This phase uses three binary assessments derived from the Microsoft Responsible AI Impact Assessment Template §3.6. |
There was a problem hiding this comment.
See previous comments about SU's :)
There was a problem hiding this comment.
Addressed — covered by the same Phase 2 redesign in Comment #20. All "sensitive uses" references in phase-reference.md replaced with "Risk Classification" terminology, "Prohibited Uses Gate" naming, and NIST AI RMF 1.0 subcategory annotations. The companion handoff-pipeline.md received the same terminology updates including the review dimension language change ("evaluated with documented mitigations").
…k-classification - rename instruction file to reflect NIST AI RMF risk classification alignment - update collection manifests with new file path - regenerate plugin symlinks for renamed file 🔄 - Generated by Copilot
…ess characteristics - replace 6 Microsoft RAI principles with 7 NIST trustworthiness characteristics - rewrite standards, identity, impact assessment, security model, and backlog handoff - update agent definition, prompts, and documentation for NIST alignment - migrate state schema to riskClassification - regenerate plugin outputs 🏛️ - Generated by Copilot
Re: Disclaimer text update (5 files)Addressed. All 5 files updated with the expanded disclaimer text. The updated disclaimer replaces the generic 1-line AI notice with a multi-sentence version specifying professional review requirements and output types (risk assessments, security models, planning artifacts). One modification from the suggested text: "sensitive use screenings" updated to "risk classification screenings" to align with the terminology redesign across this PR.
|
|
Thank you for the review! All feedback incorporated — see inline replies on each comment thread for specific before/after details and file references. |
NIST AI RMF 1.0 Sole Embedded Standard — Complete AlignmentThis PR migrates the RAI Planner from Microsoft's 6 proprietary RAI principles to NIST AI RMF 1.0's 7 trustworthiness characteristics as the sole embedded standard. All proprietary framework content has been removed. 29 files changed | 953 insertions | 744 deletions | 2 commits Architecture Changes
NIST 7 Trustworthiness Characteristics
Terminology Changes
Files Changed
Breaking Changes
Validation
Comment Resolution Checklist
|
- add author: Microsoft to 6 RAI planning doc pages for CI frontmatter validation - format tables in 4 RAI planning instruction files - format tables in 4 RAI planning doc files - fix symlink index for 2 plugin instruction files 🔧 - Generated by Copilot
4 participants
Description
Implements Issue #1281: RAI Planner Updates — Guide Alignment, Scoring Removal, and UX Improvements.
The RAI Planner agent expands from a 5-phase to a 6-phase workflow, replacing numeric scoring with qualitative assessment, aligning terminology and structure with the Microsoft Responsible AI Impact Assessment Guide, and introducing artifact signing infrastructure. Additionally, this PR establishes a config-driven AI artifact validation pipeline for enforcing footer and disclaimer standards across instruction files. 46 files changed across agent definitions, instructions, prompts, documentation, plugins, collection metadata, config, CI workflows, and scripts.
Key Changes
Basic/Standard/Comprehensive). Subsequent phases renumbered.Low/Moderate/High), a review quality checklist, maturity indicators, and audience adaptation profiles. Renamedrai-scorecard.md→rai-review-summary.md.T-RAI-{NNN}.Sign-RaiArtifacts.ps1for SHA-256 manifest generation with optional Sigstore cosign keyless signing, backed by 262-line Pester test suite. Installed cosign v3.0.5 in devcontainer..github/config/disclaimers.yml,.github/config/footer-with-review.yml) defining footer text, human review checkboxes, and tiered artifact classification with scope-aware glob patterns. CreatedValidate-AIArtifacts.ps1(612 lines) for CI enforcement, a JSON Schema for config validation, a reusable GitHub Actions workflow (ai-artifact-validation.yml), and a 20-test Pester suite. Wired intopr-validation.ymland thelint:allnpm chain.Related Issue(s)
Closes #1281
Type of Change
Select all that apply:
Code & Documentation:
Infrastructure & Configuration:
AI Artifacts:
prompt-builderagent and addressed all feedback.github/instructions/*.instructions.md).github/prompts/*.prompt.md).github/agents/*.agent.md).github/skills/*/SKILL.md)> Note for AI Artifact Contributors:
>
> * Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review
.github/agents/before creating new ones.> * Skills: Must include both bash and PowerShell scripts. See Skills.
> * Model Versions: Only contributions targeting the latest Anthropic and OpenAI models will be accepted. Older model versions (e.g., GPT-3.5, Claude 3) will be rejected.
> * See Agents Not Accepted and Model Version Requirements.
Other:
.ps1,.sh,.py)Sample Prompts (for AI Artifact Contributions)
User Request:
Invoke
RAI Plannerin the VS Code chat pane and use one of three entry prompts:/rai-capture— Start a new conversational RAI assessment from scratch/rai-plan-from-prd— Generate an RAI plan from an existing PRD/rai-plan-from-security-plan— Generate an RAI plan from an existing security planExecution Flow:
T-RAI-{NNN}threats.rai-review-summary.md, dual-format backlog (ADO + GitHub), and optional signed artifact manifest.Output Artifacts:
.copilot-tracking/rai-plans/{session}/state.json— Session state with phase progression.copilot-tracking/rai-plans/{session}/rai-review-summary.md— Qualitative review summary (replaces scored scorecard).copilot-tracking/rai-plans/{session}/rai-backlog-*.md— Dual-format work item backlog.copilot-tracking/rai-plans/{session}/rai-manifest.json— SHA-256 artifact manifest (optional signing)Success Indicators:
rai-review-summary.mdinstead ofrai-scorecard.mdT-RAI-{NNN}format consistentlyTesting
npm run plugin:generate) — 14 pluginsnpm run lint:ai-artifacts> Note: Manual testing was performed along side automated validation and sandbox evaluation as the primary verification methods.
GHCP Artifact Maturity
> [!WARNING]
> This PR includes experimental GHCP artifacts that may have breaking changes.
.github/agents/rai-planning/rai-planner.agent.md.github/prompts/rai-planning/rai-capture.prompt.md.github/prompts/rai-planning/rai-plan-from-prd.prompt.md.github/prompts/rai-planning/rai-plan-from-security-plan.prompt.md.github/instructions/rai-planning/rai-identity.instructions.md.github/instructions/rai-planning/rai-standards.instructions.md.github/instructions/rai-planning/rai-security-model.instructions.md.github/instructions/rai-planning/rai-impact-assessment.instructions.md.github/instructions/rai-planning/rai-backlog-handoff.instructions.md.github/instructions/rai-planning/rai-capture-coaching.instructions.md.github/instructions/rai-planning/rai-sensitive-uses-triggers.instructions.mdGHCP Maturity Acknowledgment
Checklist
Required Checks
AI Artifact Contributions
/prompt-analyzeto review contributionprompt-builderreviewRequired Automated Checks
The following validation commands must pass before merging:
npm run lint:mdnpm run spell-checknpm run lint:frontmatternpm run validate:skillsnpm run lint:md-linksnpm run lint:psnpm run lint:ai-artifacts> Repository template used:
.github/PULL_REQUEST_TEMPLATE.md