Skip to content

Add orca-custom-framework skill - #9

Merged
dror-orca merged 4 commits into
mainfrom
feat/orca-custom-framework
Jun 4, 2026
Merged

dror-orca merged 4 commits into
mainfrom
feat/orca-custom-framework

Conversation

@dror-orca

Copy link
Copy Markdown
Contributor

Summary

  • New skill: orca-custom-framework — creates custom compliance frameworks in Orca Security from existing controls, alert lists, or security themes
  • Uses Orca MCP tools (read-only) to gather controls, then the REST API (POST /api/compliance/frameworks) to create frameworks
  • Suggests custom discovery alerts (POST /api/sonar/rules) when coverage gaps are found
  • Tested live: created "Supply Chain Security Controls" (39 controls, 6 sections, 28% initial score)

Files added/modified

File Description
skills/orca-custom-framework/SKILL.md Full skill definition with 3-phase flow, API reference, Supply Chain example
test-data/sample-orca-custom-framework.json Fabricated test data (12 controls, 5 sections, 5 coverage gaps)
promptfooconfig.yaml 3 eval cases: control counting, risk section ranking, gap detection
README.md Added skill to overview table, workflows, and details section
test-data/README.md Documented new test data file

How the skill works

  1. Read (MCP tools) — queries get_enabled_compliance_frameworks, get_compliance_framework_control_tests, discovery_search to gather rule_id values
  2. Organize (Claude logic) — groups rules into themed sections with priority weights
  3. Create (REST API) — POST /api/compliance/frameworks with sections and tests
  4. Gap detection — identifies missing domains and suggests custom discovery alerts via POST /api/sonar/rules

Test plan

  • Run promptfoo eval — 3 new test cases should pass
  • Verify skill triggers on natural language ("create a custom framework for...")
  • Live test: create a framework with a new theme against a real Orca environment
  • Verify framework appears in Orca Console under Compliance
  • Test coverage gap detection and custom alert suggestion flow

dror added 2 commits May 14, 2026 16:10
The README links to CONTRIBUTING.md but the file was missing.
Adds the document covering bug reporting, feature requests,
development setup, PR process, and style guide.
…orks

New skill that creates custom compliance frameworks in Orca Security from
existing framework controls, alert lists, or security themes. Uses MCP tools
for reading controls and the Orca REST API for framework creation.

- SKILL.md with full processing logic, API reference, and Supply Chain example
- Test data and 3 promptfoo eval cases (control counting, risk ranking, gap detection)
- Custom discovery alert suggestion for coverage gaps (POST /api/sonar/rules)
- README updated with skill entry, workflow, and detailed documentation

@orca-security-eu orca-security-eu Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

dror added 2 commits May 26, 2026 10:00
…e only

The skill creates whatever framework the user asks for. Supply Chain
references now only appear in the Example section and usage samples,
not in the processing logic steps.
Resolves conflicts with #11 (promptfoo test suite removal):
- Accept deletion of promptfooconfig.yaml
- Accept deletion of test-data/ directory (including new sample-orca-custom-framework.json)
@dror-orca
dror-orca merged commit 5dfcfa1 into main Jun 4, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants