Name	Name	Last commit message	Last commit date
parent directory ..
archive	archive
DESIGN.md	DESIGN.md
README.md	README.md
REVIEW_RESPONSE.md	REVIEW_RESPONSE.md

Multilingual Batch Scanner for SkillSpector

SkillSpector is a static+LLM security analyzer for AI agent skill definitions. This module extends it to scan directories of skills in parallel, with automatic language detection and targeted LLM gap-fill for non-English skills. Zero changes to upstream src/skillspector/.

Contents: What it does · Quickstart · All Commands · Running Tests · For PR Reviewers

What it does

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 7

Finds all SKILL.md-containing directories under the input root
Detects language per skill (en / zh / ja / ko)
Runs the full SkillSpector graph pipeline per skill in parallel
For non-English skills, applies LLM gap-fill for 8 vulnerability rules that English-keyword static patterns cannot detect
Produces an aggregated report sorted by risk score

Quickstart

Prerequisites

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install SkillSpector in development mode
pip install -e .

# Copy and edit the environment template
cp contrib/batch_scan/.env.example .env

The .env file needs these keys (see .env.example for the full template):

Variable	Required	Purpose
`SKILLSPECTOR_PROVIDER`	Yes	`openai` for DeepSeek/OpenAI-compatible
`SKILLSPECTOR_MODEL`	Yes	e.g. `deepseek-v4-flash`
`OPENAI_API_KEY`	For single-key	Standard OpenAI-compatible key
`OPENAI_BASE_URL`	For single-key	e.g. `https://api.deepseek.com/v1`
`SKILLSPECTOR_API_KEYS`	For multi-key	Pipe-delimited: `key\|base_url\|model`, one per line

⚠️ Parallel LLM scanning requires multiple API keys. With --workers 4 and 1 key, you hit rate limits immediately. Configure at least as many keys as workers — 10 keys for --workers 8 is safe. The ApiKeyPool handles automatic failover when a key is rate-limited. If you only have 1 key, use --workers 1 or --no-llm.

Static-only (fast, no API keys needed)

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --no-llm

Full LLM scan

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 7

Test with built-in fixtures

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8

23 skills designed to exercise every detection rule.

Output formats

Format	Flag	Use case
Terminal (Rich)	`-f terminal` (default)	Human review
JSON	`-f json -o report.json`	CI pipelines
Markdown	`-f markdown -o report.md`	PR comments

Example: terminal output (23 fixtures, 8 workers)

SkillSpector Batch Scan — 23 skill(s) in ./tests/fixtures  (8 workers, 10 API keys)

  [1/23] malicious_skill → 100/100 CRITICAL (14 issue(s))
  [8/23] sdi/sdi1_mismatch → 97/100 CRITICAL (6 issue(s))
  [11/23] sdi/sdi4_divergence → 100/100 CRITICAL (8 issue(s))
  [19/23] ssd/ssd1_semantic_injection → 100/100 CRITICAL (4 issue(s))
  [5/23] mcp_poisoned_tool → 100/100 CRITICAL (16 issue(s))

╭──────────────────────────────────────────────────────────────────╮
│ SkillSpector Batch Scan Report                                   │
╰────────────────── v2.2.3  |  Multilingual Enhanced ──────────────╯

Total: 23 skill(s) scanned

                Skills by Risk Score (23 completed)
┏━━━━━━━━━━━━━━━━━━━━┳━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Skill              ┃ LR ┃   Score ┃ Severity ┃ Issues ┃ Lang ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━╇━━━━━━┩
│ chef-assistant     │ ✓  │ 100/100 │ CRITICAL │     14 │ en   │
│ reаd_data          │ ✓  │ 100/100 │ CRITICAL │     16 │ en   │
│ ...                │    │         │          │        │      │
│ safe-greeting      │ ✓  │   0/100 │ LOW      │      0 │ en   │
│ code-reviewer      │ ✓  │   0/100 │ LOW      │      0 │ en   │
└────────────────────┴────┴─────────┴──────────┴────────┴──────┘

15 skill(s) with HIGH or CRITICAL risk — review immediately
6 skill(s) with LOW risk — likely safe

LR column: Language Reliability. ✓ = English (full static + LLM coverage). ⚠ = non-English (gap-fill applied, 8 extra rules covered).

Example: JSON output (excerpt)

{
  "batch": {
    "scanned_at": "2026-06-19T01:20:00+00:00",
    "total_skills": 23,
    "scan_mode": "multilingual-enhanced",
    "enhancements": {
      "language_detection": "unicode-script-ratio",
      "gap_fill_applied": 0,
      "gap_fill_findings": 0
    }
  },
  "skills": [
    {
      "skill": {
        "name": "malicious_skill",
        "source": "malicious_skill",
        "source_group": ".",
        "language": "en",
        "scanned_at": "2026-06-19T01:20:05+00:00"
      },
      "risk_assessment": {
        "score": 100,
        "severity": "CRITICAL",
        "recommendation": "DO NOT INSTALL"
      },
      "issues": [
        {
          "id": "E1",
          "message": "Skill executes shell commands without user consent",
          "severity": "CRITICAL",
          "confidence": 1.0,
          "language_compatible": true
        }
      ],
      "scan_mode": "multilingual-enhanced",
      "enhancements": {
        "gap_fill_applied": false,
        "gap_fill_findings": 0,
        "english_keyword_rules_skipped": 0
      }
    }
  ]
}

LLM vs static comparison (same 23 fixtures, 8 workers)

Skill	`--no-llm`	LLM mode	What LLM caught
`ssd1_semantic_injection`	0/100 (0)	100/100 (4)	Semantic injection invisible to static
`ssd2_novel_phrasing`	0/100 (0)	100/100 (3)	Novel phrasing bypasses keyword match
`ssd3_nl_exfiltration`	0/100 (0)	60/100 (3)	NL-veiled data exfiltration
`ssd4_narrative_deception`	10/100 (1)	100/100 (9)	Deceptive narrative framing
`sdi4_divergence`	13/100 (2)	100/100 (8)	Intent-behavior mismatch
`sdi1_mismatch`	52/100 (4)	97/100 (6)	+2 additional LLM findings
`sdi3_scope_creep`	71/100 (3)	100/100 (9)	Hidden scope expansion
`sqp2_missing_warnings`	26/100 (2)	58/100 (3)	Missing safety guardrails
`malicious_skill`	100/100 (6)	100/100 (14)	+8 additional LLM findings
`mcp_poisoned_tool`	100/100 (8)	100/100 (16)	+8 additional LLM findings
`safe_skill`	0/100 (0)	0/100 (0)	Clean stays clean ✓
`ssd_clean`	0/100 (0)	0/100 (0)	Clean stays clean ✓

Key insight: LLM semantic analyzers (SSD/SDI/SQP) catch entire vulnerability categories that English-keyword static patterns miss completely. Clean skills remain clean — no false-positive inflation. For skills already flagged by static rules, LLM finds 2–8 additional issues per skill.

Quick comparison: upstream vs batch

# Upstream — scan one skill
skillspector scan ./tests/fixtures/malicious_skill/ -f json -o upstream.json

# Batch — scan all skills
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o batch.json

Key differences in batch output:

scan_mode: "multilingual-enhanced" — provenance marker
enhancements.gap_fill_applied — true if LLM gap-fill was used
enhancements.english_keyword_rules_skipped — count of static rules bypassed
skill.language — detected language tag

All Commands

Scan (LLM mode)

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 7    # default
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 1    # sequential, easy to read
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 20   # high throughput

Scan (static-only, no API keys)

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --no-llm
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --no-require-llm --no-llm  # skip LLM even for non-English

Output formats

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal                # default (Rich)
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o report.json
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f markdown -o report.md

Fixture test (built-in 23 skills)

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8 --no-llm
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o report.json --workers 8

Language override

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang auto --workers 4    # detect (default)
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang zh -f terminal --workers 4
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang ja -f terminal --workers 4
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang ko -f terminal --workers 4
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang en -f terminal --workers 4   # skip gap-fill

Debugging

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --workers 1 -V             # single worker + verbose
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --workers 4 -V
skillspector scan ./tests/fixtures/malicious_skill/ --no-llm                   # verify upstream works

Compare upstream vs batch

skillspector scan ./tests/fixtures/malicious_skill/ -f json -o upstream.json
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o batch.json --workers 4

CI

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o report.json --workers 8
if [ $? -eq 0 ]; then echo "All clean"; fi

Tuning `--workers`

Scenario	Workers	Peak concurrent LLM requests
Free-tier API key	1	10–15
Paid basic	4 (default)	25–40
Enterprise / multi-key	7–10	50–80
Debugging	1 + `-V`	Sequential, easy to read

Language options

--lang auto    # Unicode script-ratio detection (default)
--lang zh      # Force Chinese
--lang ja      # Force Japanese
--lang ko      # Force Korean
--lang en      # Force English (skip gap-fill)

Debugging

# Single worker + verbose output — easiest to read
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --workers 1 -V

# Verify upstream still works
skillspector scan ./tests/fixtures/malicious_skill/ --no-llm

Edge cases

# Static-only + skip LLM requirement even for non-English skills
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --no-require-llm --no-llm

Exit codes

Code	Meaning
0	All safe (no HIGH/CRITICAL)
1	≥1 skill has HIGH or CRITICAL risk
2	Scan errors occurred

CI usage:

python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o report.json
if [ $? -eq 0 ]; then
    echo "All clean"
fi

Troubleshooting

Symptom	Fix
"No LLM API key configured"	Set up `.env` or use `--no-llm`
Connection errors / 429	Reduce `--workers`
Skills timing out (90s)	Check network; the scanner skips and continues
"Event loop is closed"	Harmless, suppressed
model_info token limit warning	Harmless, 128K default used

Known Limitations

No checkpoint/resume. A failure at skill 847 of 1000 loses all progress.
Language detection covers 4 scripts. Arabic, Hindi, Cyrillic are classified as English and lose gap-fill coverage.
No SARIF output. Upstream supports it; this contrib adds terminal/JSON/Markdown.
Gap-fill quality not benchmarked for non-English. No ground-truth comparison exists.
parse_response JSON recovery is best-effort. When the LLM returns malformed JSON, the analyzer returns empty findings (no crash). This is a graceful-degradation choice: a single malformed response won't block the pipeline, but the user won't know which findings were lost.

See DESIGN.md for architecture details and docs/archive/FUTURE_WORK.md for suggested directions.

Running Tests

# === All 164 tests ===

# Unit tests — random order (seed=42, 120 tests)
python contrib/batch_scan/tests/tests-pro/random_numbered.py

# Pool wiring smoke test (4 checks)
python contrib/batch_scan/tests/test_pool_wiring.py

# Monkey-patch invasiveness (14 tests)
python contrib/batch_scan/tests/test_monkeypatch_invasiveness.py

# Monkey-patch fragility (26 tests)
python contrib/batch_scan/tests/test_monkeypatch_fragility.py

# === Convenience ===

# All review-themed tests in one command
python -m unittest \
  contrib.batch_scan.tests.test_monkeypatch_invasiveness \
  contrib.batch_scan.tests.test_monkeypatch_fragility -v
python contrib/batch_scan/tests/test_pool_wiring.py

# Mutation test — 30 injected bugs across 4 risk areas
python contrib/batch_scan/tests/tests-pro/mutation_max.py

# Sequential pytest (if pytest installed)
pytest contrib/batch_scan/tests/tests-pro/ -v

For PR Reviewers

Since last review: pool is now fully wired (dual-patch closes from-import bypass), 44 new thematic tests answer Issues #1–#2 directly, and all 164 tests pass against upstream NVIDIA/SkillSpector@ab0431f (130+ commits, zero patch conflicts).

What changed in production code (1 file)

runner.py#L70-L91 — set_api_pool() now patches both llm_utils.get_chat_model and llm_analyzer_base.get_chat_model. Previously only the former was patched; llm_analyzer_base's from ... import created a local reference that bypassed the pool entirely. Graph analyzers (95% of LLM calls) now go through PooledChatModel. set_api_pool(None) restores both modules.

How each review concern was addressed

Issue	Answer	Proof
#1 — Pool dead code	`set_api_pool()` dual-patch	`test_pool_wiring.py`: 3 paths verified → PooledChatModel
#2 — Patches invasive	Context manager + explicit `setup_deepseek_compat()`	`test_monkeypatch_invasiveness.py`: 14 tests — import isolation, thread isolation, 50-instance concurrency
#2 — Patches fragile	`_verify_patch_targets()` guard before apply	`test_monkeypatch_fragility.py`: 26 tests — each of 7 patches individually verified, deep deps checked, atomicity proven
#3 — Risky code untested	120 unit tests across 4 risk areas	`tests/tests-pro/` — pool (45), gap-fill (41), patches (24), annotation (10)

Full response with before/after tables: REVIEW_RESPONSE.md

Test suite at a glance (164 total)

tests/
├── test_pool_wiring.py               ← Issue #1: 4 smoke checks
├── test_monkeypatch_invasiveness.py   ← Issue #2: 14 tests (thread isolation)
├── test_monkeypatch_fragility.py     ← Issue #2: 26 tests (guard verification)
├── tests-pro/
│   ├── test_api_pool.py              ← Issue #3: 45 tests (acquire/backoff)
│   ├── test_gap_fill.py              ← Issue #3: 41 tests (JSON parsing)
│   ├── test_runner_patches.py        ← Issue #3: 24 tests (context manager)
│   └── test_annotation.py            ← Issue #3: 10 tests (language compat)
└── docs/
    ├── TEST_DESIGN.md                ← WHY each suite was designed
    ├── TEST_GUIDE.md                 ← WHAT each file covers (run commands)
    └── BUGS_FOUND.md                 ← 16 bugs found, 3 test bugs fixed

Design context

DESIGN.md — architecture, concurrency model, dual-patch mechanism
archive/PITFALLS.md — thread safety, from-import pitfall, DeepSeek constraints
archive/FUTURE_WORK.md — future direction + code conventions

Next: DESIGN.md — architecture & concurrency model · REVIEW_RESPONSE.md — PR #100 review response · CONTRIBUTING.md — dev setup & code conventions

Navigation Menu

Uh oh!

README.md

Multilingual Batch Scanner for SkillSpector

What it does

Quickstart

Prerequisites

Static-only (fast, no API keys needed)

Full LLM scan

Test with built-in fixtures

Output formats

Example: terminal output (23 fixtures, 8 workers)

Example: JSON output (excerpt)

LLM vs static comparison (same 23 fixtures, 8 workers)

Quick comparison: upstream vs batch

All Commands

Scan (LLM mode)

Scan (static-only, no API keys)

Output formats

Fixture test (built-in 23 skills)

Language override

Debugging

Compare upstream vs batch

CI

Tuning `--workers`

Language options

Debugging

Edge cases

Exit codes

Troubleshooting

Known Limitations

Running Tests

For PR Reviewers

What changed in production code (1 file)

How each review concern was addressed

Test suite at a glance (164 total)

Design context

Uh oh!

FilesExpand file tree

docs

Directory actions

More options

Directory actions

More options

Latest commit

History

docs

Folders and files

parent directory

README.md

Multilingual Batch Scanner for SkillSpector

What it does

Quickstart

Prerequisites

Static-only (fast, no API keys needed)

Full LLM scan

Test with built-in fixtures

Output formats

Example: terminal output (23 fixtures, 8 workers)

Example: JSON output (excerpt)

LLM vs static comparison (same 23 fixtures, 8 workers)

Quick comparison: upstream vs batch

All Commands

Scan (LLM mode)

Scan (static-only, no API keys)

Output formats

Fixture test (built-in 23 skills)

Language override

Debugging

Compare upstream vs batch

CI

Tuning --workers

Language options

Debugging

Edge cases

Exit codes

Troubleshooting

Known Limitations

Running Tests

For PR Reviewers

What changed in production code (1 file)

How each review concern was addressed

Test suite at a glance (164 total)

Design context

Tuning `--workers`