feat(skills): auto-activate skills by fuzzy-matching user prompt#253
feat(skills): auto-activate skills by fuzzy-matching user prompt#253Prithvi1994 wants to merge 5 commits intompfaffenberger:mainfrom
Conversation
Skills were only activated if the LLM proactively called activate_skill(), which rarely happened unless the user explicitly mentioned "skill". This plugin hooks into get_model_system_prompt and scores all installed skill descriptions against the user prompt using rapidfuzz token_set_ratio (already a project dependency). Skills scoring >= 65 have their full SKILL.md content auto-injected into the system prompt before the agent runs. Key design decisions: - handled=False so claude-code/antigravity handlers still fire - Capped at MAX_AUTO_ACTIVATE=3 skills to protect context window - Degrades gracefully on any error (returns None, never crashes agent) - Uses existing plugin/callback architecture (no core changes) Fixes: skills not activating unless prompt contains the word "skill"
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 45 minutes and 43 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughNew Auto Skill Activator plugin that discovers installed skills, scores their relevance to the user prompt (LLM steering model with fuzzy fallback), and auto-injects qualifying SKILL.md contents into the model system prompt; includes a compaction hook to detect removed injections and comprehensive unit tests. Changes
Sequence DiagramsequenceDiagram
actor User
participant PluginSystem as Plugin System
participant Callback as Auto-Activate Callback
participant Discovery as Skill Discovery
participant Scorer as Scoring Engine
participant Loader as Skill Content Loader
participant Prompt as System Prompt
User->>PluginSystem: request model with user prompt
PluginSystem->>Callback: invoke get_model_system_prompt
Callback->>Discovery: discover installed skills & metadata
Discovery-->>Callback: skill list
Callback->>Callback: filter disabled / missing SKILL.md
loop each eligible skill
Callback->>Scorer: score prompt vs skill metadata
Scorer-->>Callback: relevance score (0-100)
end
Callback->>Callback: select top N by threshold & MAX_AUTO_ACTIVATE
loop each selected skill
Callback->>Loader: load SKILL.md content
Loader-->>Callback: skill content
Callback->>Prompt: append skill content with relevance %
end
Callback-->>PluginSystem: return modified instructions + user_prompt
PluginSystem-->>User: model prompt ready
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/plugins/test_auto_skill_activator.py`:
- Around line 139-145: The string value passed in skill_contents for the
"github-pr" key uses a raw newline inside single quotes which causes a
SyntaxError; update the literal in the test (where _run is called and
skill_contents={"github-pr": "..."} and the similar occurrence around lines
158-159) to a valid Python multiline string by either using a triple-quoted
string ("""...""") or replacing the raw newline with an explicit "\n" escape so
the module parses correctly.
- Around line 62-66: The test test_synonym_scores_above_threshold is failing
because _score_prompt_against_skill with token_set_ratio returns 50 for "push my
docker app" vs "deploy docker container deployment" which is below
AUTO_ACTIVATE_THRESHOLD; change the test input to increase token overlap (e.g.,
use a prompt like "deploy my docker container" or "deploy docker container") so
the token_set_ratio rises above AUTO_ACTIVATE_THRESHOLD when calling
_score_prompt_against_skill.
- Around line 106-130: The patches target attributes on register_callbacks but
those functions are imported inside register_callbacks.register_callbacks(), so
patching fails; update the patch targets to the modules that actually define
get_skills_enabled, get_skill_directories, get_disabled_skills, discover_skills,
parse_skill_metadata, and load_full_skill_content (i.e., patch the original
defining modules where those functions live) in the test block that patches
these names around register_callbacks.register_callbacks(), and make the same
replacements for the second patch block at lines 283–293 so both sets of patches
point to the real defining modules instead of register_callbacks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d6a2cc72-0f3f-42c5-b4d5-32ef63b95908
📒 Files selected for processing (3)
code_puppy/plugins/auto_skill_activator/__init__.pycode_puppy/plugins/auto_skill_activator/register_callbacks.pytests/plugins/test_auto_skill_activator.py
- Fix test_synonym_scores_above_threshold: 'push my docker app' scores 50 (below threshold 65) with token_set_ratio; use 'deploy my docker container' which shares tokens with the skill description and scores 94 - Fix SyntaxError: raw newlines in single-quoted strings replaced with \\n - Fix mock patch targets: functions are imported inside the function body so patching register_callbacks.<name> raises AttributeError; patch the defining modules directly (agent_skills.config, .discovery, .metadata)
|
I like the idea, but I think it would be much more powerful to leverage a smaller model like Haiku, one of the GPT Nanos, or even an open source one like Gemma 4 / GLM Flash to choose which skill to inject - if any. We would also need the background model to note if a compaction occurs at any given time to re-inject the skill if necessary. Then we could further generalize this concept beyond skill injection. Background steering agents. |
|
Great feedback — I'm on board with the direction. Here's my plan:
I'll push an updated commit with items 1 and 2 first, then follow up with the generalization. Thanks for the direction! |
…action re-injection - Replace rapidfuzz token_set_ratio with lightweight LLM call (Haiku/GPT-4o-mini/Gemma) for semantic skill relevance scoring - Add compaction re-injection hook via message_history_processor_end callback to re-evaluate and re-inject skills after context compaction - Keep rapidfuzz as fallback when steering model is unavailable - Add STEERING_MODEL_DEFAULT config and auto_skill_steering_model setting - Update tests for new LLM-based scoring approach + compaction re-injection Per mpfaffenberger feedback: mpfaffenberger#253 (comment)
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
code_puppy/plugins/auto_skill_activator/register_callbacks.py (1)
46-48: Module-level mutable state lacks thread safety for concurrent agents.If multiple agents run concurrently in the same process,
_last_activated_skillsand_last_user_promptcould race. Consider using a thread-local or session-keyed dictionary if concurrent agent support is planned.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@code_puppy/plugins/auto_skill_activator/register_callbacks.py` around lines 46 - 48, The module-level mutable state variables _last_activated_skills and _last_user_prompt are not thread-safe and can race when multiple agents run concurrently; change them to a concurrency-safe mechanism (e.g., use threading.local() or contextvars.ContextVar, or a session-keyed dict keyed by agent/session id) within register_callbacks.py so each agent has isolated state; locate usages of _last_activated_skills and _last_user_prompt in this module (functions that read/write them) and replace reads/writes with the chosen thread-local/session-scoped accessor to ensure no cross-agent contamination.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@code_puppy/plugins/auto_skill_activator/register_callbacks.py`:
- Around line 309-313: The handler in register_callbacks.py is returning
"handled": False which prevents model_utils.py (see result.get("handled") at
line ~99) from accepting the PreparedPrompt; change the response so the callback
returns "handled": True (or refactor the callback chain) so the injected prompt
is used—update the return dict in register_callbacks.py that builds the
PreparedPrompt (keys "instructions", "user_prompt", "handled") to set handled to
True (or implement a proper chaining mechanism in model_utils.py to allow
multiple handlers to contribute).
- Around line 119-141: The current use of ThreadPoolExecutor with
Future.result(timeout=15) can leak threads because the underlying asyncio task
(scoring_agent.run) keeps running after the Future times out; replace this
pattern by enforcing the timeout inside the asyncio task so it gets canceled
instead of leaving a running thread: when running in the async context (the
branch where loop.is_running() is true), run scoring_agent.run(user_message)
inside a thread but wrap the coroutine with asyncio.wait_for(..., timeout=15)
(or run a helper that calls asyncio.run(asyncio.wait_for(scoring_agent.run(...),
15))) so the coroutine is canceled on timeout, and update the exception handling
to catch asyncio.TimeoutError and handle cancellation/cleanup; alternatively use
ProcessPoolExecutor with explicit termination if cancellation inside the worker
is insufficient. Ensure you modify the code paths that call scoring_agent.run
(the ThreadPoolExecutor block and the else branch if you also need timeouts
there) and keep the existing logging/timeout handling consistent.
---
Nitpick comments:
In `@code_puppy/plugins/auto_skill_activator/register_callbacks.py`:
- Around line 46-48: The module-level mutable state variables
_last_activated_skills and _last_user_prompt are not thread-safe and can race
when multiple agents run concurrently; change them to a concurrency-safe
mechanism (e.g., use threading.local() or contextvars.ContextVar, or a
session-keyed dict keyed by agent/session id) within register_callbacks.py so
each agent has isolated state; locate usages of _last_activated_skills and
_last_user_prompt in this module (functions that read/write them) and replace
reads/writes with the chosen thread-local/session-scoped accessor to ensure no
cross-agent contamination.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 96f1bb48-b8d9-4530-89bb-a261bafd34a4
📒 Files selected for processing (2)
code_puppy/plugins/auto_skill_activator/register_callbacks.pytests/plugins/test_auto_skill_activator.py
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/plugins/test_auto_skill_activator.py
… thread leakage - Change handled: False → handled: True so model_utils.py actually accepts the injected prompt (critical — feature was non-functional) - Replace module-level mutable state with ContextVar for thread-safe concurrent agent support - Replace ThreadPoolExecutor + Future.result(timeout) with asyncio.wait_for() to properly cancel async tasks on timeout, preventing thread leakage
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
code_puppy/plugins/auto_skill_activator/register_callbacks.py (1)
49-52: Consider using a factory for the ContextVar default to prevent accidental mutation.The
default=[]creates a single shared list object. While the current code correctly uses.set()to replace values rather than mutating, a future modification like_last_activated_skills.get().append(...)would unexpectedly mutate the shared default.🛡️ Defensive alternative using factory pattern
-_last_activated_skills: ContextVar[List[str]] = ContextVar( - "_last_activated_skills", default=[] -) +_last_activated_skills: ContextVar[List[str]] = ContextVar("_last_activated_skills") + +def _get_activated_skills() -> List[str]: + try: + return _last_activated_skills.get() + except LookupError: + return []Alternatively, document explicitly that the returned list must not be mutated.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@code_puppy/plugins/auto_skill_activator/register_callbacks.py` around lines 49 - 52, The ContextVar _last_activated_skills currently uses default=[] which creates a shared mutable list; change its default to an immutable value (e.g., default=()) and update the type hint accordingly (e.g., ContextVar[Tuple[str, ...]] _last_activated_skills = ContextVar("_last_activated_skills", default=()) so callers cannot accidentally share/mutate the same list, and when a mutable list is needed explicitly call _last_activated_skills.set(list(...)) or convert the tuple to a list before modifying; leave _last_user_prompt unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@code_puppy/plugins/auto_skill_activator/register_callbacks.py`:
- Around line 286-294: The LLM-returned skill name in the loop over scored can
differ in formatting from keys in skill_path_map causing silent misses; fix by
normalizing both sides and looking up using a normalized map: build a
normalized_skill_map (e.g., map normalized_key -> original_key/path) from
skill_path_map (normalize via lowercasing and a consistent replacement/strip
strategy), then in the loop normalize item["name"] and check against
normalized_skill_map instead of raw skill_path_map, append the canonical
name/path to matching and update the logger.info to include both the LLM name
and the matched canonical skill name for clarity (referencing variables scored,
AUTO_ACTIVATE_THRESHOLD, skill_path_map, matching, and logger.info).
---
Nitpick comments:
In `@code_puppy/plugins/auto_skill_activator/register_callbacks.py`:
- Around line 49-52: The ContextVar _last_activated_skills currently uses
default=[] which creates a shared mutable list; change its default to an
immutable value (e.g., default=()) and update the type hint accordingly (e.g.,
ContextVar[Tuple[str, ...]] _last_activated_skills =
ContextVar("_last_activated_skills", default=()) so callers cannot accidentally
share/mutate the same list, and when a mutable list is needed explicitly call
_last_activated_skills.set(list(...)) or convert the tuple to a list before
modifying; leave _last_user_prompt unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 791266a3-388f-43e1-854b-3745b99933a8
📒 Files selected for processing (1)
code_puppy/plugins/auto_skill_activator/register_callbacks.py
The LLM may return reformatted skill names (e.g. 'GitHub PR Workflow' instead of 'github-pr-workflow'), causing silent lookup failures. Normalize both sides to lowercase before matching.
|
@mpfaffenberger , Please review the changes when you get a chance. Thank you. |
|
Hey @Prithvi1994 , i reviewed the code just now. Looks good from my end. |
3 participants
Problem
Skills in Code Puppy are injected into the system prompt as an XML list of names + descriptions, with guidance text telling the LLM to call
activate_skill()when a match is detected.In practice, skills rarely activate unless the user explicitly says "skill" in their prompt. The LLM doesn't proactively match task intent against skill descriptions — it needs a very direct signal.
This is a real usability gap: a user asking "help me create a pull request" won't benefit from an installed
github-pr-workflowskill, even though it's a perfect match.Root Cause
Agents are activated deterministically —
invoke_agentis an explicit tool call. Skills rely entirely on LLM discretion — passive XML in the prompt with a vague instruction to "match". There's no pre-run matching layer.Solution
This PR adds a new plugin
auto_skill_activatorthat hooks intoget_model_system_promptand:name + description + tagsagainst the user's prompt usingrapidfuzz.token_set_ratio(already a project dependency)AUTO_ACTIVATE_THRESHOLD) have their fullSKILL.mdcontent auto-injected into the system prompt before the agent runsMAX_AUTO_ACTIVATE) to protect context windowhandled=Falsesoclaude-code, antigravity, and other model handlers still fireWhy This Approach
base_agent.py,model_utils.py,callbacks.py)rapidfuzzis already inpyproject.tomlFiles Changed
code_puppy/plugins/auto_skill_activator/__init__.pycode_puppy/plugins/auto_skill_activator/register_callbacks.pytests/plugins/test_auto_skill_activator.pyTest Coverage
handled=Falsealways setSummary by CodeRabbit