Add Inspect-native evaluation foundation for CodeGuard

[thomas-bartlett]

Summary

Adds an Inspect-native foundation for evaluating how Project CodeGuard affects model-generated code.

Inspect owns model execution, logs, and metrics. This project adds the small reusable pieces needed to load CodeGuard guidance, validate generated output, and attach benchmark-specific security scoring. New benchmarks can be added as focused sibling adapters without building a separate execution or reporting harness.

SecurityEval is the first adapter. The initial experiment compares baseline, generic secure-coding, and CodeGuard-guided outputs on 23 cases where pinned Bandit reliably detects the reference implementation's target CWE. This calibrated slice keeps the initial metric interpretable; it is not the intended limit. Future work can expand SecurityEval with additional analyzers or scorers and add other benchmarks through the same Inspect-native structure.

Safety and Current Limitations

• The current evaluation is one-shot and tool-free: no agent shell, Docker sandbox, or generated-code execution.
• CodeGuard is supplied as system guidance, so this does not test native skill activation.
• Bandit detections are not proof of security, and output validation checks structure rather than functional correctness.
• Host-side parsing is appropriate for this trusted benchmark; hostile models or sensitive hosts should use a disposable VM.
• No real provider/model smoke run has been performed yet.
• The README documents the temporary Click override required by the pinned Inspect release.

Validation

• 65 tests passed on Python 3.11 and 3.14
• Compile and Inspect task-discovery checks passed
• pip-audit found no known vulnerabilities
• git diff --check passed

Related

• Refs Add evaluation harness for Project CodeGuard #86
• Related to Discussion AI assisted Code Generation: Testing Harness Literature Review #70

[bact]

Suggested change

DISCUSSION_URL = "https://github.com/cosai-oasis/project-codeguard/discussions/70"

We probably don't need this but can keep for the purpose of being a seed.

[bact]

Suggested change

"discussion_url": DISCUSSION_URL,

We probably don't need this but can keep for the purpose of being a seed.

[bact]

Thank you. Looks solid for the job and we can build further on this.

Btw, to have this eval registered on Inspect Evals at https://ukgovernmentbeis.github.io/inspect_evals/ we need a small YAML file in the Inspect Eval Register. Something like this:

title: "CodeGuard Password Storage Eval"
description: |
  Evaluate password storage security practice
contributors:
  - thomas-bartlett
tags:                                      # optional; the upstream repo name is added automatically
  - CodeGuard
  - Coding
  - Password
  - Security
tasks:
  - name: password_storage                 # must match the @task function name
    task_path: evaluations/codeguard-inspect/codeguard_harness/tasks/password_storage.py
source:
  repository_url: https://github.com/cosai-oasis/project-codeguard
  repository_commit: <40-char SHA>         # grab from the commit page on GitHub; tags and branches aren't accepted
  comment: "Project CodeGuard is an open-source, model-agnostic security framework that embeds secure-by-default practices into AI coding agent workflows."

(use example from https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/register/example_eval.yaml )