feat: Add HarmActionsEval for evaluation of action-level safety in AI agents

[prane-eth]

Background

AI agents have a growing adoption across the industry, including critical applications. AI agents that have access to tools can currently call tools directly with no centralized validation layer that inspects these calls before execution, allowing harmful or disallowed tool calls to be executed without oversight. Existing agent benchmarks evaluate the safety of final responses instead of actions.

HarmActionsEval benchmark evaluates actions. It found that 80% of the LLMs tested executed actions at the first attempt for over 95% of the harmful prompts.

Related work: https://github.com/Pro-GenAI/Agent-Action-Guard. It received 962 downloads on PyPI, and 247 clones on GitHub in the first week.

Changes

I integrated HarmActionsEval to allow evaluation of agent safety at action-level. HarmActionsEval is a dataset-based evaluation benchmark. I added the dataset, benchmark, test cases, and documentation.

All the test cases passed using: pytest tests/test_frameworks/test_harmactionseval.py
Example file at "examples/harmactionseval_evaluation.py" has been executed successfully.

[vercel]

@prane-eth is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.