GitHub App for Automated AI Security Testing (looking for community feedback)

[ksiabani]

Hi everyone 👋

We’d love to get your thoughts on an idea we’re exploring: a GitHub App that helps projects like DATAGEN stay secure as they grow.

What it would do

1. Scan & detect risks against the OWASP LLM Top 10 and OWASP Agentic Top 10.
2. Map the agentic system: generate a graph of agents, prompts, and connected tools.
3. Simulate & attack: run contextual adversarial prompts/conversations in a sandbox.
4. Record findings: share attack transcripts for any successful exploits.
5. Suggest mitigations: prompt-hardening changes.
6. Reference standards: tie recommendations back to OWASP guidelines.

Additional notes

• It would run periodically (e.g., weekly) to keep results fresh.
• It would always open a PR with findings + proposed fixes.
• It would be free for OSS projects and stay that way.

Why this could help DATAGEN

• Reduce the burden on maintainers by getting ready-to-review security PRs.
• Give the community a transparent view of security posture with reproducible reports and fixes.
• Keep DATAGEN resilient against evolving AI threats without requiring extra setup.
• Build even more trust with users by showing security is continuously tested and improved.
• Comply with recommendations from major LLM providers (OpenAI, Google, Microsoft, etc.) for thorough testing of GenAI apps, including Red Teaming (see OpenAI’s guidance).

👉 Would this be useful for the DATAGEN community?
👉 Are there specific risks or areas you’d like the app to cover?

We’d love to co-design this with you. Your input will shape whether and how we bring this to life here.

[starpig1129]

Thanks for sharing this great proposal! The concept of a GitHub App to automate security processes and reduce maintainer burden sounds incredibly valuable.

I'm very interested in this service and would love to learn more about the specific implementation details and future plans. However, due to my current tight schedule, I might not be able to contribute much extra time for detailed work at the moment.

[ksiabani]

Thanks a lot for the encouragement 🙏. Totally understand that time is limited. No extra workload is expected from maintainers right now. Our plan is to prototype the GitHub App on a few OSS projects and share the results.

Would you be open if we included DATAGEN as one of the early pilot projects?
That way you’ll get periodic security PRs with no setup required, and you can simply review or ignore them at your convenience.

[starpig1129]

Thanks for the follow-up! I'm definitely open to including DATAGEN in the early pilot program. The idea of improving my project's security without adding to my workload is very appealing.

Before I give the green light, however, I'd like to understand how the GitHub App will work. Could you please share some details on its operational model? For example, what permissions will it require? How will it integrate with my project? And is there any documentation or a workflow you can share?

[ksiabani]

Great to hear, thanks for being open to a pilot 🙌

Here’s how the GitHub App will work at a high level:

Permissions

• Read: repository contents, pull requests, checks
• Write: checks (to report results), pull request comments, and create PRs with findings/fixes
• No access to secrets or tokens

Integration

• Maintainers install the App from the GitHub Marketplace and select which repos it can access
• Once installed, it runs on a schedule (e.g. weekly)
• Each run scans the repo for agentic AI patterns, simulates them in a sandbox, and checks against OWASP LLM + Agentic Top 10

Workflow

1. The App opens a PR with:
   • A report of risks found
   • Example attack transcripts (adversarial conversations)
   • Suggested mitigations (prompt hardening)
   • References to OWASP remediation docs
2. Maintainers can review and merge like any other PR
3. If merged, the fixes become part of the codebase; if not, no impact on the project

We’re preparing short documentation and workflow diagrams. I’ll share those here once ready so you can see exactly what the PRs will look like.

[starpig1129]

Thanks so much for the clear and detailed explanation. The workflow you described makes perfect sense, and it's reassuring to know the app won't need access to our secrets.

I'm looking forward to seeing the documentation and diagrams, and am excited to see the results this app can bring!