Request: Run eval on remote promptfoo server

[underyx]

Apologies about having such a vague request, I have a vague idea of what problems we'd run into if we tried to set up promptfoo-action.

I think our eval datasets are considered sensitive enough that we don't want them in GitHub Actions and/or don't want all contributors of a repo to have access to them. To be more precise, this is primarily because we wrap promptfoo with a script that pulls data from S3 buckets and other sources in our infra. The script then generates a promptfooconfig.yaml, rendering the vars and assertions parts of the file. In short, to be able to run in CI, we'd need to let GHA access various sensitive data.

We currently only run promptfoo locally (with AWS SSO controlling access to data), but we'd like to host an instance to accomplish the following:

1. Share results team-wide
2. Generate datasets on a cron schedule instead of locally on every promptfoo execution
3. Trigger evals on the web UI without needing to launch our dev env

If we have this server, and it actually accomplishes the above three things (haven't looked into how exactly this would work), then it might be a logical architecture to reuse that server for running evals for us. I'm imagining that GHA would have 1) a promptfoo API key in secrets 2) the name of the remote dataset to run on and 3) the prompts that we just modified. Then this action would be able to use those three pieces of info to trigger a remote eval; the server would have 1) all provider API keys 2) the datasets themselves. The server could return just the scores without raw data to GHA which the comments could display. The comment could also include a link out to the web UI to review the full results.