[OPIK-2940] [FE][BE][DOCS] Video: Add support for Video LLM-as-a-judge, Datasets and Playground - WIP DO NOT MERGE #3988
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Collaborator
|
✅ Test environment is now available! Access Information
The deployment has completed successfully and the version has been verified. |
Nimrod007
changed the title
vincentkoc
added
test-environment
Contributor
Collaborator
|
✅ Test environment is now available! Access Information
The deployment has completed successfully and the version has been verified. |
…to feat/video-eval * 'feat/video-eval' of https://github.com/comet-ml/opik: Update base version to 1.9.10 [OPIK-2856] [FE] Hide All time option in metrics tab and support optional date filtering (#4052) Update TypeScript SDK version to 1.9.9
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Warning
Large base64 data such as video should ideally be moved to a blob store and appropriate routes, handlers and retention configured. For test purposes this will work but in large volumes could be unstable. See SDK chanegs in #3998 for how we might use attachments in traces and spans.
Note
Additional work might be required to ensure video output from supported SDKs/LLMs is captured and can be routed for the online evaluation (not yet tested, i.e. Wan 2.5). See #3998 for some support via attachments.
Note
Frontend UI with
image (+)andvideo (+)is crowded and should really be using afile (+)with smart mime/ext detection, however if we have a url based approach it might not be aparent so would/might want to improve the UX here later.Details
Add support for video based online and SDK based evals (LLM-as-a-judge), video datsets (base64, url), and normalizing functions to a "media" handler ubiquitous of video or image for simplification.
Change checklist
Issues
Testing
Testing requires a local inference model, for this I have been using Ollama with vLLM endpoint. For cloud testing you can tunnel using ngrok. Ollama support added here ollama/ollama#12962 local build can be provided if not merged.
Documentation
Updated
Examples
Important
Traces with attachments for videos (minio/s3) are aware to the online evaluation flow but not exposed in an accessiable manner. See current fields from #3998