Enable visual prompts for YOLOE in the custom frontend demo (#761)

* feat: add YOLOE with visual prompts

* feat: add precision option for YOLOE models

* fix: default model and precision args

* style: formatting

* docs: improve model and precision description

* small typo fix

* unrelated chore: corrected ordering

* feat: add encoding and high-res visualization

* refactor: model arg renaming

* docs: fix model arg name in readme

* docs: add argument changing instructions

* chore: upgrade depthai and depthai-nodes versions

* feat: add bbox drawing functionality

* fix: remove device argument

* feat: preserve background in overlay node

* feat: add yolo-world fp16 version

* feat: add notification toasts

* chore: update depthai-nodes version

* feat: improve toast notifications

* feat: make YOLOE fp16 default

* feat: add unified YOLOE model with both text and image prompts

* style: formatting

---------

Co-authored-by: klemen1999 <[email protected]>

Author: sokovninn

.pre-commit-config.yaml

@@ -16,5 +16,6 @@ repos:
     rev: 0.7.10
     hooks:
       - id: mdformat
+        args: [--number]
         additional_dependencies:
           - mdformat-gfm==0.3.6

custom-frontend/dynamic-yolo-world/README.md

@@ -1,6 +1,6 @@
 # Dynamic YOLO World/YOLOE
 
-This example demonstrates an advanced use of a custom frontend. On the DepthAI backend, it runs either the **YOLO-World** (default) or **YOLOE** model on-device, with configurable class labels and confidence threshold — both controllable via the frontend.
+This example demonstrates an advanced use of a custom frontend. On the DepthAI backend, it runs either **YOLOE** (default) or **YOLO-World** on-device, with configurable class labels and confidence threshold — both controllable via the frontend.
 The frontend, built using the `@luxonis/depthai-viewer-common` package, displays a real-time video stream with detections. It is combined with the [default oakapp docker image](https://hub.docker.com/r/luxonis/oakapp-base), which enables remote access via WebRTC.
 
 > **Note:** This example works only on RVC4 in standalone mode.
@@ -16,30 +16,34 @@ Running this example requires a **Luxonis device** connected to your computer. R
 Here is a list of all available parameters:
 
 ```
--d DEVICE, --device DEVICE
-					Optional name, DeviceID or IP of the camera to connect to. (default: None)
--fps FPS_LIMIT, --fps-limit FPS_LIMIT
-					FPS limit. (default: None)
--ip IP, --ip IP       IP address to serve the frontend on. (default: None)
--p PORT, --port PORT  Port to serve the frontend on. (default: None)
--n MODEL_NAME, --model-name MODEL_NAME
-					Name of the model to use: yolo-world or yoloe (default: yolo-world)
+-fps FPS_LIMIT, --fps_limit FPS_LIMIT
+                    FPS limit. (default: None)
+ -ip IP, --ip IP       IP address to serve the frontend on. (default: None)
+ -p PORT, --port PORT  Port to serve the frontend on. (default: None)
+ -m MODEL, --model MODEL
+                    Name of the model to use: yolo-world or yoloe (default: yoloe)
+ --precision PRECISION
+                    Model precision for YOLOE models: int8 (faster) or fp16 (more accurate) (default: fp16)
 ```
 
 ### Model Options
 
-This example supports two different YOLO models:
+This example supports two YOLO models:
 
-- **YOLO-World** (default): An open-vocabulary object detection model that supports both text-based class definitions and image-based prompting (upload an image to detect similar objects)
-- **YOLOE**: A fast and efficient object detection model with enhanced visualization features including instance segmentation
+- **YOLOE** (default): Supports both text prompts and image prompts (visual prompts). The model outputs 160 classes in total: indices 0–79 correspond to text prompts, and indices 80–159 correspond to image prompts. When only one prompt type is provided, dummy inputs are sent for the other and ignored by the model.
+- **YOLO-World**: Open-vocabulary detection with text prompts and optional image prompting (CLIP visual encoder).
+
+Notes:
+
+- Backend function `extract_image_prompt_embeddings(image, max_num_classes=80, model_name, mask_prompt=None)` accepts an optional `mask_prompt` of shape `(80,80)` or `(1,1,80,80)` for `yoloe`. When `None`, a default central mask is used.
 
 ### Prerequisites
 
 Before running the example you’ll need to first build the frontend. Follow these steps:
 
 1. Install FE dependencies: `cd frontend/ && npm i`
-1. Build the FE: `npm run build`
-1. Move back to origin directory: `cd ..`
+2. Build the FE: `npm run build`
+3. Move back to origin directory: `cd ..`
 
 ## Standalone Mode (RVC4 only)
 
@@ -55,9 +59,13 @@ oakctl app run .
 
 Once the app is built and running you can access the DepthAI Viewer locally by opening `https://<OAK4_IP>:9000/` in your browser (the exact URL will be shown in the terminal output).
 
-This will run the example with default argument values (YOLO-World model). If you want to change these values you need to edit the `oakapp.toml` file (refer [here](https://docs.luxonis.com/software-v3/oak-apps/configuration/) for more information about this configuration file).
+This will run the example with default argument values (YOLOE model). If you want to change these values you need to edit the `backend-run.sh` file to pass the arguments to the backend. Example:
+
+```bash
+python3.12 /app/backend/src/main.py --model yoloe --precision fp16 --fps_limit 10
+```
 
 ### Remote access
 
 1. You can upload oakapp to Luxonis Hub via oakctl
-1. And then you can just remotly open App UI via App detail
+2. And then you can just remotely open App UI via App detail

custom-frontend/dynamic-yolo-world/backend/src/depthai_models/yolo_world_l_fp16.RVC4.yaml

@@ -0,0 +1,2 @@
+model: yolo-world-l:640x640-host-decoding-fp16
+platform: RVC4

custom-frontend/dynamic-yolo-world/backend/src/depthai_models/yoloe_v8_l.RVC4.yaml

@@ -0,0 +1,2 @@
+model: yoloe-v8-l:640x640-text-visual-prompt:e23da0f
+platform: RVC4