[HW Accel Support]: Onnx Detector running on GPU but still High CPU Usage

[stillbevens]

Describe the problem you are having

Hello, I have my detector configure to run on GPU and see it in nvidia-smi 'frigate.detector.onnx' but it's CPU usage is way higher than GPU usage. Maybe I fundamentally misunderstand how the pipelining works....

My two streams are just opensource/internet accessible cameras as I am just starting to play around with and learn frigate before buying my own cameras, etc.

Version

0.16.0-d96efdb

Frigate config file

mqtt:
  enabled: False

detectors:
  onnx:
    type: onnx

model:
  model_type: yolo-generic
  width: 320 # <--- should match the imgsize set during model export
  height: 320 # <--- should match the imgsize set during model export
  input_tensor: nchw
  input_dtype: float
  path: /config/model_cache/yolov9-e.onnx
  labelmap_path: /labelmap/coco-80.txt

go2rtc:
  streams:
    petersburg: ffmpeg:http://89.104.109.62:8081/mjpg/video.mjpg#video=h264#hardware
    beach: ffmpeg:http://97.68.104.34/mjpg/video.mjpg#video=h264#hardware

ffmpeg:
      hwaccel_args: preset-nvidia

cameras:
  petersburg: # <------ Name the camera
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia
      inputs:
        - path: rtsp://127.0.0.1:8554/petersburg # <----- The stream you want to use for detection
          roles:
            - detect
            - record
    detect:
      enabled: True # <---- disable detection until you have a working camera feed

    zones: {}
    review: {}

  beach: # <------ Name the camera
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia
      inputs:
        - path: rtsp://127.0.0.1:8554/beach # <----- The stream you want to use for detection
          roles:
            - detect
            - record
    detect:
      enabled: True # <---- disable detection until you have a working camera feed
    zones: {}
    review: {}

objects:
  track:
    - car
    - person
    - face
    - license_plate

review:
  alerts:
    labels:
      - car
      - person
      - face
      - license_plate

  detections:
    labels:
      - car
      - person
      - face
      - license_plate

record:
  enabled: True
  expire_interval: 60
  sync_recordings: False
  retain:
    days: 0
    mode: all
  export:
    timelapse_args: -vf setpts=0.04*PTS -r 30
  preview:
    quality: medium
  alerts:
    pre_capture: 5
    post_capture: 5
    retain:
      days: 14
      mode: motion
  detections:
    pre_capture: 5
    post_capture: 5
    retain:
      days: 14
      mode: motion

snapshots:
  enabled: True
  clean_copy: True
  timestamp: True
  bounding_box: True
  crop: True
  height: 175
  required_zones: []
  retain:
    default: 10
    objects:
      person: 15
  quality: 70

semantic_search:
  enabled: True
  reindex: True
  model_size: large

face_recognition:
  enabled: True
  unknown_score: 0.8
  detection_threshold: 0.7
  recognition_threshold: 0.9
  min_area: 500
  min_faces: 1
  save_attempts: 100
  blur_confidence_filter: True
  model_size: large

lpr:
  enabled: True
  device: GPU
  model_size: large
  detection_threshold: 0.7
  min_area: 1000
  recognition_threshold: 0.9
  min_plate_length: 4
  format: None
  match_distance: 1
  known_plates: {}
  enhancement: 0
  debug_save_plates: False

version: 0.16.0-beta4

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    stop_grace_period: 30s
    image: ghcr.io/blakeblackshear/frigate:0.16.0-beta4-tensorrt
    shm_size: "512mb"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ports:
      - "8971:8971"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp"
      - "8555:8555/udp"

Relevant Frigate log output

N/A

Relevant go2rtc log output

N/A

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"stream_groups":[],"streams":[{"codec_long_name":"H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10","width":1280,"height":720,"display_aspect_ratio":"16:9","avg_frame_rate":"25/1"}]}}]

[{"return_code":0,"stderr":"","stdout":{"programs":[],"stream_groups":[],"streams":[{"codec_long_name":"H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10","width":1920,"height":1080,"display_aspect_ratio":"16:9","avg_frame_rate":"25/1"}]}}]

Install method

Docker Compose

Object Detector

TensorRT

Network connection

Wired

Camera make and model

AXIS P3344 Network Camera, AXIS Q6078-E PTZ Camera

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Everything is working, I just don't understand why CPU usage is so high when I have the detectors configured for GPU (Nvidia)

[NickM-27]

The CPU is still used to prepare the inputs for the model, you likely want to check the cameras and tune motion detection