[HW Accel Support]: Issue with Nvidia Tesla P4 and any GPU features

[kanemari]

Describe the problem you are having

I have had a stable Frigate setup running on Proxmox with a USB Coral for years. Its great, thank you for all your hard work.

Over time I added an Nvidia P106 GPU (full passthrough) to test tensor object identification and more recently face recognition, etc. It all works, but my main issue was this GPU having no HW decoder and the CPU being under pressure adding timestamps to recorded video (6 streams @ 1080p). I needed something else reasonably cheap, so went with a tesla P4.

I recently added the Tesla P4 (vGPU) and tried to get HW acceleration working. I realise the built-in ffmpeg requires drivers higher than what this card supports, and compiled my own ffmpeg binaries that work. The problem is that by enabling the GPU in the docker compose file, I get maybe 10 minutes of the system working before it crashes my entire proxmox server with a kernel panic.

Ive checked the kernel logs from journalctl in both the VM and host, there is no error at all before it crashes.

I have tried running this setup with just the GPU enabled in the docker config and all other HW acceleration / GPU features off in the frigate config, but still something runs on the GPU (I can see the process in nvidia-smi) - pushes it only up to around 4% utilisation. and it crashes very quickly. if I remove the GPU from the docker-compose file it runs fine again.

I have run other VMs with CUDA on this setup and done benchmarking tests, and they are all stable. I really dont think its a driver problem in itself, but more of a compatibility problem with Frigate. I want to try and rebuild any GPU / CUDA related components with CUDA 12.2.2 (the highest supported by this GPU / driver) but there doesnt appear to be a documented way to run frigate outside of docker.

Do you have any suggestions?

Version

0.16.0-c2f8de9

Frigate config file

mqtt:
  host: 192.168.x.x
  user: frigate
  password: xxx

go2rtc:
  streams:
    15_garage:
      - rtsp://thingino:[email protected]/ch1
      - ffmpeg:15_garage#audio=opus
    15_intercom:
      - rtsp://admin:[email protected]/Streaming/Channels/101
      - ffmpeg:15_intercom#audio=aac
    15_driveway:
      - rtsp://ringcam:[email protected]:38554/54e019b76dd5_live
      #- ffmpeg:15_driveway#audio=aac
    15_backyard:
      - rtsp://ringcam:[email protected]:38554/9c7613b33f56_live
      #- ffmpeg:15_backyard#audio=aac
    15a_garage:
      - rtsp://tapocam:[email protected]/stream2
      - ffmpeg:15a_garage#audio=opus#audio=aac
    15a_intercom:
      - rtsp://admin:[email protected]/Streaming/Channels/101
      - ffmpeg:15a_intercom#audio=aac
  #log:
  #  exec: trace

  webrtc:
    candidates:
      - 192.168.x.x:8555
      - 192.168.x.x:8555
      - stun:8555

cameras:
  15a_garage:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/15a_garage
          input_args: preset-rtsp-restream-low-latency
          roles:
            - detect
            - audio
        - path: rtsp://tapocam:[email protected]/stream1
          input_args: preset-rtsp-generic
          roles:
            - record
    detect:
      width: 640
      height: 360
      fps: 10
    mqtt:
      enabled: false
      timestamp: true
      bounding_box: true
      crop: true
      #height: 1080
      required_zones: []
    snapshots:
      enabled: true
      timestamp: true
      bounding_box: true
      crop: false
      #height: 175
      #required_zones: []
      retain:
        default: 7
        objects:
          person: 14
    record:
      enabled: true
      retain:
        days: 2
        # Optional: Mode for retention. Available options are: all, motion, and active_objects
        #   all - save all recording segments regardless of activity
        #   motion - save all recordings segments with any detected motion
        #   active_objects - save all recording segments with active/moving objects
        # NOTE: this mode only applies when the days setting above is greater than 0
        mode: all
      alerts:
        retain:
          days: 14
        pre_capture: 2
        post_capture: 4
      detections:
        retain:
          days: 7
        pre_capture: 2
        post_capture: 4
    motion:
      mask: 0,0.434,0,0,1,0,1,0.457,0.647,0.141,0.31,0.132
      threshold: 35
      contour_area: 10
      improve_contrast: true
    review:
      alerts:
        labels:
          - fire_alarm
          - scream
          - yell
          - person
          - car
          - dog
          - cat
      detections:
        labels:
          - bark
          - speech

    objects:
      filters:
        person:
          mask: 0,0.51,0.197,0.372,0.238,1,0,1
  15_garage:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/15_garage
          input_args: preset-rtsp-restream-low-latency
          roles:
            - detect
            - audio
        - path: rtsp://thingino:[email protected]/ch0
          input_args: preset-rtsp-generic
          roles:
            - record
    detect:
      width: 640
      height: 360
      fps: 10
    mqtt:
      enabled: false
      timestamp: true
      bounding_box: true
      crop: true
      #height: 1080
      required_zones: []
    snapshots:
      enabled: true
      timestamp: true
      bounding_box: true
      crop: false
      #height: 175
      #required_zones: []
      retain:
        default: 7
        objects:
          person: 14
    record:
      enabled: true
      retain:
        days: 2
        # Optional: Mode for retention. Available options are: all, motion, and active_objects
        #   all - save all recording segments regardless of activity
        #   motion - save all recordings segments with any detected motion
        #   active_objects - save all recording segments with active/moving objects
        # NOTE: this mode only applies when the days setting above is greater than 0
        mode: all
      alerts:
        retain:
          days: 14
        pre_capture: 2
        post_capture: 4
      detections:
        retain:
          days: 7
        pre_capture: 2
        post_capture: 4
    motion:
      mask: 0,0.425,0.292,0.133,0.63,0.131,0.998,0.385,1,0,0,0
      threshold: 35
      contour_area: 10
      improve_contrast: true
    objects:
      filters:
        person:
          mask: 
            0.688,0.248,0.718,0.137,0.777,0.106,0.792,0.161,0.773,0.209,0.768,0.248,0.767,0.355
    review:
      alerts:
        labels:
          - fire_alarm
          - scream
          - yell
          - person
          - car
          - dog
          - cat
      detections:
        labels:
          - bark
          - speech

  15a_intercom:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/15a_intercom
          input_args: preset-rtsp-restream-low-latency
          roles:
            - detect
            - audio
        - path: rtsp://admin:[email protected]/Streaming/Channels/101
          input_args: preset-rtsp-generic
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 10
    mqtt:
      enabled: false
      timestamp: true
      bounding_box: true
      crop: true
      #height: 720
      required_zones: []
    snapshots:
      enabled: true
      timestamp: true
      bounding_box: true
      crop: false
      #height: 175
      #required_zones: []
      retain:
        default: 7
        objects:
          person: 14
    record:
      enabled: true
      retain:
        days: 1
        # Optional: Mode for retention. Available options are: all, motion, and active_objects
        #   all - save all recording segments regardless of activity
        #   motion - save all recordings segments with any detected motion
        #   active_objects - save all recording segments with active/moving objects
        # NOTE: this mode only applies when the days setting above is greater than 0
        mode: all
      alerts:
        retain:
          days: 14
        pre_capture: 2
        post_capture: 4
      detections:
        retain:
          days: 7
        pre_capture: 2
        post_capture: 4
    objects:
    # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - car
        - dog
        - cat
      filters:
        dog: {}
        car:
          mask: 
            0.209,0.748,0.437,0.807,0.749,0.829,0.932,0.808,1,0.79,1,0,0,0,0,0.66
    motion:
      mask: 0,0,1,0,1,0.69,0.844,0.693,0.619,0.683,0.357,0.654,0,0.578
      threshold: 40
      contour_area: 20
      improve_contrast: true
    review:
      alerts:
        labels:
          - fire_alarm
          - scream
          - yell
      detections:
        labels:
          - bark
          - speech
          - person
          - car
          - dog
          - cat

  15_intercom:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/15_intercom
          input_args: preset-rtsp-restream-low-latency
          roles:
            - detect
            - audio
        - path: rtsp://admin:[email protected]/Streaming/Channels/101
          input_args: preset-rtsp-generic
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 10
    mqtt:
      enabled: false
      timestamp: true
      bounding_box: true
      crop: true
      #height: 720
      required_zones: []
    snapshots:
      enabled: true
      timestamp: true
      bounding_box: true
      crop: false
      #height: 175
      #required_zones: []
      retain:
        default: 7
        objects:
          person: 14
    record:
      enabled: true
      retain:
        days: 1
        # Optional: Mode for retention. Available options are: all, motion, and active_objects
        #   all - save all recording segments regardless of activity
        #   motion - save all recordings segments with any detected motion
        #   active_objects - save all recording segments with active/moving objects
        # NOTE: this mode only applies when the days setting above is greater than 0
        mode: all
      alerts:
        retain:
          days: 14
        pre_capture: 2
        post_capture: 4
      detections:
        retain:
          days: 7
        pre_capture: 2
        post_capture: 4
    objects:
    # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - car
        - dog
        - cat
      filters:
        dog: {}
        car:
          mask: 0.519,0.82,0.753,0.822,1,0.778,1,0,0,0,0,0.663,0.241,0.776
    motion:
      mask: 0,0,1,0,1,0.705,0.873,0.699,0.552,0.676,0.217,0.627,0,0.589
      threshold: 40
      contour_area: 20
      improve_contrast: true
    review:
      alerts:
        labels:
          - fire_alarm
          - scream
          - yell
      detections:
        labels:
          - bark
          - speech
          - person
          - car
          - dog
          - cat

  15_driveway:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/15_driveway
          input_args: preset-rtsp-restream-low-latency
          roles:
            - detect
            - audio
        - path: rtsp://ringcam:[email protected]:38554/54e019b76dd5_live
          input_args: preset-rtsp-generic
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 10
    mqtt:
      enabled: false
      timestamp: true
      bounding_box: true
      crop: true
      #height: 1080
      required_zones: []
    snapshots:
      enabled: true
      timestamp: true
      bounding_box: true
      crop: false
      #height: 175
      #required_zones: []
      retain:
        default: 7
        objects:
          person: 14
    record:
      enabled: true
      retain:
        days: 2
        # Optional: Mode for retention. Available options are: all, motion, and active_objects
        #   all - save all recording segments regardless of activity
        #   motion - save all recordings segments with any detected motion
        #   active_objects - save all recording segments with active/moving objects
        # NOTE: this mode only applies when the days setting above is greater than 0
        mode: all
      alerts:
        retain:
          days: 14
        pre_capture: 2
        post_capture: 4
      detections:
        retain:
          days: 7
    objects:
    # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - car
        - dog
        - cat
      # Optional: mask to prevent all object types from being detected in certain areas (default: no mask)
      # Checks based on the bottom center of the bounding box of the object.
      # NOTE: This mask is COMBINED with the object type specific mask below
      #mask: 0,0,1000,0,1000,200,0,200
      filters:
        person:
          # Optional: minimum width*height of the bounding box for the detected object (default: 0)
          #min_area: 18000
          # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
          max_area: 50000
          # Optional: minimum width/height of the bounding box for the detected object (default: 0)
          #min_ratio: 0.4
          # Optional: maximum width/height of the bounding box for the detected object (default: 24000000)
          #max_ratio: 1.0
          # Optional: minimum score for the object to initiate tracking (default: shown below)
          #min_score: 0.7
          # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
          threshold: 0.75
          # Optional: mask to prevent this object type from being detected in certain areas (default: no mask)
          # Checks based on the bottom center of the bounding box of the object
          #mask: 0,1080,570,1080,570,625,0,625
          mask:
            #- 243,1080,657,1080,658,592,236,593
            - 0.272,0.536,0.347,0.585,0.367,0.66,0.375,0.849,0.299,1,0.189,1,0.19,0.707,0.226,0.582
            - 0.371,0.338,0.375,0.249,0.379,0.177,0.42,0.16,0.466,0.146,0.471,0.253,0.462,0.344,0.414,0.351
        cat:
          max_area: 10000
        dog:
          max_area: 10000
        car:
          mask: 
            0.144,1,0.323,0.483,0.408,0.472,0.489,0.114,0.678,0.111,0.874,1,1,1,1,0,0,0,0,1
    motion:
      mask: 
        0.197,0.217,0.287,0.098,0.296,0.066,0.355,0.034,0.361,0.063,0.42,0.039,0.479,0.022,0.492,0.136,0.676,0.137,0.677,0.015,0.707,0.024,0.718,0.046,0.781,0.125,0.91,0.301,1,0.464,1,0,0,0,0,0.429
      threshold: 35
      contour_area: 15
      improve_contrast: true
    zones: {}
    review:
      alerts:
        labels:
          - fire_alarm
          - scream
          - yell
          - person
          - car
          - dog
          - cat
      detections:
        labels:
          - bark
          - speech

  15_backyard:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/15_backyard
          input_args: preset-rtsp-restream-low-latency
          roles:
            - detect
            - audio
        - path: rtsp://ringcam:[email protected]:38554/9c7613b33f56_live
          input_args: preset-rtsp-generic
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 10
    mqtt:
      enabled: false
      timestamp: true
      bounding_box: true
      crop: true
      #height: 1080
      required_zones: []
    snapshots:
      enabled: true
      timestamp: true
      bounding_box: true
      crop: false
      #height: 175
      #required_zones: []
      retain:
        default: 7
        objects:
          person: 14
    record:
      enabled: true
      retain:
        days: 2
        # Optional: Mode for retention. Available options are: all, motion, and active_objects
        #   all - save all recording segments regardless of activity
        #   motion - save all recordings segments with any detected motion
        #   active_objects - save all recording segments with active/moving objects
        # NOTE: this mode only applies when the days setting above is greater than 0
        mode: all
      alerts:
        retain:
          days: 14
        pre_capture: 2
        post_capture: 4
      detections:
        retain:
          days: 7
        pre_capture: 2
        post_capture: 4
    objects:
    # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - dog
        - cat
      # Optional: mask to prevent all object types from being detected in certain areas (default: no mask)
      # Checks based on the bottom center of the bounding box of the object.
      # NOTE: This mask is COMBINED with the object type specific mask below
      #mask: 0,0,1000,0,1000,200,0,200
      filters:
        person:
          mask: 
            0.766,0.478,0.844,0.52,0.862,0.626,0.808,0.677,0.742,0.764,0.698,0.66,0.718,0.53
          # Optional: minimum width*height of the bounding box for the detected object (default: 0)
          #min_area: 18000
          # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
          #max_area: 150000
          # Optional: minimum width/height of the bounding box for the detected object (default: 0)
          #min_ratio: 0.4
          # Optional: maximum width/height of the bounding box for the detected object (default: 24000000)
          #max_ratio: 1.2
          # Optional: minimum score for the object to initiate tracking (default: shown below)
          #min_score: 0.7
          # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
          #threshold: 0.8
          # Optional: mask to prevent this object type from being detected in certain areas (default: no mask)
          # Checks based on the bottom center of the bounding box of the object
          #mask: 179,887,537,991,630,684,272,612
    motion:
      threshold: 35
      contour_area: 15
      improve_contrast: true
      mask: 
        0,0.597,0.047,0.534,0.189,0.124,0.243,0.047,0.435,0.019,0.555,0.022,0.601,0.031,0.658,0.009,0.699,0.044,0.732,0.057,0.838,0.169,0.879,0.229,0.943,0.326,0.973,0.357,1,0.305,1,0,0,0
    review:
      alerts:
        labels:
          - fire_alarm
          - scream
          - yell
          - person
          - car
          - dog
          - cat
      detections:
        labels:
          - bark
          - speech

detectors:
  coral:
    type: edgetpu
    device: usb
#  tensorrt:
#    type: tensorrt
#    device: 0

#model:
#  path: /config/model_cache/tensorrt/yolov7-320.trt
#  input_tensor: nchw
#  input_pixel_format: rgb
#  width: 320
#  height: 320

ffmpeg:
  global_args: -hide_banner -loglevel warning -threads 1
  #hwaccel_args: preset-nvidia
  #hwaccel_args: cuda -c:v h264_nvenc
  input_args: preset-rtsp-generic
  output_args:
    detect: -avioflags direct -threads 1 -fps_mode vfr -muxdelay 0 -flags 
      low_delay -flush_packets 1 -f rawvideo -pix_fmt yuv420p
    #record: preset-record-generic-audio-aac
    # uses preset-record-generic-audio-aac but adds timestamp
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 
      -strftime 1 -vf 
      "drawtext=text='%{localtime}':fontcolor=white:fontsize=24:x=10:y=10:box=1:[email protected]"
      -c:a aac
  path: '5.0'
  #path: /usr/lib/btbn-ffmpeg/

birdseye:
  enabled: true
  width: 1920
  height: 1080
  quality: 8
  restream: true
  # Optional: Mode of the view. Available options are: objects, motion, and continuous
  #   objects - cameras are included if they have had a tracked object within the last 30 seconds
  #   motion - cameras are included if motion was detected in the last 30 seconds
  #   continuous - all cameras are included always
  mode: continuous

audio:
  enabled: true
  listen:
    - bark
    - fire_alarm
    - scream
    - speech
    - yell

timestamp_style:
  # "tl" (top left), "tr" (top right), "bl" (bottom left), "br" (bottom right)
  position: tl
  format: '%d/%m/%Y %H:%M:%S'
  color:
    red: 255
    green: 255
    blue: 255
  thickness: 2
  #           None (No effect),
  #           "solid" (solid background in inverse color of font)
  #           "shadow" (shadow for font)
  effect: shadow

objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
    - car
    - dog
    - cat
  # Optional: mask to prevent all object types from being detected in certain areas (default: no mask)
  # Checks based on the bottom center of the bounding box of the object.
  # NOTE: This mask is COMBINED with the object type specific mask below
  #mask: 0,0,1000,0,1000,200,0,200
  # Optional: filters to reduce false positives for specific object types
  #filters:
  #  person:
      # Optional: minimum width*height of the bounding box for the detected object (default: 0)
  #    min_area: 10000
      # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
  #    max_area: 250000
      # Optional: minimum width/height of the bounding box for the detected object (default: 0)
  #    min_ratio: 0.4
      # Optional: maximum width/height of the bounding box for the detected object (default: 24000000)
  #    max_ratio: 1.2
      # Optional: minimum score for the object to initiate tracking (default: shown below)
  #    min_score: 0.65
      # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
  #    threshold: 0.75
      # Optional: mask to prevent this object type from being detected in certain areas (default: no mask)
      # Checks based on the bottom center of the bounding box of the object
      #mask: 0,0,1000,0,1000,200,0,200

detect:
  enabled: true

face_recognition:
  enabled: false
  model_size: large

lpr:
  enabled: false
  device: GPU
  match_distance: 1
  known_plates:
    Kane:
      - "NKSI"
    Sophia:
      - "CLZ-101"
classification:
  bird:
    enabled: false

semantic_search:
  enabled: false
  model_size: large
  reindex: false

genai:
  enabled: false
  provider: gemini
  api_key: xx
  model: gemini-1.5-flash

tls:
  enabled: false

logger:
  default: info
  #logs:
    #frigate.ptz.onvif: error
    #frigate.ptz.autotrack: debug
    #frigate.record: debug

record:
  # Optional: Sync recordings with disk on startup and once a day (default: shown below).
  sync_recordings: true
  # Optional: Recording Preview Settings
  preview:
    # Optional: Quality of recording preview (default: shown below).
    # Options are: very_low, low, medium, high, very_high
    quality: very_high
  expire_interval: 5
  enabled: true
  retain:
    days: 0
    # Optional: Mode for retention. Available options are: all, motion, and active_objects
    #   all - save all recording segments regardless of activity
    #   motion - save all recordings segments with any detected motion
    #   active_objects - save all recording segments with active/moving objects
    # NOTE: this mode only applies when the days setting above is greater than 0
    mode: all
  alerts:
    retain:
      days: 14
    pre_capture: 2
    post_capture: 4
  detections:
    retain:
      days: 7
    pre_capture: 2
    post_capture: 4

version: 0.16-0
camera_groups:
  Intercoms:
    order: 1
    icon: LuCamera
    cameras:
      - 15a_intercom
      - 15_intercom
  Birdseye:
    order: 2
    icon: LuEye
    cameras: birdseye

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    stop_grace_period: 30s # allow enough time to shut down the various services
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    #image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "256mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # Passes the USB Coral, needs to be modified for other versions
      #- /dev/apex_0:/dev/apex_0 # Passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      #- /dev/video11:/dev/video11 # For Raspberry Pi 4B
      #- /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
    #deploy:
    #  resources:
    #    reservations:
    #      devices:
    #        - driver: nvidia
             #device_ids: ['0'] # this is only needed when using multiple GPUs
    #          count: 1 # number of GPUs
    #          capabilities: [gpu]
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/kane/frigate/config:/config
      - /mnt/frigate:/media/frigate
      #- /home/kane/frigate/ffmpeg-binaries:/usr/lib/btbn-ffmpeg:ro
      #- /usr/local/cuda-12.2/targets/x86_64-linux/lib:/usr/lib/host-cuda-lib:ro
      #- /usr/local/cuda-12.2/lib64:/usr/lib/host-cuda-lib64:ro
      #- /usr/local/lib:/usr/lib/host-local-lib:ro
      #- /usr/local/lib/x86_64-linux-gnu:/usr/lib/host-local-gnu:ro
      #- /usr/lib/x86_64-linux-gnu:/usr/lib/host-usr-gnu:ro
      #- /lib/x86_64-linux-gnu:/usr/lib/host-gnu:ro
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 536870912
    ports:
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      #- YOLO_MODELS=yolov7-320,yolov7x-640
      #- USE_FP16=false
      #- NVIDIA_VISIBLE_DEVICES=ALL
      #- NDIVIA_DRIVER_CAPABILITIES=compute,video,utility
     - FRIGATE_RTSP_PASSWORD="password"

Relevant Frigate log output

No errors - logs get wiped on restart (kernel panic freezes host)

Relevant go2rtc log output

No errors - logs get wiped on restart (kernel panic freezes host)

FFprobe output from your camera

not a camera issue

Install method

Proxmox via Docker

Object Detector

Coral

Network connection

Wired

Camera make and model

Multiple - but no camera issue

Screenshots of the Frigate UI's System metrics pages

Nothing noteworthy

Any other information that may be helpful

highest supported driver version for the Tesla P4 is 535.261.03 - CUDA 12.2.2 - Nvidia container toolkit 1.14.3

[inkeep-ai-assistant[bot]]

Your setup and troubleshooting steps are thorough. Based on Frigate documentation and the details you provided:

• The recommended way to use Nvidia GPUs for hardware acceleration (video decode and/or object detection) in Frigate is by using the official Docker images with the -tensorrt tag for standard Nvidia GPUs, or -tensorrt-jp6 for Jetson devices(1)(2)(3)(4).

• The Tesla P4 is supported for object detection and hardware video decoding, but you must ensure that the host driver version is compatible with the CUDA libraries used by the Frigate container image. For TensorRT, the minimum driver version must be >=545 and the GPU must be Compute Capability 5.0+(2).

• To enable hardware decoding in Frigate, add the following to your config.yml:

  ffmpeg:
    hwaccel_args: preset-nvidia

  and ensure your container is running with the Nvidia runtime(5)(1).

• If the built-in ffmpeg in the Frigate image isn't compatible with your GPU/driver, custom ffmpeg binaries are not officially supported, and there is no documented way to run Frigate outside of Docker(1).

• If you see ffmpeg processes in nvidia-smi but experience kernel panics, this points to a possible low-level driver or virtualization issue, not a Frigate configuration bug. Frigate itself cannot crash the host kernel(5)(2).

• For object detection, you must use the correct detector configuration:

  detectors:
    tensorrt:
      type: tensorrt
      device: 0

  The TensorRT detector uses .trt model files generated specifically for your hardware(3)(4).

• Mixing multiple detectors (e.g., Coral and TensorRT/GPU) is not supported for the same object detection task(3).

Key points:

• The Frigate container must use the Nvidia runtime and compatible drivers.
• Only the documented Docker images (not bare-metal or custom ffmpeg) are supported(1)(2)(3).
• If your GPU (Tesla P4) is not working with the default Frigate image, and kernel panics occur even with minimal Frigate config, this is almost certainly a host/virtualization/driver issue and not a Frigate software bug.
• There is no supported, documented way to run Frigate natively outside of Docker(1).

For more details on hardware acceleration and troubleshooting, see:

• Video Decoding
• Recommended hardware
• Supported Hardware
• Troubleshooting GPU

📚 Sources:

1. Installation
2. Recommended hardware
3. Supported Hardware
4. Supported Hardware
5. Video Decoding

---

Have another question? Just tag @inkeep.

[kanemari]

is this AI response the only response I am going to get?

it might be worth adding that I am running Ollama without any issue using nvidia-container-toolkit. I have also compiled my own ffmpeg binary (using CUDA 12.2 & v525 drivers) and left it running encoding one of the RTSP streams for hours and it works fine without causing a kernel panic. I also more recently tried a v570+ driver that was made available for my aging card, and it worked seemingly fine for around 10-15 minutes before it crashed the PVE server again.

so I dont think its a low level issue with my system because it works for other VMs passing through the vGPU - I think its something specific that frigate does, and I just want some help or pointers trying to work out what is causing it.

it also seems I am not alone - but this other thread has too many variables - #3652

[NickM-27]

To be clear are you trying to build the dev container or just a local frigate environment? Your log message says you are running the dev container which may not be what you are trying to do

[kanemari]

just trying for a local frigate environment. the 'contributing' pages seem to have pointed me in the right direction. I cloned the repo, checked out v0.16.1, and started the 'official build' script. hopefully that will result in what I am after.

ideally I wanted to try to run frigate outside of docker just to test my specific case where using it with an Nvidia vGPU randomly crashes my PVE server with no indication of any errors. not sure a locally built docker container will end up any different, but worth a try. the vGPU is the only hardware decoder I have in that system, and with 6 permanent and an additional 3 ad-hoc feeds I need it.

[kanemari]

@NickM-27 - I think I may have almost worked my way through this. Is there any debug option to find out all the arguments that get passed to ffmpeg for each stream? because of all the presets its hard to trace exactly what options need to be supported when trying to compile my own ffmpeg binary. Im at the stage where I can get it working in the CLI but it errors out within frigate because of conflicting options with cuda and GPU / CPU filters.

[NickM-27]

/api/stats shows it, if they're removed though then you'll potentially be losing some performance

[kanemari]

thanks for that - all working now. there were multiple issues in the end and I am not sure if I even needed a custom ffmpeg binary.

but anyway it ended up working with a custom ffmpeg binary that was using CUDA 12.2.2, and minor tweaks to the record output_args in the config. the main omission was adding h264_nvenc to the output, since with nothing specified it was using CPU to reencode the timestamp into the recordings by default. then needed to tweak other -vf options to get it to work. I had one stream from a dodgy camera that was saying it was outputting 300fps, so I had to hard code fps down to 25 to get it to work uniformly across all 6 cameras. took more than an hour just working our why that one was failing when all the others worked - stupid Hikvision camera.

    record: -vf "fps=25,format=nv12,drawtext=text='%{localtime}':fontcolor=white:fontsize=24:x=10:y=10:box=1:[email protected]"
      -c:v h264_nvenc -c:a aac
      -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1

the irony is, I recently found a vGPU unlocked v570 / CUDA 12.8 capable driver for my card as well, so while this was all fun it will probably work out of the box now.

[demonlj]

hi, I also want to using P106-100 for object detection. Does it work?
my hardware were dell t3630, E-2174g with DDR4 16G. using E-2174g integrated graphics for hw decoding, using P106-100 for object detection.
I am not sure wether debian host should install driver for p106-100?