[Support]: amggpu crash when using rocm in docker #16982

foxdvd · 2025-03-06T14:24:10Z

foxdvd
Mar 6, 2025

Checklist

I have updated to the latest available Frigate version.
I have cleared the cache of my browser.
I have tried a different browser to see if it is related to my browser.
I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

GPU and other components are good, stress test pass ok.
I tried it in latest Debian, Arch, Ubuntu - no luck.

GPU: AMD Radeon MI50 16GB
CPU: E5-2690 v2 x 2
RAM: 256GB DDR3
SSD: 1T SATA x 6
MB: Supermicro

Steps to reproduce

Start docker container
Look dmesg & docker logs

Version

latest

In which browser(s) are you experiencing the issue with?

No response

Frigate config file

mqtt:
enabled: false

record:
enabled: true
retain:
days: 7
mode: motion
alerts:
retain:
days: 14
pre_capture: 12
post_capture: 24
detections:
retain:
days: 14
pre_capture: 12
post_capture: 24
ffmpeg:
hwaccel_args: preset-vaapi

detectors:
rocm:
type: rocm

model:
model_type: yolonas
path: /config/model_cache/yolo_nas_l.onnx
input_pixel_format: rgb
width: 320
height: 320
  input_tensor: nchw

objects:
track:
  - person
    - cat
    - dog


detect:
fps: 5
enabled: true
min_initialized: 3
max_disappeared: 25
annotation_offset: 0
stationary:
interval: 50
threshold: 50
max_frames:
default: 3000
objects:
person: 2000
cat: 700
dog: 700

camera_kitchen:
enabled: true
ffmpeg:
inputs:
- path: rtsp://user:[email protected]:554/
roles:
- detect
- record
detect:
enabled: true
width: 1920
height: 1080

docker-compose file or Docker CLI command

services:
frigate:
ipc: host
security_opt:
- seccomp:unconfined
container_name: frigate
privileged:
restart: no
    image: ghcr.io/blakeblackshear/frigate:stable-rocm
shm_size: "8096mb"
devices:
- /dev/dri
- /dev/kfd
volumes:
- /etc/localtime:/etc/localtime:ro
- /opt/frigate/config:/config
- /opt/frigate/storage:/media/frigate
- type: tmpfs
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "8971:8971"
#        - "8554:8554" # RTSP feeds
#        - "8555:8555/tcp" # WebRTC over tcp
#        - "8555:8555/udp" # WebRTC over udp
environment:
FRIGATE_RTSP_PASSWORD: "password"
LIBVA_DRIVER_NAME: "radeonsi"

Relevant Frigate log output

...
2025-03-06 14:47:09.717086519  [2025-03-06 14:47:09] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2025-03-06 14:47:09.892904737  [2025-03-06 14:47:09] frigate.api.fastapi_app        INFO    : FastAPI started
2025-03-06 14:47:15.004280768  [INFO] Starting go2rtc healthcheck service...
2025-03-06 14:47:21.645925672  [2025-03-06 14:47:09] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: loading parsed model from /config/model_cache/yolo_nas_l.mxr
2025-03-06 14:47:21.646542326  [2025-03-06 14:47:21] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: model loaded
2025-03-06 14:47:28.189527407  Memory access fault by GPU node-2 (Agent handle: 0x11558c60) on address 0x460e40c000. Reason: Page not present or supervisor privilege.
...

Relevant go2rtc log output

---

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

HiWatch (FHD)

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

[  +0.000016] amdgpu 0000:0c:00.0: amdgpu:  for process frigate.detecto pid 15045 thread frigate.detecto pid 15045)
[  +0.000004] amdgpu 0000:0c:00.0: amdgpu:   in page starting at address 0x000000460e40c000 from IH client 0x1b (UTCL2)
[  +0.000008] amdgpu 0000:0c:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00801031
[  +0.000003] amdgpu 0000:0c:00.0: amdgpu:       Faulty UTCL2 client ID: TCP (0x8)
[  +0.000002] amdgpu 0000:0c:00.0: amdgpu:       MORE_FAULTS: 0x1
[  +0.000003] amdgpu 0000:0c:00.0: amdgpu:       WALKER_ERROR: 0x0
[  +0.000001] amdgpu 0000:0c:00.0: amdgpu:       PERMISSION_FAULTS: 0x3
[  +0.000002] amdgpu 0000:0c:00.0: amdgpu:       MAPPING_ERROR: 0x0
[  +0.000002] amdgpu 0000:0c:00.0: amdgpu:       RW: 0x0
[  +0.000010] amdgpu 0000:0c:00.0: amdgpu: [gfxhub0] no-retry page fault (src_id:0 ring:24 vmid:8 pasid:32794)
[  +0.000004] amdgpu 0000:0c:00.0: amdgpu:  for process frigate.detecto pid 15045 thread frigate.detecto pid 15045)
[  +0.000002] amdgpu 0000:0c:00.0: amdgpu:   in page starting at address 0x000000460e40c000 from IH client 0x1b (UTCL2)
[  +0.000005] amdgpu 0000:0c:00.0: amdgpu: [gfxhub0] no-retry page fault (src_id:0 ring:24 vmid:8 pasid:32794)
[  +0.000003] amdgpu 0000:0c:00.0: amdgpu:  for process frigate.detecto pid 15045 thread frigate.detecto pid 15045)
[  +0.000002] amdgpu 0000:0c:00.0: amdgpu:   in page starting at address 0x000000460e40c000 from IH client 0x1b (UTCL2)
[  +0.000005] amdgpu 0000:0c:00.0: amdgpu: [gfxhub0] no-retry page fault (src_id:0 ring:24 vmid:8 pasid:32794)
[  +0.000003] amdgpu 0000:0c:00.0: amdgpu:  for process frigate.detecto pid 15045 thread frigate.detecto pid 15045)
[  +0.000003] amdgpu 0000:0c:00.0: amdgpu:   in page starting at address 0x000000460e40c000 from IH client 0x1b (UTCL2)
[  +0.000004] amdgpu 0000:0c:00.0: amdgpu: [gfxhub0] no-retry page fault (src_id:0 ring:24 vmid:8 pasid:32794)
[  +0.000003] amdgpu 0000:0c:00.0: amdgpu:  for process frigate.detecto pid 15045 thread frigate.detecto pid 15045)
[  +0.000002] amdgpu 0000:0c:00.0: amdgpu:   in page starting at address 0x000000460e40c000 from IH client 0x1b (UTCL2)
[  +0.000004] amdgpu 0000:0c:00.0: amdgpu: [gfxhub0] no-retry page fault (src_id:0 ring:24 vmid:8 pasid:32794)
[  +0.000003] amdgpu 0000:0c:00.0: amdgpu:  for process frigate.detecto pid 15045 thread frigate.detecto pid 15045)
[  +0.000002] amdgpu 0000:0c:00.0: amdgpu:   in page starting at address 0x000000460e40c000 from IH client 0x1b (UTCL2)

NickM-27 · 2025-03-06T14:49:04Z

NickM-27
Mar 6, 2025
Collaborator Sponsor

This is not a bug, you have not defined the proper env variables for gfx version.

0 replies

foxdvd · 2025-03-06T16:01:48Z

foxdvd
Mar 6, 2025
Author

If I set version (HSA_OVERRIDE_GFX_VERSION: "9.0.0" or HSA_OVERRIDE_GFX_VERSION: "9.0.6"), segfault appear

2025-03-06 18:59:05.872950365  Fatal Python error: Segmentation fault
2025-03-06 18:59:05.872953701
2025-03-06 18:59:05.872955237  Thread 0x000073e5160816c0 (most recent call first):
2025-03-06 18:59:05.872962524    File "/usr/lib/python3.9/threading.py", line 312 in wait
2025-03-06 18:59:05.872975575    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2025-03-06 18:59:05.872977228    File "/usr/lib/python3.9/threading.py", line 892 in run
2025-03-06 18:59:05.872982152    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2025-03-06 18:59:05.872995900    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2025-03-06 18:59:05.872997078
2025-03-06 18:59:05.873001955  Current thread 0x000073e5ebeae740 (most recent call first):
2025-03-06 18:59:05.873031989    File "/usr/local/lib/python3.9/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 483 in _create_inference_session
2025-03-06 18:59:05.873063501    File "/usr/local/lib/python3.9/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419 in __init__
2025-03-06 18:59:05.873079668    File "/opt/frigate/frigate/detectors/plugins/onnx.py", line 44 in __init__
2025-03-06 18:59:05.873099514    File "/opt/frigate/frigate/detectors/__init__.py", line 18 in create_detector
2025-03-06 18:59:05.873114856    File "/opt/frigate/frigate/object_detection.py", line 68 in __init__
2025-03-06 18:59:05.873131166    File "/opt/frigate/frigate/object_detection.py", line 121 in run_detector
2025-03-06 18:59:05.873154556    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
2025-03-06 18:59:05.873187713    File "/opt/frigate/frigate/util/process.py", line 41 in run_wrapper
2025-03-06 18:59:05.873208521    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
2025-03-06 18:59:05.873229930    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2025-03-06 18:59:05.873252201    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2025-03-06 18:59:05.873269404    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2025-03-06 18:59:05.873287540    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2025-03-06 18:59:05.873305247    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2025-03-06 18:59:05.873318979    File "/opt/frigate/frigate/util/process.py", line 30 in start
2025-03-06 18:59:05.873338104    File "/opt/frigate/frigate/object_detection.py", line 204 in start_or_restart
2025-03-06 18:59:05.873353596    File "/opt/frigate/frigate/object_detection.py", line 172 in __init__
2025-03-06 18:59:05.873381012    File "/opt/frigate/frigate/app.py", line 350 in start_detectors
2025-03-06 18:59:05.873392130    File "/opt/frigate/frigate/app.py", line 598 in start
2025-03-06 18:59:05.873411853    File "/opt/frigate/frigate/__main__.py", line 60 in main
2025-03-06 18:59:05.873430702    File "/opt/frigate/frigate/__main__.py", line 64 in <module>
2025-03-06 18:59:05.873449825    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2025-03-06 18:59:05.873471661    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main

1 reply

NickM-27 Mar 6, 2025
Collaborator Sponsor

The m150 should not be that old, 9.0.0 is vega64, did you follow the instructions in the docs to figure out the correct version?

foxdvd · 2025-03-06T16:04:28Z

foxdvd
Mar 6, 2025
Author

MI 50 is gfx906

1 reply

NickM-27 Mar 6, 2025
Collaborator Sponsor

I see, maybe check the packages on the host and see if something can be updated. There is unfortunately nothing that can be done on the frigate side

foxdvd · 2025-03-06T16:38:08Z

foxdvd
Mar 6, 2025
Author

I switch rocm to onnx detector - it work for 1 minute and crash with this:

2025-03-06 19:19:39.063269539  [2025-03-06 19:15:47] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2025-03-06 19:19:39.063326073  [2025-03-06 19:15:47] frigate.api.fastapi_app        INFO    : FastAPI started
2025-03-06 19:19:40.363849573  Process detector:rocm_onnx:
2025-03-06 19:19:40.363855721  Traceback (most recent call last):
2025-03-06 19:19:40.363857697    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2025-03-06 19:19:40.363862818      self.run()
2025-03-06 19:19:40.363865296    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2025-03-06 19:19:40.363867167      return run(*args, **kwargs)
2025-03-06 19:19:40.363892256    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2025-03-06 19:19:40.363895020      self._target(*self._args, **self._kwargs)
2025-03-06 19:19:40.363898128    File "/opt/frigate/frigate/object_detection.py", line 145, in run_detector
2025-03-06 19:19:40.363900599      detections = object_detector.detect_raw(input_frame)
2025-03-06 19:19:40.363902918    File "/opt/frigate/frigate/object_detection.py", line 95, in detect_raw
2025-03-06 19:19:40.363932124      return self.detect_api.detect_raw(tensor_input=tensor_input)
2025-03-06 19:19:40.363934863    File "/opt/frigate/frigate/detectors/plugins/onnx.py", line 69, in detect_raw
2025-03-06 19:19:40.363937289      (_, x_min, y_min, x_max, y_max, confidence, class_id) = prediction
2025-03-06 19:19:40.363939306  ValueError: too many values to unpack (expected 7)

Config is

detectors:
rocm_onnx:
type: onnx

model:
model_type: yolonas
input_tensor: nchw
input_dtype: float
path: /config/model_cache/yolo_nas_l_coco.onnx
input_pixel_format: rgb
width: 640
height: 640
labelmap_path: /config/model_cache/labels.txt

1 reply

NickM-27 Mar 6, 2025
Collaborator Sponsor

that would be an issue with the model, presumably it was output in the wrong format

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Support]: amggpu crash when using rocm in docker #16982

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 4 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

[Support]: amggpu crash when using rocm in docker #16982

Uh oh!

Uh oh!

foxdvd Mar 6, 2025

Checklist

Describe the problem you are having

Steps to reproduce

Version

In which browser(s) are you experiencing the issue with?

Frigate config file

docker-compose file or Docker CLI command

Relevant Frigate log output

Relevant go2rtc log output

Operating system

Install method

Network connection

Camera make and model

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Replies: 4 comments · 3 replies

Uh oh!

NickM-27 Mar 6, 2025 Collaborator Sponsor

Uh oh!

foxdvd Mar 6, 2025 Author

Uh oh!

NickM-27 Mar 6, 2025 Collaborator Sponsor

Uh oh!

foxdvd Mar 6, 2025 Author

Uh oh!

NickM-27 Mar 6, 2025 Collaborator Sponsor

Uh oh!

foxdvd Mar 6, 2025 Author

Uh oh!

NickM-27 Mar 6, 2025 Collaborator Sponsor

foxdvd
Mar 6, 2025

Replies: 4 comments 3 replies

NickM-27
Mar 6, 2025
Collaborator Sponsor

foxdvd
Mar 6, 2025
Author

NickM-27 Mar 6, 2025
Collaborator Sponsor

foxdvd
Mar 6, 2025
Author

NickM-27 Mar 6, 2025
Collaborator Sponsor

foxdvd
Mar 6, 2025
Author

NickM-27 Mar 6, 2025
Collaborator Sponsor