[Support]: 0.15 Go2rtc continuous drops and reconnect on camera with rotated image

[leccelecce]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

New error in 0.15 dev.

I have a camera mounted upside down with no ability to rotate the image in-camera, so I use ffmpeg to rotate the image, once for recording, and once with a downsized stream for detection.

Since upgrading I get continuous (every 4s) errors from go2rtc on the rotated and downsized feed:

2024-10-21 16:56:26.247967946 [vf#0:0 @ 0x5573ded67380] Error while filtering: Cannot allocate memory

There's 32GB on this machine and 16GB limit on the Frigate container, which it isn't close to. It's an Intel I7-10700t using the built-in GPU so possibly there is a shared memory limit yet, but I can't get memory stats from the iGPU.

I did have a go at making the downsized feed base itself off the rotated feed (to avoid two sets of rotations) but couldn't get that to work.

Steps to reproduce

n/a

Version

0.15.0-8364e68

In which browser(s) are you experiencing the issue with?

No response

Frigate config file

ffmpeg:
  hwaccel_args: preset-intel-qsv-h264

...

go2rtc:
...
    hikcam4:
      - rtsp://[email protected]:554/Streaming/Channels/101
    hikcam4_rotated:
      - ffmpeg:hikcam4#video=h264#hardware=vaapi#rotate=270#audio=copy
    hikcam4_resized:
      - ffmpeg:hikcam4#video=h264#hardware=vaapi#rotate=270#width=1920#height=1080#raw=-fpsmax 5

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    image: ghcr.io/blakeblackshear/frigate:8364e68-amd64 # 0.15 dev build 21/01/2024
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    shm_size: "512mb"
    deploy:
      resources:
        limits:
          memory: 16384M
    network_mode: host
    volumes:
...
    ports:
...
    devices:
      - /dev/bus/usb/002:/dev/bus/usb/002
      - /dev/dri/renderD128:/dev/dri/renderD128

Relevant Frigate log output

No errors in log

Relevant go2rtc log output

2024-10-21 16:56:22.778727382  16:56:22.778 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:hikcam4#video=h264#hardware=vaapi#rotate=270#audio=copy
2024-10-21 16:56:22.778745083  16:56:22.778 DBG [streams] retry=0 to url=ffmpeg:hikcam4#video=h264#hardware=vaapi#rotate=270#audio=copy
2024-10-21 16:56:22.778876661  16:56:22.778 DBG [exec] run cmd="/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_flags allow_profile_mismatch -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/hikcam4?video&audio -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -c:a copy -vf format=vaapi|nv12,hwupload,transpose_vaapi=2,scale_vaapi=out_color_matrix=bt709:out_range=tv:format=nv12 -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/f5193c37b8e8008aa1bb8b800cafd1df" url="exec:/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_flags allow_profile_mismatch -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/hikcam4?video&audio -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -c:a copy -vf \"format=vaapi|nv12,hwupload,transpose_vaapi=2,scale_vaapi=out_color_matrix=bt709:out_range=tv:format=nv12\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/f5193c37b8e8008aa1bb8b800cafd1df"
2024-10-21 16:56:22.780077139  16:56:22.779 DBG [rtsp] handle error="read tcp 127.0.0.1:8554->127.0.0.1:44180: read: connection reset by peer"
2024-10-21 16:56:22.780091087  16:56:22.780 DBG [rtsp] disconnect stream=hikcam4
2024-10-21 16:56:22.789382501  16:56:22.789 DBG [rtsp] new consumer stream=hikcam4
2024-10-21 16:56:25.687731088  16:56:25.687 DBG [exec] run launch=2.908829138s
2024-10-21 16:56:26.247967946  [vf#0:0 @ 0x5573ded67380] Error while filtering: Cannot allocate memory
2024-10-21 16:56:26.252373638  [vf#0:0 @ 0x5573ded67380] Task finished with error code: -12 (Cannot allocate memory)
2024-10-21 16:56:26.252378761  [vf#0:0 @ 0x5573ded67380] Terminating thread with return code -12 (Cannot allocate memory)
2024-10-21 16:56:26.547077601  16:56:26.546 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:hikcam4#video=h264#hardware=vaapi#rotate=270#audio=copy
2024-10-21 16:56:26.547102365  16:56:26.547 DBG [streams] retry=0 to url=ffmpeg:hikcam4#video=h264#hardware=vaapi#rotate=270#audio=copy
2024-10-21 16:56:26.547175028  16:56:26.547 DBG [exec] run cmd="/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_flags allow_profile_mismatch -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/hikcam4?video&audio -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -c:a copy -vf format=vaapi|nv12,hwupload,transpose_vaapi=2,scale_vaapi=out_color_matrix=bt709:out_range=tv:format=nv12 -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/f5193c37b8e8008aa1bb8b800cafd1df" url="exec:/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_flags allow_profile_mismatch -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/hikcam4?video&audio -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -c:a copy -vf \"format=vaapi|nv12,hwupload,transpose_vaapi=2,scale_vaapi=out_color_matrix=bt709:out_range=tv:format=nv12\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/f5193c37b8e8008aa1bb8b800cafd1df"
2024-10-21 16:56:26.548248936  16:56:26.548 DBG [rtsp] handle error="read tcp 127.0.0.1:8554->127.0.0.1:44196: read: connection reset by peer"
2024-10-21 16:56:26.548321053  16:56:26.548 DBG [rtsp] disconnect stream=hikcam4
2024-10-21 16:56:26.558803450  16:56:26.558 DBG [rtsp] new consumer stream=hikcam4

Operating system

Debian

Install method

Docker Compose

Network connection

Wired

Camera make and model

Hikvision DS-2CD2086G2-IU

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[NickM-27]

This is most likely just related to using a newer ffmpeg, you could try some different things:

1. add manual input args to go2rtc to set the filter_threads 1 -filter_complex_threads 1 and see if that helps
2. set the path to 5.0 so it uses the same ffmpeg build as previous versions of frigate
3. try a different ffmpeg 7.0 build and see if the issue persists

[leccelecce]

Thanks for the fast response. I can confirm 2) fixes it; will try 1) and 3) at some point as well.

[gkostov]

2. just worked for me too. No idea why ...
   Thanks for the tip!