[Bug]: memory leak when accessing an event

[jerome71]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

This is on a Raspberry Pi 5 8GB with a Coral USB TPU and a 4GB USB SSD for media storage.

I could see that, after a moment, accessing an event would take a very long time (accessing the page with the list of event works well), and would again be quick after restarting Frigate (from the UI or any other way).
Since version 0.15.0, it came to the point the Pi would become unresponsive (no SSH access, no web serving etc.), but would usually regain consciousness after a few minutes.
After analyzing logs, it appears the Pi was out of memory (and therefore wrote a swap file to the SD card, which cannot go well).
After introducing memory limits for the frigate process in the docker-compose file, the Pi remains responsive, but, often, when watching an event, frigate quits and relaunchs. After a few seconds to a minute, when reloading the page, the event plays as it should.

Steps to reproduce

1. Go to the events page
2. View 1 event
3. Go back and veiw another one
4. Continue until Frigate crashes
   ...

Version

0.15.0-cea210d

In which browser(s) are you experiencing the issue with?

Safari 18.3 (20620.2.4.11.5) on macOS 15.3.1 (Apple Silicon)

Frigate config file

(edited so that I have less data to obfuscate)
mqtt:
  enabled: true
  host: 127.0.0.1
  port: 1883
  user: mqtt_frigate
  password: password


ui:
  time_format: 24hour

auth:
  enabled: true
  reset_admin_password: false

tls:
  enabled: false

#ffmpeg:
#  path: "5.0"
#  hwaccel_args: preset-rpi-64-h264 #Enable Hardware Acceleration

semantic_search:
  enabled: false
  reindex: false

detectors:
  coral:
    type: edgetpu
    device: usb

record:
  enabled: true
  retain:
    days: 21
    mode: all

objects:
  track:
    - car
    - cat
    - dog
    - person


review:
  alerts:
    labels:
#      - car
      - person

go2rtc:
  streams:
    Aiguebelle_Dahua_PT:
      - rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    Aiguebelle_Dahua_PT_small:
      - rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=1
  webrtc:
    candidates:
      - 192.168.1.100:8555
      - stun:8555
      - 100.x.y.z:8555
  homekit:
    Aiguebelle_Dahua_PT:
      name: Aiguebelle Dahua PT
      pin: 12345678
      pairings:
        - client_id=lotsofnumbers&permissions=1


cameras:
  Aiguebelle_Dahua_PT: #Name for your comment
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Aiguebelle_Dahua_PT #The Stream you want to monitor
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/Aiguebelle_Dahua_PT_small #The Stream you want to monitor
          roles:
            - detect
    live:
      stream_name: Aiguebelle_Dahua_PT
    onvif:
      host: 192.168.51.110
      port: 80
      user: admin
      password: password
      ignore_time_mismatch: false
    detect:
      enabled: true # Detection is disabled
      fps: 2
    mqtt:
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 500
    ui:
      order: 54

version: 0.15-1
camera_groups:
  Birdseye:
    order: 1
    icon: LuBird
    cameras: birdseye
#  Aline:
#    order: 4
#    icon: LuContact
#    cameras:
#      - Aline_Paris
#      - St_Clair_Piscine
  Hameau:
    order: 2
    icon: LuWarehouse
    cameras:
      - Hameau_Entree
      - Hameau_Jardin_Gauche
      - Hameau_Facade
      - Hameau_Jardin_Droite
      - Hameau_Escalier
  Aline:
    order: 4
    icon: LuSun
    cameras:
      - St_Clair_Piscine
      - Aline_Paris
  Aiguebelle:
    order: 4
    icon: LuFish
    cameras:
      - Aiguebelle_Dahua_Fixe
      - Aiguebelle_Dahua_PT

docker-compose file or Docker CLI command

services:
  frigate-igor:
    container_name: frigate-igor
    privileged: true # this may not be necessary for all setups
    network_mode: host
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 3500M
    image: ghcr.io/blakeblackshear/frigate:0.15.0
    shm_size: "352mb"
    devices:
      - /dev/bus/usb:/dev/bus/usb #Used for Coral if Available
#      - /dev/video10 #Map the Pi hardware acceleration
    volumes:
      - /etc/localtime:/etc/localtime:ro
#      - /etc/letsencrypt/live/myIPaddress:/etc/letsencrypt/live/frigate:ro
#      - /etc/letsencrypt/archive/myIPaddress:/etc/letsencrypt/archive/frigate:ro
      - /home/igor/frigate-big/config1:/config
      - /home/igor/frigate-big/storage1:/media/frigate
      - type: tmpfs #Remove this if using a Pi with 2GB or Less RAM
        target: /tmp/cache
        tmpfs:
          size: 1000000000
#    ports:
#      - "5000:5000"
#      - "8971:8971"
#      - "8554:8554" # RTSP feeds
#      - "8555:8555/tcp" # WebRTC over tcp
#      - "8555:8555/udp" # WebRTC over udp
#    extra_hosts:
#      - "host.docker.internal:host-gateway"
    environment:
      FRIGATE_RTSP_PASSWORD: "password"

Relevant Frigate log output

Docker logs for the container when crash occurs trying to play an event
2025-02-14 16:45:28.943258673  127.0.0.1 - - [14/Feb/2025:16:45:28 +0100] "GET /vod/Aiguebelle_Dahua_PT/start/1738746000/end/1738749600/seg-141-v1.ts HTTP/1.1" 200 4864312 "https://myhostURL/review" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15" "172.225.116.187"
2025-02-14 16:45:29.147294148  127.0.0.1 - - [14/Feb/2025:16:45:29 +0100] "GET /ws HTTP/1.1" 101 21777 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15" "172.225.116.187"
2025-02-14 16:45:29.147303574  127.0.0.1 - - [14/Feb/2025:16:45:29 +0100] "GET /ws HTTP/1.1" 101 24624 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15" "172.225.116.187"
2025-02-14 16:45:29.154800254  2025/02/14 16:45:29 [error] 161#161: *905 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /vod/Aiguebelle_Dahua_PT/start/1738746000/end/1738749600/seg-142-v1.ts HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "myhostURL", referrer: "https://myhostURL/review"
2025-02-14 16:45:29.154813291  2025/02/14 16:45:29 [error] 161#161: *905 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /vod/Aiguebelle_Dahua_PT/start/1738746000/end/1738749600/seg-142-v1.ts HTTP/1.1", host: "myhostURL", referrer: "https://myhostURL/review"
2025-02-14 16:45:29.154818050  127.0.0.1 - - [14/Feb/2025:16:45:29 +0100] "GET /vod/Aiguebelle_Dahua_PT/start/1738746000/end/1738749600/seg-142-v1.ts HTTP/1.1" 500 177 "https://myhostURL/review" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15" "myMacsIPaddress"
2025-02-14 16:45:29.155823697  2025/02/14 16:45:29 [error] 168#168: *899 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /api/profile HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "myhostURL", referrer: "https://myhostURL/review"
2025-02-14 16:45:29.155832197  2025/02/14 16:45:29 [error] 168#168: *899 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/profile HTTP/1.1", host: "myhostURL", referrer: "https://myhostURL/review"
2025-02-14 16:45:29.156566771  127.0.0.1 - - [14/Feb/2025:16:45:29 +0100] "GET /api/profile HTTP/1.1" 500 177 "https://myhostURL/review" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15" "172.225.116.187"
2025-02-14 16:45:29.156902678  2025/02/14 16:45:29 [error] 168#168: *900 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /api/review/summary?timezone=Europe%2FParis&cameras=Aiguebelle_Dahua_PT HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "myhostURL", referrer: "https://myhostURL/review"
2025-02-14 16:45:29.156971715  2025/02/14 16:45:29 [error] 168#168: *900 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/review/summary?timezone=Europe%2FParis&cameras=Aiguebelle_Dahua_PT HTTP/1.1", host: "myhostURL", referrer: "https://myhostURL/review"
2025-02-14 16:45:29.157432548  127.0.0.1 - - [14/Feb/2025:16:45:29 +0100] "GET /api/review/summary?timezone=Europe%2FParis&cameras=Aiguebelle_Dahua_PT HTTP/1.1" 500 177 "https://myhostURL/review" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15" "172.225.116.187"
2025-02-14 16:45:29.157639529  2025/02/14 16:45:29 [error] 168#168: *902 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /api/preview/all/start/1738710000/end/1738800000 HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "myhostURL", referrer: "https://myhostURL/review"
2025-02-14 16:45:29.157801826  2025/02/14 16:45:29 [error] 168#168: *902 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/preview/all/start/1738710000/end/1738800000 HTTP/1.1", host: "myhostURL", referrer: "https://myhostURL/review"
2025-02-14 16:45:29.158191881  127.0.0.1 - - [14/Feb/2025:16:45:29 +0100] "GET /api/preview/all/start/1738710000/end/1738800000 HTTP/1.1" 500 177 "https://myhostURL/review" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15" "myMacsIPaddress"
2025-02-14 16:45:29.185775101  [INFO] Service Frigate exited with code 256 (by signal 9)
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service go2rtc-healthcheck: stopping
s6-rc: info: service certsync: stopping
2025-02-14 16:45:29.297256702  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
2025-02-14 16:45:29.299530478  [INFO] Service CERTSYNC exited with code 256 (by signal 15)
s6-rc: info: service certsync successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service certsync-log: stopping
s6-rc: info: service go2rtc-healthcheck successfully stopped
s6-rc: info: service certsync-log successfully stopped
2025-02-14 16:45:29.625612528  [INFO] Service NGINX exited with code 0 (by signal 0)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service nginx-log: stopping
s6-rc: info: service frigate: stopping
s6-rc: info: service frigate successfully stopped
s6-rc: info: service go2rtc: stopping
s6-rc: info: service frigate-log: stopping
s6-rc: info: service nginx-log successfully stopped
2025-02-14 16:45:29.673774158  exit with signal: terminated
s6-rc: info: service frigate-log successfully stopped
2025-02-14 16:45:29.731735133  [INFO] The go2rtc service exited with code 0 (by signal 0)
s6-rc: info: service go2rtc successfully stopped
s6-rc: info: service go2rtc-log: stopping
s6-rc: info: service go2rtc-log successfully stopped
s6-rc: info: service log-prepare: stopping
s6-rc: info: service s6rc-fdholder: stopping
s6-rc: info: service log-prepare successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service s6rc-fdholder successfully stopped
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Relevant go2rtc log output

Nothing relevant.
2025-02-14 16:50:13.636508161  [INFO] Preparing new go2rtc config...
2025-02-14 16:50:14.511770148  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2025-02-14 16:50:14.920977284  [INFO] Starting go2rtc...
2025-02-14 16:50:15.092252617  16:50:15.089 INF go2rtc platform=linux/arm64 revision=b2399f3 version=1.9.2
2025-02-14 16:50:15.092258728  16:50:15.089 INF config path=/dev/shm/go2rtc.yaml
2025-02-14 16:50:15.092265562  16:50:15.090 INF [api] listen addr=:1984
2025-02-14 16:50:15.092462395  16:50:15.092 INF [rtsp] listen addr=:8554
2025-02-14 16:50:15.093954227  16:50:15.093 INF [webrtc] listen addr=:8555
2025-02-14 16:50:23.648283746  [INFO] Starting go2rtc healthcheck service...

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Dahua DH-P5D-5F-PV

Screenshots of the Frigate UI's System metrics pages

(please note that the alert is related to the CPU being used to create the detect stream for a camera - top shows 40% CPU available on the Pi)
20240214 - General Stats - Frigate.pdf
20240214 - Storage Stats - Frigate.pdf
20240214 - Cameras Stats - Frigate.pdf

Any other information that may be helpful

Seems to crash way more than 0.14.1.
Tried removing TLS config and using ffmpeg 5 with no success.
I put the memory limits in the docker compose file so that the crash resulted in a Frigate restart instead of the Pi becoming unresponsive.
The Python process usually uses betwen 1.4GB and 1.8GB, so 3.5GB should be plenty enough, why not triggering a memory swap.

Here is the Raspberry Pi log that indicated a memory leak from the Python process: "Out of memory: Killed process 19165 (python3) total-vm:5438768kB, anon-rss:2547392kB..." from this file
Kernel Crash Raspberry Pi 20250214.txt

[NickM-27]

A memory leak is a very specific thing which means memory is allocated and then not released despite not being in use. In order to have evidence of a memory leak, we will need more in-depth information of specifically what is using memory, and if it truly is not being used any longer.

The Python process usually uses betwen 1.4GB and 1.8GB, so 3.5GB should be plenty enough, why not triggering a memory swap.

this is really pushing it, especially for the number of cameras that you have.

[jerome71]

A memory leak is a very specific thing which means memory is allocated and then not released despite not being in use. In order to have evidence of a memory leak, we will need more in-depth information of specifically what is using memory, and if it truly is not being used any longer.

That is why I joined the Pi log that showed the Python process was using over 5GBs and was terminated by the system at the request of Tailscale
Also, the behavior (Pi not becoming unresponsive but Frigate restarting) after putting a memory limit in the docker-compose file seems to indicate that the process is actually putting the Pi out of memory.

What else can I provide?

The Python process usually uses betwen 1.4GB and 1.8GB, so 3.5GB should be plenty enough, why not triggering a memory swap.

this is really pushing it, especially for the number of cameras that you have.

I have 9 cameras (only kept the config for one here, for readability and to reduce the number of things I needed to obfuscate). The issue happens when I watch an event for that very camera (and also for others).
Correction: using docker stats, I could see that the frigate process is actually using close to 3.5GB. So I increased the limit to 5.5GB (the screenshot is before this increase)

[jerome71]

Here is the increase in memory usage when I watch events one after the other ...until Frigate restarts (more than probably after hitting the pemory usage limit), via "docker stats frigate" (frigate-igor, the name of the container in my case)

then

then, after Frigate's automatic restart (note the different PID)