[Support]: Out of memory: Killed process (frigate.capture) #16710
Replies: 20 comments 10 replies
-
|
Sounds similar to my long-standing issue with wireless Hikvision cameras - #10814 - though I was able to reproduce it simply by disconnecting the ethernet cable on a wired camera too. 9 times out of 10 connection loss for a camera is harmless, but 1 in 10 times it causes something to crap out. Wifi cameras I suspect are just more prone to triggering. |
Beta Was this translation helpful? Give feedback.
-
|
I'm having oom-killer events as well. Running a simple installation, no gen ia with 3 wired cameras on a 6 GB Proxmox LXC. I'm having these issues most of the time when using the web interface , especially the video and object life cycle in the explore section are prone to do this. |
Beta Was this translation helpful? Give feedback.
-
|
It's becoming more and more of a problem to the point where I may have to drop Frigate entirely. |
Beta Was this translation helpful? Give feedback.
-
|
The above issue seems to be because docker-compose v3+ can't handle setting limits on swap usage, only memory usaged - https://docs.docker.com/engine/containers/resource_constraints/. To get around this issue I've got a script checking the memory usage of the Frigate container every 60s. If it's above 4GB memory usage the container is rebooted. This obviously isn't the best solution as if there's a recording in progress this will get interrupted. It is however the more preferred option over oom-killer events. |
Beta Was this translation helpful? Give feedback.
-
|
HI. I have just started experiencing this issue, I have been running frigate on my Lenovo m700 with 16 gb of memory for about 2 years, however I have only noticed this in the last few days I did change my docker compose file to increase the shm_size: to 128mb as I was noticing errors in the frigate log and asking to increase it. I am not sure if this is the cause I am no linux or docker expert, but do wonder if increasing this setting might be the issue This script you have, could you provide it, or any other finding you might have Thank You |
Beta Was this translation helpful? Give feedback.
-
|
I have the same problem with my Amcrest WIFI doorbell camera. Hangs with the graceful exit message. I am running both my cameras through Go2RTC and downsampling the higher-bitrate stream to 1280x960 for my detect stream (the substream is only 720x576 and doesn't give great detect results.) I have no idea what is happening and don't even really know where to look - but I've just solved my problem by watching for either of my camera feeds to die in either Frigate or Go2RTC with the following script. If it dies I'm using a go2rtc API call to restart Go2RTC (only). This has caused a Go2RTC restart to occur about 5 times a day over the last 4 days. I lose about 8 seconds of recording during that restart but that is much shorter than restarting the whole FRIGATE process. Thus far it has worked like a champ. No more "dead camera" in my otherwise working Frigate instance. Frigate is running inside an privileged Proxmox LXC container (because I'm lazy and couldn't get the Coral card passthru to work inside the LXC correctly.) The host system is an old laptop and has Plex, Immich and Frigate running on it and all are using Intel GPU (shared) for video processing. It is probably not enough resources but it works for me. Anyway - I run this shell script inside the container with root permissions and have it set up as a systemd cron job that runs ever 10 seconds and looks for the camera to be missing. If it is then it issues the API call and sends a notification to Home Assistant. I'm not sure if the sleep/wait thing is really required but I didn't want systemd to launch another watch instance while Go2RTC was restarting - so I just have it wait. This seems to kill the cron process so then I restart the cron process with the last two lines. Super kludge and I don't really know how to code. ChatGPT wrote the "jq" steps for me. But they work... #!/bin/sh
GO2RTC_URL="http://192.168.xxx.yyy:1984/api/streams"
FRIGATE_URL="http://192.168.xxx.yyy:5000/api/stats"
FRIGATE_CAMERAS_DOWN=$(curl -s $FRIGATE_URL | jq -r '.cameras | to_entries[] | select(.value.camera_fps == 0.0) | .key')
GORTC_CAMERAS_DOWN=$(curl -s $GO2RTC_URL | jq -r 'to_entries[] | select(.value.producers == null or .value.producers == []) | select(.key | test("rest_command|other_non_camera_items") | not) | .key')
switch=0
# Check if any FRIGATE cameras are down
if [ -n "$FRIGATE_CAMERAS_DOWN" ]; then
echo "Camera(s) down: $FRIGATE_CAMERAS_DOWN"
switch=$((switch+1))
fi
# Check if any GO2RTC cameras are down
if [ -n "$GO2RTC_CAMERAS_DOWN" ]; then
echo "Camera(s) down: $GO2RTC_CAMERAS_DOWN"
switch=$((switch+1))
fi
if [ "$switch" -gt 0 ];
then
switch=$((0))
echo "Restarting Go2RTC"
curl -X POST http://192.168.xxx.yyy:1984/api/restart
curl -X POST http://192.168.xxx.zzz:8123/api/services/notify/mobile_app_<myphoneid> -H "Authorization: Bearer <TokenfromHA>" -H "Content-Type: application/json" -d '{"message": "A camera stream may be down - Go2RTC has restarted!", "title": "Frigate Alert"}'
sleep 300 &
wait
systemctl enable --now frigate-check.timer
systemctl restart frigate-check.timer
fithen inside the Frigate Config file under my Go2RTC config I added: No original work here - flagrantly plagiarized from other people's work - just sharing what I have working. Didn't include the steps to get the systemd cron process running - ask the goog or ai. |
Beta Was this translation helpful? Give feedback.
-
|
I have only been running Frigate for 5 days and this has happen to me 2 times already. I have a single TP-Link C320WS connected by Ethernet and every time the network goes down, for whatever reason, I get an And there is also the memory issue, last time it consumed over 10GB of RAM before the OOM came to kill Here is the system log for the last time it happen: Note: Yes, I'm aware that running a docker container in a privileged LXC is far from ideal, but it was the "fastest" way to get it working (for now) and I don't think it is related wit the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Same: |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 for me and probably other user it's critical issue. It's ok when it affects only itself like restarting container but not whole system. I migrated from QNAP to UnRAID which much better specs -switched from i965 to iHD (J4125 -> i5-12500) but still it's unstable. |
Beta Was this translation helpful? Give feedback.
-
|
Been experiencing random freezes on an UnRAID server that could only be recovered from a force shutdown the past couple of months as a result of this. Recently discovered it was caused by Frigate and limited its container memory to 2GB though it still manages to cause a full server freeze every few weeks. Frigate is unfortunately unusable with this issue present. Log Dump |
Beta Was this translation helpful? Give feedback.
-
|
I have also been suffering from the same problem for several months now. Since I run several Docker containers, including Frigate, I started measuring the memory usage and the problem clearly comes from Frigate:
Since I have been unable to fix the problem and my entire server kept crashing, bringing down Home Assistant, Z2M... I finally set up a Home Assistant automation which restarts the Frigate service every time it hits
So again, not a solution to the problem, but my system does not crash anymore... |
Beta Was this translation helpful? Give feedback.
-
|
Was having the same problem with lockups and memory usage, so added this to the "Extra Parameters" for the Docker containers settings in Unraid. It still consumes memory but stops my Unraid locking up when it is exhausted. Not a fix but a workaround until I can track down the cause. |
Beta Was this translation helpful? Give feedback.
-
|
For anyone else stuggling with this issue adding
to your docker compose file will make the oom-killer kill every process in the docker container when frigate.capture runs away, which will then cause the container to be restarted. |
Beta Was this translation helpful? Give feedback.
-
|
Any idea how to add this to the Docker container/app in Unraid? |
Beta Was this translation helpful? Give feedback.
-
|
There is another thread here #14571 on a similar issue (possibly the same) where I suggest changing some of the kswapd (paging) parameters. It looks similar to me because the oom logs report a large amount of inactive memory, which is ultimately available for use, but freeing it does take some time (from one of the above posts): We can see that memory pressure is high because there are instances of "direct" memory access: This is where kswapd was unable to free up inactive memory fast enough, and the memory was actually blocked waiting for allocation while the freeing happened. This is generally bad and indicates memory pressure. My suggestion in the other thread is to change the kswapd watermarks, which force it to be more aggressive when it comes to reclaiming memory. I don't think this is possible at a cgroup level, its a system-wide setting. |
Beta Was this translation helpful? Give feedback.
-
|
Funny, after weeks of thinking what is the problem and discovering its OOM I see that its actually very common issue :D #18502 I will post here my own recap what I experienced:
What I see, normally single frigate.capture process consumes about 200MB and whole Frigate container consumes about 3GB. dmesgDid anyone discover what mechanism in code actually causes the high memory consumption? |
Beta Was this translation helpful? Give feedback.
-
|
#19110 may improve what you are seeing |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much. But since you created #19110 you managed to recreate the problem yourself? From the pull request description I see you likely have good idea what was actually happening. |
Beta Was this translation helpful? Give feedback.
-
|
No, I'm not able to reproduce it, I just worked through and figured out an idea of what may be happening based on various user reports. |
Beta Was this translation helpful? Give feedback.
-
|
Just seen 0.16.0 Beta 4 includes fix #19110. I've got this running now and will report back after a few days if I still experience the issue. 
|
Beta Was this translation helpful? Give feedback.
-
|
I think I figured it out, or at least I am very close. First lets look how ffmpeg command looks like for As you know, your IP camera outputs for example 20fps, but for Frigate detection, only 5fps (default setting) is used. How? Because these parts in the commands above: Each frame has its own timestamp and this is how frigate determines the timing and fps. But wifi cameras have the bad habit to drop connection and be kinda unreliable. This can easily cause lost frames and from the point of view of ffmpeg skips in timestamps. These timestamps skips can cause ffmpeg to suddenly output higher fps than set above.
As then all these frames get into frigate.capture where it triggers This also explains why I did some testing by trying to change timestamps in video stream via ffmpeg in another container and then send it to Frigate. I ran the stream and then suddenly changed the timestamps 5 minute ahead so Frigate ffmpeg think it lost time. And while using Maybe some other setting for ffmpeg can also help with this problem. ChatGPT recommended Honestly, I wouldnt imagine that timestamp skip can make ffmpeg to output so many frames even if it should be limited to 5fps. But now we know where to focus. Hope I didnt make any mistakes and I am not totally wrong, but this seems to me the most likely explanation. |
Beta Was this translation helpful? Give feedback.
-
|
yes, this is the correct understanding of what happens, and the fix for OOM is to not try and drain the pipe into memory when we detect that this has happened |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 So what's the solution? Frigate doesn't support Wifi cameras either? |
Beta Was this translation helpful? Give feedback.
-
|
They are supported, just can have more issues |
Beta Was this translation helpful? Give feedback.
-
|
Ok, if someone wants to try to replicate themselves, this is as close as I got. I created separate Docker container (but you can do it in host or LXC or whatever). I got this from ChatGPT and while it works I didnt bother to learn to much about how. I iterated so much with ChatGPT and didnt really know what I am looking for that it was basically just faster doing "vibe coding". This should create artificial stream 1080p 20fps that gets outputed to rtsp://localhost:8554/mystream. First 800 frames (40 seconds) are normal and should be enough time so Frigate gets stable. After that it changes timestamps 30 seconds ahead so it basically creates a gap so confuse ffmpeg client in Frigate and make it to output more fps. So basically ffmpeg creates the video stream that goes to MediaMTX. Then it goes to Frigate. I usually pipe it through go2rtc. rtsp://192.168.1.100:48554/mystream is the IP to my RTSP in the separate docker. 48554 port outside 8554 inside. Frigate docker is best if without any hardware acceleration and I also think it can be better replicated if CPU usage is high, so I limited cpu in Docker Compose. Well, this is what I got. If someone wants to try the reproduction I am happy to help. |
Beta Was this translation helpful? Give feedback.
-
|
I'm having good luck with the latest 0.16.0 beta 4, so far no OOM problems. |
Beta Was this translation helpful? Give feedback.
-
|
I also tried 0.16.0 beta 4 over the last couple of days and it seems to have resolved the OOM Kills for me as well |
Beta Was this translation helpful? Give feedback.
-
|
Extra news for me - I will try to update to latest beta 🔥 |
Beta Was this translation helpful? Give feedback.
-
|
I've been on 0.16.0 Beta 4 since 12th June and not had a single OOM issue since. Previously it was a few times a week but zero occurrences in 3 weeks is looking promising. |
Beta Was this translation helpful? Give feedback.
-
|
Agree, looks like this works nicely. I just dont really understand how it can be caught by this check: |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
My UnRAID server has been crashing lately. The logs during the crash show the issue to be with Frigate. See the below logs.
This has happened across multiple versions of Frigate. I thought it may be a 0.15beta issue so reverted back to 0.14 and the issue persisted.
I have tried capping the memory of the container to 4G (seen commented in compose) but that didn't seem to make a difference either.
I'm yet to find a pattern to the issue, it is seemingly random.
Unfortunately getting any Frigate logs is proving difficult as the whole system is generally unresponsive until it is rebooted.
I understand ReoLink cameras can sometimes be problematic but as this is a restream of a Scrypted broadcast (used for the HomeAssistant integration) I'm not sure that would be a problem here.
Any help is appreciated.
Regards,
Jack
Version
0.15.0-cea210d
What browser(s) are you using?
No response
Frigate config file
Relevant Frigate log output
Relevant go2rtc log output
FFprobe output from your camera
Frigate stats
{"cameras":{"doorbell":{"camera_fps":10.1,"process_fps":10.1,"skipped_fps":0.0,"detection_fps":0.0,"detection_enabled":true,"pid":447,"capture_pid":449,"ffmpeg_pid":925,"audio_rms":0.0,"audio_dBFS":0.0}},"detectors":{"tensorrt":{"inference_speed":5.71,"detection_start":0.0,"pid":394}},"detection_fps":0.0,"gpu_usages":{"NVIDIA GeForce GTX 1060 6GB":{"gpu":"0.0%","mem":"5.04%","enc":"0.0%","dec":"0.0%"}},"cpu_usages":{"frigate.full_system":{"cpu":"6.5","mem":"42.7"},"1":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service"},"15":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise s6-linux-init-shutdownd"},"16":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B"},"24":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise s6rc-oneshot-runner"},"25":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise s6rc-fdholder"},"26":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise certsync"},"27":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise certsync-log"},"28":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise frigate"},"29":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise frigate-log"},"30":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise go2rtc"},"31":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise go2rtc-healthcheck"},"32":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise go2rtc-log"},"33":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise nginx"},"34":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise nginx-log"},"44":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --"},"46":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules"},"94":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate"},"95":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/certsync"},"96":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx"},"97":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc"},"105":{"cpu":"2.1","cpu_average":"1","mem":"0.1","cmdline":"/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml"},"113":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"bash ./run.user go2rtc-healthcheck"},"114":{"cpu":"1.5","cpu_average":"1","mem":"1.5","cmdline":"python3 -u -m frigate"},"131":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: master process nginx"},"177":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: worker process"},"178":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: worker process"},"179":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: worker process"},"185":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: worker process"},"195":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: cache manager process"},"314":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"bash ./run.user certsync"},"378":{"cpu":"1.1","cpu_average":"1","mem":"0.8","cmdline":"frigate.recording_manager"},"387":{"cpu":"0.3","cpu_average":"0","mem":"0.7","cmdline":"frigate.review_segment_manager"},"394":{"cpu":"0.1","cpu_average":"6","mem":"1.6","cmdline":"frigate.detector.tensorrt"},"397":{"cpu":"0.1","cpu_average":"0","mem":"2.4","cmdline":"frigate.embeddings_manager"},"418":{"cpu":"0.5","cpu_average":"0","mem":"0.8","cmdline":"frigate.output "},"423":{"cpu":"0.0","cpu_average":"0","mem":"0.1","cmdline":"/usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -threads 1 -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:"},"431":{"cpu":"0.0","cpu_average":"0","mem":"0.1","cmdline":"/usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:"},"447":{"cpu":"2.5","cpu_average":"36","mem":"0.8","cmdline":"frigate.process:doorbell"},"449":{"cpu":"1.9","cpu_average":"1","mem":"0.7","cmdline":"frigate.capture:doorbell"},"915":{"cpu":"0.6","cpu_average":"0","mem":"0.1","cmdline":"/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -user_agent FFmpeg Frigate/0.15.0-cea210d -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/doorbell?video&audio -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/doorbell@%Y%m%d%H%M%S%z.mp4"},"925":{"cpu":"2.3","cpu_average":"2","mem":"0.6","cmdline":"/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.15.0-cea210d -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8554/doorbell_sub -r 10 -vf fps=10,scale_cuda=w=640:h=480,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:"},"13667":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"sh"},"13764":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"bash"},"13933":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"sleep 60"},"13951":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"sleep 30s"}},"service":{"uptime":34716,"version":"0.15.0-cea210d","latest_version":"0.15.0","storage":{"/media/frigate/recordings":{"total":11682023.0,"used":9725586.2,"free":1954786.9,"mount_type":"fuse.shfs"},"/media/frigate/clips":{"total":11682023.0,"used":9725586.2,"free":1954786.9,"mount_type":"fuse.shfs"},"/tmp/cache":{"total":953.7,"used":5.1,"free":948.6,"mount_type":"tmpfs"},"/dev/shm":{"total":128.0,"used":23.2,"free":104.8,"mount_type":"tmpfs"}},"temperatures":{},"last_updated":1740078235},"processes":{"recording":{"pid":378},"review_segment":{"pid":387},"go2rtc":{"pid":105},"embeddings":{"pid":397}}}Install method
Docker Compose
docker-compose file or Docker CLI command
Object Detector
TensorRT
Network connection
Wired
Camera make and model
ReoLink DB_566128M5MP_W
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions