[Config Support]: tmpfs /tmp/cache fills up and doesn't release space, causing recording failures #20758
Unanswered
0xjams
asked this question in
Config Support
[Config Support]: tmpfs /tmp/cache fills up and doesn't release space, causing recording failures
#20758
Replies: 10 comments 7 replies
-
|
(1), (2), (3), (4), (5), (6), (7) Analysis of Your tmpfs IssueBased on your symptoms and the available documentation, here's what's happening: The Core Problem: Orphaned File HandlesThe 7.5GB of "ghost" space in Root Cause: go2rtc Codec MismatchYour go2rtc logs show a critical issue: For the Configuration IssuesProblem 1: Audio configuration mismatch output_args:
record: preset-record-generic # This preset doesn't handle audio
inputs:
- path: rtsp://127.0.0.1:8554/patio_lateral_cam
roles:
- record
- audio # But you're trying to use audio hereProblem 2: go2rtc audio transcoding patio_lateral_cam:
- rtsp://REDACTED:[email protected]:554/cam/realmonitor?channel=5&subtype=0
- ffmpeg:patio_lateral_cam#audio=opusBut your camera likely doesn't provide AAC audio, and you're not handling it correctly(1). SolutionsFix 1: Correct Audio ConfigurationIf you want audio for patio_lateral_cam:
ffmpeg:
output_args:
record: preset-record-generic-audio-copy # Change thisFix 2: Fix go2rtc Audio HandlingIf your NVR doesn't provide AAC audio, update your go2rtc config to properly transcode: patio_lateral_cam:
- rtsp://REDACTED:[email protected]:554/cam/realmonitor?channel=5&subtype=0
- "ffmpeg:patio_lateral_cam#video=copy#audio=copy#audio=aac"Or if you don't need audio at all, remove the audio transcoding and role: patio_lateral_cam:
- rtsp://REDACTED:[email protected]:554/cam/realmonitor?channel=5&subtype=0 # Remove audio lineAnd remove the inputs:
- path: rtsp://127.0.0.1:8554/patio_lateral_cam
roles:
- record # Remove audio roleFix 3: Increase tmpfs SizeYour calculation for tmpfs may be insufficient. The documentation provides this formula for calculating required tmpfs size(6): # For each camera:
# (width * height * 1.5 * fps + 270480) / 1048576 * <number of cameras> + 40MB
# For your 5 recording cameras, assuming worst case 2560x1920 @ 15fps:
python -c 'print("{:.2f}MB".format(((2560 * 1920 * 1.5 * 15 + 270480) / 1048576) * 5 + 40))'Your 8GB may be too small if you have high-resolution cameras(6). Consider increasing to 16GB: tmpfs:
size: 16000000000 # 16GBAddressing Your Specific QuestionsQ: Why does tmpfs space remain allocated after FFmpeg crashes? When FFmpeg crashes before closing file handles, Linux keeps the space allocated until the process exits(6). The recording maintainer can't clean these up because the files are still "open" from the kernel's perspective(5). Q: Could the go2rtc codec mismatch be causing crashes? Yes, absolutely(1). The codec mismatch between video (H264) and audio (ANY/undefined) is causing FFmpeg to fail when trying to write segments(1). Q: Should the recording maintainer clean up orphaned segments? The maintainer does clean up corrupt segments automatically(5), but it cannot clean up files with open handles from crashed processes(5). You're seeing warnings like "Discarding a corrupt recording segment" in similar issues(7). Q: Manual garbage collection without restart? No documented method exists to force cleanup without restarting. The container restart is currently the only way to release orphaned file handles(6). Recommended Action Plan
The audio codec issue is almost certainly your root cause. Once fixed, the FFmpeg crashes should stop and tmpfs should no longer fill up with orphaned files. ** 📚 Sources:
Have another question? Just tag @inkeep. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Increasing the size of tmpfs is just delaying the eventual crash. I started at 2GB, and every time I increased the size, the problem would still occur eventually. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
We will need to see a full copy of logs starting at Frigate startup |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Here are the docker logs since the last time the cron job restarted Frigate: Here are the go2rtc logs too: Here are some details: twice during the day, my event-related recordings stopped (regardless of the camera), and I noticed that the problem happened when my Amcrest Doorbell camera started showing connection problems. After I restarted it, Frigate recovered. I was wondering if there's a way to simply alert when it starts failing or maybe just disable the camera temporarily. I think this event is causing a domino effect. I suspect that when that camera starts failing, ffmpeg starts crashing, and the more that happens, the sooner tmpfs gets full, and it all crashes. Any suggestions? I don't know if this could be a viable fix? But I don't want to keep trying things without having an idea of what the problem truly is. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Heads up @NickM-27 I am also seeing this /tmp/cache fill up issue with Amcrest cameras. Looking at other issues on GitHub there seems to be a common thread of the cache not flushing. Happy to contribute logs if I can find them. I've restarted the system several times so there's nothing left at the moment. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hi everyone: Update: Config: Logs: Here are the Go2RTC logs in case they are needed: |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
it looks like your cameras are having some timeout issues and then it runs into some kind of crash, I have never seen this crash before. |
Beta Was this translation helpful? Give feedback.
-
|
In any case, this has been refactored in 0.17, so you can try using that unreleased version or for now work on fixing the cameras timeout issues |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much! I'll give the newer version a try, and I'll also look into the timeout issue. I'll report my results |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
After further investigation, I've discovered that the tmpfs storage exhaustion is a symptom, not the root cause. The actual issue is that Frigate's recording process ( Evidence
Kernel Oops During Process Exit: Zombie Process After Crash: # ps aux | grep frigate
root 77549 1.0 0.0 0 0 ? Zl 08:26 1:07 [frigate.recordi] <defunct>Docker Unable to Stop Container: Required Manual Intervention: kill -9 76583 # containerd-shim PID holding the zombieChain of Events
System Info
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I've seen this issue albeit without camera timeout errors on 0.16.2. Interestingly, I am testing 0.17.0-de2144f alongside 0.16.2 - using go2rtc's restreams in 0.16.2 as the input for 0.17's go2rtc install. At the point 0.16.2 started giving out of space errors, 0.17 didn't miss a beat. Happy to provide logs if it helps. The camera configs are the same for both installs. However, 0.16.2 uses PCI/USB Corals whereas the 0.17 install uses Intel iGPU only. |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone: I have an update. I tested 0.17.0-de2144f, I still got a SegFault, but the log looks different: |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
The segfault doesn't make sense - it's crashing on a cv2 function, something we don't maintain: frigate/frigate/util/services.py Line 698 in 97b29d1 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your response @hawkeye217, I started testing The pattern above has repeated frequently. It also relates to CV just like in the version where the SegFault took place. These OpenCV warnings didn't appear at first, but right now they are showing up every ~30 seconds or so. The warnings started showing up at the ~23-hour mark. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
Your camera is sending corrupt data to Frigate. If this is a wifi camera, you'll want to try to improve the connection. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! That was it; rebooting the camera fixed the warnings. That particular Amcrest doorbell model has given people a lot of headaches, from what I've read. |
Beta Was this translation helpful? Give feedback.
-
|
Great! Glad it's working for you now. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
Description:
I'm experiencing an issue where my tmpfs partition (/tmp/cache) fills up and space is not released, eventually preventing new recordings from being created. FFmpeg crashes with "No space left on device" errors, but the space remains allocated even after the crashes.
Environment:
Symptoms:
[ERROR:[email protected]] global grfmt_webp.cpp:386 write Only 0 or 9850 bytes are written
[out#0/segment @ 0x5606ab9ba100] Could not write header (incorrect codec parameters ?): No space left on device
0 /tmp/cache/[email protected]
0 /tmp/cache/[email protected]
[... many more 0-byte files ...]
80K /tmp/cache/preview_frames
7.5G /tmp/cache (total reported by du)
error="streams: codecs not matched: video:H264 => audio:ANY" stream=patio_lateral_cam
Troubleshooting completed:
Questions:
Temporary workaround:
Restarting the container releases the space, but the issue recurs. I even created a daily cron for this, but sometimes it fills up quicker than the cron's frequency
Version
0.16.2-4d58206
Frigate config file
Relevant Frigate log output
Relevant go2rtc log output
error="streams: codecs not matched: video:H264 => audio:ANY" stream=patio_lateral_camFrigate stats
No response
Operating system
UNRAID
Install method
Docker Compose
docker-compose file or Docker CLI command
Object Detector
Coral
Screenshots of the Frigate UI's System metrics pages
No response
Any other information that may be helpful
I tried using the AI on the website to debug this, but it didn't help much. I've been tweaking my config for days and nothing changes.
Beta Was this translation helpful? Give feedback.
All reactions