QNAP: sherpa install of Bazaar, Lidarr, Readarr etc writing to /tmp folder until it's full and then cannot access QNAP server via web interface. Login stuck on "Loading..."

[jonnypuma]

QNAP owner here.

Having issues after installing -arr applications with sherpa on my QNAP server, where the /tmp folder is getting filled up entirely, reaching 100%. This leads to no longer being able to open the QNAP web interface and being stuck on the "Loading..." screen page, and also getting error messages running CLI commands. Only way to resolve is to reboot the server with CLI command, but a while after rebooting the applications will fill up the /tmp folder again.

Running the sudo lsof /tmp/ command, it seems like the -arr applications are writing to the /tmp folder and filling it up. When I kill these screen processes for the applications, the web interfaces of the apps are no longer able to be reached. Example:
screen 31103 admin 3r FIFO 0,24 0t0 192094 /tmp/screens/S-admin/31103.OReadarr

Why are these processes being written to the /tmp folder? Can't they be written to another folder in the app qpkg folder instead?

I never had this problem when using myqnap.org qpkgs, and these screen processes never show up when I run myqnap.org qpkgs of the -arr apps. However, the myqnap.org qpkgs have their own issues, which made me switch to sherpa in the first place. Seems I might have to switch back unfortunately.

[OneCDOnly]

Hi @jonnypuma and thanks for reporting this.

Can you please advise which files are filling up your /tmp directory? It's not the screen files. Please include the file sizes so I can confirm your claim.

Yes, if you kill a screen instance, the daemon running within that instance also gets killed. So, don't do that. 😉

Thank you.

[jonnypuma]

Hi OneCDOnly,

Thanks for taking the time and replying to a novice like myself. Unfortunately, I'm linux illiterate pretty much, so apologies for my ignorance on the subject. Seems you are right about the screen processes not being the issue (see below).

However, I only experience the tmp folder becoming full after starting sherpa installed apps. It usually takes some hours to slowly fill up.

It seems it's 'none' that starts to fill up after starting the -aar applications. It will start at 79% and slowly climb until it reaches 400M. This is the output df -h right after starting the apps:
Filesystem Size Used Available Use% Mounted on
none 400.0M 317.4M 82.6M 79% / <---- this is stable at 79% until I start sherpa installed applications. 15 mins later it shows 80% used. Keeps climbing slowly.
devtmpfs 7.8G 4.0K 7.8G 0% /dev
tmpfs 64.0M 1.1M 62.9M 2% /tmp
tmpfs 7.8G 148.0K 7.8G 0% /dev/shm
tmpfs 16.0M 0 16.0M 0% /share
/dev/sdi5 7.8M 28.0K 7.8M 0% /mnt/boot_config
tmpfs 16.0M 0 16.0M 0% /mnt/snapshot/export
/dev/md9 493.5M 151.3M 342.2M 31% /mnt/HDA_ROOT
cgroup_root 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/md13 417.0M 380.7M 36.3M 91% /mnt/ext
tmpfs 32.0M 27.2M 4.8M 85% /samba_third_party
tmpfs 1.0M 0 1.0M 0% /share/external/.nd
tmpfs 1.0M 0 1.0M 0% /share/external/.cm
tmpfs 1.0M 0 1.0M 0% /mnt/hm/temp_mount
tmpfs 48.0M 40.0K 48.0M 0% /share/CACHEDEV1_DATA/.samba/lock/msg.lock
df: /mnt/ext/opt/samba/private/msg.sock: Permission denied
tmpfs 29.0M 20.4M 8.6M 70% /mnt/ext/opt/FileStation5
/dev/loop0 29.0M 20.4M 8.6M 70% /mnt/ext/opt/FileStation5
tmpfs 10.0M 0 10.0M 0% /tmp/wfm

This is the output of sudo lsof /tmp/ right after starting applications:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
php-fpm-q 1786 admin 3u REG 0,24 0 1614 /tmp/.ZendSem.CK5sCA (deleted)
php-fpm-q 1787 httpdusr 3u REG 0,24 0 1614 /tmp/.ZendSem.CK5sCA (deleted)
php-fpm-q 1791 httpdusr 3u REG 0,24 0 1614 /tmp/.ZendSem.CK5sCA (deleted)
crond 7300 admin cwd DIR 0,24 60 1674 /tmp/cron/crontabs
cs_daemon 8241 admin cwd DIR 0,24 2780 1 /tmp
python 10763 admin 1u REG 0,24 0 952 /tmp/nvs-django-o.log
python 10763 admin 4u REG 0,24 0 952 /tmp/nvs-django-o.log
python 10763 admin 5u REG 0,24 0 953 /tmp/nvs-django-e.log
python 10763 admin 7w REG 0,24 82568 2612 /tmp/nvsrequest_log
recent-fi 11660 admin cwd DIR 0,24 2780 1 /tmp
recent-fi 11662 admin cwd DIR 0,24 2780 1 /tmp
porter 14326 admin 3w REG 0,24 68710 1178 /tmp/.qid.log
pn_daemon 14692 admin 3w REG 0,24 3600 1654 /tmp/.push_notification_daemon.log
screen 14779 admin 3r FIFO 0,24 0t0 636743 /tmp/screens/S-admin/14779.Bazarr
p2pagent 15323 admin 1w REG 0,24 96 1942 /tmp/.p2pagent.log
p2pagent 15323 admin 2w REG 0,24 96 1942 /tmp/.p2pagent.log
p2pagent 15323 admin 6w REG 0,24 96 1942 /tmp/.p2pagent.log
tunnelage 15414 admin 5w REG 0,24 21933 2159 /tmp/.tunnel_agent.log
qulogd 16569 admin 10u REG 0,24 0 645512 /tmp/smb.status
smbstatus 16571 admin 1u REG 0,24 0 645512 /tmp/smb.status
qmmsd 18645 admin 3w REG 0,24 0 2259 /tmp/qmsd.log
screen 18668 admin 3r FIFO 0,24 0t0 637373 /tmp/screens/S-admin/18668.OReadarr
php-fpm-p 21258 admin 3u REG 0,24 0 2569 /tmp/.ZendSem.mtO58P (deleted)
php-fpm-p 21262 admin 3u REG 0,24 0 2569 /tmp/.ZendSem.mtO58P (deleted)
php-fpm-p 21263 admin 3u REG 0,24 0 2569 /tmp/.ZendSem.mtO58P (deleted)
ql_daemon 21987 admin 3w REG 0,24 1571 1183 /tmp/.qlicense_daemon.log
multipath 23616 admin 3wW REG 0,24 5 2726 /tmp/multipathd.pid
xl2tpcd 26620 admin 0r FIFO 0,24 0t0 2999 /tmp/xl2tpd/xl2tpc-control
screen 31530 admin 3r FIFO 0,24 0t0 639978 /tmp/screens/S-admin/31530.OLidarr
portmap 32458 admin 6u REG 0,24 0 1542 /tmp/.rf_1000_0a6e57132-5d81-4186-92ab-528bf1787033.lock (deleted)

Checking QNAP resource manager right after starting packages:
OReadarr screen process 2.3 MB <-- you are correct, seems to be stable at this size
OReadaar process 202.5 MB
bash process 3.0 MB <-- seems to be stable at this size

OLidaar screen process 2.3 MB <-- you are correct, seems to be stable at this size
OLidaar process 317.1
bash process 3.1 MB <-- seems to be stable at this size

OBazaar screen process 2.3 MB <-- you are correct, seems to be stable at this size
python 3 219.4 MB
python 3 20.8 MB

Not sure if this tells you anything at all.

[OneCDOnly]

Cheers mate, yes that helps. 👍🏽

So, it sounds like something is writing to a location it shouldn't. Based-on past experience, an application is either writing into $HOME for the internal admin user, or it's created a directory off root (/) and is writing files there. Both of these locations wouldn't be an issue with a regular Linux distro, but are problematic in QTS due to its non-persistent nature.

To figure-out which application is causing this, please start only one of the suspected sherpa QPKGs, then monitor the filesystem usage at the specific locations mentioned above. I'll include commands so you can check the usage. These will give you a quick size summary.

To examine the root filesystem (/) only:

du -hsx /

To examine $HOME for the admin user only:

du -hsx /root

Keep monitoring these locations. Does the total size used for one (or both) increase?

If-not, please start the next suspected QPKG, and monitor again.

As you can see, this is a slow process. It requires patient, methodical checks to narrow-down the culprit.

When you've found the problem application, we can move onto the next step.

[jonnypuma]

Cheers!

I will give it a go and report back, thanks again!

[jonnypuma]

Update, so I started with Bazarr, and ran du -hsx / at 1 hour intervals. Usage increased from 316MB to 326MB usage in about 3 hours and climbing. du -hsx /root was stable at 36K the entire time, so nothing is writing to that.

Then I stopped Bazarr, waited 30 mins. Ran du -hsx / - it still showed 326M used. I started Lidarr, and after 1 hour ran du -hsx / and it now showed 327MB used. Went to bed and 7 hours later it now shows 331MB used. So it's steadily climbing also with Lidarr but at a slower pace it seems.

du -hsx /root usage is stable at 36K

[jonnypuma]

Now it's up to 340MB used after another 10 hours of Lidarr running

[OneCDOnly]

Let's try looking for files modified on the root filesystem (/) in the last 10 minutes:

find / -xdev -mmin -10 -exec du -sk {} \; 2>/dev/null | sort -nk1 | tail -n30

If nothing is shown, please extend that 10 minutes to 60 minutes and search again:

find / -xdev -mmin -60 -exec du -sk {} \; 2>/dev/null | sort -nk1 | tail -n30

Can you please paste back what you see?

[OneCDOnly]

I've edited the commands to sort the results by file size, and show the largest files last. Sizes are displayed in kilobytes and only the last 30 modified files are listed.

[jonnypuma]

This is the output from last 10 min command:
0 /var/lock/.hal_util.conf.lck
0 /var/lock/lvm
0 /var/lock/smart_tmp.lck
4 /etc/bandwidth_record.conf
4 /etc/lvm/cache
4 /etc/lvm/cache/.cache
4 /var/ledvalue
4 /var/ledvalue_formatting_count
8 /var/log/hda_monitor.log
8 /var/log/network/default.log
12 /var/cache/netmgr/netmgr-redis.rdb
16 /var/log/log.nmbd
32 /var/log/network/slaacrrd.log
44 /var/log/hal_util_net.log
48 /var/cache/netmgr
68 /var/log/network/netmgr-redis.log
72 /var/lock
104 /var/log/network/default.log.bak
276 /var/log/storage_lib.log
408 /var/log/hal_lib.log
440 /var/log/hal_daemon.log
992 /var/log/network
1172 /tmp
28964 /var

And this is from the 60 min command:
4 /etc/lvm/cache
4 /etc/lvm/cache/.cache
4 /var/ledvalue
4 /var/ledvalue_formatting_count
4 /var/run/network/ncsi_wgc3001
8 /var/log/hda_monitor.log
8 /var/log/network/default.log
8 /var/log/qulog/qulogd.log
12 /var/cache/netmgr/netmgr-redis.rdb
12 /var/run/network/nm.curr
16 /var/log/log.nmbd
16 /var/log/nc.log
32 /var/log/network/gwd.log
32 /var/log/network/slaacrrd.log
44 /var/log/hal_util_net.log
48 /var/cache/netmgr
60 /var/log/network/ncaas.log
68 /var/log/network/netmgr-redis.log
72 /var/lock
80 /var/log/wget.log
104 /var/log/network/default.log.bak
148 /var/run/network
152 /var/log/log.qpkg
292 /var/log/storage_lib.log
424 /var/log/hal_lib.log
440 /var/log/hal_daemon.log
992 /var/log/network
1172 /tmp
14232 /var/log/OLidarr.screen.log
28996 /var

[jonnypuma]

I checked in the /var/log folder, and in addition to the Qlidarr.screen.log being that large and increasing, the Bazarr.screen.log there is almost 10MB in size also, which seems to correspond to the increase while Bazaar was running for about 3 hours yesterday.

[jonnypuma]

Deleted Bazarr.screen.log and it freed up 10MB again.

[OneCDOnly]

Can you please also check the total of all the screen logs?

du -hc /var/log/*.screen.log

[jonnypuma]

Sure, but since I deleted the Bazarr one it's only the Lidarr one left:
14M /var/log/OLidarr.screen.log
14M total

But I assume the total was 24MB before I deleted the Bazaar one

[OneCDOnly]

It appears the daemon logs are a lot more chatty in general operation than I thought (I don't actually use them myself in a production capacity).

I'll move the location of each screen log into a path below each application installation. And ensure it's cleared on each QPKG start.

And I'll post back when this is ready to go. 👍🏽

[jonnypuma]

Great! Thank you so much for your help and patience!

[OneCDOnly]

I've now updated the QPKGs and released them into the unstable branch only for-now.

Can you please switch to the unstable sherpa branch and try them out?

sherpa follow unstable
sherpa upgrade new
rm /var/log/*.screen.log

When you've had a chance to check the space available on root (/) again after letting the QPKGs run for some time, please let me know if the issue is resolved. I'll then release the new QPKG versions into the stable branch for everyone.

Thank you. 🤓

[jonnypuma]

Ok, I've switched and upgraded the apps, but got an error for Sonarr:
• This package action failed (and why):
upgrade OSonarr QPKG in 2 seconds (for more information: /etc/init.d/sonarr.sh log)

Started three of the apps I've got installed Bazarr, Lidarr and Readarr. Will report back the results after letting them run for some hours.

[OneCDOnly]

Can you please check the service log for OSonarr and advise what went wrong?

[jonnypuma]

Good news is that the fix seems to have worked. No increase in usage now when running Bazarr, Lidarr and Readarr. :)

Is /etc/init.d/sonarr.sh the correct one to check? Asking as it doesn't seem to contain any logs or error messages.

I tried to upgrade only Sonarr, this time there was no error but it stated that there was no new version available:

sherpa v250206-unstable

• This package action was skipped (and why):
upgrade OSonarr QPKG (no new QPKG is available)

[OneCDOnly]

That's great news, so I've now merged those packages into the stable branch. 👍🏽

To get you back on the stable branch too:

sherpa reset

You won't need to upgrade anything.

With regard to Sonarr, yes that's the service script. If you run it with log as the first parameter, like this:

/etc/init.d/sonarr.sh log

... you'll be shown each previous run session for the application. Each one starts with a grey line, and concludes with a green line if the requested action was successful. If it concludes with a red line, something went wrong. You're looking for the last red line session in the log.

[jonnypuma]

Here is the output of that command:
1 > source: osonarr.sh, action: start, time: Tue 18 Mar 2025 8:55:10 PM CET, load: 6.18
2 - package: 241223, service: 241223, library: 241223
3 = 'OSonarr' QPKG is disabled. Please enable it first with: /share/CACHEDEV1_DATA/.qpkg/OSonarr/osonarr.sh enable
4 = source: osonarr.sh, action: start, time: Tue 18 Mar 2025 8:55:10 PM CET, result: FAILED, elapsed: 192ms, load: 6.18
5 •
6 > source: sonarr.sh, action: start, time: Fri 21 Mar 2025 12:43:13 PM CET, load: 4.30
7 - package: 250321, service: 250321, library: 250321

Regarding Line 3: not possible to enable Sonarr with that command, the filname in that folder is sonarr.sh without the 'o' prefix.

[OneCDOnly]

It looks like that message was written by the previous version of OSonarr QPKG. This recently changed and now the QPKG service script is called sonarr.sh

It then seems the next action shown is also a start (after the upgraded version was installed), but this finished abnormally (there's no conclusion line to that start operation). Which is odd.

Let's proceed anyway. If you'd like to start OSonarr:

/etc/init.d/sonarr.sh enable
/etc/init.d/sonarr.sh start

[jonnypuma]

I believe that start operation was a manual start by me from the QNAP web gui, so not related to the sherpa upgrade action. Sorry, shouldd have mentioned that.

I've been running the sherpa sonarr version now for a day with no increase in the tmp folder usage.

[OneCDOnly]

Starting a QPKG by the UI also enables it, so it shouldn't be that. If the QPKG was stopped via the UI (which also disables it), then the NAS rebooted, it's normal for QTS to try and start the QPKG even though it's disabled, which would generate that message. 👍🏽

I think you're all up-and-running now. Thanks again for reporting these issues, you've helped make the project better for everyone. If there are any other problems, please don't hesitate to mention them or ask for help.

Cheers! 🤓

[jonnypuma]

Not a problem, thanks for all the help and patience!