Monitoring Foreman performance with PCP

[OttaviaB]

URL of the document

https://docs.theforeman.org/nightly/Monitoring_Project/index-foreman-el.html

Describe the issue

I have tried configuring the monitoring of Foreman performance with Performance Co-Pilot (PCP) by following the documentation.

When I do this on an almost empty Foreman, everything seems to work fine.

However, on systems with a little bit more load (but perfectly functional ones) it works for a few days until the redis database crashes which then causes the dynflow-sidekiq workers to also crash. The redis log is attached.

redis.log

Here is some additional info on the systems that crash:

# lscpu | grep -E '^CPU|^Thread|^Core'
CPU op-mode(s):                          32-bit, 64-bit
CPU(s):                                  16
CPU family:                              6
Thread(s) per core:                      1
Core(s) per socket:                      4

# free -h
               total        used        free      shared  buff/cache   available
Mem:            46Gi        26Gi       3.2Gi       2.3Gi        20Gi        20Gi
Swap:          4.0Gi       3.5Gi       513Mi

# df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                              4.0M     0  4.0M   0% /dev
tmpfs                                  24G 1012K   24G   1% /dev/shm
tmpfs                                 9.4G  894M  8.6G  10% /run
/dev/mapper/vg_system-lv_root          12G  7.5G  4.6G  62% /
/dev/mapper/vg_system-lv_tmp          5.0G   38M  5.0G   1% /tmp
/dev/mapper/vg_data-lv_var            4.0T  3.3T  709G  83% /var
/dev/mapper/vg_system-lv_log           15G  4.4G   11G  29% /var/log
/dev/sda1                             488M  377M   76M  84% /boot
.
.
.

Foreman version: 3.16.

Managed hosts: approx. 65.

Smart proxies: 3.

Total size of Pulp synced content:

# du -sh /var/lib/pulp/
3.1T	/var/lib/pulp/

Anything else to add? (context, suggestions for a fix, etc.)

I am not sure if this is the cause for the above issues, but:

By configuring PCP like in the documentation, pmproxy writes its data to the same redis database as Foreman. Couldn't this be problematic?

See also my previous post in the Foreman community forum: https://community.theforeman.org/t/monitoring-foreman-performance-with-pcp/46112/3