podman pmda and pcp containers in general only find root-managed containers

[Fodoj]

As a normal user:

5f87707b1cef  docker.io/library/redis:alpine       docker-entrypoint...  2 hours ago  Up 2 hours ago  0.0.0.0:8000->8065/tcp, 0.0.0.0:8001->8066/tcp, 0.0.0.0:3001->3001/tcp  redis
f0ad97f94c2d  docker.io/library/postgres:9.6.9     docker-entrypoint...  2 hours ago  Up 2 hours ago  0.0.0.0:8000->8065/tcp, 0.0.0.0:8001->8066/tcp, 0.0.0.0:3001->3001/tcp  postgres

As root (sudo podman ps):

2297307970dc  localhost/langtool:latest  /bin/sh -c /start...  6 days ago  Up 6 days ago  0.0.0.0:8081->8081/tcp  exciting_swirles

pminfo shows only the one running as root:

pminfo --fetch podman.container.name

result:

podman.container.name
    inst [0 or "2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844"] value "exciting_swirles"

and for containers:

pminfo --fetch containers       

result:

containers.engine
    inst [0 or "2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844"] value "podman"

containers.name
    inst [0 or "2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844"] value "exciting_swirles"

containers.pid
    inst [0 or "2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844"] value 138562

containers.cgroup
    inst [0 or "2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844"] value "/libpod-2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844.scope"

containers.state.running
    inst [0 or "2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844"] value 1

containers.state.paused
    inst [0 or "2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844"] value 0

containers.state.restarting
    inst [0 or "2297307970dcaed7b0c1e95c7cd6ae5532bd9d902f7dfcb8bf17c8c138b35844"] value 0

I did not do any specific pcp configuration, just installed default packages on my Fedora 32 Workstation.

[natoscott]

@Fodoj there's no way to query podman for other users on a system (AFAIK). The PMDA runs as root, so as you see it can access only the root users containers. We could add an option to the PMDA to run as an unprivileged user - but you will still only be able to see that user's containers.

[natoscott]

Actually, we could possibly do better than that and make pmdapodman behave like pmdaproc, such that it performs podman queries as the user running the client monitoring tool connected to pmcd.

[apitanga]

+1 to this. We are encouraging users to run rootless containers these days, so this pmda is less useful if it only reports on containers created by the root user

[andreasgerstmayr]

Actually, we could possibly do better than that and make pmdapodman behave like pmdaproc, such that it performs podman queries as the user running the client monitoring tool connected to pmcd.

What about container monitoring with the pmproxy REST API/Grafana? Ideally I would like to see all containers of all users. As a regular linux user I also see all processes of other users (unless hidepid is set).

[natoscott]

Related (thanks Marko): https://podman.io/blogs/2020/08/01/deprecate-and-remove-varlink-notice.html

[sercanyaldiz]

Hello @natoscott ,
I am using pcp-pmda-podman , version 5.3.7 together with pcp version 5.3.7 and I am unable to find non-root manged containers.

Isnt this issue resolved by: #1284 ?

[natoscott]

@natoscott yep - check you have /run/user/*/podman/podman.sock file(s) for the non-root user(s) - pmdapodman communicates over these sockets to the podman REST API. Also check there's no selinux or other security module preventing access.

[sercanyaldiz]

@natoscott the sock file is present:
srw-rw----. 0 Apr 20 09:28 /run/user/1000/podman/podman.sock

Also the podman.socket is enabled and active.
● podman.socket - Podman API Socket
Loaded: loaded (/usr/lib/systemd/system/podman.socket; enabled; vendor preset: disabled)
Active: active (listening) since Wed 2023-04-19 20:24:17 CEST; 1 week 6 days ago
Until: Wed 2023-04-19 20:24:17 CEST; 1 week 6 days ago
Triggers: ● podman.service
Docs: man:podman-system-service(1)
Listen: /run/podman/podman.sock (Stream)
CGroup: /system.slice/podman.socket
Apr 19 20:24:17 home systemd[1]: Listening on Podman API Socket.

$ pminfo --fetch containers
containers.engine
No value(s) available!
containers.name
No value(s) available!
containers.pid
No value(s) available!
containers.cgroup
No value(s) available!
containers.state.running
No value(s) available!
containers.state.paused
No value(s) available!
containers.state.restarting
No value(s) available!

I have the following running containers

podman ps -q
cf24d60248a5
7862962360ac
7b97311b3892
6b347bfb4e03
3118aeb8964b

[natoscott]

OK, so let's check the APIs the PCP code is using. Can you see what results you get from...

$ curl -s --unix-socket /run/user/1000/podman/podman.sock http://d/v1.0.0/libpod/info | jq

And if that's working, try these endpoints too (substitute in the above command from v1 onwards)

/v3.0.0/libpod/pods/json
/v3.0.0/libpod/containers/json
/v3.0.0/libpod/containers/stats?stream=false

The above is for the podman PMDA (as referenced in the PR you linked earlier).
To activate those metrics, its:

# touch /var/lib/pcp/pmdas/podman/.NeedInstall
# systemctl restart pmcd

For the 'root' PMDA which services container.* we look in a containers.json file
below /var/lib/containers/storage/overlay-containers - I assume that must exist
because you're getting the root metrics. But, hmm, I'm not sure this code will
find user containers unless they're in that location too. Can you see any file in
the users HOME (maybe below .containers or something) that looks similar?

cheers.

[sercanyaldiz]

@natoscott thank you sir! Really appreciate for helping me.

I know for sure that I installed podman pmdas by executing ./Install after dnf install pmdas-podman confused why it didnt work right away.

But I followed your instructions below and I now I do see the metrics coming in.

# touch /var/lib/pcp/pmdas/podman/.NeedInstall
# systemctl restart pmcd

● pmcd.service - Performance Metrics Collector Daemon
     Loaded: loaded (/usr/lib/systemd/system/pmcd.service; disabled; vendor preset: disabled)
     Active: active (running) since Thu 2023-05-04 20:51:46 CEST; 17min ago
       Docs: man:pmcd(1)
   Main PID: 670596 (pmcd)
      Tasks: 8 (limit: 406740)
     Memory: 25.6M
        CPU: 906ms
     CGroup: /system.slice/pmcd.service
             ├─670596 /usr/libexec/pcp/bin/pmcd -A
             ├─670600 /var/lib/pcp/pmdas/root/pmdaroot
             ├─670601 /var/lib/pcp/pmdas/proc/pmdaproc -d 3
             ├─670603 /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
             ├─670605 /var/lib/pcp/pmdas/linux/pmdalinux
             ├─670606 /var/lib/pcp/pmdas/kvm/pmdakvm -d 95
             ├─670607 python3 /var/lib/pcp/pmdas/libvirt/pmdalibvirt.python
             └─670922 /var/lib/pcp/pmdas/podman/pmdapodman -d 33
May 04 20:51:46 systemd[1]: Starting Performance Metrics Collector Daemon...
May 04 20:51:46 systemd[1]: Started Performance Metrics Collector Daemon.
May 04 20:51:46 pmcd[670645]: Installing podman PMDA ...

Is .NeedInstall a special way of installation?

When I want to fetch the container names , I do got wrong values.

$ pminfo --fetch podman.container.name
podman.container.name
    inst [0 or "626bc7b270ab95c2040a2373b24fcc870ed4da0904023fc80c621890a6061c4c"] value "626bc7b270ab95c2040a2373b24fcc870ed4da0904023fc80c621890a6061c4c"
    inst [1 or "6b347bfb4e03f1d4db2d4b0b9d3763f6cd1389044e437b9a5c4e22dd55272069"] value "626bc7b270ab95c2040a2373b24fcc870ed4da0904023fc80c621890a6061c4c"
    inst [2 or "7862962360acc1621aef51b3f23e1eed8626963000a7dd4a7d49ad579f38e1e1"] value "626bc7b270ab95c2040a2373b24fcc870ed4da0904023fc80c621890a6061c4c"
    inst [3 or "7b97311b3892557db9fa098aa0943a7987f5bd875e24d833ef259f151d3c8781"] value "626bc7b270ab95c2040a2373b24fcc870ed4da0904023fc80c621890a6061c4c"
    inst [4 or "**cf24d60248a5e8d7909ec5dd0d44d50913331c98c11e0033bab5f653bdf89eba**"] value "626bc7b270ab95c2040a2373b24fcc870ed4da0904023fc80c621890a6061c4c"

When I request the data with curl I do get the correct values;

 curl -s --unix-socket /run/user/1000/podman/podman.sock http://d/v3.0.0/libpod/containers/json | jq
[
  {
    "AutoRemove": false,
    "Command": null,
    "Created": "2023-03-30T08:34:24.799258169+02:00",
    "CreatedAt": "",
    "Exited": false,
    "ExitedAt": 1681940209,
    "ExitCode": 0,
    "Id": "**cf24d60248a5e8d7909ec5dd0d44d50913331c98c11e0033bab5f653bdf89eba**",
    "Image": "registry.redhat.io/rhel9/grafana:latest",
    "ImageID": "fd8fc14372b7b67a2bf96c620fb35cbffab4499e4c380d7d431b566fde01cf4f",
    "IsInfra": false,
    "Labels": {
      "architecture": "x86_64",
      "build-date": "2023-02-22T14:01:27",
      "com.redhat.component": "grafana-container",
      "com.redhat.license_terms": "https://www.redhat.com/agreements",
      "description": "Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus, InfluxDB and Performance Co-Pilot.",
      "distribution-scope": "public",
      "help": "cat /README.md",
      "io.buildah.version": "1.27.3",
      "io.k8s.description": "Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus, InfluxDB and Performance Co-Pilot.",
      "io.k8s.display-name": "Grafana",
      "io.openshift.expose-services": "3000:grafana",
      "io.openshift.tags": "grafana,monitoring,dashboard",
      "maintainer": "Grafana Maintainers <grafana-maint@redhat.com>",
      "name": "rhel9/grafana",
      "release": "110",
      "summary": "Grafana is an open source, feature rich metrics dashboard and graph editor",
      "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhel9/grafana/images/7-110",
      "usage": "podman run -d --name grafana -p 3000:3000 -v grafana-data:/var/lib/grafana registry.redhat.io/rhel9/grafana",
      "vcs-ref": "9d2153d82ddf485a5c15318366cf5a98affde1d4",
      "vcs-type": "git",
      "vendor": "Red Hat, Inc.",
      "version": "7"
    },
    "Mounts": [
      "/var/lib/grafana"
    ],
    "**Names**": [
      "grafana"
    ],
    "Namespaces": {},
    "Networks": [],
    "Pid": 79670,
    "Pod": "",
    "PodName": "",
    "Ports": [
      {
        "host_ip": "",
        "container_port": 3000,
        "host_port": 3000,
        "range": 1,
        "protocol": "tcp"
      }
    ],
    "Size": null,
    "StartedAt": 1681983068,
    "State": "running",
    "Status": ""
  },

Seems like I found a bug. After some time , I am getting the following errors in journalog

May 04 21:30:07 systemd-coredump[681449]: [🡕] Process 681394 (pmdapodman) of user 0 dumped cor>
░░ Subject: Process 681394 (pmdapodman) dumped core
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ Documentation: man:core(5)
░░
░░ Process 681394 (pmdapodman) crashed and dumped core.
░░
░░ This usually indicates a programming error in the crashing program and
░░ should be reported to its vendor as a bug.
May 04 21:31:06 systemd-coredump[681698]: [🡕] Process 681645 (pmdapodman) of user 0 dumped cor>
░░ Subject: Process 681645 (pmdapodman) dumped core
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ Documentation: man:core(5)
░░
░░ Process 681645 (pmdapodman) crashed and dumped core.
░░
░░ This usually indicates a programming error in the crashing program and
░░ should be reported to its vendor as a bug.