tips for diagnosing podman speed issues

[open768]

hi, im relatively new to podman and have joined a project where it is used in a ci/cd pipeline to build docker images.

one of the issues is that sometimes podman becomes incredibly slow. Even when doing a simple podman ps, it can take minutes.

is this a known issue? I'm not finding much in the way of resources on how to diagnose podman when experiencing these delays and would appreciate some pointers to resources that allow me to do a root cause analysis.

is there perhaps a diagnostics tool for podman that would identify problems?

[Luap99]

see #26041 and #25862 for example, most likely some process is holding the lock for a long time

[open768]

thanks @Luap99 , it seems strange that strace (as discussed in #26041) should be used for diagnostics, is there a podman specific diagnostics command that would help in situations like this?

[baude]

i believe podman system locks might show some things. that said, other podman commands may also require a lock, therefore making things even more confusing ? the biggest offender we seen are when build is doing a commit because that takes a while. we are currently working on a possible solution for that one.

[open768]

$ podman system locks

No lock conflicts have been detected.

[open768]

even checking system locks takes a large amount of time:

$time podman system locks

No lock conflicts have been detected.

WARN[0152] Found incomplete layer "2712ba.......", deleting it

real 3m27.612s
user 0m0.094s
sys 0m0.128s