-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Description
Issue Description
podman version
Client: Podman Engine
Version: 4.9.3
API Version: 4.9.3
Go Version: go1.22.2
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64
dpkg-query -W podman
podman 4.9.3+ds1-1ubuntu0.2
uname -a
Linux amnezia-vpn-gw 6.8.0-87-generic #88-Ubuntu SMP PREEMPT_DYNAMIC Sat Oct 11 09:28:41 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 24.04 with podman + pasta networking, wireguard server running under rootless user. In all 3 cases WG tunnel is working without any issues, but there's a different routing behavior between 2 pasta-cases.
Works fine with slirp4netns and with passt 2025_09_19.623dbf6-58-g2c6590d, but not with 0.0~git20240220.1e6f92b-1
tap0 interface name in container is maintained across all the cases to keep iptables rule working - just its name.
-A POSTROUTING -s 10.12.12.0/24 -o tap0 -j MASQUERADE
Steps to reproduce the issue
podman run --detach --replace --name amnez-test --rm --network "pasta:-I,tap0,-U,auto,-u,3400:51820" -v amnez-test:/etc/amnezia/amneziawg/ --cap-add net_admin --sysctl net.ipv4.conf.all.src_valid_mark=1 --sysctl net.ipv4.ip_forward=1 test:latesttest container image is actually custom-built Wireguard based on latest Alpine with setcap cap_net_admin=eip awg and setcap cap_net_admin=eip $(readlink -f /usr/sbin/iptables) and some scripts (not published yet due to this issue I did hit), but I believe the issue lays in Ubuntu's stock pasta
Describe the results you received
Works fine with slirp4netns (net_raw isn't needed actually - only added for testing):
podman run --detach --replace --name amnez-test --rm -p 3400:51820/udp -v amnez-test:/etc/amnezia/amneziawg/ --cap-add net_admin --cap-add net_raw --sysctl net.ipv4.conf.all.src_valid_mark=1 --sysctl net.ipv4.ip_forward=1 test:latestWorks a bit worse with latest compiled passt - consumes a lot of CPU (have perf recording) - passt 2025_09_19.623dbf6-58-g2c6590d:
podman run --detach --replace --name amnez-test --rm --network "pasta:-I,tap0,-U,auto,-u,3400:51820" -v amnez-test:/etc/amnezia/amneziawg/ --cap-add net_admin --sysctl net.ipv4.conf.all.src_valid_mark=1 --sysctl net.ipv4.ip_forward=1 test:latestDoesn't work with 0.0~git20240220.1e6f92b-1 version installed from Ubuntu 24.04 repo - in container I see only outgoing packets to WAN, but nothing returns:
passt --version
passt unknown version
apt info passt
Package: passt
Version: 0.0~git20240220.1e6f92b-1I believe there's some issue with conntrack probably - it seems returning packets are dropped as invalid with 0.0~git20240220.1e6f92b-1 pasta.
Attached two pcaps reflecting curl https://google.com/ - the only difference is passt version to run them.
not_working_pasta_git20240220.pcap.gz
working_pasta_2025_09_19.623dbf6-58-g2c6590d.pcap.gz
Describe the results you expected
Proper routing with 0.0~git20240220.1e6f92b-1 version - maybe some different config is required. CPU issues to solve - currently in this specific case pasta has less performance than slirp4netns.
podman info output
host:
arch: amd64
buildahVersion: 1.33.7
cgroupControllers:
- cpu
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon_2.1.10+ds1-1build2_amd64
path: /usr/bin/conmon
version: 'conmon version 2.1.10, commit: unknown'
cpuUtilization:
idlePercent: 99.54
systemPercent: 0.37
userPercent: 0.1
cpus: 8
databaseBackend: sqlite
distribution:
codename: noble
distribution: ubuntu
version: "24.04"
eventLogger: journald
freeLocks: 2046
hostname: amnezia-vpn-gw
idMappings:
gidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 165536
size: 65536
uidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 165536
size: 65536
kernel: 6.8.0-87-generic
linkmode: dynamic
logDriver: journald
memFree: 202928128
memTotal: 984088576
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns_1.4.0-5_amd64
path: /usr/lib/podman/aardvark-dns
version: aardvark-dns 1.4.0
package: netavark_1.4.0-4_amd64
path: /usr/lib/podman/netavark
version: netavark 1.4.0
ociRuntime:
name: crun
package: crun_1.14.1-1_amd64
path: /usr/bin/crun
version: |-
crun version 1.14.1
commit: de537a7965bfbe9992e2cfae0baeb56a08128171
rundir: /run/user/1001/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/local/bin/pasta
package: Unknown
version: |
pasta 2025_09_19.623dbf6-58-g2c6590d
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: false
path: /run/user/1001/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns_1.2.1-1build2_amd64
version: |-
slirp4netns version 1.2.1
commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.5
swapFree: 0
swapTotal: 0
uptime: 1h 52m 52.00s (Approximately 0.04 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries: {}
store:
configFile: /home/andrico/.config/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 1
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/andrico/.local/share/containers/storage
graphRootAllocated: 11221196800
graphRootUsed: 8563613696
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 37
runRoot: /run/user/1001/containers
transientStore: false
volumePath: /home/andrico/.local/share/containers/storage/volumes
version:
APIVersion: 4.9.3
Built: 0
BuiltTime: Thu Jan 1 00:00:00 1970
GitCommit: ""
GoVersion: go1.22.2
Os: linux
OsArch: linux/amd64
Version: 4.9.3Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Proxmox VM
Additional information
Samples: 69K of event 'cycles:P', Event count (approx.): 60760428494
Children Self Command Shared Object Symbol
+ 88.80% 0.21% pasta.avx2 /proc/kcore 0xffff7fff9edfb130 k [k] entry_SYSCALL_64_after_hwframe
+ 87.83% 0.60% pasta.avx2 /proc/kcore 0xffff7fff9ec3e14f k [k] do_syscall_64
+ 82.81% 0.62% pasta.avx2 /proc/kcore 0xffff7fff9da01e36 k [k] x64_sys_call
+ 33.61% 0.00% pasta.avx2 /usr/local/bin/pasta.avx2 0x6409999ee396 ! [.] 0x00006409999ee396
+ 31.31% 0.21% pasta.avx2 /proc/kcore 0xffff7fff9e8dcc61 k [k] ___sys_recvmsg
+ 30.51% 0.20% pasta.avx2 /proc/kcore 0xffff7fff9e8d97c6 k [k] sock_recvmsg
+ 30.32% 0.27% pasta.avx2 /proc/kcore 0xffff7fff9e8d99b3 k [k] ____sys_recvmsg
+ 30.18% 0.00% pasta.avx2 /usr/lib/x86_64-linux-gnu/libc.so.6 0x7e1be872be2d ! [.] 0x00007e1be872be2d
+ 30.14% 0.00% pasta.avx2 /usr/local/bin/pasta.avx2 0x6409999f2a12 ! [.] 0x00006409999f2a12
+ 29.88% 0.10% pasta.avx2 /proc/kcore 0xffff7fff9ea38c21 k [k] inet_recvmsg
+ 29.24% 0.04% pasta.avx2 /proc/kcore 0xffff7fff9e8ddc0d k [k] __x64_sys_recvmsg
+ 29.14% 0.03% pasta.avx2 /proc/kcore 0xffff7fff9e8ddb82 k [k] __sys_recvmsg
+ 27.99% 0.09% pasta.avx2 /proc/kcore 0xffff7fff9e9f0574 k [k] tcp_recvmsg
+ 27.34% 0.45% pasta.avx2 /proc/kcore 0xffff7fff9e9eef4b k [k] tcp_recvmsg_locked
+ 27.02% 0.00% pasta.avx2 /usr/lib/x86_64-linux-gnu/libc.so.6 0x7e1be8729894 ! [.] 0x00007e1be8729894
+ 26.45% 0.23% pasta.avx2 /proc/kcore 0xffffffff9e8fd20c k [k] skb_copy_datagram_iter
+ 26.13% 2.17% pasta.avx2 /proc/kcore 0xffff7fff9e8f8034 k [k] __skb_datagram_iter
+ 24.02% 0.08% pasta.avx2 /proc/kcore 0xffff7fff9deebefc k [k] __x64_sys_writev
+ 23.92% 0.06% pasta.avx2 /proc/kcore 0xffff7fff9deebde1 k [k] do_writev
+ 23.68% 0.53% pasta.avx2 /proc/kcore 0xffff7fff9e8f7e48 k [k] simple_copy_to_iter
+ 23.54% 0.40% pasta.avx2 /proc/kcore 0xffff7fff9deebb91 k [k] vfs_writev
+ 22.34% 0.08% pasta.avx2 /proc/kcore 0xffff7fff9deea459 k [k] do_iter_readv_writev
+ 22.25% 0.20% pasta.avx2 /proc/kcore 0xffff7fff9e71603f k [k] tun_chr_write_iter
+ 21.63% 0.39% pasta.avx2 /proc/kcore 0xffff7fff9e715cac k [k] tun_get_user
+ 20.11% 19.90% pasta.avx2 /proc/kcore 0xffffffff9e26e578 k [k] _copy_to_iter
+ 19.89% 0.00% pasta.avx2 /usr/local/bin/pasta.avx2 0x6409999f1f2f ! [.] 0x00006409999f1f2f
+ 18.63% 0.20% pasta.avx2 /proc/kcore 0xffff7fff9e71085d k [k] tun_rx_batched
+ 17.96% 0.12% pasta.avx2 /proc/kcore 0xffff7fff9e917321 k [k] netif_receive_skb
+ 17.64% 0.09% pasta.avx2 /proc/kcore 0xffff7fff9e917245 k [k] __netif_receive_skb
+ 17.34% 0.07% pasta.avx2 /proc/kcore 0xffff7fff9e9171c1 k [k] __netif_receive_skb_one_core
+ 17.04% 0.39% pasta.avx2 /proc/kcore 0xffff7fff9e9d5211 k [k] ip_rcv
+ 16.34% 0.19% pasta.avx2 /proc/kcore 0xffff7fff9e8dd4ca k [k] ___sys_sendmsg
+ 15.31% 0.18% pasta.avx2 /proc/kcore 0xffff7fff9e8d9fae k [k] ____sys_sendmsg
+ 15.31% 0.00% pasta.avx2 [unknown] 0 ! [k] 0000000000000000
+ 14.99% 0.12% pasta.avx2 /proc/kcore 0xffff7fff9ea38ab2 k [k] inet_sendmsg
+ 14.57% 0.00% pasta.avx2 /usr/lib/x86_64-linux-gnu/libc.so.6 0x7e1be872bf97 ! [.] 0x00007e1be872bf97
+ 14.22% 0.02% pasta.avx2 /proc/kcore 0xffff7fff9e8dd9d1 k [k] __x64_sys_sendmmsg
+ 14.11% 0.10% pasta.avx2 /proc/kcore 0xffff7fff9e8dd83e k [k] __sys_sendmmsg
+ 12.46% 0.34% pasta.avx2 /proc/kcore 0xffff7fff9ea27852 k [k] udp_sendmsg
+ 12.46% 0.21% pasta.avx2 /proc/kcore 0xffff7fff9e9da2af k [k] ip_output
+ 11.64% 0.14% pasta.avx2 /proc/kcore 0xffff7fff9e9da149 k [k] ip_finish_output
+ 11.47% 0.20% pasta.avx2 /proc/kcore 0xffff7fff9e9da046 k [k] __ip_finish_output
+ 11.07% 0.41% pasta.avx2 /proc/kcore 0xffffffff9e9de0a9 k [k] ip_finish_output2
+ 10.38% 0.43% pasta.avx2 /proc/kcore 0xffff7fff9e9151db k [k] __dev_queue_xmit
+ 9.80% 0.00% pasta.avx2 [unknown] 0x100000016 ! [k] 0x0000000100000016
+ 8.40% 0.13% pasta.avx2 /proc/kcore 0xffff7fff9ea2188c k [k] udp_send_skb
+ 8.21% 0.05% pasta.avx2 /proc/kcore 0xffff7fff9e9dcb28 k [k] ip_send_skb
+ 8.10% 0.41% pasta.avx2 /proc/kcore 0xffff7fff9e9d6c72 k [k] ip_forward
+ 7.39% 0.17% pasta.avx2 /proc/kcore 0xffff7fff9e914bb2 k [k] dev_hard_start_xmit
+ 7.26% 0.00% pasta.avx2 /usr/local/bin/pasta.avx2 0x6409999c7b6c ! [.] 0x00006409999c7b6c
+ 7.03% 0.48% pasta.avx2 /proc/kcore 0xffff7fff9e9c6bf3 k [k] nf_hook_slow
+ 5.93% 0.06% pasta.avx2 /proc/kcore 0xffff7fff9e9d6674 k [k] ip_forward_finish
+ 5.21% 0.00% pasta.avx2 /usr/lib/x86_64-linux-gnu/libc.so.6 0x7e1be871ba91 ! [.] 0x00007e1be871ba91
+ 5.10% 0.07% pasta.avx2 /proc/kcore 0xffff7fff9e9213ba k [k] neigh_connected_output