Skip to content

podman + pasta + rootless wireguard server: works with latest passt, but not with Ubuntu 24.04 stock one #27541

@andrico21

Description

@andrico21

Issue Description

podman version
Client:       Podman Engine
Version:      4.9.3
API Version:  4.9.3
Go Version:   go1.22.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

dpkg-query -W podman
podman  4.9.3+ds1-1ubuntu0.2

uname -a
Linux amnezia-vpn-gw 6.8.0-87-generic #88-Ubuntu SMP PREEMPT_DYNAMIC Sat Oct 11 09:28:41 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Ubuntu 24.04 with podman + pasta networking, wireguard server running under rootless user. In all 3 cases WG tunnel is working without any issues, but there's a different routing behavior between 2 pasta-cases.

Works fine with slirp4netns and with passt 2025_09_19.623dbf6-58-g2c6590d, but not with 0.0~git20240220.1e6f92b-1

tap0 interface name in container is maintained across all the cases to keep iptables rule working - just its name.
-A POSTROUTING -s 10.12.12.0/24 -o tap0 -j MASQUERADE

Steps to reproduce the issue

podman run --detach --replace --name amnez-test --rm --network "pasta:-I,tap0,-U,auto,-u,3400:51820" -v amnez-test:/etc/amnezia/amneziawg/ --cap-add net_admin --sysctl net.ipv4.conf.all.src_valid_mark=1 --sysctl net.ipv4.ip_forward=1 test:latest

test container image is actually custom-built Wireguard based on latest Alpine with setcap cap_net_admin=eip awg and setcap cap_net_admin=eip $(readlink -f /usr/sbin/iptables) and some scripts (not published yet due to this issue I did hit), but I believe the issue lays in Ubuntu's stock pasta

Describe the results you received

Works fine with slirp4netns (net_raw isn't needed actually - only added for testing):

podman run --detach --replace --name amnez-test --rm -p 3400:51820/udp -v amnez-test:/etc/amnezia/amneziawg/ --cap-add net_admin --cap-add net_raw --sysctl net.ipv4.conf.all.src_valid_mark=1 --sysctl net.ipv4.ip_forward=1 test:latest

Works a bit worse with latest compiled passt - consumes a lot of CPU (have perf recording) - passt 2025_09_19.623dbf6-58-g2c6590d:

podman run --detach --replace --name amnez-test --rm --network "pasta:-I,tap0,-U,auto,-u,3400:51820" -v amnez-test:/etc/amnezia/amneziawg/ --cap-add net_admin --sysctl net.ipv4.conf.all.src_valid_mark=1 --sysctl net.ipv4.ip_forward=1 test:latest

Doesn't work with 0.0~git20240220.1e6f92b-1 version installed from Ubuntu 24.04 repo - in container I see only outgoing packets to WAN, but nothing returns:

passt --version
passt unknown version

apt info passt
Package: passt
Version: 0.0~git20240220.1e6f92b-1

I believe there's some issue with conntrack probably - it seems returning packets are dropped as invalid with 0.0~git20240220.1e6f92b-1 pasta.

Attached two pcaps reflecting curl https://google.com/ - the only difference is passt version to run them.
not_working_pasta_git20240220.pcap.gz
working_pasta_2025_09_19.623dbf6-58-g2c6590d.pcap.gz

Describe the results you expected

Proper routing with 0.0~git20240220.1e6f92b-1 version - maybe some different config is required. CPU issues to solve - currently in this specific case pasta has less performance than slirp4netns.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.54
    systemPercent: 0.37
    userPercent: 0.1
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2046
  hostname: amnezia-vpn-gw
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.8.0-87-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 202928128
  memTotal: 984088576
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/local/bin/pasta
    package: Unknown
    version: |
      pasta 2025_09_19.623dbf6-58-g2c6590d
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 1h 52m 52.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/andrico/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrico/.local/share/containers/storage
  graphRootAllocated: 11221196800
  graphRootUsed: 8563613696
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 37
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/andrico/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Proxmox VM

Additional information

Samples: 69K of event 'cycles:P', Event count (approx.): 60760428494
  Children      Self  Command     Shared Object                        Symbol
+   88.80%     0.21%  pasta.avx2  /proc/kcore                          0xffff7fff9edfb130 k [k] entry_SYSCALL_64_after_hwframe
+   87.83%     0.60%  pasta.avx2  /proc/kcore                          0xffff7fff9ec3e14f k [k] do_syscall_64
+   82.81%     0.62%  pasta.avx2  /proc/kcore                          0xffff7fff9da01e36 k [k] x64_sys_call
+   33.61%     0.00%  pasta.avx2  /usr/local/bin/pasta.avx2            0x6409999ee396     ! [.] 0x00006409999ee396
+   31.31%     0.21%  pasta.avx2  /proc/kcore                          0xffff7fff9e8dcc61 k [k] ___sys_recvmsg
+   30.51%     0.20%  pasta.avx2  /proc/kcore                          0xffff7fff9e8d97c6 k [k] sock_recvmsg
+   30.32%     0.27%  pasta.avx2  /proc/kcore                          0xffff7fff9e8d99b3 k [k] ____sys_recvmsg
+   30.18%     0.00%  pasta.avx2  /usr/lib/x86_64-linux-gnu/libc.so.6  0x7e1be872be2d     ! [.] 0x00007e1be872be2d
+   30.14%     0.00%  pasta.avx2  /usr/local/bin/pasta.avx2            0x6409999f2a12     ! [.] 0x00006409999f2a12
+   29.88%     0.10%  pasta.avx2  /proc/kcore                          0xffff7fff9ea38c21 k [k] inet_recvmsg
+   29.24%     0.04%  pasta.avx2  /proc/kcore                          0xffff7fff9e8ddc0d k [k] __x64_sys_recvmsg
+   29.14%     0.03%  pasta.avx2  /proc/kcore                          0xffff7fff9e8ddb82 k [k] __sys_recvmsg
+   27.99%     0.09%  pasta.avx2  /proc/kcore                          0xffff7fff9e9f0574 k [k] tcp_recvmsg
+   27.34%     0.45%  pasta.avx2  /proc/kcore                          0xffff7fff9e9eef4b k [k] tcp_recvmsg_locked
+   27.02%     0.00%  pasta.avx2  /usr/lib/x86_64-linux-gnu/libc.so.6  0x7e1be8729894     ! [.] 0x00007e1be8729894
+   26.45%     0.23%  pasta.avx2  /proc/kcore                          0xffffffff9e8fd20c k [k] skb_copy_datagram_iter
+   26.13%     2.17%  pasta.avx2  /proc/kcore                          0xffff7fff9e8f8034 k [k] __skb_datagram_iter
+   24.02%     0.08%  pasta.avx2  /proc/kcore                          0xffff7fff9deebefc k [k] __x64_sys_writev
+   23.92%     0.06%  pasta.avx2  /proc/kcore                          0xffff7fff9deebde1 k [k] do_writev
+   23.68%     0.53%  pasta.avx2  /proc/kcore                          0xffff7fff9e8f7e48 k [k] simple_copy_to_iter
+   23.54%     0.40%  pasta.avx2  /proc/kcore                          0xffff7fff9deebb91 k [k] vfs_writev
+   22.34%     0.08%  pasta.avx2  /proc/kcore                          0xffff7fff9deea459 k [k] do_iter_readv_writev
+   22.25%     0.20%  pasta.avx2  /proc/kcore                          0xffff7fff9e71603f k [k] tun_chr_write_iter
+   21.63%     0.39%  pasta.avx2  /proc/kcore                          0xffff7fff9e715cac k [k] tun_get_user
+   20.11%    19.90%  pasta.avx2  /proc/kcore                          0xffffffff9e26e578 k [k] _copy_to_iter
+   19.89%     0.00%  pasta.avx2  /usr/local/bin/pasta.avx2            0x6409999f1f2f     ! [.] 0x00006409999f1f2f
+   18.63%     0.20%  pasta.avx2  /proc/kcore                          0xffff7fff9e71085d k [k] tun_rx_batched
+   17.96%     0.12%  pasta.avx2  /proc/kcore                          0xffff7fff9e917321 k [k] netif_receive_skb
+   17.64%     0.09%  pasta.avx2  /proc/kcore                          0xffff7fff9e917245 k [k] __netif_receive_skb
+   17.34%     0.07%  pasta.avx2  /proc/kcore                          0xffff7fff9e9171c1 k [k] __netif_receive_skb_one_core
+   17.04%     0.39%  pasta.avx2  /proc/kcore                          0xffff7fff9e9d5211 k [k] ip_rcv
+   16.34%     0.19%  pasta.avx2  /proc/kcore                          0xffff7fff9e8dd4ca k [k] ___sys_sendmsg
+   15.31%     0.18%  pasta.avx2  /proc/kcore                          0xffff7fff9e8d9fae k [k] ____sys_sendmsg
+   15.31%     0.00%  pasta.avx2  [unknown]                            0                  ! [k] 0000000000000000
+   14.99%     0.12%  pasta.avx2  /proc/kcore                          0xffff7fff9ea38ab2 k [k] inet_sendmsg
+   14.57%     0.00%  pasta.avx2  /usr/lib/x86_64-linux-gnu/libc.so.6  0x7e1be872bf97     ! [.] 0x00007e1be872bf97
+   14.22%     0.02%  pasta.avx2  /proc/kcore                          0xffff7fff9e8dd9d1 k [k] __x64_sys_sendmmsg
+   14.11%     0.10%  pasta.avx2  /proc/kcore                          0xffff7fff9e8dd83e k [k] __sys_sendmmsg
+   12.46%     0.34%  pasta.avx2  /proc/kcore                          0xffff7fff9ea27852 k [k] udp_sendmsg
+   12.46%     0.21%  pasta.avx2  /proc/kcore                          0xffff7fff9e9da2af k [k] ip_output
+   11.64%     0.14%  pasta.avx2  /proc/kcore                          0xffff7fff9e9da149 k [k] ip_finish_output
+   11.47%     0.20%  pasta.avx2  /proc/kcore                          0xffff7fff9e9da046 k [k] __ip_finish_output
+   11.07%     0.41%  pasta.avx2  /proc/kcore                          0xffffffff9e9de0a9 k [k] ip_finish_output2
+   10.38%     0.43%  pasta.avx2  /proc/kcore                          0xffff7fff9e9151db k [k] __dev_queue_xmit
+    9.80%     0.00%  pasta.avx2  [unknown]                            0x100000016        ! [k] 0x0000000100000016
+    8.40%     0.13%  pasta.avx2  /proc/kcore                          0xffff7fff9ea2188c k [k] udp_send_skb
+    8.21%     0.05%  pasta.avx2  /proc/kcore                          0xffff7fff9e9dcb28 k [k] ip_send_skb
+    8.10%     0.41%  pasta.avx2  /proc/kcore                          0xffff7fff9e9d6c72 k [k] ip_forward
+    7.39%     0.17%  pasta.avx2  /proc/kcore                          0xffff7fff9e914bb2 k [k] dev_hard_start_xmit
+    7.26%     0.00%  pasta.avx2  /usr/local/bin/pasta.avx2            0x6409999c7b6c     ! [.] 0x00006409999c7b6c
+    7.03%     0.48%  pasta.avx2  /proc/kcore                          0xffff7fff9e9c6bf3 k [k] nf_hook_slow
+    5.93%     0.06%  pasta.avx2  /proc/kcore                          0xffff7fff9e9d6674 k [k] ip_forward_finish
+    5.21%     0.00%  pasta.avx2  /usr/lib/x86_64-linux-gnu/libc.so.6  0x7e1be871ba91     ! [.] 0x00007e1be871ba91
+    5.10%     0.07%  pasta.avx2  /proc/kcore                          0xffff7fff9e9213ba k [k] neigh_connected_output

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions