eBPF failed on Amazon Linux 2023.9.20251105 #11379
Description
previously work version: Amazon Linux 2023.9.20250929 kernel: 6.12.46-66.121.amzn2023.x86_64
Failed on new OS version: Amazon Linux 2023.9.20251105 kernel: 6.12.53-69.119.amzn2023.x86_64
EKS version: v1.34.1-eks-c39b1d0
Containerd://2.1.4
Calico version: v3.30.4
operator version: v1.38.7
Error log: from calico node:
2025-11-13 13:37:12.316 [INFO][83] felix/maps.go 571: Loaded map file descriptor. fd=0x22 name="cali_v4_ct_nats"
2025-11-13 13:37:12.316 [INFO][83] felix/maps.go 655: Map already exists with correct parameters. name="cali_v4_ct_nats"
libbpf: Error in bpf_create_map_xattr(cali_v4_nat_fe3):Invalid argument(-22). Retrying without BTF.
libbpf: Error in bpf_create_map_xattr(cali_v4_routes):Invalid argument(-22). Retrying without BTF.
2025-11-13 13:37:12.340 [INFO][83] felix/int_dataplane.go 2805: Using BPF program-based conntrack liveness scanner. ipVersion=IPV4
2025-11-13 13:37:12.350 [INFO][83] felix/options.go 57: proxy.WithMinSyncPeriod(1s)
I1113 13:37:12.350798 83 config.go:199] "Starting service config controller"
I1113 13:37:12.350820 83 config.go:105] "Starting endpoint slice config controller"
I1113 13:37:12.350831 83 shared_informer.go:313] Waiting for caches to sync for service config
I1113 13:37:12.350838 83 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
2025-11-13 13:37:12.351 [INFO][83] felix/health.go 204: Health of component changed name="BPFEndpointManager" newReport="non-ready" oldReport="non-ready"
2025-11-13 13:37:12.351 [INFO][83] felix/bpf_ep_mgr.go 593: HostNetworkedNATMode is 2
2025-11-13 13:37:12.352 [INFO][83] felix/bpf_ep_mgr.go 3374: /proc/sys/net/ipv4/conf/all/rp_filter set to 0
2025-11-13 13:37:12.361 [INFO][83] felix/bpf_ep_mgr.go 628: Created bpfin.cali:bpfout.cali veth pair.
2025-11-13 13:37:12.363 [INFO][83] felix/maps.go 737: Loaded map file descriptor. fd=0x8 name="/sys/fs/bpf/tc/globals/cali_progs3"
2025-11-13 13:37:12.364 [INFO][83] felix/maps.go 737: Loaded map file descriptor. fd=0x9 name="/sys/fs/bpf/tc/globals/cali_jump3"
2025-11-13 13:37:12.365 [INFO][83] felix/maps.go 737: Loaded map file descriptor. fd=0xa name="/sys/fs/bpf/tc/globals/xdp_cali_progs3"
2025-11-13 13:37:12.366 [INFO][83] felix/maps.go 737: Loaded map file descriptor. fd=0xb name="/sys/fs/bpf/tc/globals/xdp_cali_jump3"
libbpf: prog 'calico_sendmsg_v46': BPF program load failed: Invalid argument
libbpf: prog 'calico_sendmsg_v46': -- BEGIN PROG LOAD LOG --
processed 5 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'calico_sendmsg_v46': failed to load: -22
libbpf: failed to load object '/usr/lib/calico/bpf/connect_balancer_no_log_co-re_v46.o'
2025-11-13 13:37:12.368 [PANIC][83] felix/int_dataplane.go 959: BPFConnTimeLBEnabled but failed to attach connect-time load balancer, bailing out. error=error loading /usr/lib/calico/bpf/connect_balancer_no_log_co-re_v46.o:error loading program: error loading object invalid argument
panic: (*logrus.Entry) 0xc00046c540
goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc00046c4d0, 0x0, {0xc00083a840, 0x52})
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:260 +0x485
github.com/sirupsen/logrus.(*Entry).Log(0xc00046c4d0, 0x0, {0xc001004db8?, 0x422ff60?, 0xc000e027a0?})
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:304 +0x48
github.com/sirupsen/logrus.(*Entry).Panic(...)
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:342
github.com/projectcalico/calico/felix/dataplane/linux.NewIntDataplaneDriver({{0xc00007e014, 0x30}, {0xc0008fe140, 0xf}, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, ...})
/go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:959 +0x4114
github.com/projectcalico/calico/felix/dataplane.StartDataplaneDriver(0xc000ebd508, 0xc0003db890, {0x0, 0x0}, 0xc000ebbf40, 0xc000ebbf50, 0xc000683180, 0x0)
/go/src/github.com/projectcalico/calico/felix/dataplane/driver.go:423 +0x2af7
github.com/projectcalico/calico/felix/daemon.Run({0x4966eb2, 0x15}, {0x5031ff8, 0x7}, {0x50457f0, 0x18}, {0x50898e0, 0x28})
/go/src/github.com/projectcalico/calico/felix/daemon/daemon.go:441 +0x2469
main.main()
/go/src/github.com/projectcalico/calico/node/cmd/calico-node/main.go:129 +0x786
W1113 13:37:13.472251 165 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice