Unable to create route for IP; the node it belongs to was not recorded in IPAM

[wrouesnel]

Expected Behavior

calico BGP mesh is not making load-balancer assigned routes available on nodes in the cluster

Current Behavior

I have a standard RKE2 cluster applied, configured with the default IP pool range.

To this I have added the following IP pool:

---
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  serviceLoadBalancerIPs:
    - cidr: 192.168.123.0/24
---
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: loadbalancer-ip-pool
spec:
  cidr: 192.168.123.0/24
  blockSize: 29
  natOutgoing: true
  disabled: false
  assignmentMode: Automatic
  allowedUses:
    - LoadBalancer

When envoy-gateway comes up with this configuration I can see it's picked up an IP from the pool, and calicoctl confirms this:

# sample of the envoy gateway service diagram
status:
  loadBalancer:
    ingress:
      - ip: 192.168.123.80
        ipMode: VIP

Calico is running with a full mesh config figuration for BGP and appears to be healthy. However, no routes for 192.168.123.80 are appearing anywhere in the cluster. The only clue I have is Felix logs which seem to imply it can't assign the route to a node:

$ kubectl calico node status
Calico process is running.

IPv4 BGP status
+-----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS   |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+-----------------+-------------------+-------+----------+-------------+
| 192.168.122.90  | node-to-node mesh | up    | 11:30:14 | Established |
| 192.168.122.192 | node-to-node mesh | up    | 11:30:14 | Established |
| 192.168.122.143 | node-to-node mesh | up    | 11:40:09 | Established |
+-----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

calico-node-mdstm calico-node 2025-11-17 11:32:23.818 [WARNING][397] felix/l3_route_resolver.go 687: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=192.168.123.80
calico-node-mdstm calico-node 2025-11-17 11:32:23.820 [INFO][397] felix/int_dataplane.go 2201: Received *proto.RouteUpdate update from calculation graph. msg=types:REMOTE_WORKLOAD dst:"192.168.123.80/29" dst_node_name:"load-balancer"
calico-node-mdstm calico-node 2025-11-17 11:46:23.911 [WARNING][1085] felix/l3_route_resolver.go 687: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=192.168.123.80
calico-node-mdstm calico-node 2025-11-17 11:46:23.916 [INFO][1085] felix/int_dataplane.go 2201: Received *proto.RouteUpdate update from calculation graph. msg=types:REMOTE_WORKLOAD dst:"192.168.123.80/29" dst_node_name:"load-balancer"

I don't understand why Felix seems to believe this node cannot be assigned this IP? The one guess I can make is that because the actual nodes are on a different subnet (192.168.122.0/24) that it's refusing too? But should this be the case? Externally I can set routing to direct packets towards the K8S nodes (which are libvirt VMs on an if-bridge at the moment)

Context

Currently I'm testing a cluster in a virtual environment where I'm trying to set it up more how I want the real deployment to go. So specifically the chain I want is:

-> 192.168.122.1 (the VM network) -> 192.168.122.220 (kubernetes master node) -> 192.168.123.80 (envoy-gateway load balancer)

This should work AFAIK with my host set with a static route for 192.168.123.0/24 via 192.168.122.220. The problem is...the Calico nodes don't have any routes for the LoadBalancer IP (I can see a lot of routes for various workloads).

Your Environment

• Calico version v3.30.2
• Calico dataplane nftables
• Orchestrator version: RKE2 v1.33.4+rke2r1
• Operating System and version: Fedora 43

[caseydavenport]

@wrouesnel thanks for raising!

First thing - I think that WARNING log is a red herring. Felix shouldn't be programming these routes into the kernel (I'll get a PR out to fix the warning log).

For LoadBalancer IPs, these are allocated by Calico and given to Services, but aren't actually programmed as routes into the Node routing tables - rather, they are handled by kube-proxy (Or Calico's BPF proxy in BPF mode) as they related to Service load balancing.

Calico will advertise the routes over BGP to external peers, but since kube-proxy handles this in iptables/nftables, there's no need to program ip routes.

You should be able to see the nftables rules kube-proxy has programmed for this IP using nft

Once Calico is peered with an external / upstream router, then the router will direct traffic to that LB IP address to the Kubernetes cluster nodes with a destination of the LB IP. The nftables rules will do the service load balancing, resolving the traffic to a pod IP and performing DNAT to get it the rest of the way.

[caseydavenport]

Marking as a bug since I think the logging behavior here isn't great

[wrouesnel]

Okay so I can see the kube-proxy routes programmed in nftables so that's working, so the mystery at this point is why routing traffic towards the nodes isn't making it into nftables for distribution (basically I can tcpdump traffic with the right destination IP off a source node, but it doesn't seem to go anywhere).

EDIT: Okay so investigating this a bit more, if I go to the node running the actual service pod then a curl for the external IP connects successfully. But any other node in the cluster does not, nor externally. So kube-proxy is working but only on that one node it seems.

EDIT 2: Okay mystery solved but I would swear the docs were very unclear about this. At least the way I was doing things, I had envoy setting up the load balancer with spec.externalTrafficPolicy: Local. Google generally suggests this should work, but I suspect the proviso of that is "IF BGP IS ADVERTISING YOUR ROUTES SO TRAFFIC HITS THE CALICO NODE".

The way I'm hacking things together (which for various reasons may be a lot like what production looks like) I just have a single route on my machine pointing to one of the test VMs (namely the head node). So it looks like kube-proxy is actually looking for a node-local backend to route to in that case but there's only a single instance running and it's not on the head node.

EDIT3: So I did try an alternative - run envoy-proxy as a daemonset, and leave the externalTrafficPolicy to "local" and this...didn't work though I couldn't really say why because it feels like it should. So Cluster routing it is while I'm in this mode.