RPi3B+ PXE Early Terminate to nowhere

[Lukanite]

The Raspberry Pi 3B+ sometimes fails to boot, and performs an early terminate. However, this early terminate is sent to the wrong port, never gets picked up by the server, so the server tries to resend packets later (the Pi does request some other files which are successfully served). The Pi isn't listening, though, and due to incomplete files cannot boot. Here's an example:

File being terminated is kernel7.img
Server is communicating with Pi on port 44059, but Pi somehow decides to terminate the transfer by trying to send a packet to port 49171

Setup:

• RPi3B+ with official power supply
• Bootcode.bin (SHA256 ending in DED0)
• Managed switches with DHCP proxy server
• DNSMasq 2.79 TFTP server

[maxnet]

Are you capturing on the dnsmasq server, or on (a mirror port of) the Ethernet switch?

If the last, it could be related to #991
Bootcode does not deal well with with any packet loss.
If the dnsmasq server did send data block # 3664, but it got dropped somewhere along the way, the bootcode will give up waiting after a second, and continue downloading the next file.

Can see if this is the case by (also) capturing on the server, as that do should show all packets it tries to send, even if they get dropped later due to congested links or other reasons.

[Lukanite]

This was captured on a mirror port connected to the Raspberry Pi, and judging by the timings I think that could be right - I'll have to do more testing.

Even then, though, I'm surprised that it doesn't even "give up" on the right port :P

[Lukanite]

Yep, it's caused by a packet that the server tries to send but the Pi never receives.

Testing was fun because it never occurred after 5PM - had to test it when everyone else was in the office :P

[JamesH65]

@ghollingworth FYI

[futzle]

I still see this behaviour for (current) bootcode.bin from commit 9f49835 (SHA256 sum of file ends in be46). It feels like this is actually multiple issues:

• When it times out, Pi sends "early terminate" packet to wrong port (it seems to be using the source port of the TFTP connection, not the destination port). Result: the TFTP server doesn't see the termination, retries sending the packet and eventually gives up.
• When it times out, Pi does not retry TFTP of the file. Result: the Pi stops trying to boot.

I don't know if fixing the first issue will help with packet loss. I suspect that retrying needs to be implemented on the Pi.

At the moment I simply power cycle the Pi (from the PoE switch; I'm not a masochist) until it's lucky enough to boot the kernel. From then it's fine.

[arcivanov]

I have a sneaking suspicion it's the same bug in all of boot netcode in all models.

raspberrypi/rpi-eeprom#74 (comment)