Skip to content

Commit fae0026

Browse files
tomerdbztomerdbz
committed
issue: 4480494 Fix TSO congestion control bugs
Resolve 30-second throughput ramp-up issue for TSO-enabled TCP connections by fixing multiple congestion control bugs and implementing TSO-aware initial window and RTO recovery parameters. Problem: Applications using TSO experienced ~30 seconds of near-zero throughput before achieving line-rate. Investigation revealed multiple interrelated bugs in XLIO's congestion control implementation: 1. ssthresh Reset Bug: ssthresh was unconditionally reset to 10*MSS (14,600 bytes) during SYN-ACK processing in tcp_in.c, forcing TCP into congestion avoidance mode immediately instead of slow start. 2. Slow Start Algorithm Bug: Both LWIP and CUBIC modules implemented incorrect linear cwnd growth (cwnd += mss) instead of RFC 5681's exponential growth (cwnd += acked). This is particularly devastating for TSO where one ACK can acknowledge 64KB (44 segments), but cwnd only grew by 1 MSS (1460 bytes). 3. RTO Recovery Bug: On retransmission timeout, cwnd was reset to 1 MSS per RFC 5681, which is too conservative for modern TSO hardware that sends 256KB super-packets, causing 20+ second recovery times. Root Cause Analysis: XLIO's CUBIC implementation is based on FreeBSD CUBIC (2007-2010), which contained the slow start bug. Modern CUBIC (Linux kernel 2024, RFC 9438) uses standard TCP slow start with exponential growth. Verification against Linux kernel source (net/ipv4/tcp_cubic.c) confirmed that XLIO's behavior was incorrect and outdated. Solution: 1. Centralized Initial Window Management: Created tcp_set_initial_cwnd_ssthresh() to consistently set TSO-aware parameters across all connection initialization paths: For TSO-enabled connections: - cwnd = TSO_max_payload / 4 (64KB with default 256KB) - ssthresh = 0x7FFFFFFF (2GB - effectively unlimited) For non-TSO connections: - cwnd = RFC 3390: min(4*MSS, max(2*MSS, 4380 bytes)) - ssthresh = 10 * MSS 2. Fixed Slow Start Algorithm: Changed cwnd increment from mss to acked in both CC modules: Before (WRONG): pcb->cwnd += pcb->mss; // Linear growth After (CORRECT): pcb->cwnd += pcb->acked; // Exponential per RFC 5681 This matches modern implementations including Linux CUBIC and is critical for TSO where one ACK can acknowledge many segments. 3. TSO-Aware RTO Recovery: Created tcp_reset_cwnd_on_congestion() for TSO-aware RTO handling: For TSO connections (deviates from RFC 5681): - cwnd = 26KB (10% of TSO max) instead of 1 MSS - ssthresh = 64KB (25% of TSO max) for slow start For non-TSO connections (RFC 5681 compliant): - cwnd = 1 MSS - ssthresh = max(FlightSize/2, 2*MSS) Rationale: RFC 5681 predates aggressive TSO. Even Linux CUBIC (which follows RFC) suffers from slow RTO recovery with large TSO. Our TSO-aware approach balances fast recovery with congestion safety. Implementation Details: - Consolidated initialization logic in 6 locations: * tcp_pcb_init() - initial PCB setup * tcp_pcb_recycle() - PCB reuse after TIME_WAIT * tcp_connect() - client-side connection initiation * tcp_in.c SYN-ACK handler * lwip_conn_init() - LWIP CC module initialization * cubic_conn_init() - CUBIC CC module initialization - Fixed slow start in 2 locations: * cc_lwip.c:lwip_ack_received() - changed mss to acked * cc_cubic.c:cubic_ack_received() - changed mss to acked - Centralized RTO recovery in 2 locations: * cc_lwip.c:lwip_cong_signal() - calls helper * cc_cubic.c:cubic_cong_signal() - calls helper Performance Impact: Before: 20+ seconds to reach line-rate (200-300 Gbps) After: Line-rate achieved in <1 second Technical Rationale: 1. Very High ssthresh (2GB): Follows industry best practices for high-BDP networks, allowing slow start to discover optimal window rather than artificially limiting growth. Standard TCP behavior for modern data center deployments. 2. TSO-Independence: TSO max payload is determined by hardware capabilities, not negotiated MSS. Initial window should similarly be independent of MSS for TSO connections. 3. Initial cwnd (64KB): Balances aggressive throughput with conservative buffer management. Exceeds RFC 6928's 10 segments (~15KB) but appropriate for XLIO's controlled environment where TSO hardware handles segmentation and applications target high-throughput scenarios. 4. RTO Recovery (26KB): More conservative than initial window (25% vs 10% of TSO max) to balance fast recovery with safety. While this deviates from RFC 5681 (1 MSS), it recognizes the reality that 1460 bytes is artificially small when hardware sends 256KB super-packets. RFC Compliance Notes: - Initial window and slow start: Compliant with RFC 5681/3390 spirit, optimized for TSO hardware. - RTO recovery: Intentionally deviates from RFC 5681 for TSO to address modern hardware reality. Non-TSO connections remain RFC-compliant. Comparison to Modern Implementations: - Linux CUBIC (2024): Uses standard slow start (cwnd += acked) - Linux CUBIC RTO: Follows RFC 5681 (cwnd = 1 MSS), which causes same slow recovery issue with aggressive TSO that we fix here - FreeBSD CUBIC (2007-2010): Had slow start bug (cwnd += mss) that XLIO inherited and we now fix References: - RFC 3390: Increasing TCP's Initial Window - RFC 5681: TCP Congestion Control (slow start, RTO behavior) - RFC 6928: Increasing TCP's Initial Window (10 segments) - RFC 9438: CUBIC for Fast and Long-Distance Networks (2023, obsoletes RFC 8312) - Linux kernel: net/ipv4/tcp_cubic.c (verified modern CUBIC behavior) https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cubic.c - Excentis: "Optimizing TCP Congestion Avoidance Parameters for Gigabit Networks" - recommends very high ssthresh for fast networks Verification: - GDB debugging confirmed ssthresh overwrite during SYN-ACK processing - Post-fix: cwnd=64KB and ssthresh=2GB maintained through connection setup - Enhanced debug logs confirmed exponential cwnd growth during slow start - Throughput testing: 200-300 Gbps achieved in <1s (previously 20+s) Signed-off-by: Tomer Cabouly <[email protected]>
1 parent 6689188 commit fae0026

File tree

5 files changed

+173
-62
lines changed

5 files changed

+173
-62
lines changed

‎src/core/lwip/cc_cubic.c‎

Lines changed: 17 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,17 @@ static void cubic_ack_received(struct tcp_pcb *pcb, uint16_t type)
119119
/* Use the logic in NewReno ack_received() for slow start. */
120120
if (pcb->cwnd <= pcb->ssthresh /*||
121121
cubic_data->min_rtt_ticks == 0*/) {
122-
pcb->cwnd += pcb->mss;
122+
/* Slow start: Increment cwnd by the number of bytes acknowledged.
123+
* RFC 5681 / RFC 9438: Standard TCP slow start with exponential growth.
124+
* Modern CUBIC (Linux 2024, RFC 9438) uses the same slow start as standard TCP.
125+
*
126+
* Note: This XLIO implementation is based on FreeBSD CUBIC (2007-2010), which
127+
* originally had a bug (cwnd += mss). Fixed to match modern behavior (cwnd += acked).
128+
*
129+
* Critical for TSO where one ACK can acknowledge many segments (e.g., 64KB = 44
130+
* segments).
131+
*/
132+
pcb->cwnd += pcb->acked;
123133
} else if (cubic_data->min_rtt_ticks > 0) {
124134
ticks_since_cong = ticks - cubic_data->t_last_cong;
125135

@@ -212,24 +222,8 @@ static void cubic_cong_signal(struct tcp_pcb *pcb, uint32_t type)
212222
break;
213223

214224
case CC_RTO:
215-
/* Set ssthresh to half of the minimum of the current
216-
* cwnd and the advertised window */
217-
if (pcb->cwnd > pcb->snd_wnd) {
218-
pcb->ssthresh = pcb->snd_wnd / 2;
219-
} else {
220-
pcb->ssthresh = pcb->cwnd / 2;
221-
}
222-
223-
/* The minimum value for ssthresh should be 2 MSS */
224-
if ((u32_t)pcb->ssthresh < (u32_t)2 * pcb->mss) {
225-
LWIP_DEBUGF(TCP_FR_DEBUG,
226-
("tcp_receive: The minimum value for ssthresh %" U16_F
227-
" should be min 2 mss %" U16_F "...\n",
228-
pcb->ssthresh, 2 * pcb->mss));
229-
pcb->ssthresh = 2 * pcb->mss;
230-
}
231-
232-
pcb->cwnd = pcb->mss;
225+
/* Use centralized TSO-aware congestion recovery logic */
226+
tcp_reset_cwnd_on_congestion(pcb, true);
233227

234228
/*
235229
* Grab the current time and record it so we know when the
@@ -251,13 +245,10 @@ static void cubic_conn_init(struct tcp_pcb *pcb)
251245
{
252246
struct cubic *cubic_data = pcb->cc_data;
253247

254-
pcb->cwnd = ((pcb->cwnd == 1) ? (pcb->mss * 2) : pcb->mss);
255-
pcb->ssthresh = pcb->mss * 3;
256-
/*
257-
* Ensure we have a sane initial value for max_cwnd recorded. Without
258-
* this here bad things happen when entries from the TCP hostcache
259-
* get used.
260-
*/
248+
if (pcb->cwnd == 1) {
249+
tcp_set_initial_cwnd_ssthresh(pcb);
250+
}
251+
261252
cubic_data->max_cwnd = pcb->cwnd;
262253
}
263254

‎src/core/lwip/cc_lwip.c‎

Lines changed: 22 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -82,8 +82,19 @@ static void lwip_ack_received(struct tcp_pcb *pcb, uint16_t type)
8282
}
8383
} else if (type == CC_ACK) {
8484
if (pcb->cwnd < pcb->ssthresh) {
85-
if ((u32_t)(pcb->cwnd + pcb->mss) > pcb->cwnd) {
86-
pcb->cwnd += pcb->mss;
85+
/* Slow start: Increment cwnd by the number of bytes acknowledged.
86+
* RFC 5681: "During slow start, a TCP increments cwnd by at most SMSS
87+
* bytes for each ACK received that cumulatively acknowledges new data."
88+
* This means cwnd grows by N*MSS when N segments are ACKed, giving
89+
* exponential growth (e.g., cwnd doubles per RTT if all segments are ACKed).
90+
*
91+
* Fixed from incorrect linear growth (cwnd += mss) to proper exponential
92+
* growth (cwnd += acked). This matches modern TCP implementations including
93+
* Linux CUBIC and is critical for TSO where one ACK can acknowledge many
94+
* segments (e.g., 64KB = 44 segments at 1460 MSS).
95+
*/
96+
if ((u32_t)(pcb->cwnd + pcb->acked) > pcb->cwnd) {
97+
pcb->cwnd += pcb->acked;
8798
}
8899
LWIP_DEBUGF(TCP_CWND_DEBUG, ("tcp_receive: slow start cwnd %" U32_F "\n", pcb->cwnd));
89100
} else {
@@ -99,28 +110,9 @@ static void lwip_ack_received(struct tcp_pcb *pcb, uint16_t type)
99110

100111
static void lwip_cong_signal(struct tcp_pcb *pcb, uint32_t type)
101112
{
102-
/* Set ssthresh to half of the minimum of the current
103-
* cwnd and the advertised window */
104-
if (pcb->cwnd > pcb->snd_wnd) {
105-
pcb->ssthresh = pcb->snd_wnd / 2;
106-
} else {
107-
pcb->ssthresh = pcb->cwnd / 2;
108-
}
109-
110-
/* The minimum value for ssthresh should be 2 MSS */
111-
if ((u32_t)pcb->ssthresh < (u32_t)2 * pcb->mss) {
112-
LWIP_DEBUGF(TCP_FR_DEBUG,
113-
("tcp_receive: The minimum value for ssthresh %" U16_F
114-
" should be min 2 mss %" U16_F "...\n",
115-
pcb->ssthresh, 2 * pcb->mss));
116-
pcb->ssthresh = 2 * pcb->mss;
117-
}
118-
119-
if (type == CC_NDUPACK) {
120-
pcb->cwnd = pcb->ssthresh + 3 * pcb->mss;
121-
} else if (type == CC_RTO) {
122-
pcb->cwnd = pcb->mss;
123-
}
113+
/* Use centralized TSO-aware congestion recovery logic */
114+
bool is_rto = (type == CC_RTO);
115+
tcp_reset_cwnd_on_congestion(pcb, is_rto);
124116
}
125117

126118
static void lwip_post_recovery(struct tcp_pcb *pcb)
@@ -130,7 +122,12 @@ static void lwip_post_recovery(struct tcp_pcb *pcb)
130122

131123
static void lwip_conn_init(struct tcp_pcb *pcb)
132124
{
133-
pcb->cwnd = ((pcb->cwnd == 1) ? (pcb->mss * 2) : pcb->mss);
125+
/* Only set cwnd if it's still uninitialized (placeholder value of 1).
126+
* Otherwise, preserve the value set by tcp_set_initial_cwnd_ssthresh().
127+
*/
128+
if (pcb->cwnd == 1) {
129+
tcp_set_initial_cwnd_ssthresh(pcb);
130+
}
134131
}
135132

136133
#endif // TCP_CC_ALGO_MOD

‎src/core/lwip/tcp.c‎

Lines changed: 127 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -504,6 +504,121 @@ bool tcp_recved(struct tcp_pcb *pcb, u32_t len, bool do_output)
504504
return false;
505505
}
506506

507+
/**
508+
* Set initial congestion window and slow start threshold.
509+
*
510+
* For TSO-enabled connections:
511+
* - cwnd: 0.25 * TSO_max_payload (to fill pipe immediately)
512+
* - ssthresh: Very high value (allows slow start to discover optimal window)
513+
*
514+
* For non-TSO connections:
515+
* - cwnd: RFC 3390 compliant
516+
* - ssthresh: 10 * MSS (allows moderate growth)
517+
*
518+
* @param pcb the tcp_pcb for which to set initial window parameters
519+
*/
520+
void tcp_set_initial_cwnd_ssthresh(struct tcp_pcb *pcb)
521+
{
522+
if (tcp_tso(pcb)) {
523+
/* TSO enabled: aggressive initial window
524+
* Use 25% of TSO max payload as initial cwnd.
525+
* Default TSO max is 256KB, so initial cwnd = 64KB.
526+
* This provides enough BDP for high-speed networks.
527+
*/
528+
pcb->cwnd = pcb->tso.max_payload_sz / 4;
529+
530+
/* Set ssthresh very high (following industry best practice).
531+
* This keeps TCP in slow start mode until network conditions
532+
* dictate otherwise, allowing exponential growth to discover
533+
* the optimal sending rate.
534+
*/
535+
pcb->ssthresh = 0x7FFFFFFF; /* 2GB - effectively unlimited */
536+
} else {
537+
/* Non-TSO: RFC 3390 compliant initial window
538+
* IW = min(4*MSS, max(2*MSS, 4380 bytes))
539+
*/
540+
if (pcb->mss * 4 <= 4380) {
541+
pcb->cwnd = pcb->mss * 4;
542+
} else {
543+
pcb->cwnd = (pcb->mss * 2 > 4380) ? pcb->mss * 2 : 4380;
544+
}
545+
546+
/* Set ssthresh higher than IW to allow slow start growth */
547+
pcb->ssthresh = pcb->mss * 10;
548+
}
549+
}
550+
551+
/**
552+
* Reset cwnd and ssthresh after a congestion event (RTO or fast retransmit).
553+
*
554+
* This is more conservative than initial window settings, as a congestion
555+
* event indicates network issues.
556+
*
557+
* For TSO connections (TSO-optimized, deviates from RFC 5681):
558+
* - Reset cwnd to 10% of TSO max (26KB) instead of 1 MSS
559+
* - Set ssthresh to 25% of TSO max (64KB) to enable slow start recovery
560+
* - Rationale: RFC 5681 (cwnd=1 MSS) is too conservative for modern TSO hardware
561+
* that sends 256KB super-packets. Even Linux CUBIC suffers from slow RTO
562+
* recovery with aggressive TSO. Our approach balances fast recovery with safety.
563+
*
564+
* For non-TSO connections:
565+
* - Follow RFC 5681: cwnd = 1 MSS, ssthresh = max(FlightSize/2, 2*MSS)
566+
*
567+
* @param pcb the tcp_pcb experiencing congestion
568+
* @param is_rto true if this is an RTO event, false for fast retransmit
569+
*/
570+
void tcp_reset_cwnd_on_congestion(struct tcp_pcb *pcb, bool is_rto)
571+
{
572+
if (tcp_tso(pcb)) {
573+
/* TSO-aware recovery (deviates from RFC 5681 and modern CUBIC):
574+
*
575+
* RFC 5681 & Modern CUBIC (Linux 2024): cwnd = 1 MSS (1460 bytes)
576+
* Our TSO approach: cwnd = 26KB (10% of TSO max)
577+
*
578+
* Rationale:
579+
* - RFC 5681 was written before aggressive TSO (256KB super-packets)
580+
* - Linux CUBIC follows RFC but suffers slow RTO recovery with large TSO
581+
* - 1460 bytes is artificially small when hardware sends 256KB packets
582+
* - 26KB restart allows reasonable BDP while still being conservative
583+
*
584+
* This ensures cwnd < ssthresh after RTO, allowing exponential growth
585+
* from 26KB → 64KB during recovery. The 64KB threshold is the same as
586+
* our initial connection window, providing consistent behavior.
587+
*/
588+
u32_t tso_recovery_cwnd = pcb->tso.max_payload_sz / 10; // 26KB
589+
u32_t tso_recovery_ssthresh = pcb->tso.max_payload_sz / 4; // 64KB
590+
591+
/* For very large windows before congestion, respect TCP's halving rule */
592+
u32_t eff_wnd = LWIP_MIN(pcb->cwnd, pcb->snd_wnd);
593+
u32_t halved_wnd = eff_wnd >> 1;
594+
595+
/* Use the larger of: halved window or our TSO recovery threshold */
596+
pcb->ssthresh = LWIP_MAX(halved_wnd, tso_recovery_ssthresh);
597+
598+
if (is_rto) {
599+
/* RTO: Conservative restart */
600+
pcb->cwnd = tso_recovery_cwnd;
601+
} else {
602+
/* Fast retransmit: Less conservative, add 3*MSS for quick recovery */
603+
pcb->cwnd = pcb->ssthresh + 3 * pcb->mss;
604+
}
605+
} else {
606+
/* Non-TSO: RFC 5681 standard behavior */
607+
u32_t eff_wnd = LWIP_MIN(pcb->cwnd, pcb->snd_wnd);
608+
pcb->ssthresh = eff_wnd >> 1;
609+
610+
if (pcb->ssthresh < (u32_t)(2 * pcb->mss)) {
611+
pcb->ssthresh = 2 * pcb->mss;
612+
}
613+
614+
if (is_rto) {
615+
pcb->cwnd = pcb->mss;
616+
} else {
617+
pcb->cwnd = pcb->ssthresh + 3 * pcb->mss;
618+
}
619+
}
620+
}
621+
507622
/**
508623
* Connects to another host. The function given as the "connected"
509624
* argument will be called when the connection has been established.
@@ -553,8 +668,10 @@ err_t tcp_connect(struct tcp_pcb *pcb, const ip_addr_t *ipaddr, u16_t port, bool
553668

554669
pcb->advtsd_mss = tcp_send_mss(pcb);
555670
pcb->mss = pcb->advtsd_mss;
556-
pcb->cwnd = 1;
557-
pcb->ssthresh = pcb->mss * 10;
671+
672+
/* Set initial congestion window and slow start threshold */
673+
tcp_set_initial_cwnd_ssthresh(pcb);
674+
558675
pcb->connected = connected;
559676

560677
/* Send a SYN together with the MSS option. */
@@ -946,7 +1063,10 @@ void tcp_pcb_init(struct tcp_pcb *pcb, u8_t prio, void *container)
9461063
}
9471064
cc_init(pcb);
9481065
#endif
949-
pcb->cwnd = 1;
1066+
1067+
/* Set initial congestion window and slow start threshold */
1068+
tcp_set_initial_cwnd_ssthresh(pcb);
1069+
9501070
iss = tcp_next_iss();
9511071
pcb->snd_wl2 = iss;
9521072
pcb->snd_nxt = iss;
@@ -994,7 +1114,10 @@ void tcp_pcb_recycle(struct tcp_pcb *pcb)
9941114
#if TCP_CC_ALGO_MOD
9951115
cc_init(pcb);
9961116
#endif
997-
pcb->cwnd = 1;
1117+
1118+
/* Set initial congestion window and slow start threshold */
1119+
tcp_set_initial_cwnd_ssthresh(pcb);
1120+
9981121
iss = tcp_next_iss();
9991122
pcb->acked = 0;
10001123
pcb->snd_wl2 = iss;

‎src/core/lwip/tcp.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -405,6 +405,8 @@ void register_ip_route_mtu(ip_route_mtu_fn fn);
405405
/*Initialization of tcp_pcb structure*/
406406
void tcp_pcb_init(struct tcp_pcb *pcb, u8_t prio, void *container);
407407
void tcp_pcb_recycle(struct tcp_pcb *pcb);
408+
void tcp_set_initial_cwnd_ssthresh(struct tcp_pcb *pcb);
409+
void tcp_reset_cwnd_on_congestion(struct tcp_pcb *pcb, bool is_rto);
408410

409411
void tcp_arg(struct tcp_pcb *pcb, void *arg);
410412
void tcp_ip_output(struct tcp_pcb *pcb, ip_output_fn ip_output);

‎src/core/lwip/tcp_in.c‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -577,14 +577,12 @@ static err_t tcp_process(struct tcp_pcb *pcb, tcp_in_data *in_data)
577577
pcb->mss = LWIP_MIN(pcb->mss, tcp_send_mss(pcb));
578578
#endif /* TCP_CALCULATE_EFF_SEND_MSS */
579579

580-
/* Set ssthresh again after changing pcb->mss (already set in tcp_connect
581-
* but for the default value of pcb->mss) */
582-
pcb->ssthresh = pcb->mss * 10;
583-
#if TCP_CC_ALGO_MOD
580+
/* Re-initialize cwnd/ssthresh after MSS negotiation.
581+
* For TSO: maintain aggressive values independent of negotiated MSS.
582+
* For non-TSO: scale with negotiated MSS per RFC 3390.
583+
*/
584+
tcp_set_initial_cwnd_ssthresh(pcb);
584585
cc_conn_init(pcb);
585-
#else
586-
pcb->cwnd = ((pcb->cwnd == 1) ? (pcb->mss * 2) : pcb->mss);
587-
#endif
588586
rseg = pcb->unacked;
589587
pcb->unacked = rseg->next;
590588

0 commit comments

Comments
 (0)