Hello,
While experimenting with bbr protocol, I manipulated the network conditions by maintaining a high RTT for about one second before abruptly reducing it. Some packets sent during the high RTT phase experienced long delays in reaching the destination, while later packets, benefiting from the lower RTT, arrived earlier. This out-of-order arrival triggered the receiver to generate duplicate acknowledgments (dup ACKs). Due to the low RTT, these dup ACKs quickly reached the sender. Upon receiving three dup ACKs, the sender initiated a fast retransmission for an earlier packet that was not lost but was simply taking longer to arrive. Interestingly, despite the fast-retransmitted packet experienced a lower RTT, the original delayed packet still arrived first. When the receiver received this packet, it sent an ACK for the next packet in sequence. However, upon later receiving the fast-retransmitted packet, an issue arose in its logic for updating the acknowledgment number. As a result, even after the next expected packet was received, the acknowledgment number was not updated correctly. The receiver continued sending dup ACKs, ultimately forcing bbr into the retransmission timeout (RTO) phase.
I generated this issue in linux kernel version 5.15.0-117-generic with Ubuntu 20.04. I attempted to confirm whether the issue persists with the latest Linux kernel. However, I discovered that the behavior of bbr has changed in the most recent kernel version, where it now sends chunks of packets instead of sending them one by one over time. As a result, I was unable to reproduce the specific sequence of events that triggered the bug we identified. Consequently, I could not confirm whether the bug still exists in the latest kernel.
I believe that the issue (if still exists) will have to be resolved in the location net/ipv4/tcp_input.c or something like that. There are so many authors here that I do not know who to CC here. So, sending this email to you. Sorry if this is not the best way to report this issue.
Thanks Shehab
________________________________________ From: Ahmed, Shehab Sarar Sent: Saturday, February 1, 2025 1:01 PM To: stable@vger.kernel.org Cc: regressions@lists.linux.dev Subject: TCP Fast Retransmission Issue
Hello,
While experimenting with bbr protocol, I manipulated the network conditions by maintaining a high RTT for about one second before abruptly reducing it. Some packets sent during the high RTT phase experienced long delays in reaching the destination, while later packets, benefiting from the lower RTT, arrived earlier. This out-of-order arrival triggered the receiver to generate duplicate acknowledgments (dup ACKs). Due to the low RTT, these dup ACKs quickly reached the sender. Upon receiving three dup ACKs, the sender initiated a fast retransmission for an earlier packet that was not lost but was simply taking longer to arrive. Interestingly, despite the fast-retransmitted packet experienced a lower RTT, the original delayed packet still arrived first. When the receiver received this packet, it sent an ACK for the next packet in sequence. However, upon later receiving the fast-retransmitted packet, an issue arose in its logic for updating the acknowledgment number. As a result, even after the next expected packet was received, the acknowledgment number was not updated correctly. The receiver continued sending dup ACKs, ultimately forcing bbr into the retransmission timeout (RTO) phase.
I generated this issue in linux kernel version 5.15.0-117-generic with Ubuntu 20.04. I attempted to confirm whether the issue persists with the latest Linux kernel. However, I discovered that the behavior of bbr has changed in the most recent kernel version, where it now sends chunks of packets instead of sending them one by one over time. As a result, I was unable to reproduce the specific sequence of events that triggered the bug we identified. Consequently, I could not confirm whether the bug still exists in the latest kernel.
I believe that the issue (if still exists) will have to be resolved in the location net/ipv4/tcp_input.c or something like that. There are so many authors here that I do not know who to CC here. So, sending this email to you. Sorry if this is not the best way to report this issue.
Thanks Shehab
On 25/02/01 07:09PM, Ahmed, Shehab Sarar wrote:
Hello,
Hello,
While experimenting with bbr protocol, I manipulated the network conditions by maintaining a high RTT for about one second before abruptly reducing it. Some packets sent during the high RTT phase experienced long delays in reaching the destination, while later packets, benefiting from the lower RTT, arrived earlier. This out-of-order arrival triggered the receiver to generate duplicate acknowledgments (dup ACKs). Due to the low RTT, these dup ACKs quickly reached the sender. Upon receiving three dup ACKs, the sender initiated a fast retransmission for an earlier packet that was not lost but was simply taking longer to arrive. Interestingly, despite the fast-retransmitted packet experienced a lower RTT, the original delayed packet still arrived first. When the receiver received this packet, it sent an ACK for the next packet in sequence. However, upon later receiving the fast-retransmitted packet, an issue arose in its logic for updating the acknowledgment number. As a result, even after the next expected packet was received, the acknowledgment number was not updated correctly. The receiver continued sending dup ACKs, ultimately forcing bbr into the retransmission timeout (RTO) phase.
I generated this issue in linux kernel version 5.15.0-117-generic with Ubuntu 20.04. I attempted to confirm whether the issue persists with the latest Linux kernel. However, I discovered that the behavior of bbr has changed in the most recent kernel version, where it now sends chunks of packets instead of sending them one by one over time. As a result, I was unable to reproduce the specific sequence of events that triggered the bug we identified. Consequently, I could not confirm whether the bug still exists in the latest kernel.
I believe that the issue (if still exists) will have to be resolved in the location net/ipv4/tcp_input.c or something like that. There are so many authors here that I do not know who to CC here. So, sending this email to you. Sorry if this is not the best way to report this issue.
does this cause problems for real applications? To me it sounds a bit like a constructed issue, but I'm also not really proficient about the stack mentioned about :p
I'm asking because this is important for how we treat this report, see the "Reporting Regression"[0] document for more details on what we consider to be an regression.
Thanks Shehab
Cheers, Christian
On 25/02/02 10:26AM, Christian Heusel wrote:
On 25/02/01 07:09PM, Ahmed, Shehab Sarar wrote:
Hello,
Hello,
While experimenting with bbr protocol, I manipulated the network conditions by maintaining a high RTT for about one second before abruptly reducing it. Some packets sent during the high RTT phase experienced long delays in reaching the destination, while later packets, benefiting from the lower RTT, arrived earlier. This out-of-order arrival triggered the receiver to generate duplicate acknowledgments (dup ACKs). Due to the low RTT, these dup ACKs quickly reached the sender. Upon receiving three dup ACKs, the sender initiated a fast retransmission for an earlier packet that was not lost but was simply taking longer to arrive. Interestingly, despite the fast-retransmitted packet experienced a lower RTT, the original delayed packet still arrived first. When the receiver received this packet, it sent an ACK for the next packet in sequence. However, upon later receiving the fast-retransmitted packet, an issue arose in its logic for updating the acknowledgment number. As a result, even after the next expected packet was received, the acknowledgment number was not updated correctly. The receiver continued sending dup ACKs, ultimately forcing bbr into the retransmission timeout (RTO) phase.
I generated this issue in linux kernel version 5.15.0-117-generic with Ubuntu 20.04. I attempted to confirm whether the issue persists with the latest Linux kernel. However, I discovered that the behavior of bbr has changed in the most recent kernel version, where it now sends chunks of packets instead of sending them one by one over time. As a result, I was unable to reproduce the specific sequence of events that triggered the bug we identified. Consequently, I could not confirm whether the bug still exists in the latest kernel.
I believe that the issue (if still exists) will have to be resolved in the location net/ipv4/tcp_input.c or something like that. There are so many authors here that I do not know who to CC here. So, sending this email to you. Sorry if this is not the best way to report this issue.
does this cause problems for real applications? To me it sounds a bit like a constructed issue, but I'm also not really proficient about the stack mentioned about :p
I'm asking because this is important for how we treat this report, see the "Reporting Regression"[0] document for more details on what we consider to be an regression.
Thanks Shehab
Cheers, Christian
(forgot the link)
[0]: https://docs.kernel.org/admin-guide/reporting-regressions.html#what-is-a-reg...
linux-stable-mirror@lists.linaro.org
-
Ahmed, Shehab Sarar -
Christian Heusel