Willem de Bruijn wrote:
Richard Gobert wrote:
{inet,ipv6}_gro_receive functions perform flush checks (ttl, flags, iph->id, ...) against all packets in a loop. These flush checks are used in all merging UDP and TCP flows.
These checks need to be done only once and only against the found p skb, since they only affect flush and not same_flow.
This patch leverages correct network header offsets from the cb for both outer and inner network headers - allowing these checks to be done only once, in tcp_gro_receive and udp_gro_receive_segment. As a result, NAPI_GRO_CB(p)->flush is not used at all. In addition, flush_id checks are more declarative and contained in inet_gro_flush, thus removing the need for flush_id in napi_gro_cb.
This results in less parsing code for non-loop flush tests for TCP and UDP flows.
To make sure results are not within noise range - I've made netfilter drop all TCP packets, and measured CPU performance in GRO (in this case GRO is responsible for about 50% of the CPU utilization).
perf top while replaying 64 parallel IP/TCP streams merging in GRO: (gro_receive_network_flush is compiled inline to tcp_gro_receive) net-next: 6.94% [kernel] [k] inet_gro_receive 3.02% [kernel] [k] tcp_gro_receive
patch applied: 4.27% [kernel] [k] tcp_gro_receive 4.22% [kernel] [k] inet_gro_receive
perf top while replaying 64 parallel IP/IP/TCP streams merging in GRO (same results for any encapsulation, in this case inet_gro_receive is top offender in net-next) net-next: 10.09% [kernel] [k] inet_gro_receive 2.08% [kernel] [k] tcp_gro_receive
patch applied: 6.97% [kernel] [k] inet_gro_receive 3.68% [kernel] [k] tcp_gro_receive
Signed-off-by: Richard Gobert richardbgobert@gmail.com
+static inline int inet_gro_flush(const struct iphdr *iph, const struct iphdr *iph2,
struct sk_buff *p, bool outer)+{
- const u32 id = ntohl(*(__be32 *)&iph->id);
- const u32 id2 = ntohl(*(__be32 *)&iph2->id);
- const u16 ipid_offset = (id >> 16) - (id2 >> 16);
- const u16 count = NAPI_GRO_CB(p)->count;
- const u32 df = id & IP_DF;
- int flush;
- /* All fields must match except length and checksum. */
- flush = (iph->ttl ^ iph2->ttl) | (iph->tos ^ iph2->tos) | (df ^ (id2 & IP_DF));
- if (outer && df)
return flush;if (flush) return 1;To be able to avoid the two flush | below? Or to avoid adding a branch
if (flush | (outer && df)) return 1;
- /* When we receive our second frame we can make a decision on if we
* continue this flow as an atomic flow with a fixed ID or if we use* an incrementing ID.*/- if (count == 1 && df && !ipid_offset)
NAPI_GRO_CB(p)->ip_fixedid = true;- if (NAPI_GRO_CB(p)->ip_fixedid && df)
return flush | ipid_offset;- return flush | (ipid_offset ^ count);
And then simply
if (NAPI_GRO_CB(p)->ip_fixedid) return ipid_offset; else return ipid_offset ^ count;Since NAPI_GRO_CB(p)->ip_fixedid is only set if DF is set on the first two segments, and df ^ id2 & IP_DF is tested above, no need to test that again.
+}
I like the idea, it is more readable. We just need to return flush, and not 1 to make it correct (since flush could be 0 while outer && df set):
if (flush | (outer && df)) return flush;
Not setting NAPI_GRO_CB(p)->ip_fixedid when flush==1 is a slight change from the previous behaviour. AFAIU it is ok since it doesn't change GRO logic - p will be flushed from gro_list and NAPI_GRO_CB becomes irrelevant. Removing the DF check is nice, I also think we can avoid a branch while keeping the code readable as follows:
return ipid_offset ^ (count * !NAPI_GRO_CB(p)->ip_fixedid);