On Mon, 4 Mar 2024 18:01:49 -0800 Mina Almasry wrote:
+Intro +=====
+Device memory TCP (devmem TCP) enables receiving data directly into device +memory (dmabuf). The feature is currently implemented for TCP sockets.
+Opportunity +-----------
+A large amount of data transfers have device memory as the source and/or
s/amount/number/
+destination. Accelerators drastically increased the volume of such transfers.
s/volume/prevalence/
+Some examples include:
+- Distributed training, where ML accelerators, such as GPUs on different hosts,
- exchange data among them.
s/among them//
+- Distributed raw block storage applications transfer large amounts of data with
- remote SSDs, much of this data does not require host processing.
+Today, the majority of the Device-to-Device data transfers the network are
"Today" won't age well.
+implemented as the following low level operations: Device-to-Host copy, +Host-to-Host network transfer, and Host-to-Device copy.
+The implementation is suboptimal, especially for bulk data transfers, and can
/The implementation/The flow involving host copies/
+put significant strains on system resources such as host memory bandwidth and +PCIe bandwidth.
+Devmem TCP optimizes this use case by implementing socket APIs that enable +the user to receive incoming network packets directly into device memory.
+More Info +---------
- slides, video
- https://netdevconf.org/0x17/sessions/talk/device-memory-tcp.html
- patchset
- [RFC PATCH v3 00/12] Device Memory TCP
- https://lore.kernel.org/lkml/20231106024413.2801438-1-almasrymina@google.com...
Won't age well? :)
+Interface +=========
+Example +-------
+tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setting up +the RX path of this API.
+NIC Setup +---------
+Header split, flow steering, & RSS are required features for devmem TCP.
+Header split is used to split incoming packets into a header buffer in host +memory, and a payload buffer in device memory.
+Flow steering & RSS are used to ensure that only flows targeting devmem land on +RX queue bound to devmem.
+Enable header split & flow steering:
+::
You can put the :: at the end of the text, IIRC, like this:
Enable header split & flow steering::
- # enable header split (assuming priv-flag)
- ethtool --set-priv-flags eth1 enable-header-split on
Olek added the "set" in commit 50d73710715d ("ethtool: add SET for TCP_DATA_SPLIT ringparam"), no need for the priv flag any more.