From: Leon Romanovsky leonro@nvidia.com
P2PDMA documentation describes ACS controls as path-wide, although Request and Completion controls apply to different transaction directions at the path divergence. Functions sharing one PCIe link may instead have no internal peer route even with redirect controls clear.
Document the Requester and Completer roles, the directional controls and the forwarding requirements at shared ancestors for host routes. Explain that unusable forwarding leaves direct routes available, while clearing redirect controls cannot create an unsupported internal sibling route.
Signed-off-by: Leon Romanovsky leonro@nvidia.com --- Documentation/admin-guide/kernel-parameters.txt | 11 ++++---- Documentation/driver-api/pci/p2pdma.rst | 35 +++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 5 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 0b3c140babd94..f7e116703ffcd 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5251,11 +5251,12 @@ Kernel parameters Specify one or more PCI devices (in the format specified above) separated by semicolons. Each device specified will have the PCI ACS - redirect capabilities forced off which will - allow P2P traffic between devices through - bridges without forcing it upstream. Note: - this removes isolation between devices and - may put more devices in an IOMMU group. + P2P Request Redirect, Completion Redirect, and + Egress Control features forced off. This may + allow P2P traffic through bridges that would + otherwise be redirected upstream. Note: this + removes isolation between devices and may + cause affected devices to share an IOMMU group. config_acs= Format: <ACS flags>@<pci_dev>[; ...] diff --git a/Documentation/driver-api/pci/p2pdma.rst b/Documentation/driver-api/pci/p2pdma.rst index 80f8fec9b0e90..91e6367151f1d 100644 --- a/Documentation/driver-api/pci/p2pdma.rst +++ b/Documentation/driver-api/pci/p2pdma.rst @@ -15,6 +15,41 @@ then based on the ACS settings the transaction can route entirely within the PCIe hierarchy and never reach the root port. The kernel will evaluate the PCIe topology and always permit P2P in these well-defined cases.
+The client is the PCIe Requester and the provider is the Completer, whether +the client reads or writes provider memory. For devices on separate links +below the same Root Port, their paths first meet in a Switch, where they +diverge. +A Request enters that Switch through the Downstream Port above the +Requester, the Requester-side port. If ACS P2P Request Redirect is enabled +there, the Request is routed upstream; otherwise it is routed back +downstream through the Downstream Port leading to the Completer, the +Completer-side port. The path of Completions from a read likewise depends on +ACS P2P Completion Redirect in the Completer-side port. An enabled Egress +Control is conservatively treated as a Request redirect. + +Below the divergence, the route toward the other branch is already upstream, +so those P2P redirect controls do not affect it. Redirect controls for the +reverse transaction directions do not affect the mapping. P2P DMA is routed +through the host bridge when either applicable port redirects. If an ACS +Control register needed for a route cannot be read, that route is rejected. + +Functions sharing one PCIe link can instead route internally within their +device. An ACS capability without Request or Completion Redirect support +means that the corresponding internal route does not exist. Device-specific +ACS quirks can report the same restriction for functions without ACS, and +SR-IOV functions without ACS cannot support internal peer traffic. These +restrictions apply only between functions sharing the link, including ARI +functions and VFs; they do not redirect traffic to a separate device. + +When traffic continues upstream beyond the divergence, ACS Upstream Forwarding +must be enabled, or provided by a device-specific quirk, at every shared +Downstream and Root Port on the way to the host bridge. This includes ports +above a shared Switch Upstream Port as well as ports shared by sibling +functions. Missing forwarding support or unreadable forwarding controls reject +the host route while leaving direct routes available. The host bridge must +also support P2P. Clearing redirect controls cannot create an internal route +that the device does not support. + This evaluation assumes clients issue strictly ordered Requests carrying an Untranslated address. Its result is not defined when clients use Relaxed Ordering or issue ATS-translated Requests because those TLP attributes can