From: Leon Romanovsky leonro@nvidia.com

P2PDMA selects a mapping without receiving the Request's ordering or Address Type attributes. Its ACS handles only strictly ordered Requests carrying an Untranslated address.

Document that the result is not defined for Relaxed Ordering or ATS-translated Requests because those TLP attributes can select different routes through the fabric.

Reviewed-by: Logan Gunthorpe logang@deltatee.com Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- Documentation/driver-api/pci/p2pdma.rst | 7 +++++++ 1 file changed, 7 insertions(+)

diff --git a/Documentation/driver-api/pci/p2pdma.rst b/Documentation/driver-api/pci/p2pdma.rst index 63cff9e4d2c9..80f8fec9b0e9 100644 --- a/Documentation/driver-api/pci/p2pdma.rst +++ b/Documentation/driver-api/pci/p2pdma.rst @@ -15,6 +15,13 @@ then based on the ACS settings the transaction can route entirely within the PCIe hierarchy and never reach the root port. The kernel will evaluate the PCIe topology and always permit P2P in these well-defined cases.

+This evaluation assumes clients issue strictly ordered Requests carrying an +Untranslated address. Its result is not defined when clients use Relaxed +Ordering or issue ATS-translated Requests because those TLP attributes can +select different routes through the fabric. Unless ACS Translation Blocking +is enabled, a Port with ACS Direct Translated P2P enabled routes a +Translated Request directly to the peer regardless of the redirect controls. + However, if the P2P transaction reaches the host bridge then it might have to hairpin back out the same root port, be routed inside the CPU SOC to another PCIe root port, or routed internally to the SOC.

From: Leon Romanovsky leonro@nvidia.com

pci_acs_flags_enabled() and the Intel SPT PCH quirk use ACS registers without checking config-space read errors. A failed read may leave control state indeterminate yet allow the device to satisfy requested isolation controls.

Return false when either ACS capability or control state cannot be read. An unknown state cannot prove isolation.

Reviewed-by: Logan Gunthorpe logang@deltatee.com Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/pci/pci.c | 3 ++- drivers/pci/quirks.c | 6 ++++-- 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index b2879a6be5f8..f7d94ecf9157 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3594,7 +3594,8 @@ static bool pci_acs_flags_enabled(struct pci_dev *pdev, u16 acs_flags) */ acs_flags &= (pdev->acs_capabilities | PCI_ACS_EC);

- pci_read_config_word(pdev, pos + PCI_ACS_CTRL, &ctrl); + if (pci_read_config_word(pdev, pos + PCI_ACS_CTRL, &ctrl)) + return false; return (ctrl & acs_flags) == acs_flags; }

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index de9bbccda21f..d5c3e6802840 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -4992,10 +4992,12 @@ static int pci_quirk_intel_spt_pch_acs(struct pci_dev *dev, u16 acs_flags) return -ENOTTY;

/* see pci_acs_flags_enabled() */ - pci_read_config_dword(dev, pos + PCI_ACS_CAP, &cap); + if (pci_read_config_dword(dev, pos + PCI_ACS_CAP, &cap)) + return 0; acs_flags &= (cap | PCI_ACS_EC);

- pci_read_config_dword(dev, pos + INTEL_SPT_ACS_CTRL, &ctrl); + if (pci_read_config_dword(dev, pos + INTEL_SPT_ACS_CTRL, &ctrl)) + return 0;

return pci_acs_ctrl_enabled(acs_flags, ctrl); }

From: Leon Romanovsky leonro@nvidia.com

P2PDMA documentation describes ACS controls as path-wide, although Request and Completion controls apply to different transaction directions and only affect peer-versus-upstream decisions at the path divergence.

Document the fixed client and provider roles, the divergence port checked for each TLP direction, and the conservative handling of unreadable ACS state. Clarify which controls disable_acs_redir changes.

Reviewed-by: Logan Gunthorpe logang@deltatee.com Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- Documentation/admin-guide/kernel-parameters.txt | 15 +++++++++------ Documentation/driver-api/pci/p2pdma.rst | 13 +++++++++++++ 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 68647ff4bdd2..bc83e07dd5fc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5291,12 +5291,15 @@ Kernel parameters disable_acs_redir=<pci_dev>[; ...] Specify one or more PCI devices (in the format specified above) separated by semicolons. - Each device specified will have the PCI ACS - redirect capabilities forced off which will - allow P2P traffic between devices through - bridges without forcing it upstream. Note: - this removes isolation between devices and - may put more devices in an IOMMU group. + Each device specified will have the PCI ACS P2P + Request Redirect, Completion Redirect, and Egress + Control features forced off. This may allow P2P + traffic through bridges that would otherwise be + redirected upstream. This may allow P2P traffic + through bridges that would otherwise be redirected + upstream and thus this removes isolation between + devices and may cause affected devices to share + an IOMMU group. config_acs= Format: <ACS flags>@<pci_dev>[; ...] diff --git a/Documentation/driver-api/pci/p2pdma.rst b/Documentation/driver-api/pci/p2pdma.rst index 80f8fec9b0e9..42b18610bf7d 100644 --- a/Documentation/driver-api/pci/p2pdma.rst +++ b/Documentation/driver-api/pci/p2pdma.rst @@ -15,6 +15,19 @@ then based on the ACS settings the transaction can route entirely within the PCIe hierarchy and never reach the root port. The kernel will evaluate the PCIe topology and always permit P2P in these well-defined cases.

+The client remains the PCIe requester when it reads or writes provider memory. +Where the paths diverge, the kernel therefore evaluates P2P Request Redirect +and Egress Control on the client-side port, and P2P Completion Redirect on the +provider-side port for completions from a read. An enabled Egress Control is +conservatively treated as a Request redirect. + +Below the divergence, the route toward the other branch is already upstream, +so those P2P redirect controls do not affect it. Redirect controls for the +reverse transaction directions do not affect the mapping. P2P DMA is routed +through the host bridge when either applicable port redirects. If an ACS +Control register cannot be read, P2P DMA is rejected because the kernel cannot +establish a usable route. + This evaluation assumes clients issue strictly ordered Requests carrying an Untranslated address. Its result is not defined when clients use Relaxed Ordering or issue ATS-translated Requests because those TLP attributes can

From: Leon Romanovsky leonro@nvidia.com

calc_map_type_and_dist() returns one mapping type per provider and client, valid only for strictly ordered Requests carrying an Untranslated address. Clients that use Relaxed Ordering or ATS cannot ask what the fabric would do with their traffic.

Add enum pci_p2pdma_tlp_flags to name a class and pci_p2pdma_map_type_tlp() to ask about one. The topology walk and the ACS Control reads do not depend on the class, so decide all of them from the one walk and cache them together, four bits each. Every class still answers alike; the controls that tell them apart come next.

Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/pci/p2pdma.c | 118 +++++++++++++++++++++++++++++++++------------ include/linux/pci-p2pdma.h | 49 +++++++++++++++++-- 2 files changed, 133 insertions(+), 34 deletions(-)

diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 841c86be31bb..8ed02d4a5c43 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -508,7 +508,7 @@ enum pci_acs_p2pdma_state { * bus addressing be assumed. */ static enum pci_acs_p2pdma_state -pci_acs_p2pdma_request(u16 ctrl) +pci_acs_p2pdma_request(u16 ctrl, unsigned int tlp_flags) { return ctrl & (PCI_ACS_RR | PCI_ACS_EC) ? PCI_ACS_P2PDMA_REDIRECT : PCI_ACS_P2PDMA_DIRECT; @@ -520,7 +520,7 @@ pci_acs_p2pdma_request(u16 ctrl) * affects a Completion. */ static enum pci_acs_p2pdma_state -pci_acs_p2pdma_completion(u16 ctrl) +pci_acs_p2pdma_completion(u16 ctrl, unsigned int tlp_flags) { return ctrl & PCI_ACS_CR ? PCI_ACS_P2PDMA_REDIRECT : PCI_ACS_P2PDMA_DIRECT; @@ -577,13 +577,16 @@ struct pci_p2pdma_acs_path { * the peer's bus addresses. */ static enum pci_p2pdma_map_type -pci_p2pdma_route(const struct pci_p2pdma_acs_path *path) +pci_p2pdma_route(const struct pci_p2pdma_acs_path *path, + unsigned int tlp_flags) { if (path->unreadable) return PCI_P2PDMA_MAP_NOT_SUPPORTED;

- if (pci_acs_p2pdma_request(path->req_ctrl) == PCI_ACS_P2PDMA_DIRECT && - pci_acs_p2pdma_completion(path->cpl_ctrl) == PCI_ACS_P2PDMA_DIRECT) + if (pci_acs_p2pdma_request(path->req_ctrl, tlp_flags) == + PCI_ACS_P2PDMA_DIRECT && + pci_acs_p2pdma_completion(path->cpl_ctrl, tlp_flags) == + PCI_ACS_P2PDMA_DIRECT) return PCI_P2PDMA_MAP_BUS_ADDR;

return PCI_P2PDMA_MAP_THRU_HOST_BRIDGE; @@ -591,7 +594,9 @@ pci_p2pdma_route(const struct pci_p2pdma_acs_path *path)

/* * Name the ports that keep this path off a direct route, so that the admin - * can hand them to pci=disable_acs_redir=. + * can hand them to pci=disable_acs_redir=. That parameter clears the + * redirect controls, which decide the default class, so that is the class + * this reports on. */ static void pci_p2pdma_warn_path(struct pci_dev *client, struct pci_dev *provider, @@ -610,9 +615,10 @@ static void pci_p2pdma_warn_path(struct pci_dev *client, }

seq_buf_init(&acs_list, buf, sizeof(buf)); - if (pci_acs_p2pdma_completion(path->cpl_ctrl) != PCI_ACS_P2PDMA_DIRECT) + if (pci_acs_p2pdma_completion(path->cpl_ctrl, 0) != + PCI_ACS_P2PDMA_DIRECT) seq_buf_print_bus_devfn(&acs_list, a_child); - if (pci_acs_p2pdma_request(path->req_ctrl) != PCI_ACS_P2PDMA_DIRECT) + if (pci_acs_p2pdma_request(path->req_ctrl, 0) != PCI_ACS_P2PDMA_DIRECT) seq_buf_print_bus_devfn(&acs_list, b_child);

/* Drop the final semicolon; the list is not empty here. */ @@ -780,6 +786,31 @@ static unsigned long map_types_idx(struct pci_dev *client) return (pci_domain_nr(client->bus) << 16) | pci_dev_id(client); }

+/* + * One cache entry holds the routing of every TLP class, four bits each, + * indexed by the &enum pci_p2pdma_tlp_flags combination that selects it. An + * absent entry reads back as PCI_P2PDMA_MAP_UNKNOWN in every class. + */ +static_assert(PCI_P2PDMA_MAP_THRU_HOST_BRIDGE < 16); + +static unsigned long +pci_p2pdma_map_types_pack(const enum pci_p2pdma_map_type *type) +{ + unsigned long val = 0; + unsigned int flags; + + for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) + val |= (unsigned long)type[flags] << (flags * 4); + + return val; +} + +static enum pci_p2pdma_map_type +pci_p2pdma_map_types_unpack(unsigned long val, unsigned int tlp_flags) +{ + return (val >> (tlp_flags * 4)) & 0xf; +} + /* * Calculate the P2PDMA mapping type and distance between two PCI devices. * @@ -809,6 +840,10 @@ static unsigned long map_types_idx(struct pci_dev *client) * check Request Redirect and Egress Control on the client-side port, and * Completion Redirect for read Completions on the provider-side port. * + * Those controls apply to different TLPs, so every class named by &enum + * pci_p2pdma_tlp_flags is decided from the one walk and cached together; + * @tlp_flags selects which one is returned. + * * If ACS redirects traffic at either divergence port, return * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE. If the ACS Control register cannot be * read, return PCI_P2PDMA_MAP_NOT_SUPPORTED. Otherwise, return @@ -822,14 +857,16 @@ static unsigned long map_types_idx(struct pci_dev *client) */ static enum pci_p2pdma_map_type calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, - int *dist, bool verbose) + int *dist, unsigned int tlp_flags, bool verbose) { - enum pci_p2pdma_map_type map_type = PCI_P2PDMA_MAP_THRU_HOST_BRIDGE; + enum pci_p2pdma_map_type map_type[PCI_P2PDMA_TLP_CLASSES]; struct pci_dev *a = provider, *b = client, *bb; struct pci_dev *a_child = NULL, *b_child = NULL; struct pci_p2pdma_acs_path path = {}; struct pci_p2pdma *p2pdma; bool cpu_p2pdma, host_whitelisted = false; + bool host_fallback = false; + unsigned int flags; int dist_a = 0; int dist_b = 0;

@@ -863,6 +900,8 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, * request can only get to the peer through the host bridge. */ *dist = dist_a + dist_b; + for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) + map_type[flags] = PCI_P2PDMA_MAP_THRU_HOST_BRIDGE; goto map_through_host_bridge;

check_paths_acs: @@ -882,18 +921,24 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, }

/* - * Below a shared upstream bridge, a path whose divergence ports do not - * redirect routes the request directly. + * The walk and the config reads above serve every class; only the + * decision below depends on the kind of TLP being routed. */ - map_type = pci_p2pdma_route(&path); - if (map_type == PCI_P2PDMA_MAP_BUS_ADDR) - goto done; + for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) { + map_type[flags] = pci_p2pdma_route(&path, flags); + if (map_type[flags] == PCI_P2PDMA_MAP_THRU_HOST_BRIDGE) + host_fallback = true; + }

- if (verbose) - pci_p2pdma_warn_path(client, provider, &path, a_child, b_child); + if (verbose && map_type[0] != PCI_P2PDMA_MAP_BUS_ADDR) + pci_p2pdma_warn_path(client, provider, &path, a_child, + b_child);

- /* An unreadable control does not establish an upstream redirect. */ - if (path.unreadable) + /* + * Nothing needs the host bridge: the classes that did not get a direct + * route have no fallback that would use it. + */ + if (!host_fallback) goto done;

map_through_host_bridge: @@ -906,16 +951,19 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, if (verbose) pci_warn(client, "cannot be used for peer-to-peer DMA as the client and provider (%s) do not share an upstream bridge or whitelisted host bridge\n", pci_name(provider)); - map_type = PCI_P2PDMA_MAP_NOT_SUPPORTED; + for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) + if (map_type[flags] == PCI_P2PDMA_MAP_THRU_HOST_BRIDGE) + map_type[flags] = PCI_P2PDMA_MAP_NOT_SUPPORTED; } done: rcu_read_lock(); p2pdma = rcu_dereference(provider->p2pdma); if (p2pdma) xa_store(&p2pdma->map_types, map_types_idx(client), - xa_mk_value(map_type), GFP_ATOMIC); + xa_mk_value(pci_p2pdma_map_types_pack(map_type)), + GFP_ATOMIC); rcu_read_unlock(); - return map_type; + return map_type[tlp_flags]; }

/** @@ -956,7 +1004,7 @@ int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients, return -1; }

- map = calc_map_type_and_dist(provider, pci_client, &distance, + map = calc_map_type_and_dist(provider, pci_client, &distance, 0, verbose);

pci_dev_put(pci_client); @@ -1221,22 +1269,28 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) EXPORT_SYMBOL_GPL(pci_p2pmem_publish);

/** - * pci_p2pdma_map_type - Determine the mapping type for P2PDMA transfers + * pci_p2pdma_map_type_tlp - Determine the mapping type for P2PDMA transfers * @provider: P2PDMA provider structure * @dev: Client device that initiates the transfer + * @tlp_flags: &enum pci_p2pdma_tlp_flags describing the TLPs @dev will issue * * Determines how peer-to-peer DMA transfers should be mapped between * the provider and the client device. The mapping type indicates whether * the transfer can be done directly through PCI switches or must go * through the host bridge. + * + * ACS routes a peer-to-peer transaction by the attributes its TLPs carry, so + * the answer depends on @tlp_flags. A caller that passes flags its traffic + * does not match gets a mapping the fabric will not deliver. */ -enum pci_p2pdma_map_type pci_p2pdma_map_type(struct p2pdma_provider *provider, - struct device *dev) +enum pci_p2pdma_map_type +pci_p2pdma_map_type_tlp(struct p2pdma_provider *provider, struct device *dev, + unsigned int tlp_flags) { - enum pci_p2pdma_map_type type = PCI_P2PDMA_MAP_NOT_SUPPORTED; struct pci_dev *pdev = to_pci_dev(provider->owner); + unsigned long cache_index, cached = 0; + enum pci_p2pdma_map_type type; struct pci_p2pdma *p2pdma; - unsigned long cache_index; struct pci_dev *client; int dist;

@@ -1253,12 +1307,14 @@ enum pci_p2pdma_map_type pci_p2pdma_map_type(struct p2pdma_provider *provider, p2pdma = rcu_dereference(pdev->p2pdma);

if (p2pdma) - type = xa_to_value(xa_load(&p2pdma->map_types, - cache_index)); + cached = xa_to_value(xa_load(&p2pdma->map_types, + cache_index)); rcu_read_unlock(); + type = pci_p2pdma_map_types_unpack(cached, tlp_flags);

if (type == PCI_P2PDMA_MAP_UNKNOWN) - return calc_map_type_and_dist(pdev, client, &dist, true); + return calc_map_type_and_dist(pdev, client, &dist, tlp_flags, + true);

return type; } diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h index dd17501ba1b6..36045b1b730c 100644 --- a/include/linux/pci-p2pdma.h +++ b/include/linux/pci-p2pdma.h @@ -28,6 +28,33 @@ struct p2pdma_provider { u64 bus_offset; };

+/** + * enum pci_p2pdma_tlp_flags - Properties of the TLPs a client will issue + * + * These describe the traffic rather than the topology, and select which ACS + * controls apply along the peer-to-peer path. A value of 0 means strictly + * ordered Requests carrying an Untranslated address. + * + * @PCI_P2PDMA_TLP_TRANSLATED: Requests carry an ATS Translated address. PCIe + * r7.0 sec 6.12.3 routes those to the peer regardless of ACS P2P Request + * Redirect and ACS P2P Egress Control wherever ACS Direct Translated P2P + * is enabled. + * @PCI_P2PDMA_TLP_RELAXED_CPL: The provider returns Completions with the + * Relaxed Ordering attribute set. PCIe r7.0 sec 6.12.1.1 never redirects + * those, so ACS P2P Completion Redirect does not gate the path. The + * Completer chooses this attribute and the specification does not require + * it to copy Relaxed Ordering from the Request into the Completion, so a + * caller passing this flag asserts that its provider does. + */ +enum pci_p2pdma_tlp_flags { + PCI_P2PDMA_TLP_TRANSLATED = 1 << 0, + PCI_P2PDMA_TLP_RELAXED_CPL = 1 << 1, +}; + +/* Every combination of the flags above selects one routing class. */ +#define PCI_P2PDMA_TLP_CLASSES \ + ((PCI_P2PDMA_TLP_TRANSLATED | PCI_P2PDMA_TLP_RELAXED_CPL) + 1) + enum pci_p2pdma_map_type { /* * PCI_P2PDMA_MAP_UNKNOWN: Used internally as an initial state before @@ -86,8 +113,9 @@ int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev, bool *use_p2pdma); ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev, bool use_p2pdma); -enum pci_p2pdma_map_type pci_p2pdma_map_type(struct p2pdma_provider *provider, - struct device *dev); +enum pci_p2pdma_map_type +pci_p2pdma_map_type_tlp(struct p2pdma_provider *provider, struct device *dev, + unsigned int tlp_flags); #else /* CONFIG_PCI_P2PDMA */ static inline int pcim_p2pdma_init(struct pci_dev *pdev) { @@ -150,7 +178,8 @@ static inline ssize_t pci_p2pdma_enable_show(char *page, return sprintf(page, "none\n"); } static inline enum pci_p2pdma_map_type -pci_p2pdma_map_type(struct p2pdma_provider *provider, struct device *dev) +pci_p2pdma_map_type_tlp(struct p2pdma_provider *provider, struct device *dev, + unsigned int tlp_flags) { return PCI_P2PDMA_MAP_NOT_SUPPORTED; } @@ -168,6 +197,20 @@ static inline struct pci_dev *pci_p2pmem_find(struct device *client) return pci_p2pmem_find_many(&client, 1); }

+/** + * pci_p2pdma_map_type - Determine the mapping type for P2PDMA transfers + * @provider: P2PDMA provider structure + * @dev: Client device that initiates the transfer + * + * Same as pci_p2pdma_map_type_tlp() for a client issuing strictly ordered + * Requests that carry an Untranslated address. + */ +static inline enum pci_p2pdma_map_type +pci_p2pdma_map_type(struct p2pdma_provider *provider, struct device *dev) +{ + return pci_p2pdma_map_type_tlp(provider, dev, 0); +} + struct pci_p2pdma_map_state { struct p2pdma_provider *mem; enum pci_p2pdma_map_type map;

From: Leon Romanovsky leonro@nvidia.com

A Downstream Port with ACS Translation Blocking enabled treats every Upstream Memory Request whose Address Type is not Untranslated as an ACS Violation, ahead of "any applicable ACS P2P control mechanisms" per PCIe r7.0 sec 6.12.1.1. P2PDMA never looks at that bit, so it reports a bus-addressable path where an ATS client's Requests would be rejected.

Add PCI_ACS_P2PDMA_BLOCKED, and because blocking is not a routing control, scan the path for it rather than the divergence port alone. A blocked Request has no host bridge fallback, since the Address Type is rejected wherever the Request is addressed.

The two routes do not pass the same ports, so scan them separately. A direct route turns around at the divergence and only passes the ports below it, while the host bridge route keeps climbing and passes that port and everything above it as well.

Reviewed-by: Logan Gunthorpe logang@deltatee.com Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/pci/p2pdma.c | 120 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 112 insertions(+), 8 deletions(-)

diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 2564d57c5e6b..8f278fcb220b 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -496,11 +496,12 @@ enum pci_acs_p2pdma_state { PCI_ACS_P2PDMA_NOT_SUPPORTED, PCI_ACS_P2PDMA_DIRECT, PCI_ACS_P2PDMA_REDIRECT, + PCI_ACS_P2PDMA_BLOCKED, };

/* * Decide how a peer-to-peer Request at an ACS-capable ingress port routes, - * from that port's ACS Control register. + * from that port's ACS Control register and the Request's Address Type. * * Linux does not read the Egress Control Vector, so Egress Control is treated * conservatively as a redirect. Per PCIe r7.0 Table 6-11 the outcomes it @@ -510,6 +511,18 @@ enum pci_acs_p2pdma_state { static enum pci_acs_p2pdma_state pci_acs_p2pdma_request(u16 ctrl, unsigned int tlp_flags) { + if (tlp_flags & PCI_P2PDMA_TLP_TRANSLATED) { + /* + * PCIe r7.0 sec 6.12.1.1: Translation Blocking makes every + * Upstream Memory Request whose Address Type is not + * Untranslated an ACS Violation, taking precedence over the + * P2P controls. Sec 7.7.12.5: Direct Translated P2P "is + * ignored if ACS Translation Blocking Enable is 1b". + */ + if (ctrl & PCI_ACS_TB) + return PCI_ACS_P2PDMA_BLOCKED; + } + return ctrl & (PCI_ACS_RR | PCI_ACS_EC) ? PCI_ACS_P2PDMA_REDIRECT : PCI_ACS_P2PDMA_DIRECT; } @@ -549,6 +562,39 @@ static bool pci_acs_p2pdma_ctrl(struct pci_dev *pdev, u16 *ctrl) return !pci_read_config_word(pdev, pos + PCI_ACS_CTRL, ctrl); }

+/* + * Report whether any port between @client and @common rejects Translated + * addresses. If @common is NULL, every port up to the host bridge is walked. + * A port whose ACS Control cannot be read counts as blocking, which withdraws + * only the Translated classes because an Untranslated Request is routed by the + * redirect controls instead. + */ +static bool pci_p2pdma_path_blocks_translation(struct pci_dev *client, + struct pci_dev *common) +{ + struct pci_dev *pdev; + u16 ctrl; + + /* + * @common is @client itself when the provider is the client or sits + * below it. The Request never travels upstream then, so no port sees + * it and none can reject its Address Type. + */ + if (client == common) + return false; + + for (pdev = pci_upstream_bridge(client); pdev && pdev != common; + pdev = pci_upstream_bridge(pdev)) { + if (!pci_acs_p2pdma_ctrl(pdev, &ctrl)) + return true; + + if (ctrl & PCI_ACS_TB) + return true; + } + + return false; +} + static void seq_buf_print_bus_devfn(struct seq_buf *buf, struct pci_dev *pdev) { if (!buf) @@ -567,14 +613,42 @@ static void seq_buf_print_bus_devfn(struct seq_buf *buf, struct pci_dev *pdev) * than upstream, so it is where the Request controls apply. * @cpl_ctrl: ACS Control of the provider-side divergence port, likewise for * the Completions travelling back. + * @tb_on_path: A port below the divergence blocks Translated addresses. + * Every route out of @client passes those, so none carries them. + * @tb_above_divergence: A port at or above the divergence blocks Translated + * addresses. Only a Request continuing to the host bridge passes + * those, so a direct route is still open to them. + * @no_common_bridge: The two paths share no upstream bridge, so no direct + * route exists for ACS to gate. * @unreadable: First port whose ACS Control could not be read, if any. */ struct pci_p2pdma_acs_path { u16 req_ctrl; u16 cpl_ctrl; + bool tb_on_path; + bool tb_above_divergence; + bool no_common_bridge; struct pci_dev *unreadable; };

+/* + * ACS Translation Blocking is not a routing control, so unlike the redirect + * controls it is not decided at the divergence alone. PCIe r7.0 sec 6.12.1.1 + * has every Downstream Port check the Address Type of each Upstream Memory + * Request it receives, ahead of "any applicable ACS P2P control mechanisms". + * A port below the divergence cannot redirect the Request anywhere it was not + * already going, but it can still reject a Translated address. + */ +static enum pci_acs_p2pdma_state +pci_p2pdma_request_state(const struct pci_p2pdma_acs_path *path, + unsigned int tlp_flags) +{ + if (tlp_flags & PCI_P2PDMA_TLP_TRANSLATED && path->tb_on_path) + return PCI_ACS_P2PDMA_BLOCKED; + + return pci_acs_p2pdma_request(path->req_ctrl, tlp_flags); +} + /* * Combine both directions into a mapping type. Only a path that routes the * Request and the Completions it generates directly can be programmed with @@ -584,15 +658,35 @@ static enum pci_p2pdma_map_type pci_p2pdma_route(const struct pci_p2pdma_acs_path *path, unsigned int tlp_flags) { + enum pci_acs_p2pdma_state req; + if (path->unreadable) return PCI_P2PDMA_MAP_NOT_SUPPORTED;

- if (pci_acs_p2pdma_request(path->req_ctrl, tlp_flags) == - PCI_ACS_P2PDMA_DIRECT && + req = pci_p2pdma_request_state(path, tlp_flags); + + /* + * Translation Blocking rejects the Address Type rather than the + * target, so a blocked Request stays blocked however it is addressed. + * No host bridge fallback keeps a Translated address working; the + * caller has to issue a different kind of Request instead. + */ + if (req == PCI_ACS_P2PDMA_BLOCKED) + return PCI_P2PDMA_MAP_NOT_SUPPORTED; + + if (!path->no_common_bridge && req == PCI_ACS_P2PDMA_DIRECT && pci_acs_p2pdma_completion(path->cpl_ctrl, tlp_flags) == PCI_ACS_P2PDMA_DIRECT) return PCI_P2PDMA_MAP_BUS_ADDR;

+ /* + * A Request that turns around at the divergence never reaches the + * ports above it, but one that keeps climbing to the host bridge + * does, so that route has to clear their Translation Blocking too. + */ + if (tlp_flags & PCI_P2PDMA_TLP_TRANSLATED && path->tb_above_divergence) + return PCI_P2PDMA_MAP_NOT_SUPPORTED; + return PCI_P2PDMA_MAP_THRU_HOST_BRIDGE; }

@@ -841,8 +935,11 @@ pci_p2pdma_map_types_unpack(unsigned long val, unsigned int tlp_flags) * redirection setting of the ports along the path. * * The client initiates Requests to provider memory. At the path divergence, - * check Request Redirect and Egress Control on the client-side port, and - * Completion Redirect for read Completions on the provider-side port. + * check Request Redirect, Egress Control, Translation Blocking and Direct + * Translated P2P on the client-side port, and Completion Redirect for read + * Completions on the provider-side port. Translation Blocking is checked on + * every client-side port instead, because it rejects a Request rather than + * routing it. * * Those controls apply to different TLPs, so every class named by &enum * pci_p2pdma_tlp_flags is decided from the one walk and cached together; @@ -850,8 +947,8 @@ pci_p2pdma_map_types_unpack(unsigned long val, unsigned int tlp_flags) * * If ACS redirects traffic at either divergence port, return * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE. If the ACS Control register cannot be - * read, return PCI_P2PDMA_MAP_NOT_SUPPORTED. Otherwise, return - * PCI_P2PDMA_MAP_BUS_ADDR. + * read, or Translation Blocking rejects the class being asked about, return + * PCI_P2PDMA_MAP_NOT_SUPPORTED. Otherwise, return PCI_P2PDMA_MAP_BUS_ADDR. * * Any two devices that have a data path that goes through the host bridge * will consult a whitelist. If the host bridge is in the whitelist, return @@ -904,8 +1001,10 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, * request can only get to the peer through the host bridge. */ *dist = dist_a + dist_b; + path.no_common_bridge = true; + path.tb_on_path = pci_p2pdma_path_blocks_translation(client, NULL); for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) - map_type[flags] = PCI_P2PDMA_MAP_THRU_HOST_BRIDGE; + map_type[flags] = pci_p2pdma_route(&path, flags); goto map_through_host_bridge;

check_paths_acs: @@ -924,6 +1023,11 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, path.unreadable = b_child; }

+ path.tb_on_path = pci_p2pdma_path_blocks_translation(client, a); + if (b_child) + path.tb_above_divergence = + pci_p2pdma_path_blocks_translation(b_child, NULL); + /* * The walk and the config reads above serve every class; only the * decision below depends on the kind of TLP being routed.

From: Leon Romanovsky leonro@nvidia.com

When P2PDMA rejects a mapping, existing warnings identify only the final ACS or host-bridge result. They omit topology, live controls, divergence ports, cache state, and intermediate routing decisions.

Emit debug-level messages for verbose calculations and cache lookups. Report both paths, decoded ACS controls, directional decisions, host fallback, and the final mapping. This keeps incidental unsupported probes quiet while allowing the diagnostics to be enabled when needed.

Reviewed-by: Logan Gunthorpe logang@deltatee.com Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/pci/p2pdma.c | 232 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 215 insertions(+), 17 deletions(-)

diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index d68ba562febd..f03127901880 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -552,23 +552,128 @@ pci_acs_p2pdma_completion(u16 ctrl, unsigned int tlp_flags) PCI_ACS_P2PDMA_DIRECT; }

+static const char *pci_acs_p2pdma_state_name(enum pci_acs_p2pdma_state state) +{ + switch (state) { + case PCI_ACS_P2PDMA_DIRECT: + return "direct"; + case PCI_ACS_P2PDMA_REDIRECT: + return "redirect"; + case PCI_ACS_P2PDMA_BLOCKED: + return "blocked"; + case PCI_ACS_P2PDMA_NOT_SUPPORTED: + return "not-supported"; + } + + return "invalid"; +} + +static const char *pci_p2pdma_map_type_name(enum pci_p2pdma_map_type type) +{ + switch (type) { + case PCI_P2PDMA_MAP_UNKNOWN: + return "unknown"; + case PCI_P2PDMA_MAP_NONE: + return "none"; + case PCI_P2PDMA_MAP_NOT_SUPPORTED: + return "not-supported"; + case PCI_P2PDMA_MAP_BUS_ADDR: + return "bus-address"; + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + return "through-host-bridge"; + } + + return "invalid"; +} + /* * Read @pdev's ACS Control register. A device without an ACS capability has * no peer-to-peer controls at all, which routes the same as having them all * clear. Returns false when the register is present but cannot be read; @ctrl * is then meaningless. */ -static bool pci_acs_p2pdma_ctrl(struct pci_dev *pdev, u16 *ctrl) +static bool pci_acs_p2pdma_ctrl(struct pci_dev *pdev, const char *what, + u16 *ctrl, bool verbose) { - int pos; + int pos, ret;

pos = pdev->acs_cap; if (!pos) { + if (verbose) + pci_dbg(pdev, + "P2PDMA ACS: %s has no ACS capability\n", what); *ctrl = 0; return true; }

- return !pci_read_config_word(pdev, pos + PCI_ACS_CTRL, ctrl); + ret = pci_read_config_word(pdev, pos + PCI_ACS_CTRL, ctrl); + if (ret) { + if (verbose) + pci_dbg(pdev, + "P2PDMA ACS: %s ACS Control read failed at %#x: %#x\n", + what, pos + PCI_ACS_CTRL, ret); + return false; + } + + if (verbose) { + pci_dbg(pdev, + "P2PDMA ACS: %s cap=%#x caps=%#06x ctrl=%#06x\n", + what, pos, pdev->acs_capabilities, *ctrl); + pci_dbg(pdev, + "P2PDMA ACS: control bits SV=%u TB=%u RR=%u CR=%u UF=%u EC=%u DT=%u\n", + !!(*ctrl & PCI_ACS_SV), !!(*ctrl & PCI_ACS_TB), + !!(*ctrl & PCI_ACS_RR), !!(*ctrl & PCI_ACS_CR), + !!(*ctrl & PCI_ACS_UF), !!(*ctrl & PCI_ACS_EC), + !!(*ctrl & PCI_ACS_DT)); + } + + return true; +} + +static void pci_p2pdma_log_path(const char *name, struct pci_dev *start, + struct pci_dev *common) +{ + struct pci_dev *pdev, *upstream; + int hop = 0, ret, type; + u16 ctrl; + + for (pdev = start; pdev; pdev = upstream, hop++) { + upstream = pci_upstream_bridge(pdev); + type = pci_is_pcie(pdev) ? pci_pcie_type(pdev) : -1; + pci_dbg(pdev, + "P2PDMA ACS: %s path hop=%d common=%u pcie=%u type=%d class=%#08x vendor=%04x device=%04x upstream=%s\n", + name, hop, pdev == common, pci_is_pcie(pdev), type, + pdev->class, pdev->vendor, pdev->device, + upstream ? pci_name(upstream) : "<none>"); + + if (pdev->subordinate) + pci_dbg(pdev, + "P2PDMA ACS: bridge bus range=%02llx-%02llx\n", + (unsigned long long)pdev->subordinate->busn_res.start, + (unsigned long long)pdev->subordinate->busn_res.end); + + if (!pdev->acs_cap) { + pci_dbg(pdev, "P2PDMA ACS: ACS capability absent\n"); + continue; + } + + ret = pci_read_config_word(pdev, pdev->acs_cap + PCI_ACS_CTRL, + &ctrl); + if (ret) { + pci_dbg(pdev, + "P2PDMA ACS: ACS cap=%#x caps=%#06x Control read failed: %#x\n", + pdev->acs_cap, pdev->acs_capabilities, ret); + continue; + } + + pci_dbg(pdev, + "P2PDMA ACS: ACS cap=%#x caps=%#06x ctrl=%#06x SV=%u TB=%u RR=%u CR=%u UF=%u EC=%u DT=%u\n", + pdev->acs_cap, pdev->acs_capabilities, ctrl, + !!(ctrl & PCI_ACS_SV), !!(ctrl & PCI_ACS_TB), + !!(ctrl & PCI_ACS_RR), !!(ctrl & PCI_ACS_CR), + !!(ctrl & PCI_ACS_UF), !!(ctrl & PCI_ACS_EC), + !!(ctrl & PCI_ACS_DT)); + } }

/* @@ -579,7 +684,8 @@ static bool pci_acs_p2pdma_ctrl(struct pci_dev *pdev, u16 *ctrl) * redirect controls instead. */ static bool pci_p2pdma_path_blocks_translation(struct pci_dev *client, - struct pci_dev *common) + struct pci_dev *common, + bool verbose) { struct pci_dev *pdev; u16 ctrl; @@ -594,11 +700,15 @@ static bool pci_p2pdma_path_blocks_translation(struct pci_dev *client,

for (pdev = pci_upstream_bridge(client); pdev && pdev != common; pdev = pci_upstream_bridge(pdev)) { - if (!pci_acs_p2pdma_ctrl(pdev, &ctrl)) + if (!pci_acs_p2pdma_ctrl(pdev, "path hop", &ctrl, verbose)) return true;

- if (ctrl & PCI_ACS_TB) + if (ctrl & PCI_ACS_TB) { + if (verbose) + pci_dbg(pdev, + "P2PDMA ACS: Translation Blocking rejects Translated Requests on this path\n"); return true; + } }

return false; @@ -972,14 +1082,22 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, enum pci_p2pdma_map_type map_type[PCI_P2PDMA_TLP_CLASSES]; struct pci_dev *a = provider, *b = client, *bb; struct pci_dev *a_child = NULL, *b_child = NULL; + struct pci_host_bridge *provider_host, *client_host; struct pci_p2pdma_acs_path path = {}; struct pci_p2pdma *p2pdma; bool cpu_p2pdma, host_whitelisted = false; + bool cache_store = false; bool host_fallback = false; unsigned int flags; int dist_a = 0; int dist_b = 0;

+ if (verbose) + pci_dbg(client, + "P2PDMA ACS: begin provider=%s client=%s cache-index=%#lx\n", + pci_name(provider), pci_name(client), + map_types_idx(client)); + /* * Note, we don't need to take references to devices returned by * pci_upstream_bridge() seeing we hold a reference to a child @@ -1010,14 +1128,32 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, * request can only get to the peer through the host bridge. */ *dist = dist_a + dist_b; + if (verbose) { + pci_dbg(client, + "P2PDMA ACS: no common upstream bridge provider-distance=%d client-distance=%d total=%d\n", + dist_a, dist_b, *dist); + pci_p2pdma_log_path("provider", provider, NULL); + pci_p2pdma_log_path("client", client, NULL); + } path.no_common_bridge = true; - path.tb_on_path = pci_p2pdma_path_blocks_translation(client, NULL); + path.tb_on_path = pci_p2pdma_path_blocks_translation(client, NULL, + verbose); for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) map_type[flags] = pci_p2pdma_route(&path, flags); goto map_through_host_bridge;

check_paths_acs: *dist = dist_a + dist_b; + if (verbose) { + pci_dbg(client, + "P2PDMA ACS: common=%s provider-divergence=%s client-divergence=%s provider-distance=%d client-distance=%d total=%d\n", + pci_name(a), + a_child ? pci_name(a_child) : "<none>", + b_child ? pci_name(b_child) : "<none>", + dist_a, dist_b, *dist); + pci_p2pdma_log_path("provider", provider, a); + pci_p2pdma_log_path("client", client, a); + }

/* * ACS P2P routing controls apply where a TLP can route toward the peer @@ -1025,17 +1161,33 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, * branch is upstream, so redirect controls do not affect the path. */ if (a_child && b_child) { - if (!pci_acs_p2pdma_ctrl(a_child, &path.cpl_ctrl)) + if (!pci_acs_p2pdma_ctrl(a_child, "completion", &path.cpl_ctrl, + verbose)) path.unreadable = a_child; - if (!pci_acs_p2pdma_ctrl(b_child, &path.req_ctrl) && - !path.unreadable) + if (!pci_acs_p2pdma_ctrl(b_child, "request", &path.req_ctrl, + verbose) && !path.unreadable) path.unreadable = b_child; + if (verbose && !path.unreadable) + pci_dbg(client, + "P2PDMA ACS: request=%s at %s completion=%s at %s\n", + pci_acs_p2pdma_state_name( + pci_p2pdma_request_state(&path, 0)), + pci_name(b_child), + pci_acs_p2pdma_state_name( + pci_acs_p2pdma_completion(path.cpl_ctrl, + 0)), + pci_name(a_child)); + } else if (verbose) { + pci_dbg(client, + "P2PDMA ACS: peer divergence is incomplete; no ACS peer-routing controls evaluated\n"); }

- path.tb_on_path = pci_p2pdma_path_blocks_translation(client, a); + path.tb_on_path = pci_p2pdma_path_blocks_translation(client, a, + verbose); if (b_child) path.tb_above_divergence = - pci_p2pdma_path_blocks_translation(b_child, NULL); + pci_p2pdma_path_blocks_translation(b_child, NULL, + verbose);

/* * The walk and the config reads above serve every class; only the @@ -1064,6 +1216,19 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, host_whitelisted = host_bridge_whitelist(provider, client, verbose);

+ if (verbose) { + provider_host = pci_find_host_bridge(provider->bus); + client_host = pci_find_host_bridge(client->bus); + pci_dbg(client, + "P2PDMA ACS: host fallback cpu-support=%u whitelist=%s provider-host=%s client-host=%s same-host=%u\n", + cpu_p2pdma, + cpu_p2pdma ? "not-consulted" : + (host_whitelisted ? "yes" : "no"), + provider_host ? dev_name(&provider_host->dev) : "<none>", + client_host ? dev_name(&client_host->dev) : "<none>", + provider_host && provider_host == client_host); + } + if (!cpu_p2pdma && !host_whitelisted) { if (verbose) pci_warn(client, "cannot be used for peer-to-peer DMA as the client and provider (%s) do not share an upstream bridge or whitelisted host bridge\n", @@ -1075,11 +1240,31 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, done: rcu_read_lock(); p2pdma = rcu_dereference(provider->p2pdma); - if (p2pdma) + if (p2pdma) { xa_store(&p2pdma->map_types, map_types_idx(client), - xa_mk_value(pci_p2pdma_map_types_pack(map_type)), - GFP_ATOMIC); + xa_mk_value(pci_p2pdma_map_types_pack(map_type)), GFP_ATOMIC); + cache_store = true; + } rcu_read_unlock(); + if (verbose) { + pci_dbg(client, + "P2PDMA ACS: final provider=%s result=%s(%d) tlp-flags=%#x distance=%d unreadable=%s cache-store=%u index=%#lx\n", + pci_name(provider), + pci_p2pdma_map_type_name(map_type[tlp_flags]), + map_type[tlp_flags], tlp_flags, *dist, + path.unreadable ? pci_name(path.unreadable) : "<none>", + cache_store, map_types_idx(client)); + pci_dbg(client, + "P2PDMA ACS: classes strict=%s relaxed=%s translated=%s translated+relaxed=%s\n", + pci_p2pdma_map_type_name(map_type[0]), + pci_p2pdma_map_type_name( + map_type[PCI_P2PDMA_TLP_RELAXED_CPL]), + pci_p2pdma_map_type_name( + map_type[PCI_P2PDMA_TLP_TRANSLATED]), + pci_p2pdma_map_type_name( + map_type[PCI_P2PDMA_TLP_TRANSLATED | + PCI_P2PDMA_TLP_RELAXED_CPL])); + } return map_type[tlp_flags]; }

@@ -1409,13 +1594,21 @@ pci_p2pdma_map_type_tlp(struct p2pdma_provider *provider, struct device *dev, enum pci_p2pdma_map_type type; struct pci_p2pdma *p2pdma; struct pci_dev *client; + bool provider_state; int dist;

- if (!pdev->p2pdma) + if (!pdev->p2pdma) { + pci_dbg(pdev, + "P2PDMA ACS: map lookup rejected; provider state is absent\n"); return PCI_P2PDMA_MAP_NOT_SUPPORTED; + }

- if (!dev_is_pci(dev)) + if (!dev_is_pci(dev)) { + dev_dbg(dev, + "P2PDMA ACS: provider=%s map lookup rejected; client is not PCI\n", + pci_name(pdev)); return PCI_P2PDMA_MAP_NOT_SUPPORTED; + }

client = to_pci_dev(dev); cache_index = map_types_idx(client); @@ -1426,8 +1619,13 @@ pci_p2pdma_map_type_tlp(struct p2pdma_provider *provider, struct device *dev, if (p2pdma) cached = xa_to_value(xa_load(&p2pdma->map_types, cache_index)); + provider_state = !!p2pdma; rcu_read_unlock(); type = pci_p2pdma_map_types_unpack(cached, tlp_flags); + pci_dbg(client, + "P2PDMA ACS: map lookup provider=%s index=%#lx tlp-flags=%#x cached=%s(%d) provider-state=%u\n", + pci_name(pdev), cache_index, tlp_flags, + pci_p2pdma_map_type_name(type), type, provider_state);

if (type == PCI_P2PDMA_MAP_UNKNOWN) return calc_map_type_and_dist(pdev, client, &dist, tlp_flags,

From: Leon Romanovsky leonro@nvidia.com

calc_map_type_and_dist() decides which ports along a path carry the routing controls, and holds every class's answer in one cache entry. Neither depends on a single register, so a table of them cannot reach the walk itself.

Drive the walk over a fabricated fabric of two devices below a switch, with fake config space supplying the ACS Control registers. Cover the ports below the divergence, the three cases where two classes of one path disagree, and the packed cache, which the fabric has no provider state to exercise indirectly.

Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/pci/p2pdma.c | 9 +- drivers/pci/pci.h | 9 + drivers/pci/pci_acs_test.c | 478 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 493 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 7543e39a997f..1f4bf0125a5c 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -1005,7 +1005,7 @@ static unsigned long map_types_idx(struct pci_dev *client) */ static_assert(PCI_P2PDMA_MAP_THRU_HOST_BRIDGE < 16);

-static unsigned long +VISIBLE_IF_KUNIT unsigned long pci_p2pdma_map_types_pack(const enum pci_p2pdma_map_type *type) { unsigned long val = 0; @@ -1016,12 +1016,14 @@ pci_p2pdma_map_types_pack(const enum pci_p2pdma_map_type *type)

return val; } +EXPORT_SYMBOL_IF_KUNIT(pci_p2pdma_map_types_pack);

-static enum pci_p2pdma_map_type +VISIBLE_IF_KUNIT enum pci_p2pdma_map_type pci_p2pdma_map_types_unpack(unsigned long val, unsigned int tlp_flags) { return (val >> (tlp_flags * 4)) & 0xf; } +EXPORT_SYMBOL_IF_KUNIT(pci_p2pdma_map_types_unpack);

/* * Calculate the P2PDMA mapping type and distance between two PCI devices. @@ -1070,7 +1072,7 @@ pci_p2pdma_map_types_unpack(unsigned long val, unsigned int tlp_flags) * ports per above. If the device is not in the whitelist, return * PCI_P2PDMA_MAP_NOT_SUPPORTED. */ -static enum pci_p2pdma_map_type +VISIBLE_IF_KUNIT enum pci_p2pdma_map_type calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, int *dist, unsigned int tlp_flags, bool verbose) { @@ -1262,6 +1264,7 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client, } return map_type[tlp_flags]; } +EXPORT_SYMBOL_IF_KUNIT(calc_map_type_and_dist);

/** * pci_p2pdma_distance_many - Determine the cumulative distance between diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index c0d2711d73f9..56f821e40637 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -7,6 +7,7 @@ #include <linux/align.h> #include <linux/bitfield.h> #include <linux/pci.h> +#include <linux/pci-p2pdma.h> #include <trace/events/pci.h>

struct pcie_tlp_log; @@ -1107,6 +1108,14 @@ enum pci_acs_p2pdma_state pci_acs_p2pdma_request(u16 ctrl, unsigned int tlp_flags); enum pci_acs_p2pdma_state pci_acs_p2pdma_completion(u16 ctrl, unsigned int tlp_flags); +unsigned long pci_p2pdma_map_types_pack(const enum pci_p2pdma_map_type *type); +enum pci_p2pdma_map_type pci_p2pdma_map_types_unpack(unsigned long val, + unsigned int tlp_flags); +enum pci_p2pdma_map_type calc_map_type_and_dist(struct pci_dev *provider, + struct pci_dev *client, + int *dist, + unsigned int tlp_flags, + bool verbose); #endif

#ifdef CONFIG_PCI_QUIRKS diff --git a/drivers/pci/pci_acs_test.c b/drivers/pci/pci_acs_test.c index ce6b9375da36..7f4f9cc04bfb 100644 --- a/drivers/pci/pci_acs_test.c +++ b/drivers/pci/pci_acs_test.c @@ -102,11 +102,489 @@ static void pci_acs_p2pdma_completion_test(struct kunit *test) c->expect); }

+/* + * Drive calc_map_type_and_dist() over a fabricated PCIe fabric matching the + * canonical topology of two devices below one switch: + * + * host bridge / root bus + * Root Port + * Switch Upstream Port + * Switch Downstream Port 0 + * Nested Switch -- provider + * Switch Downstream Port 1 + * Nested Switch -- client + * + * Fake config-space operations supply the ACS Control registers. This lets + * the cases vary both divergence ports and controls below the divergence + * without depending on real hardware. + */ +struct acs_port_cfg { + u16 ctrl; + bool fail_read; +}; + +struct acs_fabric { + struct pci_dev *rootport; + struct pci_dev *provider; + struct pci_dev *client; + struct pci_dev *dn0; /* Downstream Port 0 (provider side) */ + struct pci_dev *dn1; /* Downstream Port 1 (client side) */ + struct pci_dev *provider_leaf; + struct pci_dev *client_leaf; + struct acs_port_cfg dn0_cfg; + struct acs_port_cfg dn1_cfg; + struct acs_port_cfg provider_leaf_cfg; + struct acs_port_cfg client_leaf_cfg; + struct acs_port_cfg rootport_cfg; +}; + +static int acs_port_read(struct pci_dev *port, struct acs_port_cfg *cfg, + int where, int size, u32 *val) +{ + if (port->acs_cap && size == 2 && + where == port->acs_cap + PCI_ACS_CTRL) { + if (cfg->fail_read) + return PCIBIOS_DEVICE_NOT_FOUND; + *val = cfg->ctrl; + } + + return PCIBIOS_SUCCESSFUL; +} + +static int acs_fabric_read(struct pci_bus *bus, unsigned int devfn, + int where, int size, u32 *val) +{ + struct acs_fabric *f = bus->sysdata; + + *val = 0; + if (bus == f->rootport->bus && devfn == f->rootport->devfn) + return acs_port_read(f->rootport, &f->rootport_cfg, + where, size, val); + if (bus == f->dn0->bus && devfn == f->dn0->devfn) + return acs_port_read(f->dn0, &f->dn0_cfg, where, size, val); + if (bus == f->dn1->bus && devfn == f->dn1->devfn) + return acs_port_read(f->dn1, &f->dn1_cfg, where, size, val); + if (bus == f->provider_leaf->bus && + devfn == f->provider_leaf->devfn) + return acs_port_read(f->provider_leaf, &f->provider_leaf_cfg, + where, size, val); + if (bus == f->client_leaf->bus && devfn == f->client_leaf->devfn) + return acs_port_read(f->client_leaf, &f->client_leaf_cfg, + where, size, val); + + return PCIBIOS_SUCCESSFUL; +} + +static int acs_fabric_write(struct pci_bus *bus, unsigned int devfn, + int where, int size, u32 val) +{ + return PCIBIOS_SUCCESSFUL; +} + +static struct pci_ops acs_fabric_ops = { + .read = acs_fabric_read, + .write = acs_fabric_write, +}; + +static struct pci_bus *acs_add_bus(struct kunit *test, struct pci_bus *parent, + struct pci_dev *self, u8 nr, void *sysdata) +{ + struct pci_bus *bus = kunit_kzalloc(test, sizeof(*bus), GFP_KERNEL); + + KUNIT_ASSERT_NOT_NULL(test, bus); + bus->parent = parent; + bus->self = self; + bus->number = nr; + bus->ops = &acs_fabric_ops; + bus->sysdata = sysdata; + INIT_LIST_HEAD(&bus->devices); + return bus; +} + +static struct pci_dev *acs_add_dev(struct kunit *test, struct pci_bus *bus, + unsigned int devfn, int pcie_type) +{ + struct pci_dev *dev = kunit_kzalloc(test, sizeof(*dev), GFP_KERNEL); + + KUNIT_ASSERT_NOT_NULL(test, dev); + dev->bus = bus; + dev->devfn = devfn; + dev->pcie_cap = 0x40; + dev->pcie_flags_reg = (pcie_type << 4) | 0x2; + list_add_tail(&dev->bus_list, &bus->devices); + return dev; +} + +static void acs_build_fabric(struct kunit *test, struct acs_fabric *f) +{ + struct pci_bus *bus0, *bus1, *bus2, *bus3, *bus4, *bus5, *bus6; + struct pci_bus *bus7, *bus8; + struct pci_dev *swup, *provider_swup, *client_swup; + struct pci_dev *rootport; + struct pci_host_bridge *host; + + host = kunit_kzalloc(test, sizeof(*host), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, host); + + bus0 = acs_add_bus(test, NULL, NULL, 0, f); + /* The Root Port doubles as the whitelisted host-bridge device. */ + rootport = acs_add_dev(test, bus0, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_ROOT_PORT); + f->rootport = rootport; + rootport->vendor = PCI_VENDOR_ID_GOOGLE; + rootport->device = 0x1234; + host->bus = bus0; + bus0->bridge = &host->dev; + + bus1 = acs_add_bus(test, bus0, rootport, 1, f); + swup = acs_add_dev(test, bus1, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_UPSTREAM); + + bus2 = acs_add_bus(test, bus1, swup, 2, f); + f->dn0 = acs_add_dev(test, bus2, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_DOWNSTREAM); + f->dn1 = acs_add_dev(test, bus2, PCI_DEVFN(1, 0), + PCI_EXP_TYPE_DOWNSTREAM); + + bus3 = acs_add_bus(test, bus2, f->dn0, 3, f); + provider_swup = acs_add_dev(test, bus3, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_UPSTREAM); + bus5 = acs_add_bus(test, bus3, provider_swup, 5, f); + f->provider_leaf = acs_add_dev(test, bus5, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_DOWNSTREAM); + bus7 = acs_add_bus(test, bus5, f->provider_leaf, 7, f); + f->provider = acs_add_dev(test, bus7, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_ENDPOINT); + + bus4 = acs_add_bus(test, bus2, f->dn1, 4, f); + client_swup = acs_add_dev(test, bus4, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_UPSTREAM); + bus6 = acs_add_bus(test, bus4, client_swup, 6, f); + f->client_leaf = acs_add_dev(test, bus6, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_DOWNSTREAM); + bus8 = acs_add_bus(test, bus6, f->client_leaf, 8, f); + f->client = acs_add_dev(test, bus8, PCI_DEVFN(0, 0), + PCI_EXP_TYPE_ENDPOINT); +} + +static enum pci_p2pdma_map_type acs_walk_map(struct acs_fabric *f, + unsigned int tlp_flags) +{ + int dist; + + return calc_map_type_and_dist(f->provider, f->client, &dist, tlp_flags, + false); +} + +static void acs_walk_bus_addr_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), PCI_P2PDMA_MAP_BUS_ADDR); +} + +static void acs_walk_request_redirect_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.dn1->acs_cap = 0x100; + f.dn1->acs_capabilities = PCI_ACS_RR; + f.dn1_cfg.ctrl = PCI_ACS_RR; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); +} + +static void acs_walk_completion_redirect_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.dn0->acs_cap = 0x100; + f.dn0->acs_capabilities = PCI_ACS_CR; + f.dn0_cfg.ctrl = PCI_ACS_CR; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); +} + +static void acs_walk_egress_control_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.dn1->acs_cap = 0x100; + f.dn1->acs_capabilities = PCI_ACS_EC; + f.dn1_cfg.ctrl = PCI_ACS_EC; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); +} + +static void acs_walk_asymmetric_direct_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + /* These controls affect only the reverse transaction directions. */ + f.dn0->acs_cap = 0x100; + f.dn0->acs_capabilities = PCI_ACS_RR | PCI_ACS_EC; + f.dn0_cfg.ctrl = PCI_ACS_RR | PCI_ACS_EC; + f.dn1->acs_cap = 0x100; + f.dn1->acs_capabilities = PCI_ACS_CR; + f.dn1_cfg.ctrl = PCI_ACS_CR; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), PCI_P2PDMA_MAP_BUS_ADDR); +} + +static void acs_walk_nested_completion_redirect_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.provider_leaf->acs_cap = 0x100; + f.provider_leaf->acs_capabilities = PCI_ACS_CR; + f.provider_leaf_cfg.ctrl = PCI_ACS_CR; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), PCI_P2PDMA_MAP_BUS_ADDR); +} + +static void acs_walk_nested_request_redirect_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.client_leaf->acs_cap = 0x100; + f.client_leaf->acs_capabilities = PCI_ACS_RR; + f.client_leaf_cfg.ctrl = PCI_ACS_RR; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), PCI_P2PDMA_MAP_BUS_ADDR); +} + +static void acs_walk_translation_blocking_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.client_leaf->acs_cap = 0x100; + f.client_leaf->acs_capabilities = PCI_ACS_TB; + f.client_leaf_cfg.ctrl = PCI_ACS_TB; + + /* Untranslated Requests are unaffected by Translation Blocking. */ + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), PCI_P2PDMA_MAP_BUS_ADDR); + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, PCI_P2PDMA_TLP_TRANSLATED), + PCI_P2PDMA_MAP_NOT_SUPPORTED); +} + +static void acs_walk_relaxed_completion_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.dn0->acs_cap = 0x100; + f.dn0->acs_capabilities = PCI_ACS_CR; + f.dn0_cfg.ctrl = PCI_ACS_CR; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, PCI_P2PDMA_TLP_RELAXED_CPL), + PCI_P2PDMA_MAP_BUS_ADDR); +} + +static void acs_walk_direct_translated_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.dn1->acs_cap = 0x100; + f.dn1->acs_capabilities = PCI_ACS_RR | PCI_ACS_DT; + f.dn1_cfg.ctrl = PCI_ACS_RR | PCI_ACS_DT; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, PCI_P2PDMA_TLP_TRANSLATED), + PCI_P2PDMA_MAP_BUS_ADDR); +} + +/* + * The cache stores one packed value per client, so every class has to come + * back out under the flags that selected it. + */ +static void acs_map_types_pack_test(struct kunit *test) +{ + static const enum pci_p2pdma_map_type type[PCI_P2PDMA_TLP_CLASSES] = { + [0] = PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, + [PCI_P2PDMA_TLP_TRANSLATED] = PCI_P2PDMA_MAP_NOT_SUPPORTED, + [PCI_P2PDMA_TLP_RELAXED_CPL] = PCI_P2PDMA_MAP_BUS_ADDR, + [PCI_P2PDMA_TLP_TRANSLATED | PCI_P2PDMA_TLP_RELAXED_CPL] = + PCI_P2PDMA_MAP_UNKNOWN, + }; + unsigned long packed = pci_p2pdma_map_types_pack(type); + unsigned int flags; + + for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) + KUNIT_EXPECT_EQ(test, + pci_p2pdma_map_types_unpack(packed, flags), + type[flags]); + + /* An absent cache entry reads back as unknown in every class. */ + for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) + KUNIT_EXPECT_EQ(test, pci_p2pdma_map_types_unpack(0, flags), + PCI_P2PDMA_MAP_UNKNOWN); +} + +/* + * The provider can be an ancestor of the client, which leaves no divergence + * to evaluate. Translation Blocking still applies to every port the Request + * passes on its way up. + */ +static void acs_walk_ancestor_provider_tb_test(struct kunit *test) +{ + struct acs_fabric f = {}; + int dist; + + acs_build_fabric(test, &f); + f.client_leaf->acs_cap = 0x100; + f.client_leaf->acs_capabilities = PCI_ACS_TB; + f.client_leaf_cfg.ctrl = PCI_ACS_TB; + + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(f.dn1, f.client, &dist, 0, + false), + PCI_P2PDMA_MAP_BUS_ADDR); + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(f.dn1, f.client, &dist, + PCI_P2PDMA_TLP_TRANSLATED, + false), + PCI_P2PDMA_MAP_NOT_SUPPORTED); +} + +/* + * Without a common upstream bridge the Request still climbs towards the host + * bridge, so Translation Blocking on the way withdraws the Translated classes + * there too. + */ +static void acs_walk_no_common_bridge_tb_test(struct kunit *test) +{ + struct acs_fabric f = {}, g = {}; + int dist; + + acs_build_fabric(test, &f); + acs_build_fabric(test, &g); + f.client_leaf->acs_cap = 0x100; + f.client_leaf->acs_capabilities = PCI_ACS_TB; + f.client_leaf_cfg.ctrl = PCI_ACS_TB; + + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(g.provider, f.client, &dist, 0, + false), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(g.provider, f.client, &dist, + PCI_P2PDMA_TLP_TRANSLATED, + false), + PCI_P2PDMA_MAP_NOT_SUPPORTED); +} + +/* + * A Request between the client and itself never leaves the device, so no port + * is in a position to inspect its Address Type. Translation Blocking directly + * above the client must not withdraw the Translated classes. + */ +static void acs_walk_self_dma_tb_test(struct kunit *test) +{ + struct acs_fabric f = {}; + int dist; + + acs_build_fabric(test, &f); + f.client_leaf->acs_cap = 0x100; + f.client_leaf->acs_capabilities = PCI_ACS_TB; + f.client_leaf_cfg.ctrl = PCI_ACS_TB; + + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(f.client, f.client, &dist, 0, + false), + PCI_P2PDMA_MAP_BUS_ADDR); + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(f.client, f.client, &dist, + PCI_P2PDMA_TLP_TRANSLATED, + false), + PCI_P2PDMA_MAP_BUS_ADDR); +} + +/* + * A direct route turns around at the divergence, so Translation Blocking above + * it does not touch one. The host bridge route keeps climbing past that port, + * and the Translated classes have to go without it. + */ +static void acs_walk_tb_above_divergence_test(struct kunit *test) +{ + struct acs_fabric f = {}; + int dist; + + acs_build_fabric(test, &f); + f.rootport->acs_cap = 0x100; + f.rootport->acs_capabilities = PCI_ACS_TB; + f.rootport_cfg.ctrl = PCI_ACS_TB; + + /* Nothing redirects yet, so the Request never reaches the Root Port. */ + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(f.provider, f.client, &dist, + PCI_P2PDMA_TLP_TRANSLATED, + false), + PCI_P2PDMA_MAP_BUS_ADDR); + + /* Request Redirect sends it up past the Root Port instead. */ + f.dn1->acs_cap = 0x100; + f.dn1->acs_capabilities = PCI_ACS_RR; + f.dn1_cfg.ctrl = PCI_ACS_RR; + + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(f.provider, f.client, &dist, 0, + false), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); + KUNIT_EXPECT_EQ(test, + calc_map_type_and_dist(f.provider, f.client, &dist, + PCI_P2PDMA_TLP_TRANSLATED, + false), + PCI_P2PDMA_MAP_NOT_SUPPORTED); +} + +static void acs_walk_unreadable_control_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.dn1->acs_cap = 0x100; + f.dn1_cfg.fail_read = true; + + KUNIT_EXPECT_EQ(test, acs_walk_map(&f, 0), + PCI_P2PDMA_MAP_NOT_SUPPORTED); +} + static struct kunit_case pci_acs_test_cases[] = { KUNIT_CASE_PARAM(pci_acs_p2pdma_request_test, acs_request_gen_params), KUNIT_CASE_PARAM(pci_acs_p2pdma_completion_test, acs_completion_gen_params), + KUNIT_CASE(acs_walk_bus_addr_test), + KUNIT_CASE(acs_walk_request_redirect_test), + KUNIT_CASE(acs_walk_completion_redirect_test), + KUNIT_CASE(acs_walk_egress_control_test), + KUNIT_CASE(acs_walk_asymmetric_direct_test), + KUNIT_CASE(acs_walk_nested_completion_redirect_test), + KUNIT_CASE(acs_walk_nested_request_redirect_test), + KUNIT_CASE(acs_walk_translation_blocking_test), + KUNIT_CASE(acs_walk_ancestor_provider_tb_test), + KUNIT_CASE(acs_walk_no_common_bridge_tb_test), + KUNIT_CASE(acs_walk_self_dma_tb_test), + KUNIT_CASE(acs_walk_relaxed_completion_test), + KUNIT_CASE(acs_walk_direct_translated_test), + KUNIT_CASE(acs_walk_tb_above_divergence_test), + KUNIT_CASE(acs_walk_unreadable_control_test), + KUNIT_CASE(acs_map_types_pack_test), {} };

From: Leon Romanovsky leonro@nvidia.com

The P2PDMA documentation stated that the mapping result is not defined for Relaxed Ordering or ATS-translated Requests. It now is.

Replace that paragraph with what the three TLP-sensitive ACS controls do, the table of outcomes per class, and the interface a client uses to name its traffic. Record that a caller claiming Relaxed Ordering Completions is asserting provider behaviour the PCIe specification leaves optional.

Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- Documentation/driver-api/pci/p2pdma.rst | 64 +++++++++++++++++++++++++++++---- 1 file changed, 58 insertions(+), 6 deletions(-)

diff --git a/Documentation/driver-api/pci/p2pdma.rst b/Documentation/driver-api/pci/p2pdma.rst index 42b18610bf7d..7a7f29f455b0 100644 --- a/Documentation/driver-api/pci/p2pdma.rst +++ b/Documentation/driver-api/pci/p2pdma.rst @@ -28,12 +28,64 @@ through the host bridge when either applicable port redirects. If an ACS Control register cannot be read, P2P DMA is rejected because the kernel cannot establish a usable route.

-This evaluation assumes clients issue strictly ordered Requests carrying an -Untranslated address. Its result is not defined when clients use Relaxed -Ordering or issue ATS-translated Requests because those TLP attributes can -select different routes through the fabric. Unless ACS Translation Blocking -is enabled, a Port with ACS Direct Translated P2P enabled routes a -Translated Request directly to the peer regardless of the redirect controls. +Three of those controls act on TLP attributes that the client chooses rather +than on the topology, so the same path routes differently for different +traffic. ACS Translation Blocking rejects any Request whose Address Type is +not Untranslated, and takes precedence over every other P2P control. ACS +Direct Translated P2P routes a Translated Request to the peer regardless of +Request Redirect and Egress Control. ACS Completion Redirect leaves alone +Completions that carry the Relaxed Ordering attribute. + +A client therefore describes its traffic with ``enum pci_p2pdma_tlp_flags`` +and asks ``pci_p2pdma_map_type_tlp()``. ``pci_p2pdma_map_type()`` answers for +the default: strictly ordered Requests carrying an Untranslated address. + +The two directions are decided independently. Translation Blocking (TB), +Direct Translated P2P (DT), Request Redirect (RR) and Egress Control (EC) on +the client-side port decide the Request: + +===== ===== ======= ============ ========== +TB DT RR/EC TLP class Request +===== ===== ======= ============ ========== +set x x translated blocked +clear set x translated direct +clear clear clear translated direct +clear clear set translated redirected +x x clear untranslated direct +x x set untranslated redirected +===== ===== ======= ============ ========== + +Completion Redirect (CR) on the provider-side port decides the Completions: + +===== ========= ========== +CR TLP class Completion +===== ========= ========== +x relaxed direct +clear strict direct +set strict redirected +===== ========= ========== + +Combining the two tables gives the mapping type. A class is bus addressable +when its Request and its Completions both route directly, and anything else +goes through the host bridge. A blocked Request has neither route, because +Translation Blocking rejects the Address Type itself rather than the +destination. With no ACS control set anywhere, every class routes directly. + +Note that DT only matters where RR or EC would otherwise redirect: it +overrides them for a Translated address rather than granting a direct route +that was not already there. + +Translation Blocking is not a routing control, so it is evaluated on every +port the Request passes rather than at the divergence alone. A direct route +turns around at the divergence and only passes the ports below it, while the +host bridge route keeps climbing and passes that port and everything above it +as well. Neither route falls back to the other, because the Address Type is +rejected wherever the Request is addressed. + +The Completer chooses whether a Completion carries Relaxed Ordering, and the +PCIe specification does not require it to copy that attribute from the +Request. A caller passing ``PCI_P2PDMA_TLP_RELAXED_CPL`` is asserting that +its provider does.

However, if the P2P transaction reaches the host bridge then it might have to hairpin back out the same root port, be routed inside the CPU SOC to another

From: Leon Romanovsky leonro@nvidia.com

An importer attached to a VFIO dma-buf cannot tell whether its peer-to-peer traffic would reach the exported BAR directly, because the &struct p2pdma_provider describing it stays in vfio_pci_dma_buf.

Implement @p2pdma_provider so dma_buf_p2pdma_map_type() can reach it. Refuse once the buffer is revoked: vfio_pci_dma_buf_cleanup() revokes before it drops the device registration that keeps the provider allocated, so a revoked buffer has nothing left to describe.

Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/vfio/pci/vfio_pci_dmabuf.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c index c16f460c01d6..fcf392cfc3c5 100644 --- a/drivers/vfio/pci/vfio_pci_dmabuf.c +++ b/drivers/vfio/pci/vfio_pci_dmabuf.c @@ -99,11 +99,23 @@ static void vfio_pci_dma_buf_release(struct dma_buf *dmabuf) kfree(priv); }

+static struct p2pdma_provider * +vfio_pci_dma_buf_provider(struct dma_buf *dmabuf) +{ + struct vfio_pci_dma_buf *priv = dmabuf->priv; + + if (priv->revoked) + return NULL; + + return priv->provider; +} + static const struct dma_buf_ops vfio_pci_dmabuf_ops = { .attach = vfio_pci_dma_buf_attach, .map_dma_buf = vfio_pci_dma_buf_map, .unmap_dma_buf = vfio_pci_dma_buf_unmap, .release = vfio_pci_dma_buf_release, + .p2pdma_provider = vfio_pci_dma_buf_provider, };

/*

From: Leon Romanovsky leonro@nvidia.com

mlx5_umem_needs_ats() enables ATS for any dma-buf whose caller asked for Relaxed Ordering, on the assumption that a switch in the path has CR, RR and DT all set. It also enables it for a buffer already mapped with the peer's bus addresses, which are not translatable at all.

P2PDMA has read the ACS controls, so ask it through dma_buf_p2pdma_map_type(): enable ATS only where the path is not routed directly as it stands, but would be for a Translated Request whose Completions carry Relaxed Ordering. Exporters that name no provider keep the old assumption, since their ACS settings remain hidden.

Tested-by: Tushar Dave tdave@nvidia.com Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 36 ++------------------------- drivers/infiniband/hw/mlx5/mr.c | 47 ++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 34 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index e9ddf2e97a76..ab32742b2180 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1646,40 +1646,8 @@ static inline bool rt_supported(int ts_cap) ts_cap == MLX5_TIMESTAMP_FORMAT_CAP_FREE_RUNNING_AND_REAL_TIME; }

-/* - * PCI Peer to Peer is a trainwreck. If no switch is present then things - * sometimes work, depending on the pci_distance_p2p logic for excluding broken - * root complexes. However if a switch is present in the path, then things get - * really ugly depending on how the switch is setup. This table assumes that the - * root complex is strict and is validating that all req/reps are matches - * perfectly - so any scenario where it sees only half the transaction is a - * failure. - * - * CR/RR/DT ATS RO P2P - * 00X X X OK - * 010 X X fails (request is routed to root but root never sees comp) - * 011 0 X fails (request is routed to root but root never sees comp) - * 011 1 X OK - * 10X X 1 OK - * 101 X 0 fails (completion is routed to root but root didn't see req) - * 110 X 0 SLOW - * 111 0 0 SLOW - * 111 1 0 fails (completion is routed to root but root didn't see req) - * 111 1 1 OK - * - * Unfortunately we cannot reliably know if a switch is present or what the - * CR/RR/DT ACS settings are, as in a VM that is all hidden. Assume that - * CR/RR/DT is 111 if the ATS cap is enabled and follow the last three rows. - * - * For now assume if the umem is a dma_buf then it is P2P. - */ -static inline bool mlx5_umem_needs_ats(struct mlx5_ib_dev *dev, - struct ib_umem *umem, int access_flags) -{ - if (!MLX5_CAP_GEN(dev->mdev, ats) || !umem->is_dmabuf) - return false; - return access_flags & IB_ACCESS_RELAXED_ORDERING; -} +bool mlx5_umem_needs_ats(struct mlx5_ib_dev *dev, struct ib_umem *umem, + int access_flags);

int set_roce_addr(struct mlx5_ib_dev *dev, u32 port_num, unsigned int index, const union ib_gid *gid, diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 00e13028762a..ee8982125c8c 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -38,6 +38,7 @@ #include <linux/export.h> #include <linux/delay.h> #include <linux/dma-buf.h> +#include <linux/dma-buf-mapping.h> #include <linux/dma-resv.h> #include <rdma/frmr_pools.h> #include <rdma/ib_umem_odp.h> @@ -47,6 +48,52 @@ #include "data_direct.h" #include "dmah.h"

+MODULE_IMPORT_NS("DMA_BUF"); + +bool mlx5_umem_needs_ats(struct mlx5_ib_dev *dev, struct ib_umem *umem, + int access_flags) +{ + struct dma_buf_attachment *attach; + bool ats; + + if (!MLX5_CAP_GEN(dev->mdev, ats) || !umem->is_dmabuf) + return false; + + /* + * The Completer decides whether its Completions carry Relaxed + * Ordering, and only a Request that asked for it can expect them to. + */ + if (!(access_flags & IB_ACCESS_RELAXED_ORDERING)) + return false; + + attach = to_ib_umem_dmabuf(umem)->attach; + + dma_resv_lock(attach->dmabuf->resv, NULL); + switch (dma_buf_p2pdma_map_type(attach, 0)) { + case PCI_P2PDMA_MAP_NONE: + /* Nothing is known about the route, so fall back to the bet. */ + ats = true; + break; + case PCI_P2PDMA_MAP_BUS_ADDR: + /* + * The path is routed directly already and is programmed with + * the peer's bus addresses. Those are not translatable, so + * ATS would be wrong as well as pointless. + */ + ats = false; + break; + default: + ats = dma_buf_p2pdma_map_type(attach, + PCI_P2PDMA_TLP_TRANSLATED | + PCI_P2PDMA_TLP_RELAXED_CPL) == + PCI_P2PDMA_MAP_BUS_ADDR; + break; + } + dma_resv_unlock(attach->dmabuf->resv); + + return ats; +} + static int mkey_max_umr_order(struct mlx5_ib_dev *dev) { if (MLX5_CAP_GEN(dev->mdev, umr_extended_translation_offset))

From: Leon Romanovsky leonro@nvidia.com

An mlx5 function issues Translated Requests only through memory keys created with ATS. Enabling ATS in its PCIe configuration therefore does not make it translate every address it is handed.

Declare that where the function's P2PDMA state is set up, before P2PDMA starts treating clients with ATS enabled as translating everything. Without the declaration, mlx5 would lose bus addressing for peer-to-peer DMA whenever the IOMMU enables its ATS.

Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/infiniband/hw/mlx5/main.c | 2 ++ 1 file changed, 2 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index 373ee1f42d4a..dcf659beaba0 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -4561,6 +4561,8 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev) if (err && err != -EOPNOTSUPP) goto err_dd;

+ pcim_p2pdma_set_ats_per_mapping(mdev->pdev); + mlx5_cmd_init_async_ctx(mdev, &dev->async_ctx);

return 0;

From: Leon Romanovsky leonro@nvidia.com

A client that enables ATS for the whole device must never be handed a bus address, and Translation Blocking must reject its peer-to-peer traffic even when the caller asks for the default class. That has to hold for answers taken from the cache as well.

Cover those on the fabricated fabric, together with a client that declared per-mapping ATS keeping the per-class answers and a client without ATS answering for the Untranslated Requests it will issue.

Signed-off-by: Leon Romanovsky leonro@nvidia.com --- drivers/pci/p2pdma.c | 8 +-- drivers/pci/pci.h | 5 ++ drivers/pci/pci_acs_test.c | 123 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 133 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index ca79af2e64c7..dfef69638782 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -1595,14 +1595,15 @@ void pcim_p2pdma_set_ats_per_mapping(struct pci_dev *pdev) } EXPORT_SYMBOL_GPL(pcim_p2pdma_set_ats_per_mapping);

-static unsigned int pci_p2pdma_client_tlp_flags(struct pci_dev *client, - unsigned int tlp_flags) +VISIBLE_IF_KUNIT unsigned int +pci_p2pdma_client_tlp_flags(struct pci_dev *client, unsigned int tlp_flags) { if (client->ats_enabled) return tlp_flags | PCI_P2PDMA_TLP_TRANSLATED;

return tlp_flags & ~PCI_P2PDMA_TLP_TRANSLATED; } +EXPORT_SYMBOL_IF_KUNIT(pci_p2pdma_client_tlp_flags);

/* * A client that translates every address it is handed cannot be handed a bus @@ -1611,7 +1612,7 @@ static unsigned int pci_p2pdma_client_tlp_flags(struct pci_dev *client, * translation carries the IOVA to the host bridge, so that route has to work * as well. */ -static enum pci_p2pdma_map_type +VISIBLE_IF_KUNIT enum pci_p2pdma_map_type pci_p2pdma_client_map_type(struct pci_dev *provider, struct pci_dev *client, bool per_mapping, enum pci_p2pdma_map_type type) { @@ -1629,6 +1630,7 @@ pci_p2pdma_client_map_type(struct pci_dev *provider, struct pci_dev *client,

return PCI_P2PDMA_MAP_NOT_SUPPORTED; } +EXPORT_SYMBOL_IF_KUNIT(pci_p2pdma_client_map_type);

/** * pci_p2pdma_map_type_tlp - Determine the mapping type for P2PDMA transfers diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 5bc703ff0c86..c705603b2145 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -1117,6 +1117,11 @@ enum pci_p2pdma_map_type calc_map_type_and_dist(struct pci_dev *provider, int *dist, unsigned int tlp_flags, bool verbose); +unsigned int pci_p2pdma_client_tlp_flags(struct pci_dev *client, + unsigned int tlp_flags); +enum pci_p2pdma_map_type +pci_p2pdma_client_map_type(struct pci_dev *provider, struct pci_dev *client, + bool per_mapping, enum pci_p2pdma_map_type type); #endif

#ifdef CONFIG_PCI_QUIRKS diff --git a/drivers/pci/pci_acs_test.c b/drivers/pci/pci_acs_test.c index 6a2cd38d968f..a9a3b80ea0ac 100644 --- a/drivers/pci/pci_acs_test.c +++ b/drivers/pci/pci_acs_test.c @@ -698,6 +698,125 @@ static void acs_walk_unreadable_control_test(struct kunit *test) PCI_P2PDMA_MAP_NOT_SUPPORTED); }

+/* + * Route the way pci_p2pdma_map_type_tlp() does once it has a client: the + * client decides which flags apply, and the answer is then checked against + * what the client can be handed. + */ +static enum pci_p2pdma_map_type acs_client_map(struct acs_fabric *f, + bool per_mapping, + unsigned int tlp_flags) +{ + enum pci_p2pdma_map_type type; + int dist; + + if (!per_mapping) + tlp_flags = pci_p2pdma_client_tlp_flags(f->client, tlp_flags); + type = calc_map_type_and_dist(f->provider, f->client, &dist, tlp_flags, + false); + return pci_p2pdma_client_map_type(f->provider, f->client, per_mapping, + type); +} + +/* + * A client with ATS enabled for the whole device translates whatever it is + * handed, so it never gets bus addresses even on a direct route, and + * Translation Blocking rejects it even when its caller asks for the default + * class. + */ +static void acs_client_ats_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.client->ats_enabled = 1; + + KUNIT_EXPECT_EQ(test, acs_client_map(&f, false, 0), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); + + f.client_leaf->acs_cap = 0x100; + f.client_leaf->acs_capabilities = PCI_ACS_TB; + f.client_leaf_cfg.ctrl = PCI_ACS_TB; + + KUNIT_EXPECT_EQ(test, acs_client_map(&f, false, 0), + PCI_P2PDMA_MAP_NOT_SUPPORTED); +} + +/* + * The cache holds what the topology allows, filled when the client first + * asked. A client whose ATS was enabled after that must still be kept off the + * bus addresses the cache recorded. + */ +static void acs_client_ats_cached_test(struct kunit *test) +{ + enum pci_p2pdma_map_type type[PCI_P2PDMA_TLP_CLASSES], hit; + struct acs_fabric f = {}; + unsigned long cached; + unsigned int flags; + int dist; + + acs_build_fabric(test, &f); + for (flags = 0; flags < PCI_P2PDMA_TLP_CLASSES; flags++) + type[flags] = calc_map_type_and_dist(f.provider, f.client, + &dist, flags, false); + cached = pci_p2pdma_map_types_pack(type); + + f.client->ats_enabled = 1; + flags = pci_p2pdma_client_tlp_flags(f.client, 0); + hit = pci_p2pdma_map_types_unpack(cached, flags); + + KUNIT_EXPECT_EQ(test, + pci_p2pdma_client_map_type(f.provider, f.client, false, + hit), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); +} + +/* + * A client that declared per-mapping ATS picks the Address Type per mapping, + * so it keeps the per-class answers even with ATS enabled. + */ +static void acs_client_ats_per_mapping_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.client->ats_enabled = 1; + + KUNIT_EXPECT_EQ(test, acs_client_map(&f, true, 0), + PCI_P2PDMA_MAP_BUS_ADDR); + KUNIT_EXPECT_EQ(test, + acs_client_map(&f, true, PCI_P2PDMA_TLP_TRANSLATED), + PCI_P2PDMA_MAP_BUS_ADDR); + + f.client_leaf->acs_cap = 0x100; + f.client_leaf->acs_capabilities = PCI_ACS_TB; + f.client_leaf_cfg.ctrl = PCI_ACS_TB; + + KUNIT_EXPECT_EQ(test, acs_client_map(&f, true, 0), + PCI_P2PDMA_MAP_BUS_ADDR); + KUNIT_EXPECT_EQ(test, + acs_client_map(&f, true, PCI_P2PDMA_TLP_TRANSLATED), + PCI_P2PDMA_MAP_NOT_SUPPORTED); +} + +/* + * A client with ATS disabled cannot issue Translated Requests, so a caller + * asking about them gets the answer for the Untranslated ones it will issue. + */ +static void acs_client_no_ats_test(struct kunit *test) +{ + struct acs_fabric f = {}; + + acs_build_fabric(test, &f); + f.dn1->acs_cap = 0x100; + f.dn1->acs_capabilities = PCI_ACS_RR | PCI_ACS_DT; + f.dn1_cfg.ctrl = PCI_ACS_RR | PCI_ACS_DT; + + KUNIT_EXPECT_EQ(test, + acs_client_map(&f, false, PCI_P2PDMA_TLP_TRANSLATED), + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE); +} + static struct kunit_case pci_acs_test_cases[] = { KUNIT_CASE_PARAM(pci_acs_p2pdma_request_test, acs_request_gen_params), @@ -723,6 +842,10 @@ static struct kunit_case pci_acs_test_cases[] = { KUNIT_CASE(acs_walk_tb_above_divergence_test), KUNIT_CASE(acs_walk_unreadable_control_test), KUNIT_CASE(acs_map_types_pack_test), + KUNIT_CASE(acs_client_ats_test), + KUNIT_CASE(acs_client_ats_cached_test), + KUNIT_CASE(acs_client_ats_per_mapping_test), + KUNIT_CASE(acs_client_no_ats_test), {} };