Hi ArmNN dev team,
I am part of the team developing the ArmNN backend for the Arm NPU and have some concerns about the validation that the ArmNN core library performs on its inputs. Below is a description of how I believe validation is performed within ArmNN and the problems that I see with this. This understanding may be flawed so please correct me where I have misunderstood.
When the user creates an INetwork there is minimal validation of the data provided by the user. For example, the dimensionality of input tensors is not checked at this point. The user then calls Optimize() which performs the following steps:
1. InferTensorInfos() - this calls ValidateTensorShapesFromInputs on each Layer in the Graph which confirms that the output tensor shape set on each Layer during Network construction is consistent with the Layer's inputs. For the example of a FullyConnectedLayer, this uses the shape of the input and the shape of the weights to determine the correct output shape. This code seems to make assumptions about the dimensionality of the inputs tensors, for example FullyConnectedLayer::InferOutputShapes() indexes into the input and weight shapes without checking their dimensionality first. 2. AssignBackends() - this calls each backend's IsLayerSupported() APIs. The only data that has been validated so far is that the output shapes of each layer are correct, so the backend IsLayerSupported() APIs cannot assume anything about the shapes of the tensors. This means the backends must perform additional validation. 3. ApplyBackendOptimizations() - this gives each backend the opportunity to "optimize" each subgraph which has been assigned to it. Again, the layers passed to the backend still have not been properly validated, although the backend has had the chance to reject the layers via the IsLayerSupported() APIs.
The user then creates a LoadedNetwork from the IOptimizedNetwork which creates the Workloads. This is delegated to the backend's IWorkloadFactory which is responsible for returning an object implementing IWorkload. In the case of the default backends (reference, Neon, CL), these workloads derive from BaseWorkload, which calls Validate() on the QueueDescriptor for that workload type. This is the place that seems to perform the "proper" validation of what is supported by ArmNN. In the example of Fully Connected, FullyConnectedQueueDescriptor::Validate checks the dimensionality of all tensors, the quantisastion infos, etc. Note that there seems to be no requirement that this validation code is called at all, in the case that the backend-created workloads do not inherit BaseWorkload (this is always the case for backends which replace subgraphs with PreCompiledLayers).
The problems that this causes are as follows:
* The InferTensorInfos() code could crash as it makes assumptions that have not been validated * Every backend's IsLayerSupported APIs must duplicate the validation code in ArmNN in order to check that the layer is valid, before they even get to the point of checking if that particular backend supports it. * The "proper" ArmNN layer validation code may never be run, depending on how the backend processes the graph. Specifically, in the case of backends which replace subgraphs with PreCompiledLayers, the validation code is never run. * These problems affect both end users of the ArmNN API and backend developers
I would suggest that a better method of validation would be to validate the INetwork completely before it is processed any further. This could be done during construction of the INetwork or as the first step in Optimize(). This would simplify the backend code as it would not need to duplicate ArmNN's validation code and give a more consistent interface to end users.
Please let me know your thoughts, Thanks, Rob
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.