On Tue, Apr 19, 2022 at 12:22:08PM +0100, Mark Brown wrote:
This series provides initial support for the ARMv9 Scalable Matrix Extension (SME). SME takes the approach used for vectors in SVE and extends this to provide architectural support for matrix operations. A more detailed overview can be found in [1].
For the kernel SME can be thought of as a series of features which are intended to be used together by applications but operate mostly orthogonally:
- The ZA matrix register.
- Streaming mode, in which ZA can be accessed and a subset of SVE features are available.
- A second vector length, used for streaming mode SVE and ZA and controlled using a similar interface to that for SVE.
- TPIDR2, a new userspace controllable system register intended for use by the C library for storing context related to the ZA ABI.
A substantial part of the series is dedicated to refactoring the existing SVE support so that we don't need to duplicate code for handling vector lengths and the SVE registers, this involves creating an array of vector types and making the users take the vector type as a parameter. I'm not 100% happy with this but wasn't able to come up with anything better, duplicating code definitely felt like a bad idea so this felt like the least bad thing. If this approach makes sense to people it might make sense to split this off into a separate series and/or merge it while the rest is pending review to try to make things a little more digestable, the series is very large so it'd probably make things easier to digest if some of the preparatory refactoring could be merged before the rest is ready.
One feature of the architecture of particular note is that switching to and from streaming mode may change the size of and invalidate the contents of the SVE registers, and when in streaming mode the FFR is not accessible. This complicates aspects of the ABI like signal handling and ptrace.
This initial implementation is mainly intended to get the ABI in place, there are several areas which will be worked on going forwards - some of these will be blockers, others could be handled in followup serieses:
- SME is currently not supported for KVM guests, this will be done as a followup series. A host system can use SME and run KVM guests but SME is not available in the guests.
- The KVM host support is done in a very simplistic way, were anyone to attempt to use it in production there would be performance impacts on hosts with SME support. As part of this we also add enumeration of fine grained traps.
- There is not currently ptrace or signal support TPIDR2, this will be done as a followup series.
- No support is currently provided for scheduler control of SME or SME applications, given the size of the SME register state the context switch overhead may be noticable so this may be needed especially for real time applications. Similar concerns already exist for larger SVE vector lengths but are amplified for SME, particularly as the vector length increases.
- There has been no work on optimising the performance of anything the kernel does.
It is not expected that any systems will be encountered that support SME but not SVE, SME is an ARMv9 feature and SVE is mandatory for ARMv9. The code attempts to handle any such systems that are encountered but this hasn't been tested extensively.
Running CPU offline/online on a Neoverse-N1 server will trigger a crash. A data point is setting CONFIG_ARM64_SVE=n could avoid it.
kernel BUG at arch/arm64/kernel/cpufeature.c:1353! Internal error: Oops - BUG: 0 [#1] PREEMPT SMP CPU: 88 PID: 0 Comm: swapper/88 Not tainted 5.18.0-rc4-next-20220426-00006-gfea0cdfbc1de #60 pstate: 204001c9 (nzCv dAIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : __read_sysreg_by_encoding lr : has_cpuid_feature sp : ffff80000a827d10 x29: ffff80000a827d10 x28: 0000000000000000 x27: ffffbb3c708efb8a x26: 1ffff7678e11df71 x25: 0000000000000002 x24: 0000000000000003 x23: ffffbb3c75870e80 x22: dfff800000000000 x21: 0000000000000029 x20: ffffbb3c708efba0 x19: ffffbb3c708efb80 x18: ffffbb3c73eb7d1c x17: 000000040044ffff x16: 1fffe7fff0e51474 x15: 1fffe806c1d7b54a x14: 1fffe7fff0e5146c x13: 0000000000000004 x12: ffff77678e839850 x11: 1ffff7678e83984f x10: ffff77678e83984f x9 : ffffbb3c6d7deef0 x8 : ffffbb3c741cc27f x7 : 0000000000000001 x6 : ffff77678e83984f x5 : ffffbb3c741cc278 x4 : 0000000000000000 x3 : 1fffe7fff0e51359 x2 : 1ffff7678e11df74 x1 : 0000000000180480 x0 : 00000000001804a0 Call trace: __read_sysreg_by_encoding has_cpuid_feature verify_local_cpu_caps verify_local_cpu_capabilities check_local_cpu_capabilities secondary_start_kernel __secondary_switched Code: 17ffff34 d5380234 17ffff32 f90013f5 (d4210000) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops - BUG: Fatal exception SMP: stopping secondary CPUs Kernel Offset: 0x3b3c657a0000 from 0xffff800008000000 PHYS_OFFSET: 0x80000000 CPU features: 0x000,0021700d,19801c82 Memory Limit: none ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception ]---