And, at least in our case, the driver for display controller may be
present, but sgx kernel module may not. To make things look "normal" to userspace, I've taken the approach of writing a DRM driver that handles the KMS stuff, and has a plug-in API[1] that other drivers, such as sgx, can register their own ioctl's and that sort of thing.. there are a few rough edges to sort out to make more than a single plugin practical, but perhaps with an approach like this the vendor could implement a platform specific DRM driver to handle the mode setting and display bits, while ARM, IMG, etc (IP vendor) could each use a standard plugin interface to write their own drivers to work with the vendor provided display driver.
I remember Dave Airlie proposed something that sounded similar for the upstream DRM, with options for display, graphics and compute, so maybe this is something that could fit upstream fairly easily ?