On Fri, Aug 19, 2011 at 5:18 AM, Pauli Nieminen pauli.nieminen@linux.intel.com wrote:
On Thu, Aug 18, 2011 at 09:58:07PM -0500, Rob Clark wrote:
From: Rob Clark rob@ti.com
To allow the potential use of overlays to display video content, a few extra parameters are required:
+ source buffer in different format (for example, various YUV formats) and size as compared to destination drawable + multi-planar formats where discontiguous buffers are used for different planes. For example, luma and chroma split across multiple memory banks or with different tiled formats. + flipping between multiple back buffers, perhaps not in order (to handle video formats with B-frames) + cropping during swap.. in case of video, perhaps the required hw buffers are larger than the visible picture to account for codec borders (for example, reference frames where a block/macroblock moves past the edge of the visible picture, but back again in subsequent frames).
Current solutions use the GPU to do a scaled/colorconvert into a DRI2 buffer from the client context. The goal of this protocol change is to push the decision to use overlay or GPU blit to the xorg driver.
Eventually this should replace Xv. With a few additions, like attributes, it could perhaps be possible to implement the client side Xv API on top of dri2.
Note: video is not exactly the same as 3d, there are a number of other things to consider (scaling, colorconvert, multi-planar formats). But on the other hand the principle is similar (direct rendering from hw video codecs). And a lot infrastructure of connection, authentication, is same. So there are two options, either extend DRI2 or add a new protocol which duplicates some parts. I'd like to consider extending DRI2 first, but if people think the requirements for video are too much different from 3d, then I could split this into a new protocol.
In either case, I will implement the xserver side infrastructure, but I wanted to get some feel for what is the preferred approach (extend dri2 or new videoproto) first.
XvPutVideo is already existing alternative to use for video playback to overlays. But XvPutVideo API isn't perfect for the job either.
We have until now actually used an unholy marriage of v4l2 and XvPutVideo.. part of the motivation to either extend dri2 or introduce a direct-video-rendering proto is have a better solution for some of the sort-comings of the current solution:
+ currently the client has to realize it is using an overlay.. and which overlay.. switching between overlay and GPU blitting is not transparent to the client + which results in a resource management issue if there are multiple videos playing, or if you need to scale beyond what can be done w/ overlay, etc + in some cases, you might want some way for the window-manager to signal to xorg driver to switch to blitting because there are some transformational effects.. ie. wobbly windows and that sort of thing.. (I'm thinking maybe setting some property on the root window?) Ideally this would be transparent to the video player app.
some of this could be accomplished w/ XvPutImage, but then you are restricted to memory that can be obtained from shmem, which is not sufficient for hw video codecs in most cases.
How would client know which formats are supported by driver? Xv expose list of supported image formats to client.
well, in current version of the patch, the client would "just have to know".. which is sort of how it is on the 3d side for dri2.
Although if there was an attribute mechanism, then I guess we could actually make it possible to have a completely generic client. One other issue to tackle then is what exactly is the buffer name. Currently for dri2 this is left as an implementation detail between the client and server. But we could decree that it is a GEM buffer flink name, or possibly that it is a dmabuf file-descriptor (see http://lists.linaro.org/pipermail/linaro-mm-sig/2011-August/000509.html )
For a first step, I was ok that the client side of the dri2 connection would have to know some implementation details of the xorg driver that it is talking too, since this is how it already is for dri2, and it could be handled by an approach like vaapi/vdpau where there is some driver specific code on the client side, above dri2.
But on the other hand, allowing for a completely generic client using the video/dri2 proto could be interesting. I guess it depends on what folk's general feeling is.. if completely generic client side is important, then I'll go back and start adding attributes and this sort of thing. I think it is doable although it would require adding more to dri2proto than what I have proposed so far.
dri2proto.txt | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 59 insertions(+), 1 deletions(-)
diff --git a/dri2proto.txt b/dri2proto.txt index df763c7..aa83b1a 100644 --- a/dri2proto.txt +++ b/dri2proto.txt @@ -163,7 +163,8 @@ and DRI2InvalidateBuffers. 6. Protocol Types
DRI2DRIVER { DRI2DriverDRI
- DRI2DriverVDPAU }
- DRI2DriverVDPAU,
- DRI2DriverXV }
These values describe the type of driver the client will want to load. The server sends back the name of the driver to use @@ -184,6 +185,10 @@ DRI2ATTACHMENT { DRI2BufferFrontLeft These values describe various attachment points for DRI2 buffers.
- In the case of video driver (DRI2DriverXV) the attachment,
- other than DRI2BufferFrontLeft, just indicates buffer
- number and has no other special significance.
DRI2BUFFER { attachment: CARD32 name: CARD32 pitch: CARD32 @@ -203,6 +208,16 @@ DRI2ATTACH_FORMAT { attachment: CARD32 format. 'attachment' describes the attachment point for the buffer, 'format' describes an opaque, device-dependent format for the buffer.
+DRI2ATTACH_VIDEO { attachment: CARD32
- format: CARD32,
- width, height: CARD32 }
- The DRI2ATTACH_VIDEO describes an attachment and the associated
- format for video buffers. 'attachment' describes the attachment
- point for the buffer, 'format' describes a fourcc value for the
- buffer.
I think you will anyway need Xv port attribute like system to expose HW features like color conversion. Width and height for buffers source buffers could be one of attributes.
If width and height are in attributes video can use DRI2GetBuffersWithFormat.
Format should include already information about excepted memory layout in buffer. Format would also include information about memory layout (packed/planar and tiled/linear)
agreed, if we go the attribute route, then DRI2GetBuffersWithFormat would be enough
⚙ ⚙ ⚙ ⚙ ⚙ ⚙
@@ -367,6 +382,15 @@ The name of this extension is "DRI2". later.
┌───
- DRI2GetVideoBuffers
- drawable: DRAWABLE
- attachments: LISTofDRI2ATTACH_VIDEO
- ▶
- width, height: CARD32
- buffers: LISTofDRI2BUFFER
+└───
+┌─── DRI2GetMSC drawable: DRAWABLE ▶ @@ -585,11 +609,21 @@ A.1 Common Types 4 CARD32 pitch 4 CARD32 cpp 4 CARD32 flags
- 4 n extra names length
- 4n LISTof extra names
└─── A DRI2 buffer specifies the attachment, the kernel memory manager name, the pitch and chars per pixel for a buffer attached to a given drawable.
- In case of multi-planar video formats, 'extra names' will give the
- list of additional buffer names if there is one buffer per plane.
- For example, I420 has one Y plane in with a 8bit luma value per
- pixel, followed by one U plane subsampled 2x2 (with one 8bit U value
- per 2x2 pixel block), followed by one V plane subsampled 2x2. This
- could either be represented as a single buffer name, or three
- separate buffer names, one each for Y, U, and V.
I think buffer format should already describe the internal memory layout for buffer.
Do you have example where client couldn't know memory layout from format, width, height and stride?
well, it isn't so much a format/width/height/stride issue. But it is likely that the different planes would not be contiguous in a single buffer. For example, some SoC's require that luma and chroma exist in different memory banks (different range of physical addresses). For TI OMAP, we use different tiling formats for luma and chroma in an NV12 buffer. Having the option for multiple buffer names per attachment point seemed to give the most flexibility in implementation.
It would be possible to stuff these all under a single buffer name, although that brings some wrinkles.. for example, does plane n+1 start immediately after the previous plane (which it would for contiguous buffers) or on next page boundary (which would be more likely for non-contiguous buffers). And if different tiling formats are used, that further complicates the kernel side code in case the buffer is mmap'd to userspace.
Having the option for either one buffer name, or one buffer name per plane avoids those complications. But I'm open to hear other opinions on the idea of 1 vs n names per attachment point.
BR, -R
┌─── DRI2ATTACH_FORMAT 4 CARD32 attachment @@ -599,6 +633,17 @@ A.1 Common Types This data type is only available with protocol version 1.1 or later.
+┌───
- DRI2ATTACH_VIDEO
- 4 CARD32 attachment
- 4 CARD32 format
- 4 CARD32 width
- 4 CARD32 height
+└───
- Used to describe the attachment and format requested from the server.
- This data type is only available with protocol version 1.? or
- later.
A.2 Protocol Requests
┌─── @@ -745,6 +790,11 @@ A.2 Protocol Requests 4 CARD32 divisor_lo 4 CARD32 remainder_hi 4 CARD32 remainder_lo
- 4 DRI2ATTACHMENT source
- 4 CARD32 x1
- 4 CARD32 y1
- 4 CARD32 x2
- 4 CARD32 y2
▶ 1 1 Reply 1 unused @@ -754,6 +804,14 @@ A.2 Protocol Requests 4 CARD32 swap_lo 5n LISTofDRI2BUFFER buffers └───
- The 'source', if not zero (DRI2BufferFrontLeft) indicates the
- attachment point of the buffer to swap w/ DRI2BufferFrontLeft.
- If zero is specified, DRI2BufferBackLeft is swapped with the
- DRI2BufferFrontLeft buffer, for compatibility.
- If 'source' is not zero, (x1,y1), (x2,y2) specify the bounding
- box in coordinates of the source buffer which should be scaled
- to (0,0), (width,height) of the destination drawable.
┌─── DRI2GetMSC -- 1.7.5.4
xorg-devel@lists.x.org: X.Org development Archives: http://lists.x.org/archives/xorg-devel Info: http://lists.x.org/mailman/listinfo/xorg-devel
fwd'ing because my original reply from other account didn't seem to be allowed by linaro-dev
---------- Forwarded message ---------- From: Rob Clark robdclark@gmail.com Date: Fri, Aug 19, 2011 at 8:43 AM Subject: Re: [PATCH dri2proto] RFC: video support for dri2 To: Pauli Nieminen pauli.nieminen@linux.intel.com Cc: xorg-devel@lists.freedesktop.org, linaro-dev@lists.linaro.org
On Fri, Aug 19, 2011 at 5:18 AM, Pauli Nieminen pauli.nieminen@linux.intel.com wrote:
On Thu, Aug 18, 2011 at 09:58:07PM -0500, Rob Clark wrote:
From: Rob Clark rob@ti.com
To allow the potential use of overlays to display video content, a few extra parameters are required:
+ source buffer in different format (for example, various YUV formats) and size as compared to destination drawable + multi-planar formats where discontiguous buffers are used for different planes. For example, luma and chroma split across multiple memory banks or with different tiled formats. + flipping between multiple back buffers, perhaps not in order (to handle video formats with B-frames) + cropping during swap.. in case of video, perhaps the required hw buffers are larger than the visible picture to account for codec borders (for example, reference frames where a block/macroblock moves past the edge of the visible picture, but back again in subsequent frames).
Current solutions use the GPU to do a scaled/colorconvert into a DRI2 buffer from the client context. The goal of this protocol change is to push the decision to use overlay or GPU blit to the xorg driver.
Eventually this should replace Xv. With a few additions, like attributes, it could perhaps be possible to implement the client side Xv API on top of dri2.
Note: video is not exactly the same as 3d, there are a number of other things to consider (scaling, colorconvert, multi-planar formats). But on the other hand the principle is similar (direct rendering from hw video codecs). And a lot infrastructure of connection, authentication, is same. So there are two options, either extend DRI2 or add a new protocol which duplicates some parts. I'd like to consider extending DRI2 first, but if people think the requirements for video are too much different from 3d, then I could split this into a new protocol.
In either case, I will implement the xserver side infrastructure, but I wanted to get some feel for what is the preferred approach (extend dri2 or new videoproto) first.
XvPutVideo is already existing alternative to use for video playback to overlays. But XvPutVideo API isn't perfect for the job either.
We have until now actually used an unholy marriage of v4l2 and XvPutVideo.. part of the motivation to either extend dri2 or introduce a direct-video-rendering proto is have a better solution for some of the sort-comings of the current solution:
+ currently the client has to realize it is using an overlay.. and which overlay.. switching between overlay and GPU blitting is not transparent to the client + which results in a resource management issue if there are multiple videos playing, or if you need to scale beyond what can be done w/ overlay, etc + in some cases, you might want some way for the window-manager to signal to xorg driver to switch to blitting because there are some transformational effects.. ie. wobbly windows and that sort of thing.. (I'm thinking maybe setting some property on the root window?) Ideally this would be transparent to the video player app.
some of this could be accomplished w/ XvPutImage, but then you are restricted to memory that can be obtained from shmem, which is not sufficient for hw video codecs in most cases.
How would client know which formats are supported by driver? Xv expose list of supported image formats to client.
well, in current version of the patch, the client would "just have to know".. which is sort of how it is on the 3d side for dri2.
Although if there was an attribute mechanism, then I guess we could actually make it possible to have a completely generic client. One other issue to tackle then is what exactly is the buffer name. Currently for dri2 this is left as an implementation detail between the client and server. But we could decree that it is a GEM buffer flink name, or possibly that it is a dmabuf file-descriptor (see http://lists.linaro.org/pipermail/linaro-mm-sig/2011-August/000509.html )
For a first step, I was ok that the client side of the dri2 connection would have to know some implementation details of the xorg driver that it is talking too, since this is how it already is for dri2, and it could be handled by an approach like vaapi/vdpau where there is some driver specific code on the client side, above dri2.
But on the other hand, allowing for a completely generic client using the video/dri2 proto could be interesting. I guess it depends on what folk's general feeling is.. if completely generic client side is important, then I'll go back and start adding attributes and this sort of thing. I think it is doable although it would require adding more to dri2proto than what I have proposed so far.
dri2proto.txt | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 59 insertions(+), 1 deletions(-)
diff --git a/dri2proto.txt b/dri2proto.txt index df763c7..aa83b1a 100644 --- a/dri2proto.txt +++ b/dri2proto.txt @@ -163,7 +163,8 @@ and DRI2InvalidateBuffers. 6. Protocol Types
DRI2DRIVER { DRI2DriverDRI
- DRI2DriverVDPAU }
- DRI2DriverVDPAU,
- DRI2DriverXV }
These values describe the type of driver the client will want to load. The server sends back the name of the driver to use @@ -184,6 +185,10 @@ DRI2ATTACHMENT { DRI2BufferFrontLeft These values describe various attachment points for DRI2 buffers.
- In the case of video driver (DRI2DriverXV) the attachment,
- other than DRI2BufferFrontLeft, just indicates buffer
- number and has no other special significance.
DRI2BUFFER { attachment: CARD32 name: CARD32 pitch: CARD32 @@ -203,6 +208,16 @@ DRI2ATTACH_FORMAT { attachment: CARD32 format. 'attachment' describes the attachment point for the buffer, 'format' describes an opaque, device-dependent format for the buffer.
+DRI2ATTACH_VIDEO { attachment: CARD32
- format: CARD32,
- width, height: CARD32 }
- The DRI2ATTACH_VIDEO describes an attachment and the associated
- format for video buffers. 'attachment' describes the attachment
- point for the buffer, 'format' describes a fourcc value for the
- buffer.
I think you will anyway need Xv port attribute like system to expose HW features like color conversion. Width and height for buffers source buffers could be one of attributes.
If width and height are in attributes video can use DRI2GetBuffersWithFormat.
Format should include already information about excepted memory layout in buffer. Format would also include information about memory layout (packed/planar and tiled/linear)
agreed, if we go the attribute route, then DRI2GetBuffersWithFormat would be enough
⚙ ⚙ ⚙ ⚙ ⚙ ⚙
@@ -367,6 +382,15 @@ The name of this extension is "DRI2". later.
┌───
- DRI2GetVideoBuffers
- drawable: DRAWABLE
- attachments: LISTofDRI2ATTACH_VIDEO
- ▶
- width, height: CARD32
- buffers: LISTofDRI2BUFFER
+└───
+┌─── DRI2GetMSC drawable: DRAWABLE ▶ @@ -585,11 +609,21 @@ A.1 Common Types 4 CARD32 pitch 4 CARD32 cpp 4 CARD32 flags
- 4 n extra names length
- 4n LISTof extra names
└─── A DRI2 buffer specifies the attachment, the kernel memory manager name, the pitch and chars per pixel for a buffer attached to a given drawable.
- In case of multi-planar video formats, 'extra names' will give the
- list of additional buffer names if there is one buffer per plane.
- For example, I420 has one Y plane in with a 8bit luma value per
- pixel, followed by one U plane subsampled 2x2 (with one 8bit U value
- per 2x2 pixel block), followed by one V plane subsampled 2x2. This
- could either be represented as a single buffer name, or three
- separate buffer names, one each for Y, U, and V.
I think buffer format should already describe the internal memory layout for buffer.
Do you have example where client couldn't know memory layout from format, width, height and stride?
well, it isn't so much a format/width/height/stride issue. But it is likely that the different planes would not be contiguous in a single buffer. For example, some SoC's require that luma and chroma exist in different memory banks (different range of physical addresses). For TI OMAP, we use different tiling formats for luma and chroma in an NV12 buffer. Having the option for multiple buffer names per attachment point seemed to give the most flexibility in implementation.
It would be possible to stuff these all under a single buffer name, although that brings some wrinkles.. for example, does plane n+1 start immediately after the previous plane (which it would for contiguous buffers) or on next page boundary (which would be more likely for non-contiguous buffers). And if different tiling formats are used, that further complicates the kernel side code in case the buffer is mmap'd to userspace.
Having the option for either one buffer name, or one buffer name per plane avoids those complications. But I'm open to hear other opinions on the idea of 1 vs n names per attachment point.
BR, -R
┌─── DRI2ATTACH_FORMAT 4 CARD32 attachment @@ -599,6 +633,17 @@ A.1 Common Types This data type is only available with protocol version 1.1 or later.
+┌───
- DRI2ATTACH_VIDEO
- 4 CARD32 attachment
- 4 CARD32 format
- 4 CARD32 width
- 4 CARD32 height
+└───
- Used to describe the attachment and format requested from the server.
- This data type is only available with protocol version 1.? or
- later.
A.2 Protocol Requests
┌─── @@ -745,6 +790,11 @@ A.2 Protocol Requests 4 CARD32 divisor_lo 4 CARD32 remainder_hi 4 CARD32 remainder_lo
- 4 DRI2ATTACHMENT source
- 4 CARD32 x1
- 4 CARD32 y1
- 4 CARD32 x2
- 4 CARD32 y2
▶ 1 1 Reply 1 unused @@ -754,6 +804,14 @@ A.2 Protocol Requests 4 CARD32 swap_lo 5n LISTofDRI2BUFFER buffers └───
- The 'source', if not zero (DRI2BufferFrontLeft) indicates the
- attachment point of the buffer to swap w/ DRI2BufferFrontLeft.
- If zero is specified, DRI2BufferBackLeft is swapped with the
- DRI2BufferFrontLeft buffer, for compatibility.
- If 'source' is not zero, (x1,y1), (x2,y2) specify the bounding
- box in coordinates of the source buffer which should be scaled
- to (0,0), (width,height) of the destination drawable.
┌─── DRI2GetMSC -- 1.7.5.4
xorg-devel@lists.x.org: X.Org development Archives: http://lists.x.org/archives/xorg-devel Info: http://lists.x.org/mailman/listinfo/xorg-devel
-
Rob Clark -
Rob Clark