On Tue, Feb 8, 2011 at 9:30 AM, Lee Jones <lee.jones@linaro.org> wrote:

Robert, Linus, what say you?

[I'm looping in Harald from Ericsson who worked with Khronos
so he can correct me for all inevitable mistakes in trying to
understand how Khronos works.]

I mainly come from the kernel for kernels' sake side of things and
my remarks are basically the same as Arnd's: what is OMX doing
handling hardware? If the OMX stuff *did* have some business
with the hardware it should be a drivers/omx or lib/omx subsystem
and everyone could make use of it from there and that's it. SCSI
and VGA and whatever hardware protocol is in the kernel after all.

That's the simple answer. However I also know from experience
why the Khronos Group folks don't see things this way.

Their problem is this: a plethora of operating systems,
including Linux, Symbian, Windows Mobile, ITRON, VxWorks,
QNX, OSE and a boatload of internal NIH systems. Some of
them with nonexistant of half-brewn frameworks to handle the
stuff that is done in kernel subsystems like V4L2, /media, or
/sound (ALSA SoC). Their objective is to unify media pipelines
across these desperately heterogeneous environments.

First premise of Khronos and OpenMAX: by describing very
strict APIs for multimedia streams they set a stage where different
companies can develop one implementation each and compete
with each other. So, define API and compete by implementation.

Second premise: for each implementation, develop once,
deploy everywhere. So each company create a large body
of code, which they port to every OS they need to support. So
that one of these OS:es happens to be Linux is just a technical
detail. For OpenMAX it's just one world where the important
stuff (i.e. the portable code and the Open* APIs) can live and

These basic premises permeates the work in Khronos so
deeply that it becomes a paradigm or ontology if you like, it
defines the world in which Open* development takes place.
If you buy into one part of Khronos APIs you usually assume
an idea of the world where all is done the Khronos way with
no compromises. E.g: for OpenMAX you know that OpenMAX
IL, AL and DL are different things but surely you expect that
you will implement all of it down to the very last bit. (Yes, I
know that this does not happen in practice, but such is the
Khronos ontology.)

Some may think that OpenMAX is just an obtrusive layer of
APIs adding nothing but complexity. If you live in an all-Linux
world where say GStreamer and ALSA, V4L2 APIs go hand
in hand this is true. But this is not the world where Khronos
lives. Anyway, that's a separate subject. Let us say if Linux
and GStreamer were dominating the world there would be
no need for OpenMAX. Likewise if say Window Mobile
dominated the world and had some framework for multimedia
for that matter. Another implicit premise of the OpenMAX work
is obviously that there will always be many OS:es involved in
this game.

OpenMAX  does not really conflict with the kernel way of doing
business at all. It's natural place is within stuff like GStreamer
or Android's counterpart, where they may or may not have
some perfect fit - this is a matter of debate, admittedly the
Linaro multimedia working group spend some time on it.
The way I understand what they do is that they implement
interfaces in say GStreamer like gst-openmax & Bellagio so
that the big portable OpenMAX shebang that every vendor
create can easily integrate into Linux.

The problem comes when you have a few DSP/ISP processors
and need some kernel services, and then there's this V4L2 API
you're supposed to use to get things done in the middle of
your world, and in some parts it's not even fully defined, things
in OpenMAX have no counterpart that maps 1-1 to V4L2 for
example. From an OpenMAX point of view that's just an obstacle
making your code bundle less portable to VxWorks and
heterogenous, requiring new .c files and #ifdefs and whatever.

So naturally, if you can get away with implementing, say,
your own userspace driver for video or audio sinks, and this can
be reused on VxWorks and ITRON, you're happy because that
means less hazzle. (In theory.)

The fact that the Linux community prefer a V4L2 API for e.g.
a camera is totally irrelevant from an OpenMAX point of view,
that's just a technical detail of Linux. Naturally, the Khronos
mindset is that apps that need photographic images need to
set up the OpenMAX IL chain and preferably conform to the
OpenMAX AL API. However that is then handled behind the
API is none of OpenMAX' business.

The goal of the portable OpenMAX is to make things like V4L2
become irrelevant to application developers. Compare how
say Qt in theory makes the fact of whether you develop for
Linux, Symbian or Windows irrelevant. There is not one word
the specs that mandates that you use the native operating
system APIs or frameworks behind the scenes at all, not even
in the lowermost DL layer. (There is some portability glue in
OpenKODE though, you can compare it to the portability code
found in stuff like glib or nspr.) Anything providing the services
will do. Needless to say, I don't think this is the view of say the
V4L2 or ALSA developers. Or even the TI people working on

I would say that the problem is exactly this: both are defining
how to do things with mediastreams, but the bridge inbetween
is undefined, which makes for all kind of strange ad hoc
solutions, usually the most comfortable one, from the
implementers point of view, meaning they signed up for
doing the OpenMAX API so they recognize that they have
to develop and deploy this, but some of them think, that even
though they are members of Linaro, they didn't sign up
for the ALSA or V4L2 APIs. Is that really so?

I leave the question open, because there was an inflamed
debate on another very related subject the other week,
which was that of OpenGL ES vs 3D hardware API's like
DRI/DRM. Again a Khronos API and similar ideas of
portable, proprietary, competing code bodies under a
standardized API, and no reuse of the community interfaces.

I don't know if Khronos and Linaro could talk about this,
say come up with something like "if you're member of both
you're actually supposed to use the Linux APIs when
implementing Khronos stuff" that kind of high-level politics is
not my department. Our problem is that right now the inverse
of that statement is assumed. Couldn't we atleast say that
the native Linux APIs are "preferable" or something like that?

Linus Walleij