Regarding using V4L to communicate with DSPs/other processors: that too could be something for Linaro to pick up: experiment with it for one particular board, see what (if anything) is needed to make this work. I expect it to be pretty easy, but again, nobody has actually done the initial work.
Actually this has been done already, unfortunately the project was (and still is) kept under a hat... STMicrolectronics (no -Ericsson in the name ;-) developed a multimedia framework that used sort-of-DSPs to decode, encode, process etc. etc. and exposed Linux-DVB/V4L/ALSA APIs. It's been used in a number of commercial projects but "Open Duckbox" picked it up as well:
http://gitorious.org/open-duckbox-project-sh4/pages/Home http://gitorious.org/open-duckbox-project-sh4/tdt/blobs/master/tdt/cvs/drive...
Having said all that, I don't think that anything of this will be useful in your development. I just mention it as a proof that it viable :-)
Cheers!
Paweł