From: dcc@ATHENA.MIT.EDU
Message-Id: <8909221607.AA07560@THANATOS.MIT.EDU>
To: toddb%tekcrl.labs.tek.com@relay.cs.net
Cc: xvideo@expo.lcs.mit.edu
Subject: re*3 Intro and questions.
Date: Fri, 22 Sep 89 12:07:49 EDT


>> The question I'm left with is whether VEX should
>> define special overlay behavior, or whether this should be left to a
>> more general overlay extension.
>
>Please propose one.  In the mean time, several things that applications
>will want to do cannot be done (in a portable manner) without this
>ability; e.g. drawing a circle around a point of interest in a video for
>training, overlaying graphic text for video post-production, overlaying
>enhancement graphics for Geographic Information Services (GIS), etc., etc.
>Gotta have it.

Yes, I totally agree that overlay capability is needed.  I think,
however, that our goals for a video extension are different.  My goal
is for the video extension to be a standard "component" extension that
expresses video semantics, that it alone may not fully define all
the capabilities that are present in a particular video workstation, and
that things that are not defined would be deferred to other standard or
proprietary extensions.  

The issue of overlays with video is similar to the issue of double
buffering with 3D graphics: overlays are frequently used in video
applications and supported by video workstations, just like double
buffering is frequently used by 3D applications and supported by 3D
workstations.  Nevertheless, double buffering was removed from the 3D
extension and made into its own extension, because it applied to more
than just 3D graphics.  I believe that this argument also applies to
overlays.

>> [new proposal for visual type: blend]
>> No I have three.  The video image (1) appears in the Active Video
>> Region which is displayed in the Blend window.  The blend image (2) is
>> drawn into the "alpha" buffer by drawing to the Blend window; it is
>> overlayed by the video image.  The graphic image (3) is drawn into the
>> frame buffer by drawing to the parent window with subwindow-mode =
>> IncludeInferiors.
>> ...
>> In this case I would create a third window with a new visual type
>> (Video), make it a child of the blend window, and draw
>> into the blend window with subwindow-mode = IncludeInferiors.
>
>This has scads of problems, and leaves important things unspecified.
>I'll fill in the best I can and show the problems that result.  Seems
>to me you describe a hierarchy like:
>
>                   -------------------------
>                   | Graphic (grandparent) |
>                   |   -------------------------
>                   |   |  Blend (parent)       |
>                   |   |   -------------------------
>                   |   |   |  video (child)        |
>                   ----|   |                       |
>                       |   |                       |
>                       ----|                       |
>                           |                       |
>                           -------------------------
> - Graphic image drawn on the grandparent is ok, but drawing with
>   subwindow-mode = IncludeInferiors is undefined in the core protocol
>   if the depth of the child does not match the depth of the parent.
>   This class of hardware very frequently has an overlay (graphic)
>   depth shallower than that of the video; frequently 8 vs. 24.

I see no problem here.  The extension would define the behavior of
IncludeInferiors drawing when inferiors have extended VisualTypes
(Blend or Video).  Different depth windows aren't a problem either,
because when you draw into the graphic window you continue to draw
into the same physical memory at the same depth regardless of whether
you are inside the bounds of a video (or blend) subwindow or not (same
goes for the blend window).

> - The blend window is a window, but it can never be visible, because hardware
>   cannot display an alpha framebuffer except indirectly.  Perhaps
>   you mean that the blend window precisely tracks the size and
>   location of the video child?  Does it also track the map state?
>   Semantics such as this lead to other very funky semantics,
>   and finally, doubt that it should be a window.

What is visibility?  I believe the blend window is visible and is an
autonomous window in all respects.  I guess that my definition of
visibility says that if pixels of a window's image are used to generate
display output (video to the workstation monitor), then those pixels are
visible.  The blend window certainly passes this test, though its
display function is more complex than an RGB lookup.

I would define no "funky" semantics:-)


> - Being a window, the blend window can accept any core X graphic requests as
>   well as those from other extensions.  This is bad.  Briefly (because
>   it has been discussed at painful length),  overlay hardware has either
>   an alpha framebuffer somewhere or it is integrated into the graphic
>   framebuffer in a chroma-key style (certain pixel values allow the "other"
>   framebuffer to show through).  The latter means either writing a new
>   ddx layer for blend visual type; or the blend window is implemented as
>   a pixmap, you touch every ddx function to see if the pixmap is
>   really a "blend" window, and then do the right thing.  This is
>   a big burden for this class of hardware.  There are similar arguments
>   if you introduce a chroma-key value to the graphic framebuffer that
>   makes it difficult for the alpha framebuffer folks.

I'm familiar with the discussion about chroma-key and alpha buffer
hardware.  I didn't see any discussion of chroma-key overlays in VEX
v3.8, can you point me to the sections that address chroma-key
overlays?  The Blend VisualType I proposed doesn't address chroma-key
overlay hardware, but VisualTypes can be defined that do (a future
discussion).

Your DDX issues aren't that serious.  First, you have a dispatch table
in the GC that could be loaded with drawing procedures that know how
to draw to the alpha buffer (or that know how to return an
Implementation error); you may have to fall back to this if the alpha
frame buffer is drastically different than the graphic frame buffer.
Second, you may be able to define device state that directs the output
of DDX to the appropriate frame buffer, this way you can reuse most of
the drawing code.  But really, all this stuff should be considered in
the hardware design.

> - The blend visual type must actually be two types: BlendStatic, and
>   BlendDynamic, so that you can express a programmable and non-programmable
>   alpha frame buffer. The video visual type must actually be six
>   types: VideoStaticGray, VideoStaticColor, VideoTrueColor,
>   VideoGrayScale, VideoPseudoColor and VideoDirectColor.  This give you
>   eight new visual types to my one new drawable type.
>

If BlendStatic means that you have a static or constrained alpha frame
buffer (i.e., pixels that are either write-only or read-only), then
this is inappropriate use of the VisualType mechanism.  Furthermore I
claim this hardware is "deficient" in the language of X11 (p351, X
Window System, by Scheifler, Gettys, and Newman, quoted below) because
it contains a frame buffer that can't support drawing:

  Implementation: The server does not implement some aspect of the
                  request. A server that generates this error for a
                  core request is deficient...

This doesn't mean that your device can't be supported by X it just
means that when drawing is performed to Blend windows, the server will
return an Implementation error.

As far as the video VisualTypes you list, my proposal talked about the
VisualTypes a particular implementation might support, and not the total
list of VisualTypes defined in VEX to be chosen from.  Nevertheless, I
claim it's the right thing to do.  The VisualType mechanism is being
used as intended and describes the display output semantics of pixels
in a window (i.e., which frame buffer(s) should a pixel be displayed 
from).

>As for
>your eight new visual types and so forth, you are kidding yourself if
>you think you aren't going to affect the semantics of some requests.
>Try solving some of the problems mentioned above and we can tally
>the score again :-)
>

I think I've explained the difference between your 8 and my 2.  It's
possible that I'm "kidding" myself, but I need more specific
criticism:-)  I think that I've addressed all the issues that you've
raised.  One advantage of the VisualType mechanism is that given any
particular server implementation, only the VisualTypes that make sense
for that device need be implemented, your mechanism has to be
implemented for all devices regardless of hardware capability, and if
it isn't then you don't adequately inform the client as to what the
device is capable of doing.


>> I agree that most video workstation hardware can support some form of
>> overlay (1 or more planes), but "alpha" buffering, as defined by VEX,
>> isn't common.  Blending is very common in broadcast video hardware,
>> but VEX is providing a mechanism to annotate a video image with
>> graphics (or vise versa), which really isn't the same.
>
>Huh?  What makes you think I'm not doing broadcast-quality video
>post-production?  Or what if I wanted to display a shaded 3-d object
>on a video background, or perhaps a video clip of a dancer on a graphics
>background?  If I have only one bit of alpha, both will have jaggies.
>I'm afraid you have a narrow view of the uses of video and graphics.
>Allow me to pass on the collective wisdom of my two daughters, of which
>I am frequently reminded: Get real, dude!

I understand what you are attempting to do, and I think it's important
and interesting.  I simply take issue with your statement that most
video workstation devices have blending (verses masking) capability.
Most may want it, and some new devices may even have it, but it's far
from being common (let alone standard) among all video workstation 
devices.


	--david