WO1996041311A2 - Peinture interactive stereoscopique - Google Patents

Peinture interactive stereoscopique Download PDF

Info

Publication number
WO1996041311A2
WO1996041311A2 PCT/US1996/009812 US9609812W WO9641311A2 WO 1996041311 A2 WO1996041311 A2 WO 1996041311A2 US 9609812 W US9609812 W US 9609812W WO 9641311 A2 WO9641311 A2 WO 9641311A2
Authority
WO
WIPO (PCT)
Prior art keywords
image
die
information
pixel
painting
Prior art date
Application number
PCT/US1996/009812
Other languages
English (en)
Other versions
WO1996041311A3 (fr
Inventor
David M. Geshwind
Original Assignee
Geshwind David M
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Geshwind David M filed Critical Geshwind David M
Priority to AU61082/96A priority Critical patent/AU6108296A/en
Publication of WO1996041311A2 publication Critical patent/WO1996041311A2/fr
Publication of WO1996041311A3 publication Critical patent/WO1996041311A3/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/334—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] using spectral multiplexing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/337—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] using polarisation multiplexing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/341—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] using temporal multiplexing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays

Definitions

  • the instant invention relates to an operator's interactive computer workstation for the conversion of two- dimensional motion pictures, video or other image information for three-dimensional display or viewing systems, as well as for the creation and integration of three-dimensional image information.
  • a stereoscopic 3-D display by creating a left and right image pair interleaved into the even and odd fields (or visa versa) of a video display, and viewing that display through LCD shuttered glasses which are flickered (180 degrees out of phase for each eye) in synchrony to the field rate of an interlaced video display, separating two 'field' images from a single video 'frame' signal.
  • the instant invention relates to techniques and devices for creating and/or delivering 3-D (stereoscopic, binocular, holographic, serial or otherwise) image information which may be derived from 2-D images or image elements converted to 3-D by the system, 3-D image elements created by the system, or integrated with pre-existing 3D image information created independent of the system.
  • 3-D stereo, binocular, holographic, serial or otherwise
  • the primary system component is an interactive 3-D computer paint system which shares many elements with 2-D computer paint systems, but which extends those elements suitably to deal with 3-D image information.
  • User interface elements such as menus, buttons, palettes, cursors, and information windows may be placed in the
  • depth information may be stored with pixels in order to implement 3-D painting and processing functions. In this way, when further painting is to be performed at a particular pixel, the depth to be painted and the depth of what is already stored there may be compared; and, painting would only be accomplished if the new information was closer to the viewer than (and should thus obscure) what was already there.
  • system components which permit the input, by a human operator, of more positional information than simply X and Y position, yet which may be constructed by combinations of simple X-Y transducers or other input elements used in combination.
  • Figure 1 depicts the use of a standard paint system and a specialized 'disinterleaving' imaging function to create stereoscopic images.
  • Figure 2 depicts a 'disinterleaved' 4 x 4 pixel brush.
  • Figure 3 depicts two-ball (3-D) mouse operational modes.
  • Figure 4 shows the parallax offsets for near, far and screen depth information in the left and right images of a 3-D paint system.
  • the instant invention relates to an operator's interactive computer workstation for the conversion of two- dimensional motion pictures, video or other image information for three-dimensional display or viewing systems. It may also be employed for the creation and processing of new 3-D image information, and for the integration of either type of image information with three-dimensional image information imported into the system.
  • Such a system could also be used for adding image information to existing true 3-D images, including 3-D photographs and computationally generated binocular 3-D graphics. For example, circles, arrows, scales, axes and text notes may be drawn 'within the 3-D space' of a 3-D video display, in much the same way that a marking pen can be used to annotate on the surface of a standard 2-D photograph.
  • the technologies described herein may also be used, either in an illustrator operated system or more automatic system, for purposes of converting 2-D medical images to 3-D. In this way, traditional diagnostic and other medical images containing complex data may use the added depth dimension to enhance visualization and increase comprehension.
  • the embodiments described herein employ LCD shuttered glasses, synchronized to the field rate of an interlaced video display.
  • Such a system may be used for both design and production of displayable and recordable end product on video.
  • Similar systems may be designed around other stereoscopic or holographic display technologies; or, the system as described herein may be used by the operator to design images that will later be produced for display, viewing or recording by any 3-D means currently in use or later developed. Such variations are considered to be within the scope of the instant invention.
  • 3-D Graphics' are 3-D only in the computational sense and result in a single 'monocular' 2-D image.
  • 3-D will generally be used to mean binocular, stereoscopic or holographic images which exhibit depth characteristics when viewed.
  • the instant invention may be implemented using standard, off-the-shelf hardware and software components, with the addition of specialized software functions and the careful application of production techniques.
  • a standard 'paint' program operating on a standard interlaced video computer display (frame buffer), may be used to convert separate 2-D image elements into a 3-D composite as follows.
  • the rearmost image element (10, a hollow square in this example), called plane zero or PO is brought into the frame buffer by running a standard 'image load' function of the paint software, or may be created using computer paint functions.
  • this PO element should contain image information (even if it is only a
  • the standard paint software is then used to load or paint a next-closer 2-D image plane, PI, (11, a vertical bar in this example) which may also have been derived by separating it from a 2-D motion picture frame, as in applicant's patent '425.
  • PI next-closer 2-D image plane
  • PI next-closer 2-D image plane
  • more forward elements will, in general, be loaded with part of the image area 'transparent' (or painted/created with 'holes') allowing PO or other relatively rearward planes to be seen through them.
  • transparency is often denoted by storing a particular numerical 'color' at those portions, often (0) for indexed or 'color-mapped' images, or (0,0,0) for full-color, R-G-B images.
  • the process is repeated and many 2-D image planes are, thus, stacked-up, back-to-front, to form a multi- planed, 3-D image when viewed through the flickering glasses.
  • the back-to-front order guarantees that obscuring of rearward layers, which will be different in the left and right images due to parallax offsets, is done correctly.
  • the last, forward-most image plane will be at the front of the CRT and the full composite image will appear to be 'inside' the video display, similar to looking into a fish tank.
  • the rearmost image plane may be restored to the plane of the CRT face. See Figure IE.
  • the composite image will then appear to come out of the CRT into 'room space'.
  • This restoration process may be run part way and arbitrarily set any position in the composite image (even between planes containing images) as the 'neutral' plane, without relative shift between the odd and even image fields.
  • some image part may 'fall off the edge' of the frame buffer and be lost.
  • the image which is spread, in opposite directions horizontally on alternate lines to create parallax, for alternating field stereo
  • the field for one eye or the other will 'fall off the edge' first. This will leave part of the image visible on only alternate lines, for a single eye, creating a ragged edge.
  • the image information may 'wrap- around' and incorrectly re-appear at the opposite edge of the frame buffer.
  • the width of the area of the display used for imaging in this embodiment may best be made less, by the amount of the maximum shifts involved, than the maximum width of the display.
  • the Z- Buffer may be initialized to a horizontally ⁇ minishing wedge to avoid images with ragged edges.
  • niinimum increment is 2 pixels (a minimum of one offset in each direction) then to avoid ragged edges while painting, the last two usable pixel columns on each side must be avoided by me next farthest plane. This restriction continues, for further planes, loosing two additional on each side. This is true, starting from the '0th' or neutral plane, for both into and out of the screen, although parallax is reversed between the two cases.
  • an extra wide frame buffer providing non-visible pixels beyond the edges of the display area, may be employed and would permit the storage and restoration of those parts of image planes temporarily pushed off the edge of the display screen by the software shift function.
  • Such a frame buffer would also avoid errors which could result from temporary wrap-around caused by shifting or unshifting the composite image.
  • the AT-VISTA is an off-the-shelf display buffer for the IBM-PC that may be software configured to provide such capabilities.
  • the "layer-shift-layer-shift-layer . . . " process may be incorporated into the program. Alternately, this process may be replaced by a more convenient functionality. This cumbersome process is required because a standard paint system 'paints' or performs its other imaging functions at the 'neutral' plane, where the odd and even image fields are not relatively shifted.
  • the paint brush (and all other painting, graphic and imaging function such as: draw a line, circle, square, polygon, text or other graphic element; load, move, copy, or modify a portion of image; as well as all other traditional 2-D paint and graphic functions) would, itself, be programmed to have its odd and even image fields offset by a variable amount.
  • Figure 2 shows a small square 4x4 pixel brush.
  • the brush is set for the zero or neutral plane with no relative offset of the odd and even lines of the brush image.
  • the odd and even lines are offset for plane 1; in 2C for plane -1; in 2D for plane 2; in 2E for plane 20.
  • This technique does have a drawback to implementation.
  • the separated brush and interleaved left and right images require that any painting function be applied separately to the odd lines and even lines of the frame buffer contents.
  • Programming algorithms that are applied to non-continuous areas of frame buffer memory is both a conceptual and a computational complication.
  • the contents of many frame buffers are manipulated by transferring rectangular areas of the image to the computer for modification and then remrning the data to display memory.
  • the overhead of initiating a transfer is often very high compared to the transfer of each additional pixel.
  • a 4x4 brush would require a single transfer of 16 pixels back and forth.
  • a separated brush as shown in Figure 2E would require 2 transfers of 12 pixels (21 and 22) or a single rectangle containing 176 pixels (23).
  • a third embodiment that eliminates or reduces me above problems can be implemented in some configurable display boards such as Truevision's AT- VISTA, which includes a 'row table' feature. These devices permit the programmed specification of which memory line will be displayed on the screen for each display line. In that case the left and right images can each be placed in continuous rectangular blocks of the display's memory.
  • the display processor is programmed, however, to display these two images in interleaved fashion.
  • the advantage is that all functions and algorithms can then be programmed to operate on (two) continuous rectangles of memory, rather than having to be made more complex in order to work with two interleaved views.
  • Another advantage of this method is that different types of interleaved and non-interleaved display formats can be accommodated for different display systems from the same frame buffer memory content, just by changing the table that maps memory to display. For example, switching between left/even right/odd configuration and a left/odd right/even configuration only requires that the table entries be shifted down or up by one line; also, a pad entry may be made at the top or bottom, and one display line lost at the bottom or top. Similarly other display formats, such as those required for the StereoGraphics brand of high frame rate alternating stereo displays can also be easily programmed.
  • This third scheme is the preferred embodiment of the instant invention.
  • the artist would be able to position the brush (or other imaging function) along the Z (depth) axis as well as the X (left to right) and Y (top to bottom) axes.
  • the brush or other imaging function
  • a depth palette may be provided, much like a standard color palette, that allows the artist to pick off a depth;
  • the above may be in the form of a depth 'ruler', as described below;
  • the artist may pick off a depth by pointing at a particular item or spot in the image
  • a numerical value may be typed in at a keyboard; 5.
  • a pen with a variable pressure tip (rather than the standard click-on tip) may be used to specify depth;
  • buttons such as a knob, foot pedal, joystick, 'data glove', 3-D input device as described below, or other control or position sensor; or,
  • a graphic of a knob, slider or other control icon may be controlled by the artist; and additionally,
  • the computer may be programmed to compute a depth by: a depth value, depth tag, or other depth specification, stored with an image element; interpolation, either spatially over a frame, or temporally between frames; by image analysis of any type; or, as described below, by consulting a depth or Z-buffer.
  • the left and right pictures and the left and right brush are interleaved for display, into the odd and even fields of a video frame, moving up or down by a single display line reverses the relationship between which brush field is displayed in which picture.
  • the left brush is painted into the left image and the right brush is painted into the right image
  • the right brush is painted into me left image and me left brush is painted into the right image.
  • the parallax shift also must be alternated or the brush will flip in and out of the screen on alternate lines.
  • Frame Mode the cursor position may only move in multiples of two lines vertically and, thus, the relationship between brush halves and picture halves is always preserved.
  • image processing related 2-D painting functions such as filtering, averaging, smearing, blurring or sharpening functions
  • these 2-D algorithms can be apphed to a single plane, or may be generalized to be applied to all the pixels in a 3-D rectangular solid, other arbitrarily shaped 3-D volume or, in particular, within the volume of a 3-D brush shape.
  • the image processing algorithms may be generalized to 3-D form as well; for example, 'blurring' or
  • the center of the kernel will be horizontally displaced by the parallax offset differently for each eye view.
  • the differential layering of image elements in the two eye views will yield different results when digital filters are applied.
  • the digital filter kernel can be applied to both the visible and obscured pixels of a plane.
  • depth (D- or Z-) buffer The above describes a convenient usable system for creating and modifying stereoscopic images.
  • the 3-D effect depends not only on horizontal parallax, but on the proper obscuring (which is subtly different for each eye) of rearward image elements by forward image elements. If the visual cues of parallax and obscuring are inconsistent, confusion results and the perception of 3-D is destroyed.
  • each pixel may have an independently specified color.
  • Blue values by using it as an index into tables of Red, Green and Blue values — called color look-up tables or color maps — with all pixels sharing a common index displayed as the same color.
  • Red Green and Blue (or Index) information In addition to die Red Green and Blue (or Index) information for each pixel, it is common to store additional information in other memory planes.
  • One, three, eight, sixteen or more memory planes (corresponding to the number of bits per pixel) of additional information may be provided, with eight being very common for moderate cost computer displays; often with 32 bits per pixel being assigned as eight bits each for Red, Green, Blue and Alpha (a non-displayed channel).
  • the Alpha channel is traditionally used for a number of purposes which can include: overlay information (e.g., text or menus); video keying or (soft) matting of the computer display signal with another signal; masking off parts of the display from further processing; temporary storage of items such as menus, icons or palettes — or, of the parts of images obscured by those items; and, pixel transparency.
  • overlay information e.g., text or menus
  • video keying or (soft) matting of the computer display signal with another signal masking off parts of the display from further processing
  • temporary storage of items such as menus, icons or palettes — or, of the parts of images obscured by those items
  • pixel transparency e.g., text or menus
  • depth information is sometimes stored for each pixel to facilitate the proper display of many intersecting objects. Rather than trying to compute the correct intersection of diverse objects, the pixels of each object calculated are only put into the display if they are closer to the viewer than the equivalent pixels already contained in the frame buffer, thus automatically creating the proper intersection between, and obscuring of, objects without their direct computation.
  • a depth memory plane is often called a Z-buffer. It is most often 16 or more bits deep, in order to provide enough resolution (2 raised to the 16th power, or roughly 64 mousand divisions) for the proper mathematical placement of objects in the computer's simulated 3-D (but still displayed as 2-D) environment.
  • While a 16-bit buffer may be used in the embodiment of the instant invention under discussion, an eight-bit Alpha channel may also be used to store depth information to great advantage, even though eight bits provide a resolution of only 256 values. Eight bits of non-display memory is routinely available for computer paint systems while 16 extra bits is not.
  • 2-D image layers are often stacked up and differentially shifted, to create a stereographic display. These 2-D image layers may be ordered and each assigned a depth or D value. This D value does not need to represent an exact depth value, just an ordering.
  • the differential odd/even field left/right offsets will have to be calculated correctly for an exact depth value; but for obscuring, only an order is required. Thus, the much cruder resolution afforded by 8 bits can be used for the instant invention.
  • a depth channel will be referred to as a D-buffer.
  • the pixel value when loading or creating a layer, in addition to the differential offset function, for each pixel, the pixel value will only be used to replace the R-G-B values already stored for that pixel in the frame buffer if the new pixel's depth is closer to the viewer and, therefore, should obscure that which is behind it.
  • the depth range behind, equal to, and in front of the selected depth for painting may be treated separately and pixel-by-pixel replacement for each range may be individually permitted or not. Not all combinations will make sense, but may still be used for effect. Additionally, individual deptii planes, or ranges of depths may be protected from replacement or not. (See section on "Protection".)
  • the value may represent the actual depth.
  • a number may be associated witii each D-buffer value which in mm indicates the actual depth which will be used to offset the pixels of that plane; in effect a depth look-up table or depth map, analogous to a color map.
  • the 256 planes specifiable with an 8-bit D-buffer do not need to be evenly spaced, and may be positioned with arbitrary resolution in the depth direction.
  • Another embodiment provides additional flexibility. Assume that a D-buffer containing 8 bits per pixel is available, although the same basic idea could be used for other numbers of bits per pixel.
  • the 8 bits will be split into two segments: the upper or most significant 3 bits (providing 8 possible numbers 0 through 7) will be reserved as a 'tag'; the lower or least significant 5 bits (providing 32 possible numbers 0 through 31) will be used for a 'value'.
  • tag 0 will be reserved for planar objects parallel to the screen. For planar elements the value will represent the depth of that plane and can be used, as is, for obscuration purposes.
  • die actual depth value used to create parallax may be stored in a look-up table.
  • Each of die 32 entries in that table would be indexed to one of the 32 possible values for planar entries as described above.
  • the depth assigned to a particular pixel would thus be derived by using the value stored at tiiat pixel as an index into the deptii table.
  • all the pixels associated with a planar element could be moved by changing a table entry rather than by changing die value at each pixel. More than one planar element could tims be assigned the same depth. If overlap occurs, then an ambiguous situation results.
  • the tag numbers could be used to arbitrate the conflict.
  • a second table could be constructed to store ordering or obscuring information to be used all the time, or only in conflict situations.
  • two separate 'depth' designations can be maintained: one, a 'distance' value would be used to determine parallax offsets; the other, an 'ordering' or 'priority' value would be used to determine which objects obscure other objects.
  • Tags 1 through 7 would each be reserved for a tilted or 'thick' image element.
  • die 32 possible values could be interpreted differently than as described above, with several alternatives: 1.
  • the 32 values associated witii a particular tag could themselves be interpreted as actual depth values.
  • the 32 values associated widi a particular tag could themselves index a table of actual depdi values (as described above), providing an image element whose individual pixels were distributed at up to 32 depths.
  • a second order table could also be used.
  • the 32 values could index a table of relative depth values, providing (probably fairly small) deptii offsets from a positioning depth value also associated witii die tag.
  • the deptii value for each pixel of the image element is calculated by adding die offset table entry associated witii die value for that pixel, from the table associated witii the pixel's tag, to the deptii position also associated with that tag.
  • the element once its 'deptii shape' at each pixel were defined, could be repositioned just by changing the one position depth rather than by changing the entire table or, worse, the depth (tag + value) stored at each pixel of the element.
  • the position could be a single value representing the back, front or middle of the element. It may also consist of several numbers representing, for example, front and back, or position and thickness. In this way the element could be bodi positioned and stretched in the depth direction by changing only two values.
  • Otiier variables associated witii a tag could indicate how die table values were to be interpreted: as actual values, table indexes, or relative offsets (1, 2 and 3 above); positive, negative or signed; linear, 1/linear, exponential or other non-linear function; etc.
  • the interpretation of the values may also change as die position brings the element closer to or further from the viewer. That, in mm, may depend on die focal length of a virtual camera or an actual camera used to photograph a flat image to which depth is to be added.
  • Image elements that are planar, but tilted with respect to the face of the display may be handled as thick or sculpted elements as described above. However, particularly for tilted planes that extend over a wide range of depth values, die limited number of possible values may produce a series of strips parallel to the face plate, rather than a smooth plane. Such planes may be marked as such witii their tag and handled as a special case. By storing positions for the edges or comers of a rectangle, and applying the parallax rules to those positions, a pair of continuous smooth image elements may be created for the left- and right-eye views by interpolating between those edges or comers.
  • Video displays in most computers, and die data files meant to be displayed tiiereupon, are often in the range of 512 pixels horizontally by 486 lines vertically.
  • Other often used horizontal resolutions are 640, 768 (or 756, with six null, black or transparent pixels on each side) or even 1024 or 1280.
  • Straightforward embodiments use offsets on pixel boundaries and, therefore, the minimum horizontal parallax shift between planes adjacent in the depth direction is one pixel (usually two pixels, one each in opposite directions for the left and right images) and, particularly at the lower end of these horizontal resolution figures, this does not permit separate planes or parts of objects to be closely spaced.
  • the current embodiment of the system utilizes the AT-VISTA display board which can be configured at many different resolutions, including a relatively high 1008 pixels horizontally (which may be increased to a nomrnal 1536).
  • a relatively high 1008 pixels horizontally which may be increased to a nomrnal 1536.
  • data pixels are repeatedly loaded into the higher resolution display's memory locations. For example, with 512-pixel resolution images, into a 1536 pixel wide screen, each pixel would be displayed three times; and for 768-pixel resolution images each pixel is displayed twice.
  • each data pixel is used for 3/2 display pixels, which is implemented by loading data pixel N in display pixel M, data pixel N+ 1 in display pixel M+2, and a 50/50 mix of data pixels N and N+ 1 in display pixel M+ 1.
  • a similar interpolative scheme is used to load 640-pixel resolution data into the 1536-pixel resolution display.
  • Each group of 5 data pixels (A dirough E) must be apportioned into 12 display pixels (1 through 12) as follows:
  • the reverse process can be used to derive interleaved or separate left and right images from the higher resolution 3-D display. For example three pixels of die 1536-pixel resolution image would be combined and averaged to a single storage pixel of a 512-pixel resolution data file.
  • Information may be loaded into a display, either from a pre-existing image or from the 'brush' of a paint system; it may be loaded from information at the same resolution as displayed or (as described above) from a different resolution tiian displayed.
  • the straight-forward approach leads to a miiumum parallax offset between planes of one pixel which may provide widely spaced planes or parts of objects in synthesized 3-D space.
  • an interpolative technique similar to that used to display information stored at different resolutions, may be used to provide sub-pixel positioning and, thereby, very fine and accurate parallax shifts which result in fine and accurate positioning in the depth direction.
  • Table 1 shows how the new information is combined with the old information for five cases: case 1 , with the new information obscuring pixels (c) dirough (g) of the old information but with no parallax offset (new information on the same plane as the old); cases 2 and 3, with additional left and right parallax shifts of one pixel; or, cases 4 and 5, with additional left and right shifts of 1/lOth pixel.
  • parallax offsets, and tiiereby depth plane spacing of 1/10 (or any fraction) of that normally achievable by display hardware of a particular resolution may be implemented by such a software technique.
  • each separately painted plane, sub-plane, or other non-planer object may be maintained as a separate object in separate buffers or sections of memory, other than the display's own memory.
  • objects may then be properly parallax offset shifted, and left and right composite images built up in the portion of memory used by the display.
  • objects may be moved in X, Y or Z without leaving 'holes' as objects reveal what is behind tiiemselves.
  • display pixels tiiat are composites of data pixels from more than one object, there is no problem due to not being unable to disentangle the composite information. Since the entire display image is reconstructed from the separate objects, the appropriate display pixel compositing is just re-computed from scratch.
  • An additional advantage is that objects may be made transparent or painted transparently; or, because objects have deptii associated with them, a light source(s) may be specified and correctly calculated shadows may be cast in 3-D-s ⁇ ace by some or all objects, to add realism to the final image. If either of these processes is done when just a composite image is stored, it is difficult, if not impossible, to undo transparency or shadow casting. Shadow casting would probably be implemented as a post-painting process. However, if the objects are stored separately, as described above, not only can tiiey be moved and re-composited, but they can be re-composited with transparency and shadows.
  • Off-die-shelf high-speed image processing sub-systems such as the Intel DVI board for the IBM-PC are available. Such devices can achieve real-time or near-real-time compositing of many separate and separately offset image elements into a unified display image, or left and right pair of images.
  • each object may be kept in its own display buffer, witii the composite image created only on die fly by properly offsetting and mixing the output signals from the multiple display buffer sections.
  • An additional advantage associated with this technique is that for many situations only a single copy of each object must be stored (with depth specification) and may be shifted in opposite directions prior to compositing for each of the left and right images. If alternate lines of an object/image are to be shifted in opposite directions for interleaved display, fine vertical detail may broken up into the two eye views, which may interfere with perception.
  • each source line may be used in each eye, although this will stretch the image vertically.
  • the human operator may be presented widi two portions of an interactive program: one geared to painting or creating objects; and, another geared toward re-arranging objects and fine-tuning the compositing process.
  • Fine-tuning the compositing process may include various options as to how object boundaries are handled. Particularly when using sub-pixel parallax shifts, pixels at the boundaries of objects may contain data from more than one object, with more than one deptii. Visibly, anti-aliasing or mixing of the visible data from multiple objects, in a pro r ⁇ t ⁇ fashion, may be used at die edges. Alternately, die visible information from the object that is closest, or the object that covers most of the pixel may be used for die entire pixel. An additional problem is to determine what depth value is to be used and stored (for offset or obscuring purposes) with such composite pixels. Again, the deptii value of a dominant pixel may be used, or the depths from several component pixels may be used. If a deptii value is to be derived from a composite, simple or weighted averaging of the several component deptii values may be used.
  • a technique may be adapted from applicants co-pending application number 07/435,487, where color information was made less saturated or 'pure' as the edges of objects were approached. I ⁇ this case the depth information will be made less 'pure' or 'fuzzier' by adding a random variation or noise to those pixels at or near the boundary of two objects separated in depth. In general, the closer to the edge, the less pure the depth will be. In practice, this will amount to picking deptiis for edge pixels tiiat are randomly scattered between the near and far depths of: the component pixels, for anti-aliasing techniques or micro-positioning; or the bordering objects for other situations.
  • a PREFERRED EMBODIMENT A '3-D Computer Paint System' has been programmed in the computer language 'C by the applicant as a preferred embodiment of the instant invention.
  • the program operates on the AT- VISTA display card installed in an IBM-PC/AT type computer.
  • the display is configured for 1008 (nominally 1024, and may be increased to a nominal 1536) horizontal by 486 vertical resolution, permitting a fine degree of horizontal parallax shifting (and, therefore, depth spacing) but configured for display on and recording by standard broadcast video equipment; as well as virtually all available electronic LCD-shuttered 3-D glasses.
  • the 'row table' feature of the AT- VISTA can be re-configured for five display modes: left/right in odd/even video fields, for 60Hz operation; right/left in odd even video fields, for 60Hz operation; left/right in over/under full frame, witii a second blanking interval, for 120Hz operation; right/left in over/under full frame, with a second blanking interval, for 120Hz operation; left image on top, right image on bottom, for glassesless viewing of both images separately (a debugging mode).
  • FIG. 1 Facilities are provided for importing/exporting both 2-D and 3-D images, full screen or partial windows, at various resolutions (512, 640, 756/768, 1008/1024, 1512/1536) with proper aspect ratio conversion.
  • both the 'canvas' upon which the image is painted and the various brushes consist of continuous left and right parts; the interleaved display, when used, is only a function of the 'row table' feature.
  • Each application of the brush, text or other function is thus carried out twice, once for each image, with the appropriate parallax offsets added to whatever normal 2-D positioning data is obtained from the mouse, data tablet or keyboard.
  • This paint system is supplied witii an unusually large number of user adjustable parameters and options, and virtually all such options may be separately adjusted for the left and right canvases. Both field and frame cursor movement, as described above, are available.
  • each pixel eight bits each are stored for Red, Green and Blue visual information.
  • the fourth eight-bit channel is used as follows. Seven bits are used for 'Z' or deptii information. Of the 128 values thus possible, 101 are used: 50 into the screen; 50 out of the screen, into room space; and, die plane of the screen (no parallax offset).
  • the paint system is left with the plane of the screen selected, it may thus be used as a standard — but, very flexible — 2-D paint system.
  • the eighth bit of the fourth channel is reserved as a protection bit, whose use will be described below.
  • the Red, Green, Blue, Depth and Protection data may each separately (and separately for each of the left and right canvases) be selected for writing to the frame buffers memory, or masked from writing to memory.
  • the user selects a single plane at a time upon which painting is to occur, or to which a flat or multi-depth function is to be referenced.
  • the selected plane may also be dynamically and continuously selected along with the X and Y position.
  • the 3-D paint system has many usual elements, menus of buttons, color and brash palettes, sub-menus, etc. but these are displayed and work in a uniquely 3-D way.
  • buttons and palettes float upon a menu plane which, in mm, floats above the canvas; this effect is achieved by relative parallax offset.
  • the text of the button floats above the plane of the button.
  • the buttons cast drop shadows in 3-D. Normally drop shadows are dark areas the same shape as the casting object offset down and right (or up, or left) and obscured by the casting object. In this case, in addition to the drop shadow offset, parallax offsets are added to the left and right shadows to make them appear behind die casting object.
  • buttons and other elements animate by depressing and returning as appropriate by altering the parallax shift of the selected element, through a number of positions, over several frame-times.
  • One such window is the 'ruler' a device with a set of tics for each plane that indicate the depth value of the selected plane (with progressively longer tics at intervals of 5, 10 and 50), whether that plane has been previously selected for painting, or whether that plane is currently set as 'protected' or masked from painting.
  • GUIs Graphic User's Interfaces
  • All options and features may be selected for the right channel, left channel, or both; and may be copied right-to-left and left-to-right. Different colors and brush halves are displayed for each eye as appropriate.
  • the cursor is positioned in Z as well as X and Y by the use of parallax offsets.
  • flat rectangular cursors used to delineate areas for processing, storing, editing, etc. have been generalized to 'bricks' which delineate a rectangular solid space in the 3-D canvass.
  • die eighth bit of the deptii channel is reserved as a pixel-by-pixel protection bit. If this option at painting is selected during a painting function, the protection bit is turned on for each pixel changed. A complementary function prevents such protection tagged pixels from being painted over.
  • each pixel should be effected only once; this is very hard to do free-hand. This technique guarantees that each pixel is effected no more than once.
  • Using more bits for a 'protection plane' provide the ability to count the number of times a pixel has been painted or 'hit' and tiius more functional flexibility. For example, such a protection counter could be used by die program to permit a pixel to be tinted heavily on its first hit, and progressively less heavily tinted on subsequent hits. Or to only perform a function after some threshold of hits has been reached.
  • the operator may paint an area of 'protection from further painting' (Fixing) without effecting the contents on the screen or depth values. Areas may be similarly 'unfixed' by painting with the protection bit turned off and, again, masking Red, Green, Blue and Depdi.
  • an 'invisible sheet of glass' at a particular deptii may be created which will protect everything behind it from being painted by a brush behind (further than) the plane of the 'glass' but which may be painted over by a brush closer than the plane of the glass. Otiier types of protection are provided. Each plane may be explicitly protected by selecting it as such on the ruler. Facilities are provided to turn on or off specific values or ranges and complex combinations may be built up this way. Then pixels are painted, or not, based on their deptii value rather than on the state of their own protection bit (although both methods may be used simultaneously).
  • Pixels may also be protected because their display (Red, Green, Blue) contents are a specific value or in a specific range of color values. Also, options are provided to restrict application of the painting function to: all depth values, one depth value, a range of depth values, all but one deptii value, all but a range of deptii values.
  • paint system brushes may be any shape within a 16 x 16 pixel rectangle. However, like the canvas itself, each brush is 'disinterleaved into a right and left half, each only 8 pixels high by 16 pixels wide (128 pixels in all, for each half). The brush itself may have a single bit, or 8, 24 or 32 bits of information stored at each pixel.
  • Each brush half may also be independently med on or off for painting.
  • the paint system has many different painting modes, some of which require complex calculation at each pixel effected. Therefore, in order to keep the operation of the program running as fast as possible, unnecessary calculations must be avoided.
  • Part of painting is dete ⁇ nining which pixels are excluded by die shape of the brash within its rectangle, by the various protection schemes, by the restriction of depth values, restriction of color values, or by the requirements of the particular painting mode (for example, luminance paint only paints if the pixel to be painted is brighter than the pixel contents already at diat position). Two schemes are described below.
  • an essentially one-bit version of the brush may be kept, which operates as a mask or picture of which pixels are to be effected.
  • the various protection schemes are checked and bits in die mask are mmed off if protection will eliminate that pixel from being effected. Only those pixels still represented in the reduced brash mask are then applied to die canvas.
  • the problem with this scheme is that the various protection schemes and many elements of the paint amount to a decision being carried out (is the protection bit on, is the deptii value separately protected, etc.).
  • each brush keeps each brush as a list of pixels.
  • Each list of bytes starts with a counter that tells how many entries are in the list: between 0 (no pixels in the brash) and 128 (all pixels on). Then the list follows with that many entries listing the location (in raster order) of each pixel in the brush. In raster order, from left to right, top to bottom, the upper left comer is 0, the lower right comer 127. Since a rectangle of image is brought into or taken from the computer's memory by the AT- VISTA display as a linear array in raster order, this scheme works well.
  • the list is short and very few pixels must be considered. No time is wasted repeatedly considering and then rejecting as not valid die many 'off pixels in a brash image. As each protection scheme is applied, rejected pixels are removed from the list since, once rejected for one reason, it is no longer necessary to determine if a pixel is to be rejected for a second reason as well.
  • the list is shortened by taking die last entry, placing it in the rejected pixels position, and decrementing the first list entry (the counter) by one. In this way, whether a pixel is 'on' or 'off never needs to be considered; off pixels are never on the list for consideration to begin with. The list can only get shorter as pixels are rejected by various tests.
  • the 3-D paint system has many different paint modes; some are similar to 2-D paint systems; others are unique to this program.
  • Standard painting may be achieved by using a one-bit 'rubber stamp-like' brash with a color chosen from the palette.
  • a brush may be a 'picture' with Red, Green and Blue (and even Transparency or Depth) values stored at each pixel.
  • This bush may man be applied to die canvas in a straightforward manner, or my be used to tint (change hue but not luminance) or shade (change luminance but not hue) die canvas.
  • the entire result may also be modulated by a transparency factor which keeps some proportion of the old pixel's display value.
  • Still other types of painting perform a calculation based only on the canvas pixels' values, based on some parameters.
  • several image processing functions can be attached to die tip of the brush to selectively adjust level, gain, contrast, saturation, color correction, etc. on a free-hand basis.
  • Otiier types look to a neighborhood around the affected pixel and perform various digital filters that may blur, soften, sharpen, etc. the image; or, may smear pixel values in the direction of the motion of the brush.
  • 'Air' brushing is a technique that lays down a transparent, soft-edged swipe of color in two dimensions on the canvas.
  • 'Spray' painting that does so but with a thick curved tube (by use of deptii values and parallax offset of individual pixels) several planes thick; it has a dense center with a 'haze' around it.
  • a similar tube ⁇ like mode witii a 'hot' center and softer 'glow' is dubbed 'neon' paint.
  • die depth values may be updated, creating a substantial object.
  • depth values may be left unchanged so tiiat no record of the spray's existence would be made, it would be 'insubstantial'. Additionally, the core may be made substantial with the 'glow' or 'haze' made insubstantial. In addition to arbitrary 16 by 16 pixel brashes. Larger computational brushes are available.
  • a 'disk' is round witii uniform density and a sharp but anti-aliased edge; a 'cone' is round witii a dense center which falls off to transparent in a controlled fashion; a 'ball' is like a cone but, again, with depth values that give it thickness and roundness in the Z direction.
  • Shape paint painting is done on die surface of a 3-D shape already in die frame buffer, changing the surface look but not the depth shape. A color and brush are picked as usual. However, instead of picking a depth and painting at a particular parallax offset, when Shape paint mode is picked, pairs of pixels with the appropriate cursor Y value are compared.
  • 'Solid' objects may be built up from contours at different depth planes (like a contour map) or may be rendered by otiier software and imported, or rendered by the 3-D paint system.
  • libraries of precomputed shapes may be made available to die operator, much like 2-D clip-art.
  • shape paint will permit the surface of those objects to be re-painted without disturbing their shape.
  • the ability to create 3-D objects by stacking painted contours, as well as to import and combine 3-D objects from other sources, and to 'Shape paint' provide a flexible mechamsm for a painter or illustrator to create 3-D objects and environments.
  • One particularly useful application for this system will be the creation of such objects and environments for inclusion in Virtual Reality simulation systems.
  • the deptii values may be 'dithered'. Dithering randomly selects one value from the closest available quantized depdi values above and below die desired deptii value. The probability of selecting one or the other is adjusted depending upon how close each available value is to the desired value. In this way a 'fuzzy' rather than stairstepped depdi image is created and when further painting or other image functions interact with the object, a fuzzy but smooth looking intersection with the new object results.
  • the POV may be shifted in 'wiggling' motion to produce 'motion (serial) parallax' which can convey much depdi information.
  • 'motion sinogram
  • the perceived wiggling of the image can be annoying, and therefore is of limited use for long entertainment presentations.
  • this method can be very useful to convey 3-D information without any glasses.
  • 3-D visualization of data can be achieved without glasses.
  • the applicant has used die Intel DVI board to composite images in real- or near- real-time utilizing the multi-plane techniques described herein.
  • Such a system can be programmed to update the POV, or parallax shifts between planes, to implement the above.
  • the POV can be updated continuously to create a display from the appropriate perspective as the viewer moves around die room; this is a very powerful 3-D (though not binocular) effect.
  • the combination of 3-D glasses and the head-tracker (or a head-mounted display) produce an even more powerful effect. In that case, not only head position but orientation may be captured and each eye may be displayed witii proper horizontal and vertical parallax offsets, since head orientation may tilt the eyes.
  • Such a system is a low-end alternative to costly full-immersion helmet-based Virtual Reality displays.
  • holes in the image may develop, and the reverse sides of objects may need to be displayed. It is just such as system as described herein that will be used to fill tiiose holes or 'paint' die reverse sides of objects so that not only will 2-D images and motion sequences be able to be adapted to 3-D stereoscopic displays, but may also be adapted for use with Virtual Reality systems.
  • Such holes (**) may be automatically covered by stretching the available material at either end, or by interpolating the material on either side of the holes.
  • the holes may be filled with values derived from interpolating from A to C.
  • the hole in the right image may be covered with material interpolated between A and B; and, die hole in the left image may be covered witii material interpolated between B and C.
  • frame-rate-doubling televisions are now commercially available. These take the 60 fields of a standard television signal and by the use of a frame store, show each twice. Such a framestore device may be easily modified to solve the flicker problem associated witii 3-D electronic LCD-shuttered glasses which deliver only 30 fields to each eye. Similarly, because die odd lines go the one eye, and the even lines to die odier, vertical resolution to each eye is halved. A television witii a frame store device would be able to fill in tiiose gaps during display. Application of the applicant's variable Spatial-Temporal Signature (STS) to this technique will prove useful. See applicant's serial number 07/077,916.
  • STS Spatial-Temporal Signature
  • a new type of low-cost electronic stereo glasses may be used where small lights (or LEDs) are flickered, alternating with each eye in synchrony to the alternating left/right pattern.
  • the light for one eye is on while the image for the other eye is displayed.
  • Altiiough not bright enough to completely disturb vision, die small flash in one eye will suppress the perception of the unwanted image in diat eye.
  • the light may be directly flashed into the visual field or, preferably, through a diffusion element.
  • half-reflective lenses may be used in diese glasses which will reflect the internal light flash but otherwise pass the television image to the eye. There will also have to be an opaque barrier to prevent cross-talk between the two lights and two eyes.
  • red fields would be alternated with green fields in order to provide thirty fields per second (e.g. , for NTSC) to each eye separately.
  • this reduces die flicker rate for each eye below the perceivable threshold, as witii 60 Hz field sequential LCD shuttered glasses.
  • the present system can increase the 'switching rate' as follows. For each field red lines and green lines will be alternated. Due to die raster pattern adjacent vertical information is widely spaced in time and therefore changing from monochromatic red to monochromatic green between lines does not effect die detail information. The switching rate is, thus, increased from 60 Hz to 15.75 KHz, far too fast for current LCD shutter technology. (Of course, once high-speed LCD shutter or otiier active technology is developed, all of the techniques herein may be applied to full color actively switched video signals.
  • lines containing both red and green may be inserted into the sequence to fill in 'holes' that would otherwise develop in one eye or the other.
  • segments of less t an one line, but sufficiently long to avoid major degradation due to composite encoding/decoding may also be used and would contain monochromatic red or monochromatic green information, possibly with the addition of segments containing both red and green information.
  • instant inventor suggests an alternative whereby monochromatic red, monochromatic green and full color segments of one line or less are alternated, with all the variations described above, for the reception of both color and 3-D information on a composite monitor.
  • red and green images can be derived as die red and green components of a full color image. These may be used instead or in combination with red and green tinted monochrome signals. Further, die blue color information may be mixed in to either the red or green side or both. In general the red and green color components, red and green tinted monochrome components, a blue component, and full color signals may be mixed to achieve an appropriate "look" for each scene of a motion picture sequence, much as standard
  • anaglyphic glasses use red and blue rather than red and green.
  • the equivalent process can be described by "swapping" die words green and blue in the discussion herein.
  • anaglyphic glasses of other color combinations in particular tuned to the NTSC I or Q hues, can be employed with those colors substituted for red and green.
  • left and right image pairs are alternated as video fields or frames.
  • interleaving of smaller image sections can be used to advantage for improved perception.
  • Individual lines or sub-lines can be alternated from die left and right images.
  • Faster flickering electronic LCD-glasses, than currently available on a low- cost basis, would be required to decode such an image.
  • flickering is not an issue.
  • die way diat die left/right information is distributed over the frame may be changed from frame to frame in a non-trivial manner, as a way to improve perception.
  • This technique called a variable STS, is described in some detail in applicant's co-pending application number 07/077,916.*** See Figure 5 for a basic high-level system diagram of the above.
  • die 3-D paint system will operate at a continuously and dynamically selected depth witii the addition of a 3-D input device. Such a device is described in the following pages.
  • mice For inputting positional information to digital computers, mice (basically upside-down trackballs) operate witii a single roller-ball or optical sensor and give two axis inputs: X (left-to-right) and Y (away /toward die user).
  • X left-to-right
  • Y away /toward die user
  • One, two or more buttons allow for selection type input as well.
  • Flat X-Y data tablets provide similar functionality, with one on/off 'button' function often tied to the pen tip.
  • the present discussion relates to a class a mouse-like input devices diat employ multiple roller-balls (or optical sensors) to provide additional input axes and a versatile user interface.
  • a twist about die Z axis i.e., the axis perpendicular to the two axes of normal mouse movement.
  • Such input could be used as a rotational parameter, for example, to point a screen cursor in a particular direction, not just position the cursor on the X-Y plane; or, for other rotational information, such as to rotate 3-D objects in a computer graphic or CAD system.
  • the input may be used as s magnitude parameter, for example to translate along the Z-axis, e.g., clockwise (CW) twist for up (with respect to the desktop, or out with respect to the screen) and counterclockwise (CCW) twist for down (or in).
  • CW clockwise
  • CCW counterclockwise
  • it may be used for sign or selection information for a 'double button' like a two way switch or knob, e.g., CW twist for 'plus' or select, CCW twist for 'minus' or cancel.
  • the two-ball mouse comprises two trackballs mounted fore and aft to (of) the center point of the Y-axis. (Alternately, the two balls could be mounted to the left and right of the center of the X axis; but, this wide mouse would probably be harder to hold.)
  • the two balls operate in tandem, giving (dieoretically, and at least approximately) equal output.
  • CW as seen from above, the rear ball will register a positive or rightward X output, die rear ball a negative or leftward X output.
  • a CCW twist will produce complementary output.
  • the output of both balls can be sent on to the host computer for integration there, or they may be integrated by die circuit in die two-ball mouse for transmission of X, Y and Z data (along witii the button hit information as well).
  • the two-ball mouse's software driver will pass on die X, Y, Z and button-state information to applications programmed to properly interpret it. Alternately, in a standard mouse compatibility mode, only die standard X, Y and button information would be presented to die application.
  • the two-ball mouse may be lifted, unwound and replaced on the pad for continued twisting.
  • an alternate interpretation of the input by the software would allow for a 'wind-up mode'.
  • die CW or CCW, perhaps chosen by whichever move started the sequence
  • the CW and CCW twists could be summed for a magnitude, die sign being specified, e.g., by a 'button' or by which twist direction started the sequence.
  • Which twist mode is in operation may be specified by the user via a button or a menu selection, or may be determined by the application depending on which software function were in operation.
  • software may allow the user to mm off wind-up mode by making a normal X or Y axis motion; pausing longer than some threshold between twists; making repeated turns in the same direction, or further than the normal back-and-fortii wind-up twists; or by some other programmed gesture.
  • die axis of twist is through a point between the two balls, and the front and rear balls act in opposition.
  • die axis of twist would pass dirough the center of either the front or rear trackball.
  • die pinned ball would see (theoretically, or at least approximately) no movement and the other ball would 'wag' left and right, along the X axis with a slight curve, for balls mounted front and rear. So, in addition to twist CW/CCW, one could pin the head ball and 'wag die tail' back and fourth, or pin the tail and wag die head back and fourth.
  • a 'rocker' pin or ball would be placed between the two trackballs. This pivot would extend somewhat beyond die two trackballs and would allow die user to pivot to have either the front or rear ball contact the mouse pad, without the other making contact. In that way, die contacted ball could be 'wagged' around the pivot, in a manner similar to above but with a smaller pivot radius.
  • the pivot point would have a small roller-ball in its tip, much like a large-bore, inkless, ballpoint pen refill.
  • the rear (or front) ball and pivot could be moved left and right along the X axis, much like the wagging mode; or, back and fourth along the Y axis.
  • a spring load in die pivot point would allow the user to apply a small amount of additional pressure to have both the front and rear balls contact the mouse pad at the same time. In that case, the dual-ball X and Y motions, as well as the twist mode, would be available.
  • tiiere would be a total of at least seven axes with the dual trackball mouse, as opposed to only two (X and Y) witii a standard mouse. These are: X and Y pivoted for the front only, X and Y pivoted for rear only, X and Y with botii balls contacting and moving in a coordinated way and, lastly, twisting with both balls contacting in an anti-coordinated way.
  • a dual trackball mouse would permit die new gestures to be added to the X-Y positioning and button clicking of the standard mouse.
  • a 'wrist flick' could be used to implement a number of useful functions such as a 'page flip' to browse through a text file as if it were a book.
  • a flick consisting of a short CW motion followed by a return CCW motion would constitute a jump ahead one page command; a CCW-CW combination, a mm back one page.
  • A(n adjustable) threshold of, say, about 5 degrees would prevent accidental page turning from minor jiggling of the mouse.
  • a larger flip would mm 2, 5 or 10 pages (again, perhaps user selectable) and a very large flip would mm 50 or 100 pages.
  • the degree of 'largeness' would be increased by: increasing the CW/CCW rotation to 10, 15 or more degrees; increasing the speed of die flip; increasing the time between the CW and CCW phases of the flip, or between two successive flips; or, to make a number of flips in quick succession with 1, 2 or 3 flips indicating 1, 10 or 100 pages, for example.
  • gestures and axis combinations can be developed with such a device and the proper software.
  • calligraphic writing may be implemented by drawing in X and Y as usual and, in addition, controlling the angle or tilt of die 'pen' by twisting the mouse.
  • an oriented chisel-like function may be implemented.
  • two separate objects may be controlled simultaneously without having to switch back and forth between functions, using a button click.
  • die two objects may also be moved togedier in a coordinated manner.
  • roller-balls More than two roller-balls may be used for additional input axes and increased system flexibility. Many configurations are possible and witiiin the scope of the present invention but, in particular, four- and five-ball configurations will be discussed blow.
  • the central pivot ball were also an active input roller-ball.
  • the central ball When used with each of the four comer balls, the central ball would provide a pair of balls and this be able to provide a twist axis as well as X and Y, for a total of 12 axes. Additionally (perhaps, depending upon the amount that the central ball is extended and the dexterity of die user) by balancing carefully, so that none of die comer balls touched the pad, the central ball alone could provide an additional set of X and Y axes, for a total of 14.
  • Such an embodiment may be particularly useful for systems used to design and produce 3-D computer animation.
  • it is often required to specify (at least for key frames) the X, Y and Z positions as well as rotations around the X, Y and Z axes, for both objects and the theoretical 'camera' .
  • These four groups of 3 parameters may each be assigned to the four comer roller-balls, each with 3 axes.
  • all of the required positioning information may be controlled gesturally.
  • the same comer roller-balls may be used to enter other information in otiier modes; mode-switching would be achieved with the active center ball, or otherwise.
  • the entire design may be rotated 45 degrees, so that the active comer roller-balls would be located at die NE, NW, SE and SW comers, with stable tilt configurations at N, S, E and W.
  • two (or more) pen/stylus or puck/cursor receptors may be put in the same housing and queried (in parallel or serially) to generate not only position information but orientation information as well; or, two trackballs may be mounted in such a way as to be covered by die palm and thus input direction information, not just X and Y positional information.
  • 3D displays such as tiiose employing passive or active glasses, lenticular screens, CRT displays, holographic film, virtual reality and serial motion parallax; trackballs, mice, styluses and data tablets, or other position or input sensors in coordinated use as described herein; and, various ways of storing image and deptii information. Accordingly, it is intended diat all matter contained in the above description or shown in the accompanying figures shall be interpreted as illustrative and not in a limiting sense.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)
  • Stereoscopic And Panoramic Photography (AREA)

Abstract

L'invention concerne un système informatique, mis en application par un illustrateur ou par un peintre, servant à créer des images binoculaires stéréoscopiques (figures 1 et 4) à partir d'éléments d'images bidimensionnelles et tridimensionnelles (figure 2). L'ampleur du décalage pour chaque objet est apparentée à la profondeur dudit objet, le sens du décalage est apparenté à l'oeil pour lequel la visualisation est destinée.
PCT/US1996/009812 1995-06-07 1996-06-07 Peinture interactive stereoscopique WO1996041311A2 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU61082/96A AU6108296A (en) 1995-06-07 1996-06-07 An interactive computer system for creating three-dimensiona l image information and for converting two-dimensional image information for three-dimensional display systems

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US48594595A 1995-06-07 1995-06-07
US48821995A 1995-06-07 1995-06-07
US08/485,945 1995-06-07
US08/488,219 1995-06-07

Publications (2)

Publication Number Publication Date
WO1996041311A2 true WO1996041311A2 (fr) 1996-12-19
WO1996041311A3 WO1996041311A3 (fr) 1997-02-13

Family

ID=27048522

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US1996/009812 WO1996041311A2 (fr) 1995-06-07 1996-06-07 Peinture interactive stereoscopique

Country Status (2)

Country Link
AU (1) AU6108296A (fr)
WO (1) WO1996041311A2 (fr)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1999012127A1 (fr) * 1997-09-02 1999-03-11 Dynamic Digital Depth Research Pty Ltd Appareil et procede de traitement d'image
WO2000060543A1 (fr) * 1999-04-05 2000-10-12 Intergraph Corporation Processeur graphique servant a effectuer l'affichage stereoscopique d'une image graphique
US6674440B1 (en) 1999-04-05 2004-01-06 3Dlabs, Inc., Inc. Ltd. Graphics processor for stereoscopically displaying a graphical image
WO2009113761A1 (fr) 2008-03-12 2009-09-17 Samsung Electronics Co., Ltd. Procédé et appareil de traitement d'image, procédé et appareil de reproduction d'image et support d'enregistrement
GB2470759A (en) * 2009-06-03 2010-12-08 Sony Comp Entertainment Europe Displaying videogame on 3D display by generating stereoscopic version of game without modifying source code
US20120013605A1 (en) * 2010-07-14 2012-01-19 Lg Electronics Inc. Mobile terminal and controlling method thereof
WO2012132333A1 (fr) * 2011-03-28 2012-10-04 Sony Corporation Appareil de traitement d'image et procédé de traitement d'image
RU2497195C2 (ru) * 2010-08-30 2013-10-27 Сони Корпорейшн Процессор сигналов, способ обработки сигналов, устройство отображения и программный продукт
CN113377028A (zh) * 2021-06-15 2021-09-10 湖南汽车工程职业学院 一种基于vr及5d的动力蓄电池测试教学系统
CN114511048A (zh) * 2022-04-20 2022-05-17 深圳市海清视讯科技有限公司 图像处理方法、装置、设备及系统

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4697178A (en) * 1984-06-29 1987-09-29 Megatek Corporation Computer graphics system for real-time calculation and display of the perspective view of three-dimensional scenes
US5347620A (en) * 1991-09-05 1994-09-13 Zimmer Mark A System and method for digital rendering of images and printed articulation

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4697178A (en) * 1984-06-29 1987-09-29 Megatek Corporation Computer graphics system for real-time calculation and display of the perspective view of three-dimensional scenes
US5347620A (en) * 1991-09-05 1994-09-13 Zimmer Mark A System and method for digital rendering of images and printed articulation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
GROSSMAN, "Look Ma, No Glasses", GAMES, April 1992, pages 12-14. *

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6496598B1 (en) 1997-09-02 2002-12-17 Dynamic Digital Depth Research Pty. Ltd. Image processing method and apparatus
WO1999012127A1 (fr) * 1997-09-02 1999-03-11 Dynamic Digital Depth Research Pty Ltd Appareil et procede de traitement d'image
WO2000060543A1 (fr) * 1999-04-05 2000-10-12 Intergraph Corporation Processeur graphique servant a effectuer l'affichage stereoscopique d'une image graphique
US6674440B1 (en) 1999-04-05 2004-01-06 3Dlabs, Inc., Inc. Ltd. Graphics processor for stereoscopically displaying a graphical image
EP2253145A4 (fr) * 2008-03-12 2013-04-17 Samsung Electronics Co Ltd Procédé et appareil de traitement d'image, procédé et appareil de reproduction d'image et support d'enregistrement
WO2009113761A1 (fr) 2008-03-12 2009-09-17 Samsung Electronics Co., Ltd. Procédé et appareil de traitement d'image, procédé et appareil de reproduction d'image et support d'enregistrement
EP2253145A1 (fr) * 2008-03-12 2010-11-24 Samsung Electronics Co., Ltd. Procédé et appareil de traitement d'image, procédé et appareil de reproduction d'image et support d'enregistrement
US8849009B2 (en) 2008-03-12 2014-09-30 Samsung Electronics Co., Ltd. Image processing method and apparatus, image reproducing method and apparatus, and recording medium
GB2470759A (en) * 2009-06-03 2010-12-08 Sony Comp Entertainment Europe Displaying videogame on 3D display by generating stereoscopic version of game without modifying source code
US9420257B2 (en) * 2010-07-14 2016-08-16 Lg Electronics Inc. Mobile terminal and method for adjusting and displaying a stereoscopic image
US20120013605A1 (en) * 2010-07-14 2012-01-19 Lg Electronics Inc. Mobile terminal and controlling method thereof
RU2497195C2 (ru) * 2010-08-30 2013-10-27 Сони Корпорейшн Процессор сигналов, способ обработки сигналов, устройство отображения и программный продукт
WO2012132333A1 (fr) * 2011-03-28 2012-10-04 Sony Corporation Appareil de traitement d'image et procédé de traitement d'image
US9779539B2 (en) 2011-03-28 2017-10-03 Sony Corporation Image processing apparatus and image processing method
CN113377028A (zh) * 2021-06-15 2021-09-10 湖南汽车工程职业学院 一种基于vr及5d的动力蓄电池测试教学系统
CN114511048A (zh) * 2022-04-20 2022-05-17 深圳市海清视讯科技有限公司 图像处理方法、装置、设备及系统

Also Published As

Publication number Publication date
WO1996041311A3 (fr) 1997-02-13
AU6108296A (en) 1996-12-30

Similar Documents

Publication Publication Date Title
US6590573B1 (en) Interactive computer system for creating three-dimensional image information and for converting two-dimensional image information for three-dimensional display systems
Agrawala et al. Artistic multiprojection rendering
US5459529A (en) Video processing for composite images
US5751927A (en) Method and apparatus for producing three dimensional displays on a two dimensional surface
Neider et al. OpenGL programming guide
US4685070A (en) System and method for displaying, and interactively excavating and examining a three dimensional volume
US4602286A (en) Video processing for composite images
US6373490B1 (en) Using remembered properties to create and regenerate points along an editable path
Raskar et al. Interacting with spatially augmented reality
Tolba et al. Sketching with projective 2D strokes
GB2113950A (en) Image composition system
US20100201623A1 (en) Method and system for displaying information
JPH0627919A (ja) カーソル生成装置及び同方法
WO1996036011A1 (fr) Systeme graphique utilisant des valeurs d'homogeneite pour la profondeur, destine au mappage a occlusion et au mappage de texture
AU7908091A (en) Computergraphic animation system
WO1992021096A1 (fr) Synthese et traitement de l'image
EP0320166A1 (fr) Traitement électronique d'images
JPH06507743A (ja) 画像合成及び処理
JP2006293792A (ja) 立体映像生成装置
EP0843939A2 (fr) Generation d'images stereo
GB2157122A (en) Image composition system
WO1996041311A2 (fr) Peinture interactive stereoscopique
JPH0793579A (ja) ３次元シミュレーション画像形成システム
Durand The “TOON” project: requirements for a computerized 2D animation system
JP4109012B2 (ja) 解像度混合ディスプレイ

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AM AT AU BB BG BR BY CA CH CN CZ DE DK EE ES FI GB GE HU IL JP KE KG KP KR KZ LK LR LT LU LV MD MG MN MW MX NO NZ PL PT RO RU SD SE SI SK TJ TT UA US UZ VN

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): KE LS MW SD SZ UG AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN ML MR NE SN TD TG

AK Designated states

Kind code of ref document: A3

Designated state(s): AM AT AU BB BG BR BY CA CH CN CZ DE DK EE ES FI GB GE HU IL JP KE KG KP KR KZ LK LR LT LU LV MD MG MN MW MX NO NZ PL PT RO RU SD SE SI SK TJ TT UA US UZ VN

AL Designated countries for regional patents

Kind code of ref document: A3

Designated state(s): KE LS MW SD SZ UG AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN ML MR NE SN TD TG

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase in:

Ref country code: CA