EP0763304A4

EP0763304A4 - Visual display systems and a system for producing recordings for visualization thereon and methods therefor

Info

Publication number: EP0763304A4
Application number: EP95924572A
Authority: EP
Inventors: Shmuel Banitt
Original assignee: Individual
Current assignee: Individual
Priority date: 1994-05-31
Filing date: 1995-05-31
Publication date: 1998-04-08
Also published as: AU2901295A; EP0763304A1; JPH10501385A; WO1995033340A1; IL113842A0; CA2191711A1

Abstract

A visual display system (100) and method for displaying three dimensional visual images wherein the display system includes a main screen (102) and at least two peripheral screens (104, 106). The main screen displays a primary two dimensional visual image of a first scene. The peripheral screens display at least two secondary visual images. Together, the three visual images produce a three-dimensional visual image. The secondary visual images are processed from images of at least second and third scenes which differ in time or significantly in space from the first scene. Each of the secondary visual images matches the primary visual image so as to form a uniform field of view therewith.

Description

VISUAL DISPLAY SYSTEMS AND A SYSTEM FOR PRODUCING RECORDINGS FOR VISUALIZATION THEREON AND METHODS

THEREFOR

FIELD OF THE INVENTION The present invention relates to visual display systems in general. The present invention relates in particular to visual display systems and a method for displaying three dimensional visual images and stimulating the feeling of viewer participation with the displayed image. The present invention also relates to a system and a method for producing the recordings of the visual images for visualization on the visual display systems.

B ^•::ROUND C -HE INVENTION There are -i.ree me is to perceive depth: a stereoscopic perception depth, a psy ological perception of depth and the use of a chcnginc, ;ocus to perceive depth. A viewer viewing a scene in which the focal point changes will perceive that the scene has depth. Stereoscopic depth perception requires that the viewer view a scene from two different locations at the same time. In real life, this means that he sees the same scene using both his left eye and right eye to see different images. When two dimensional visual images are substituted for real life, it requires the use of two images taken from different right eye and left eye locations respectively.

On the other hand, a viewer can perceive depth by virtue of a number of psychological depth perception cues which are present in both real life and in two dimensional visual images. Common cues include areal perspective where the change of coloring or softening of the shapes gives a feeling of distance to far objects, distribution of light and shade on an object, overlapping of contours as when a near object hides part of a more distant object, geometrical perspective for example in the case of the converging of the parallel rails of a railroad track as they extend in the distance, movement of an object in a visual image either towards or away from the viewer or audience, and movement for stimulating the Pulfrich effect.

This is true of all the various devices in which two dimensional visual images are projected onto screens. Such devices include home entertainment systems including televisions, motion picture theaters, training simulators, multi-media systems and so on having applications in the areas of entertainment, work, education, exhibitions, training, etc.

A number of devices have been proposed to permit a viewer or an audience to view three dimensional visual images or at ..least the illusion of three dimensional visual images-=when viewing two dimensional visual images. These fall into three groups: those which provide stereoscopic viewing, those which create the psychological illusion of depth and those which provide three dimensional visual images.

The devices providing stereoscopic viewing generally require the projection of two pictures of the same view, taken simultaneously by more than one camera from slightly different angles, to emulate the binocular vision of normal eyes. Several examples are shown in U.S. Patent Nos. 4,420,230 to McElveen, 4,714,319 to Zeevi, and 5,225,861 to Marks. The use of a vibrating mirror or screen is shown in U.S. Patent Nos. 4,130,832 to Sher and 3,814,513 to Nims et al. Other systems require special spectacles to be worn by the viewer or other devices which direct the proper view to the proper eye. All the systems are cumbersome and costly to use. The second approach is shown in U.S. Patents 4,651,219 to Rickert, 3,582,961 to Shindo and 4,000,367 to Field wherein depth perception is provided by isolating the image to be viewed. In this way, the stereoscopic cues of the space surrounding the image are suppressed so the inherent psychological depth indicators in the picture can be sensed by the observer who perceives a two dimensional visual image as a three dimensional scene. Other devices which produce a similar result are described in U.S. Patent No. 3,820,873 to Redington et al., U.S. Patent No. 4,154,514 to Harvey which utilize a curved screen, and U.S. Patent No.

4,941,041 to Kenyon which utilizes the Pulfrich illusion.

Other devices for providing three dimensional viewing are also described in the following U.S. Patents: U.S Patent No. 2,468,046 to de los Monteros discloses an apparatus for projecting and viewing images wi" a depth effect through the use of at least one mirror to reflect the image from a projector onto a screen such that a degree of displacement between the images produces the illusion of stereoscopy.

U.S. Patent 3,514,871 to Tucker discloses a wide angle visual display for a training simulator which eliminates distracting discontinuities found in many wide angle displays. It does so by providing three virtual image lenses between the viewing point and the two dimensional displays and by providing substantial duplication of the projected image on the marginal portions of the displays. This creates binocular vision at the corners of the displays which gives the illusion of three dimensionality.

U.S. Patent 5,274,405 to Webster describes a device into which the head is placed wherein images are projected both to the area of visual attention and the area of peripheral vision of each eye. The images are focused on the screens to provide sharp stereoscopic viewing throughout the field of view. There is a need for simple and economical visual display systems and method for displaying three dimensional visual images and stimulating the feeling of viewer participation with the displayed image. There is also a need for a system and a method for producing the recordings of the visual images for visualization on the visual display systems.

SUMMARY OF THE INVENTION The present invention provides visual display systems and a method for displaying three dimensional visual images and stimulating the feeling of viewer participation with the displayed image. The present invention also provides a system and a method for producing the recordings of the visual images for visualization on the visual display systems.

The visual display systems can be readily adapted for use in home entertainment systems including televisions, training simulators, multi-media systems, movie picture theaters, exhibitions, etc.

In accordance with a first aspect of the present invention, the visual display system includes a main screen and at least two peripheral screens. The main screen displays a primary two dimensional visual image of a first scene. The peripheral screens display at least two secondary visual images. Together, the three visual images produce a three dimensional visual image. The secondary visual images are processed from images of at least second and third scenes which differ in time or significantly in space from the first scene. Each of the secondary visual images matches the primary visual image so as to form a uniform field of view therewith.

The secondary visual image matching unit comprises at least one of a multi-sealer, a rate matcher, a motion matcher, an edge matcher and a color matcher. The multi- sealer scales the selected secondary visual images to match the scale of the primary visual image. The rate matcher matches the timing of a series of selected secondary visual images to the timing ^~f a series of primary visual images. The motion matv. ,er matches the camera motion in the sr^ies of selected secondary visual images to the camera m ion of a se. ies of primary visual images. The edge mate r blends each edge of the primary image with the edge >f one of the secondary images adjacent thereto. The ":olor matcher matches the color quality of the select. J. secondary images to the color quality of the primary visual image.

The primary visual image can be a real-life image or a computer generated or animated image. By virtue of the primary visual image being the subject of the displayed image, the secondary visual images need not be taken in the same manner, location or time as the primary image. Instead, the secondary visual images are matched to the primary visual image. Hence, in actual fact, each of the secondary visual images can be a real-life image or a computer generated or animated image. The secondary visual images are typically displayed at a lower resolution than the primary visual image without detracting from the effect of three dimensionality. Furthermore, computerized techniques are utilized to generate movement in the secondary visual images to further enhance the three dimensionality of the displayed image.

Depending on the particular realization of the visual display system, each of the screens can be any one of the following group: a moving picture theater screen, a television screen, a computer monitor, a LCD screen and the like.

The system can also include projection apparatus which, depending on the particular realization of the visual display system, can be any one of the following group: a photographic film projector, a CRT tube and a video projector.

Further features of the visual display systems include providing trapezoid shaped peripheral screens and a plurality of speakers for playing back a soundtrack for accompanying the visual images displayed on the system.

A system and method for producing recordings of visual images for visualization on a visual display system displaying three dimensional visual images are also disclosed. The system includes primary and secondary visual image sources, a selection unit and a secondary visual image matching unit. The primary visual image source stores primary visual images viewing first scenes. The secondary visual image sources store secondary visual images, each of which views scenes which differ from the first scenes. The selection unit selects at least two secondary visual images from the secondary visual image source such that the secondary visual images are compatible with a primary visual image. The matching unit generally matches the selected secondary visual images with the primary visual image.

The primary visual image source can be a recording from any one of the following group: a television camera, a photographic film camera, a computer database and a computer sub-routine. In a similar fashion, the secondary visual image source can be a recording from any one -of the following group: a television camera, a photographic film camera, a computer database and a computer graphic routine. The system can also include output apparatus for outputting a recording of the secondary visual images and soundtrack apparatus for preparing a soundtrack to accompany the visual images.

There is also provided, in accordance with a third preferred embodiment of the present invention, a method of generally matching the camera motion of a first camera with a second camera. The method includes the steps of (a) recording the sequence of visual images produced by said first camera on a recording device, (b) at the time of filming with said second camera, displaying said recorded sequence of visual images and (c) filming with said second camera while attempting to match the camera movement displayed in said recorded sequence of visual images.

The first and second camera can be the same camera operated at different times or two cameras viewing different scenes or the same camera viewing different scenes.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is herein described, by way of example only, with reference to the drawings, wherein:

FIG. 1 is a schematic illustration of a visual screen system for displaying three dimensional visual images constructed and operative in accordance with the teachings of the present invention; FIGS. 2A-2C show primary and secondary visual images for creating the three dimensional visual images as displayed by the visual display system of Figure 1;

FIGS. 3A-3C are schematic illustrations of three exemplary realizations of the visual display system of Figure 1;

FIG. 4 is a schematic illustration of a system for producing recordings of visual images for visualization on the visual display system of Figure 1;

FIG. 5 is a flow chart illustrating the operation of the system of Figure 4;

FIG. 6 is a block diagram of secondary image processing apparatus forming part of the system of Figure

4;

FIG. 7 is a pictorial illustration of scaling of secondary visual images, useful in understanding the apparatus of Figure 6; FIGS. 8A and 8B are pictorial illustrations of two series of secondary visual images, useful in understanding the apparatus of Figure 6;

FIG. 9 is a block diagram illustration of apparatus for matching motion, useful in understanding the apparatus of Figure 6; and

FIG. 10 is a schematic illustration of camera motion recording apparatus, useful in the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is of visual display systems for displaying three dimensional visual images, a system for producing the recordings of visual images for visualization on the visual display systems and methods therefor. The visual display systems can be adapted for use in home entertainment systems including televisions, training simulators, multi-media systems, movie picture theaters, exhibitions, etc.

The principles and operation of the systems and methods of the present invention may be better understood with reference to the accompanying drawings and description.

For the sake of clarity, the term "visual image" as used hereinbelow means and includes any visible object or group of objects which can observed by a viewer. The visual image can be a still visual image or a moving visual image and either of a real-life scene or a computer generated or animated scene. Furthermore, the term "recording" as used hereinbelow means the capture of a visual image in any form which can be used for the projection of the visual image onto a receiving surface for visualization. Hence, recording includes capturing the visual image on photographic film, such as motion picture film, as well as capturing the visual image, either permanently in a database or momentarily, in the form of electronic, magnetic, electromagnetic or other signal information which can be reproduced as a visible image on a receiving screen for visualization. Typical screens include a conventional motion picture theater screen, the receiving screen of a conventional home television set, a Super VGA computer monitor, a VGA computer monitor, a LCD screen, and so on.

Referring now to Figure 1, there is shown a schematic illustration of a visual display system, generally designated ^*3, constructed and operative according to the teac. .ngs of the present invention. Visual display system 100 is in the form of a wide angle, multi-screen system including a main screen 102 for displaying a primary visual image, generally designated P, and two peripheral screens 104 and 106 for displaying secondary left and right visual images, generally designated L and R, respectively. System 100 furthev includes a projection device 108 for proj cting a recording of visual image P on main screen 102 and projection devices 110 and 112 for projecting recordings of secondary visual images L and R on peripheral screens 104 and 106, respectively.

Peripheral screens 104 and 106 are inclined towards a viewer or audience V such that visual images L and R transform the two dimensional primary visual image P displayed on main screen 102 into a three dimensional visual image. While the example herein is illustrated with two peripheral screens, it will be appreciated that any number of peripheral screens may be provided, including above or below main screen 102. Furthermore, it should be noted that there can be a physical gap between main screen 102 and peripheral screens 104 and 106 without detracting from the effect of three dimensionality. Also, peripheral screens 104 and 106 can be trapezoid shaped to further enhance the feeling of three dimensionality depth. As will be described in greater detail below, psychological cues can be provided to secondary visual images L and R for -enhancing the feeling of viewer or audience participation with the visual image being displayed on system 100. The effect of three dimensionality can be still further enhanced through the L.. >ack of a soundtrack through a number of speakers 114 ana Λ16 extending along peripheral screens 104 and 106, respectively, and behind a viewer or audience V, to wrap around viewer or audience V. Speakers 114 and 116 are typically in addition to a speaker 118 associated with main screen 102, as known in the art. Typically, a controller 120 controls the operation of projectors 108, 110 and 112 such that secondary visual images L and R are displayed in synchronization with primary visual image P.

With reference now to Figures 2A - 2C, there is shown a number of visual images L, P and R for creating the three dimensional visual images displayed by visual display system 100. In principle, the effect of three dimensionality is achieved by the primary visual-image being picked up by the direct vision of the eye and the secondary left and right visual images being picked up substantially by the peripheral vision of the left and right eyes respectively. This means that rather than having to display three continuous visual images of a panoramic view of a scene, in actual fact, secondary visual images L and R can display substantially different subject matter than visual image P as long as there is a logical correlation or matching therebetween. However, in practice, the field of view of the direct vision of a viewer or an audience includes all of primary visual image P and the edge portions of secondary visual images L and R adjacent thereto. Therefore, the edge portions or margins of visual images L and R are preferably tailored to blend with the edge portions or margins of primary visual image P adjacent thereto, thereby ensuring that a viewer or an audience perceives a mental image of a single scene, even though it is formed from left visual image L, primary visual image P and right visual image R. With reference now to Figure 2A to summarize the above, the edge portion or margin of visual image L denoted 122 blends with the edge portion or margin of primary visual image P adjacent thereto denoted 124 to ensure that a viewer or an audience perceives a single scene. In a similar manner, the edge portion or margin of visual image R denoted 126 blends with the edge portion or margin of primary visual image P adjacent thereto denoted 128 to ensure that a viewer or an audience perceives a single image. In particular, tailoring of pair of edge portions 122 and 124 includes feature blending such that a feature which ό.' ears in edge portion l. ϊ is continued such that it appears in edge portion 122 and color blending such that a colored region which appears in edge portion 124 is continued such that it appears in edge portion 122. For example, a signpost 130 on edge portion 124 of primary visual image P is continued as a signpost 132 on edge portion 122 of visual image L. In a similar manner with reference to edge portions 126 and 128, a crowd of spectators shown as a patterned area 134 on edge portion 128 of primary visual image P is continued as a patterned area 136 on edge portion 126 of visual image R.

In contrast to edge portions 122 and 126 which have to include subject matter continuous with the subject matter of edge portions 124 and 128, respectively, visual images L and R only have to include subject matter which tends to match the subject matter of primary visual image P. It should be understood that "tends to match" entails that the subject matter of visual images L and R shares a logical correlation to the subject matter of visual image P in terms of the nature of features, size of objects, etc.

This feature of the invention enables the use of visual images taken from a library of visual images of real-life shots taken at the same location at a different time or even from a different location. For example, if primary visual image P is a live transmission of a Cup Final soccer game, then visual images L and R can be images of a crowd of spectators taken, for example, from another game played a year before. Furthermore, it enables the use of computer generated or animated visual images L and R to be used with real-life primary visual images P without detracting from the three dimensionality effects of depth and the feeling of viewer and audience participation. In this case, the library of visual images can include pre-prepared computer generated or animated scenes or computer graphic routines for preparing objects such that custom-made montages L and R can be prepared to match primary visual image P. With reference now to Figure 2B, three consecutive frames for display as visual images L, P and R are shown to illustrate the use of psychological depth perception cues embodied within secondary visual images L and R for enhancing the effect of three dimensionality. As described in the Background of the Invention, common psychological depth cues are as follows. First, areal perspective where the change of coloring or softening of the shapes gives a feeling of distance to far objects. Second, distribution of light and shade on an object. Third, overlapping of contours as when a near object hides part of a more distant object. Fourth, geometrical perspective, for example, in the case of the converging of parallel rails of a railroad track as they extend in the distance. And finally, adding movement towards and away from the viewer or audience and adding movement for creating the Pulfrich effect. As shown in Figure 2B, looking at the f. a es from top to bo÷.'om, the viewer perceives a clockwise rotating ε thrc ...jh the movement of the buildings and in particular building 138 from visual image P to visual image L. With reference now to Figure 2C, three consecutive frames for display as visual images L, P and R are shown to illustrate the use of movement for intensifying three dimensionality and the feeling of viewer or audience participation in the visual image displayed on system 100. Looking at the frames from top to bottom, the viewer perceives a zoom shot towards automobile 140 as further emphasized through the enlargement of trees 142 and 144 as they approach the viewer or audience. Also, the notion of movement is enhanced by a suitable sound-rack played through speakers 114 and 116 of system 100 as known in the art. It should r-e noted that in the case of a still picture primary visual imaσe P, secondary visual images L and R can include the aoove mentioned psychological depth perception cues including movement as long as the condition of continuity remains between the features present in pair of edge portions 122 end 124 and edge portions 126 and 128.

A further advantage of visual display system 100 is that it is not necessary to provide secondary visual images L and R of equal resolution with that of primary visual image P as demonstrated by the use of dotted lines in visual images L and R rather than the use of solid lines in visual image P in Figures 2A - 2C. This enables the use of lower resolution, and therefore typically less expensive and troublesome, technology for capturing visual images L and R than for capturing primary visual image P. For example, a television camera can be used to capture visual images L and R while a photographic film camera is used to capture primary visual image P. Alternatively, this enables the use of lower resolution, and therefore typically less expensive, technology for displaying visual images L and R than for displaying primary visual image P. For example, LCD screens can be used to display visual images L and R while a television screen is used to display primary visual image P. With reference now to Figures 3A - 3C, there are shown three exemplary realizations of system 100 where screens 102, 104 and 106 and projectors 108, 100 and 112 are implemented as one of a number of known devices depending on the particular application of system 100. For the purposes of exposition only, the three realizations of system 100 now described including a system 150 adapted for use in a movie picture theater, a system 151 adapted for use in a training simulator or multi-media system and a system 154 adapted for use in a home entertainment system including a television.

In the case of system 150 adapted for use in a movie picture theater, the recording of primary visual image P is cellulose photographic film while the recordings of secondary visual images L and R can be television shootings which are more easily produced at a lower cost. Hence, projector 108 is a stand-alone cinemagraphic-type projector 152 for projecting photographic film onto screen 102 in the form of a conventional motion picture theater screen 154 while projectors 104 and 106 are preferably stand-alone Barco projectors 156 and 158 for projecting video tapes onto screens 104 and 106 again in the form of conventional motion picture theater screens. System 150 further includes a controller 160 for ensuring the synchronous operation of projectors 152 and Barco projectors 156 and 158.

In the case of system 151 adapted for use in a training simulator or multi-media system, the recordings for visualization of primary visual image P and secondary visual images L and R are typically computer programs stored permanently in databases or momentarily, in the form of electronic, magnetic or other signal information. Hence, the recordings are computer generated or animated scenes. Screen 102 and projector 108 are preferably combined as a Super VGA high definition color computer monitor 162 while screen 104 and projector 110 and screen 106 and projector 112 can be combined as VGA color computer monitors 164 and 166. Alternatively, system 151 can employ LCD screens thereby obviating the need for projectors 110 and 112. In this instance, system 151 includes hardware and associated software 168 for the synchronous display of the recordings. A distinction between system 151 and presently available three dimensional visual image simulators is that the use of low resolution secondary visual images enables the use of less expensive hardware and less software processing. In the case of system 154 adapted for use in a home entertainment system including television, the recordings for visualization of primary visual image P and secondary visual images L and R are typically received as electromagnetic transmissions from a broadcasting company or as electric signals through cables from a TV cable company. In this instance, the electromagnetic transmissions or the electric signals include all information required for visualization of the recordings on screens 102, 104 and 106. It should be noted that due to the lower resolution of secondary visual images L and R, the amount of information required to be transmitted is, of course, more than the amount of information required for present day transmissions on a single screen, however, it is less than three times the amount of information. Hence, screen 102 and projector 108 are preferably combined as a conventional television set 170 while screen 104 and projector 110 and screen 106 and projector 112 are preferably combined as lower resolution screens and in particular LCD screens 172 and 174. With reference now to Figures 4 and 5, there are shown a system, generally designated 200, and a flow chart for producing recordings of visual images L, P and R, and preferably an accompanying soundtrack, for enabling three dimensionality when displayed on visual display system 100 (see Figure 1) . As will become apparent below, system 200 can be realized in hardware and software well within the purview of those skilled in the art. Furthermore, system 200 can be designed either to require considerable degrees of operator intervention by a film or TV editor along with a soundtrack editor or be fully computerized when integrated into a training simulator or multi-media system.

For the purposes of highlighting particular aspects of the present invention, the preparation of recordings of visual images L and R for visualization alongside a primary visual image P in the form of a movie picture in a movie picture theater is now described.

First, subject matter of the movie picture is captured on photographic film and is edited to produce a finished version of the recording of primary visual image P for visualization on main screen 102. The recording of primary visual image P is then imported using- a Telecine, another film-to-video conversion unit or directly, into a computerized video editing machine in which primary visual image P is digitized, thereby providing a primary visual image source 202. Primary visual image P can be viewed on a display 204, thereby enabling the processing of secondary visual images L and R as now described.

Unprocessed left and right secondary visual images are selected using selection apparatus 207 and retrieved from a secondary visual image source 206 having subject matter compatible with primary visual image P and displayed on left and right side screens 208 and 210, respectively. Source 206 can include other shots taken, for example, using television cameras, from the same location but at another time as the taking of primary visual image P or a different location. Alternatively, source 206 can include a library of visual images in the form of pre-prepared computer generated or animated scenes or computer graphic routines for preparing objects such that a custom-made montage can be prepared to match primary visual image P.

In the case of a more automated system 200, image processing software, for example, PHOTOSHOP, available from Adobe Systems Inc. of Mountainview, California, USA, can be employed for analyzing the subject matter primary visual image P. In particular, analysis in terms of features, colors, lighting and so on is made of edge portions 212 and 214 which are to be displayed alongside secondary visual images L and R, respectively.

The unprocessed secondary visual images L and R are processed by secondary visual image processing apparatus 216 for matching secondary visual images L and R with primary visual image P. The image processing includes generally matching the overall shading, lighting, coloring, geometry and shadows as well as object and camera movement s^~ that visual images L and R include psychological άei ι perception cues for when the recordings of visual images L and R are visualized on visual display system 100. Furthermore, both object and camera movement is incorporated within secondary vi: 1 images L and R such that the effect of t..._ee dimensionality is enhanced in the visualization of the recordings on visual display system 100.

In one embodiment, secondary visual image processing apparatus 216 tailors edge portions 218 and 220 of secondary visual images L and R t edge portions 212 and 214 of primary visual image P Such that pair of edge portions 212 and 218 and pair of edge portions 214 and 220 blend so as to form visual transitions between primary visual image P and secondary visual images L and R, respectively. This is achieved by ensuring that features and colors which appear in edge portion 212 are continued such that they appear in edge portion 218. In a similar fashion, features and colors which appear in edge portion 214 are continued such that they appear in edge portion 220. Visual images L and R can be output as video -recordings-, computer databases, etc.

And finally, a soundtrack is prepared to accompany the visual display such that its playback through speakers 114 and 116 of system 100 can be employed to enhance the effect of three dimensionality in general and the feeling of space and viewer and audience participation in particular.

Reference is now made to Figure 6 which illustrates the elements of the secondary visual image processing apparatus 216 useful in- a second embodiment of the present invention for generally matching the object and camera movement in the primary and secondary visual images. Reference is also made to Figures 7, 8A, 8B and 9 which detail the operation thereof. It will be appreciated that the secondary visual image processing apparatus 216 typically operates on each series of secondary visual images L or R separately.

Apparatus 216 typically operates in conjunction with an operator and comprises a multi-sealer 230, a rate matcher 232, a motion matcher 234 and a color matcher 236. The multi-sealer 230 scales the secondary visual images L and R so that objects therein and in the corresponding primary visual image P will have approximately the same size. Rate matcher 232 adjusts the timing of the series of secondary visual images L and R (after scaling) to approximately match that of the series of primary visual images P. The motion matcher 234 adjusts the camera motion of the series of secondary visual images L and R (after scaling and flow matching) to approximately match that of the series of primary visual images P. Finally, the color matcher 236 approximately matches the color (luminance and saturation) of the secondary visual images L and R with that of the primary visual image P. The output of the color matcher 236 is a series of secondary visual images L' and R' which match, in some way, the series of primary visual images P.

Figure 7 illustrates the operation of the multi- sealer 230. Image 240 is the original image and image 242 is the scaled image. Image 240 is divided into a plurality of sections as a function of the objects within the image. For example, image 240 is shown having a road 244, two cars 246, a fence 248, two houses 250 and some bushes 252. If the primary visual image P has the fence 248 at a different vertical location, secondary image 240 has to be changed. Multi-sealer 230 changes the size of all the sections of the image, as desired. For example, a line 254 is shown moved vertically upward from image 240 to image 242. It is labeled 254' in image 242. In order to move line 254, the sections of the image below and above it are rescaled to match the desired amount of movement. Thus, the lower section of image 240 is scaled up, in accordance with known methods, and the upper section of image 240 is scaled down, also in accordance with known methods. As can be seen, the objects 244', and 246' are larger in image 242 and the objects 248', 250' and 252' are smaller.

Rate matcher 232 attempts to match the rate of flow of objects, or the timing of their movement, in the series of secondary visual images L and R with that in the series of primary visual images P. For example, if a bird flies across the scene in the primary visual image P and into the scene in the secondary visual image L, its rate of flight should be the same in both sequences. If the camera pans the scene in the primary visual image P into the secondary visual image R, it ought to pan at the same rate. Rate matcher 232 first determines the rate of movement of an object in all three series of images and then synthesizes part of the series of secondary visual images L and R to force the rates to match. For example, and as shown in Figure 8A, it may take four secondary images 260a - 260d for a car 262 to move from the left to the right of the image. However, in the series of primary visual images P, it took five images. Thus, the rates of flow do not match. Accordingly, rate matcher 232 removes the middle images, as indicated by arrows 264, and synthesizes a series of replacement, intermediate, secondary images 266a - 266c, in accordance with known "optical flow" techniques to force the flow rate of the series of secondary visual images to match that of the primary visual images P.

Optical flow and other image processing techniques useful in the present invention are described in the book Digital Image Warping by George Wolberg, IEEE Computer Society, pg. 1990 and in the following two articles: M. Irani, B. Rousso and S. Poleg, "Detecting and Tracking Multiple Moving Objects using Temporal Integration", European Conference on Computer Vision, pp. 282 - 287, Santa Margarita Ligure, May 1992; and

M. Irani, B. Rousso and S. Poleg, "Computing Occluding and Transparent Motions", International Journal of Computer Vision, February 1993.

The book and articles are incorporated herein by reference.

Motion matcher 234 forces the camera motion of the series of secondary visual images L and R to approximately match that of the series of primary visual images P. Figure 9 illustrates the elements of the motion matcher 234 and indicates that motion matcher 234 comprises two motion detectors 270 and 272, a stabilizer 274 and a motion creator 276. Motion detector 270 determines the camera motion of the series of secondary visual images and provides motion parameters to the stabilizer 274 which, in turn, removes the camera motion from the series of seco.;;ary visual images. Motion detector 272 determines the camera motion of the series of primary images and provides the motion parameters to the motion creator 276. Motion creator 276 implements the motion of the primary ages P onto the stabilized series of secondary images L and R.

Motion detectors 270 and 272 can ^ -:y suitable elements which detect the motion of the cji„ara vis-a-vis the scene (rather than the motion of the objects within the scene) . Stabilizer 274 can be any unit which implements the inverse motion from a set of motion parameters. Motion creator 276 can be any unit which transforms a series of ε~rsbili.ed images into one having a motion defined by a ε of motion parameters. Motion detectors, stabilizers i motion creators are deεcribed in the above-mentioned r arenceε.

Color matcher 236 :: videε the operator with tools to change the colors of trie motion stabilized, rate and scale matched secondary visual images L and R to approximately match those of the primary visual image P. An exemplary color matcher is PHOTOSHOP, available from Adobe Systems Inc. Color is typically defined by its hue, saturation and luminance levels. Often, the overall luminance (or brightness) of an image may be different than another and thus, changing the luminance value of the entire image, or a portion thereof, will help match the images. Alternatively or in addition, it may be necessary to match the saturation (or greyness) of the images. Furthermore, at th edges where objects flow from the primary visual image. P to the secondary visual images L or R, it may be necessary to match the hue also, at least for certain objects. It will be appreciated that further procesεing, εuch as to match perspective or to add animation, can also be performed on the secondary visual imageε L and R.

The present invention also incorporates a method for filming a scene in order to achieve both the primary visual image sequence and the secondary visual image sequences. Typically, the primary visual image sequence is filmed with lotε of action, wherein the camera εtares directly at the performing actors. Once the action with the actors haε ended, the cameraman can then film the εecondary viεual image εequence. To do so, he rotates the line of sight of the camera to the right and left of the field of view and then filmε the εecondary scenes accordingly. Often, it is desirable to have the camera motion of the εecondary visual image sequenceε approximately match that of the primary viεual image εequence, thereby to reduce the amount of proceεεing which the motion matcher 234 haε to do.

Reference iε now briefly made to Figure 10 which illuεtrates an apparatus 280 for enabling a cameraman to approximately match the camera motion of the primary camera when filming the εecondary viεual image εequences.

The apparatus 280 typically comprises the camera

282, a monitor 284 and a video recorder 286. The cameraman utilizes the camera 282 initially to film the primary image sequence which iε diεplayed, during filming, on the monitor 284. At the εame time, in accordance with the preεent invention, the filmed scene is recorded, typically at a low resolution, on video recorder 286.

When the cameraman turns to film one of the secondary image sequenceε, whether that be in a similar location or in a completely different one, with the same camera or a different one, he displays the recorded scene from the video recorder 286 on the monitor 284. Paying attention to the camera motion therein recorded, the cameraman attempts the match the camera motion while filming the scenes for the secondary image sequence. He repeats the process for the second secondary image sequence.

While the invention haε been deεcribed with respect to a limited number of embodiments, it will be appreciated that many variations, modifications and other applications of the invention may be made.

Claims

AMENDED CLAIMS

[received by the International Bureau on 26 October 1995 (26.10.95); original claims 2 and 3 cancelled; original claims 1,4,10,14,15,22 and 23 amended; remaining claims unchanged (8 pages)]

1. A visual display system for displaying three dimensional visual images, comprising:

(a) a primary visual image source storing primary visual images showing generally real-life scenes;

(b) a secondary visual image source storing secondary visual images;

(c) a main screen for displaying a primary visual image of a first scene from said primary image source; and

(d) at least two peripheral screens inclined with respect to said main screen for displaying at least two secondary visual images from said secondary visual image source for transforming said primary visual image into a three dimensional visual image, wherein said at leaεt two secondary visual images are processed from images of at least second and third scenes which differ in time or significantly in space from said first scene; and wherein each of said at least two secondary visual images generally matches said primary visual image so as to create the appearance of display of a single scene.

4. The system as in claim 1, wherein each of said secondary visual images is a real-life image taken at a different time or different places than said primary visual images.

5. The system as in claim 1, wherein each of said secondary visual images is an animated image. 6. The system as in claim 1, wherein each of said secondary visual images is displayed at a lower resolution than said primary visual image.

7. The system as in claim 1, wherein each of said secondary visual images is a moving picture with respect to said primary visual image, wherein said primary visual image is a still picture.

8. The system as in claim 1, wherein each of said secondary visual images is a moving picture with respect to said primary visual image, wherein said primary visual image is a moving picture.

9. The system as in claim 1, wherein each of said screens is any one of the following group: a moving picture theater screen, a television screen, a computer monitor and a LCD screen.

10. The system as in claim 1, wherein each of said at least two peripheral screens are trapezoid shaped.

11. The system as in claim 1, further comprising projection means for projecting recordings of said visual images onto said screens.

12. The system as in claim 11, wherein said projection means includes any one of the following group: a photographic film projector, a CRT tube and a video projector.

13. The system as in claim 1, further comprising a plurality of speakers for playing back a soundtrack for accompanying the visual images displayed on the system. 14. A syεtem for producing recordingε of visual images for visualization on a visual display system displaying three dimensional visual images, comprising: (a) primary visual image source of primary isual images viewing first scenes of real- life; ( a secondary visual image source of secondary visual images, each of said secondary visual images viewing scenes which differ from said first scenes wherein said secondary visual images are processed from images of scenes which differ in time or significantly in space from said scenes of said primary visual images; and (c) selection means for selecting at least two secondary visual images from said secondary visual image source such that said at least two secondary visual imaget are compatible with a primary visual image; and (d) secondary visual image matching means for generally matching said selected secondary visual images with said primary visual image so as to create the appearance of a single scene.

15. The system as in claim 14, wherein said primary visual image source is provided by any one of the following group: a television camera and a photographic film camera.

16. The system as in claim 14, wherein said secondary visual image source is provided by any one of the following group: a television camera, a photographic film camera, a computer database and computer graphic sub-routines. 17. The system aε in claim 14, f rther compriεing output means for outputting a recording of each of said secondary visual images.

18. The system as in claim 14, further comprising soundtrack means for preparing a soundtrack to accompany said visual images.

19. The system as in claim 14 and wherein said secondary visual image matching means comprises at least one of:

(a) a multi-sealer for scaling at least portions said selected secondary visual images to match the scale of said primary visual image; (b) a rate matcher for matching the timing of a series of said selected secondary visual images to the timing of a corresponding series of primary visual images;

(c) a motion matcher for matching the camera motion in said a series of said selected secondary visual images to the camera motion of a series of primary visual images; and

(d) an edge matcher for blending each edge of said primary image with the one of said secondary images adjacent thereto; and

(e) a color matcher for matching the color quality of said selected secondary images to the color quality of said primary visual image.

20. The system according to claim 19 and wherein said rate matcher comprises means for synthesizing a sequence of visual images to become part of said selected secondary visual images in accordance with optical flow of a sequence of primary visual images. 21. The system according to claim 19 and wherein said motion matcher comprises:

(a) a secondary motion stabilizer for generally removing a first camera motion from said series of secondary visual images, thereby to create a series of stabilized secondary visual images;

(b) a primary motion detector for determining a second camera motion in said series of primary visual images; and

(c) a motion creator for transforming said series of stabilized secondary visual images to have said second camera motion.

22. A method for displaying three dimensional visual images, comprising the steps of:

(a) storing primary visual images showing generally real-life scenes in a primary image source;

(b) storing secondary visual images in a secondary image source;

(c) displaying a primary visual image of a first scene from said primary image source; and

(d) displaying at least two secondary visual images from said secondary visual image source on at least two peripheral screens inclined with respect to said main screen thereby to transform said primary two dimensional visual image into a three dimensional visual image, wherein said at least two secondary visual images are processed from images of at least second and third scenes which differ in time or significantly in space from said first scene; and wherein each of said at least two secondary visual images matches said primary visual image so as to create the appearance of displaying a single scene.

23. A method for producing recordings of visual images for visualization on a visual display system displaying three dimensional visual images, comprising the steps of:

(a) providing a primary visual image source of primary visual images viewing first scenes of real-life;

(b) providing a secondary visual image source of secondary visual images, each of said secondary visual images viewing scenes which differ from said first scenes, wherein said secondary visual images are processed from images of scenes which differ in time or significantly in space from said scenes of said primary visual images; and

(c) selecting at least two secondary visual images from the secondary visual image source such that the at least two secondary visual images are compatible with a primary visual image; and

(d) generally matching the at least two secondary visual images with the primary visual image so as to create the appearance of a single scene.

24. A method as in claim 23 and wherein said step of matching comprises at least one of the steps of:

(a) scaling portions of said selected secondary visual images to match the scale of said primary visual image;

(b) flow rate matching the timing of a series of said selected secondary visual images to the timing of a corresponding series of primary visual images;

(c) motion matching the camera motion in said a series of said selected secondary visual images to the camera motion of a series of primary visual images;

(d) blending each edge of said primary image with the one of said secondary images adjacent thereto; and (e) color matching the color quality of said selected secondary images ~ the color quality of said primary vi =*1 image.

25. A method according to claim 24 and wherein said step of rate matching comprises the step of synthesizing a sequence of visual images to become part of said selected secondar/ visual images in accordance with optical flow of a sequence of primary visual images.

26. A method according to claim 24 and wherein said step of motion matching comprises the steps of:

(a) generally removing a first camera motion from said series of secondary visual images, thereby to create a series of stabilized secondary visual images;

(b) determining a second camera motion in said series of primary visual images; and

(c) transforming said series of stabilized secondary visual images to have said second camera motion.

27. A method of generally matching the camera motion of a first camera with a second camera, the method comprising the steps of: (a) recording the sequence of visual images produced by said first camera on a recording device; . (b) at the time of filming with said second camera, displaying said recorded sequence of visual images; and (c) filming with said second camera while attempting to match the camera movement displayed in said recorded sequence of visual images.

28. A method according to claim 27 and wherein said first camera and said second camera are the same camera operated at different times.

29. A method according to claim 27 and wherein said first and second cameras view different scenes.