JP2009500878A - Depth illusion digital imaging - Google Patents

Abstract

Generating a sequence of display images of the scene; setting the speed of features in the scene to generate non-perspective distortion of the scene in the display image; and sequentially displaying the display image A method for generating a visual representation of a scene having a depth illusion.
[Selection] Figure 2

Description

(Related application)
This application claims the benefit under Section 119 (e) of US Patent Act No. 60/670087 of US Provisional Patent Application No. 60/670087 filed April 11, 2005, referred to as “Depth Illusion Digital Imaging”, and its disclosure. Is incorporated herein by reference.

(Field of Invention)
The present invention relates to a method for generating a visual representation of a scene that provides depth perception, and a method for controlling depth perception.

  Human stereo vision relies on depth cues, referred to as primary or physiological, and secondary or psychological, sometimes pictorial, to read 3D information from scenes received at the 2D surface of the eye or retina. Some physiological cues, such as eye accommodation or the amount of lens shape change that the eye brings to focus on an object, are provided by monocular vision, i.e. by a single eye, but many physiological cues A depth cue is a binocular cue that is a function of a person with two eyes. Binocular cues include vergence, the angle at which the eye rotates to focus on the object, and retinal distortion. Retinal distortion refers to the difference between the images of the same scene that appear in them due to the difference in the position of the left and right eyes.

  It is not generally necessary for both eyes to provide psychological depth cues, and each eye working alone can provide these cues. Psychological depth cues are used to describe the depth perception that people experience when looking at photographs and paintings, relative size, linear perspective, object height above line of sight, intervention Includes objects, shading, shadows, relative brightness, color (color stereoscopic), and atmospheric attenuation. These psychological cues are widely used in terrain representation to provide a sense of depth. Motor cues are often classified as psychological cues. However, because they cause changes in the relative displacement of the retinal image of objects at different distances, motion cues actually produce physiological responses that convey three-dimensional information. Many psychological cues can be combined with physiological cues to produce enhanced three-dimensional effects.

  There are a variety of three-dimensional stereoscopic display techniques for conveying either physiological or psychological cues that provide depth information from, for example, two-dimensional paper, video, and computer system images. As described above, printed representations generally use psychological cues. However, anaglyph prints are based on splitting the color spectrum into red and green / blue components and masking the image so that they are “routed” to the appropriate eye to represent a three-dimensional scene. Depends on the physical separation of images. The film product includes a polarizing film for separation, and a color image can be printed. In order to transmit a separate image, the film still requires glasses that generally have horizontal polarization for one eye and vertical polarization for the other. Video and computer systems use both anaglyphs and polarization, and these methods are generally implemented to achieve left / right separation in softcopy photogrammetry systems. These systems also make it possible to enhance the separation of stereoscopic images by motion and other special techniques along with technologies such as lenticular and holographic displays.

  The 3D display autostereoscopic method is based on a wide variety of concepts and methods, and it is not necessary to wear 3D glasses to achieve the synthesis of depth information in the human eye-brain system. An autostereoscopic display relies on left and right images or a stereoscopic view of alternating image pairs and includes not only lenticular displays and parallax barrier displays, but also displays based on motion parallax.

  Papers "Mapping Perceptive Depth To Regions Of Interest In Stereoscopic Images", Nick Holliman, Stereoscopic Displays and Vital EIS, PI I & T 5291, 2004, ISBN 0-8194-5194 makes the ROI in the scene depth into a stereoscopic 3D display so that the depth perceived in the defined region of interest (ROI) is improved compared to other areas in the scene. Describes how to map. For a given dynamic depth range in a stereoscopic display, the ROI is mapped to a display ratio that is larger than other areas of the scene.

  U.S. Pat. Nos. 6,320,036, 6,665,003, 6,695,109, 6,683,677, and Robinson et al., 6,695,575, all of Peleg et al. A method for generating left and right mosaic images of a scene suitable for display as a pair is described, the disclosures of which are incorporated herein by reference. US patent application Ser. No. 09/861859 to Peleg et al. Describes an alternative method of generating a depth sensation so that images featuring different disparities are displayed sufficiently fast, so that both eyes see the same image substantially simultaneously. The disclosure of which is incorporated herein by reference. The method does not use left-right image separation or masking.

  US Patent Application No. 60/661907, filed March 16, 2005, referred to as “Depth Illusion Digital Imaging” by some of the same inventors as the present application was sequentially displayed to the observer without left-right masking. Occasionally, a method and apparatus for generating a plurality of “display images” of a scene that achieves display of the scene with a desired depth perception is described, the disclosure of which is incorporated herein by reference.

  One aspect of some embodiments of the present invention is a plurality of scenes that achieve display of a scene with a desired depth perception when displayed sequentially and sufficiently fast to an observer without left-right masking. To provide a method and apparatus for generating a “display image”.

  One aspect of some embodiments of the invention relates to providing a method for controlling depth perception of a displayed scene. In some embodiments of the invention, the method is used to enhance depth perception. In some embodiments of the invention, the method is used to reduce depth perception.

  An aspect of some embodiments of the present invention is to control the depth of a feature in a scene, or the focal length at which the feature is imaged, regardless of the speed of other features in the scene or the focal length of the imaging device. It relates to generating relative motion of features to achieve perceptual enhancement. In some embodiments of the invention, feature speed control is used to reduce depth perception.

  The inventors of the present invention perceive a depth-enhanced image ("3D" image) by an observer when a scene containing a specific depth cue is presented on a normal two-dimensional display (such as a computer display or television screen). I found that I can do it.

  The inventors further have two parallax images that the nature of the depth cues that appear to induce or enhance depth perception is received from the left and right eyes when viewing a true three-dimensional scene ("stereoscopic") We found that it is preferable to be related to the nature of the brain integration. In particular, the images that each eye sees separately are parallaxally displaced from each other, but both are normal “perspective” views that are slightly different from those captured by the optical camera. However, stereoscopic vision forms a new single image that the brain “sees”, which is quite different from the normal perspective view that each eye sees. In fact, the single integrated image perceived by the brain is considerably distorted in several important respects compared to the normal perspective view seen by each eye.

  For example, when viewed with a single eye, parallel lines such as railroad tracks that run from the observer to the horizon appear to converge at the horizon. However, when viewed stereoscopically, the brain perceives the tracks as being “more parallel”. That is, as compared with a perspective view that can be seen only by one of the eyes, the distance between the tracks appears to be narrower in the foreground and wider in the distance, so that the convergence is perceived to be small. In other words, stereoscopic viewing produces contraction of nearby objects and enlargement of distant objects. Here, “near” and “far” are relative to a person's focal plane.

  The effects on the 3D perception of near and far object size changes are relatively limited, but the images in the image sequence so that the far object moves faster compared to the relative speed of the near object. The resulting scene can be used to induce depth perception enhancement images in motion sequences (ie including depth cues) if the object relationships are distorted to resemble stereoscopic viewing.

  A “motion sequence” is a series of movies or video sequences, such as a movie or video sequence, where the angle of view of the scene changes gradually (ie, the “observer” or “camera” moves relative to at least a part of the scene). That is, the scene with the depth clue described above is presented on the still frame. In embodiments of the present invention, the location where the scene is imaged can exist along any of a number of differently shaped curves. The position can move laterally with respect to the scene and / or perform zoom-in and / or zoom-out motions on the scene. For example, in some embodiments of the invention, the position exists along a three-dimensional non-planar curve, which is required to adequately describe a function of three spatial coordinates for the scene. In some embodiments of the invention, the position exists along a plane curve. In some embodiments of the invention, the position exists along a straight line.

  Note that the terms used herein to describe actual features of an actual camera, such as the optical center, optical axis, and focal plane, refer to corresponding features of a computer graphics camera as appropriate.

  Without being bound by any theory, the combination of depth cues and relative motion creates images that are sufficiently reminiscent of stereoscopic induction images seen by the brain when viewing true three-dimensional images, and the brain is a depth cues. It seems that a certain scene is recognized as a true three-dimensional image although it is presented on an ordinary two-dimensional display.

  Needless to say, it is impossible to capture an image with the depth clue by using an optical camera to capture a scene in a manner that embodies the necessary perspective distortion described above. However, in accordance with the present invention, such depth cues can be created artificially, either manually or preferably using a computer. When using a computer to form such an image, it is possible to use an algorithm that automatically delegates a specific image plane and a specific viewing angle to a specific process (such as enlargement or reduction).

  For example, an object at the desired focal length (or center of attention) is considered to be seen through a virtual lens of a specific focal length, while closer objects are seen through a lens with progressively shorter focal lengths While considering an object to be seen through a lens with progressively longer focal lengths, nearby objects can be smaller and farther objects can be larger, not only such focal length and magnification ratio, but also The enlargement aspect ratio is also determined in advance to be adjustable. In addition, the focal length of the virtual lens as a function of the distance from the camera (or the distance before or after the focal plane) so that the image around the field of view does not necessarily have the same degree of distortion as the image at the center of the field of view. It is also possible and desirable to vary not only as a function of viewing angle. Note that this change in focal length distorts the relative motion of near and far objects and thus distorts the image between frames of the motion sequence.

  In another embodiment of the present invention, an object at a desired focal length (or center of attention) is considered to be seen through a virtual lens at a specific focal length, while closer objects are progressively shorter in focal length. The size of objects in each image plane can be scaled to their exact proportions or dimensions relative to their distance from the camera, assuming that the farther objects are seen through the lens as they are seen through the lens and the focal length gradually increases. Can be maintained. Again, the focal length of the virtual lens is set to the distance from the camera (or before or after the focal plane) so that the image in the peripheral area of the field of view is not necessarily viewed through a lens with the same focal length as the image in the center of the field of view. It is possible to change not only as a function of distance) but also as a function of viewing angle.

  In each of these above embodiments, the focal length difference changes the relative motion of a near or far object relative to the camera motion rate. In the embodiments described above, the virtual lenses of different focal lengths can each be from different cameras arranged coaxially with respect to the optical axis of the lens, preferably at the same point or from any point in the scene Can be co-located at the same distance. In other words, multiple virtual cameras can be viewed as a single camera with multiple lenses, each with a different focal length and each associated with a specified plane of the scene. This method is particularly suitable for automatic or semi-automatic generation of computer images, where each camera lens automatically captures the appropriate plane of the virtual scene through any camera movement required. In some embodiments of the invention, lenses of various focal lengths (and optionally imaging surfaces) are spaced along the same optical axis. In some embodiments of the invention, the focal length of the virtual lens varies as a function of their position relative to the scene or to one or more particular features in the scene.

  The method of the present invention is particularly useful for computer “manipulation” images originally captured by cameras, or images that are completely computer generated, such as animated movies or computer games, or combinations thereof.

  In a second type of exemplary manipulation of images according to aspects of the present invention, the relative motion of the elements in the scene varies as a function of their depth. In one embodiment of the present invention, a video sequence of a computer-generated (or at least partially computer-generated or computer-operated) scene is created and the camera is in any orientation relative to at least one element in the scene. Or move in the combined direction or in other motion directions (horizontal, vertical, diagonal, rotation, zoom, or other). The relative motion of the other elements in the scene is determined when the objects in the scene are in a fixed relative position relative to each other, regardless of whether at least some of these other elements are closer or farther from the camera than the first element. It is done to move at a higher relative motion rate determined from their respective distances from one element.

  As an illustration, assume a computer-generated scene that includes a relatively stationary child, a background tree, and a foreground ball, and that the child's parent is shooting the scene with a video camera from a moving vehicle. As the vehicle moves, the parent aims the camera so that the child is maintained in the center of the camera's field of view, thus obtaining a series of camera images of the scene with the child substantially in the center of the image. In each image, the child is substantially located in the center of the scene, but due to motion parallax, the background tree and foreground ball move in the opposite direction relative to the child when viewing the video sequence of images. Motion parallax gives a small amount of depth to a video sequence. However, since the image of the video sequence is a two-dimensional image, a binocular cue that gives a sense of depth when viewing an actual scene is missing from the video sequence. As a result, the video sequence looks flat compared to the actual scene and tends to lose “liveness”.

  Embodiments of the present invention increase the speed of trees and balls (and optionally other features at various distances) to the child to provide depth sensation and realism to the displayed video sequence. . The increasing speed of the displayed image is greater than the speed of the ball and tree corresponding to and expected from their distance to the child. The increased relative speed enhances the depth sensation and increases the spatial separation between the child, the tree and the ball, thus providing a lively 3D depth in the image.

  Note that in the unenhanced video sequence acquired by the parent, the ball, tree, and child are all still stationary and their real-world relative positions have not changed. The relative movement of the ball and tree in the video sequence image is only the result of his or her movement while the parent acquires the video sequence. In contrast, a scene depth augmented image according to an embodiment of the present invention corresponds to a “real world” image in which trees and balls move relative to a child. Thus, when viewing an image of the acquired virtual scene from above, the relative position of the object moves during the video sequence (eg, between images).

  In the normal perspective view of the scene, regardless of whether it was filmed by the camera or generated by the computer, the relative position of the non-moving elements in the scene does not change with camera motion, so the radius of the circle defined by these elements is It is understood that it does not change regardless of such camera movement. While not wishing to be bound by any particular example or theory, the inventors create the enhanced perception of depth perception of the video sequence in accordance with the present invention as the background or foreground (or the foreground). ) Think of this exaggerated relative movement of elements.

  Although the above example includes only three objects, the scope of the present invention contemplates an unlimited number of objects or elements. For example, in the example described above, the same rules of relative motion can be applied to each leaf of the lawn grass, or to any or all of the other objects or elements in any scene. In fact, for every pixel in the scene, a specific relative motion rate can be specified as a function of the nominal distance from the camera or from another reference point. Furthermore, the relative movement need not be linear throughout the field of view (eg, it can be more pronounced at the center of the frame and less noticeable in the surrounding area, or vice versa).

  In addition, the depth perception illusions contemplated by the present invention are enhanced when there are a significant number of elements at various depths of the scene, but both background and foreground motions need not necessarily be present. It was revealed. The relative motion of at least some of the moving elements in the scene preferably follows the radius change rules described above.

  One embodiment of the invention further contemplates controlling the relative movement of each element and / or object in the scene automatically or semi-automatically by a computer program or algorithm.

  It is noted that changes in the speed of features in a scene according to embodiments of the present invention are not necessarily accompanied by changes in the distance between features in the scene or changes in their scale. Changes in speed can be decoupled from changes in distance and scale, but can also be performed in combination with or in concert with such changes.

  In general, the impression of the depth difference between features is increased or decreased, respectively, by increasing or decreasing the speed of the farther features in the motion sequence relative to the speed corresponding to that distance from the closer features in the sequence.

  In some embodiments of the present invention, an unexpected change in speed results in temporally distorted scene distortions and an appearance measure of plasticity that cannot be reproduced by conventional perspective imaging. Is configured to give to the scene. For example, normal perspective vision and perspective projection map straight lines in a three-dimensional scene to straight lines. On the other hand, speed changes according to embodiments of the present invention often map straight lines to curves, which may change over time during the presentation of a series of display images. The inventors have found that such a challenge to the eye-brain visual system can be effective in stimulating the brain to generate a relatively strong depth sensation enhancement.

  Any of the various methods according to embodiments of the present invention as described below can be used to independently adjust the speed of features in the scene and generate a motion sequence of the displayed images of the scene. . In some embodiments, the scene can be divided into layers that are each placed at different depths in the scene.

  The layer is not necessarily flat, for example, in some embodiments of the invention, the layer may be curved and have regions of different thickness. In some embodiments of the invention, the layer is spherical or has a spherical region. Optionally, the layer is cylindrical or has a cylindrical region. Optionally, the layer is planar.

  At least one layer has a speed such that stationary features placed in the layer move relative to other stationary features in the scene to produce a desired depth sensation in the scene according to embodiments of the invention. Or viewed through a lens with a specific focal length. At different times in the sequence, i.e. when the displayed image is displayed, the features of at least one layer are displaced from other features in the scene as a function of its assigned speed or the focal length of the lens at which it is viewed. As such, the layers are processed using, for example, a computer to project the scene to the motion sequence display.

  The thickness of the layer into which the scene is divided according to the embodiment of the present invention can be different. Optionally, each layer has a uniform thickness. In some embodiments of the invention, the thickness of the layers can be different. For example, in regions where it is desirable for the velocity to be a relatively smooth or fine function of depth, the layer is optionally relatively thin. In other areas where the speed can be a rough function of depth, the layer is optionally thick. In some embodiments of the invention, the layer approaches zero thickness and the speed of the feature in the scene or the focal length at which it is seen is a smooth function of depth.

  In an embodiment of the invention, to increase the speed of a feature in a display image of a scene, the distance at which the feature is imaged for the display image is decreased and / or the focal distance at which the feature is imaged is increased. The To reduce the apparent speed of features in a scene, the distance at which features are imaged is increased and / or the focal length at which features are imaged is reduced. By increasing the apparent speed for closer features of distant objects in the scene, the apparent difference in parallax and depth between features is increased. Similarly, by reducing the apparent speed of distant features to closer features in the scene, the parallax and depth apparent differences between features are reduced.

  It is believed that the more layers or focal lengths used (ie, the more changes in the scene), the greater the three-dimensional effect and the closer it will be. Therefore, within the possible range, the number of layers should be increased while taking into account limitations such as calculation time, and the change in focal length should be kept as long as possible.

  One aspect of some embodiments of the invention relates to generating depth cues in a displayed image of a scene by changing the scale of the features in the scene.

  The inventors have found that it can be advantageous to accompany changes in the characteristics of scene features that attempt to generate depth cues by changing the scale of the features to enhance depth perception. In particular, the inventors have found that changes in the scale of features that appear counterintuitive are advantageous in supporting depth cues. For example, features in the scene image are proportional to their depth, distant features are small and close features are large, but in order to facilitate depth perception, the inventors consider features to be proportional to their intended depth. We have found that it can be advantageous to move away from scaling.

  In particular, in facilitating a deeper sense of depth in the scene, in embodiments of the present invention, it may be advantageous to scale features far from the center of the scene more than expected from their depth. Similarly, it may be advantageous to scale closer features in the scene to a smaller scale proportional to their intended depth. We feel that this effect is associated with a perceived magnitude homeostatic phenomenon where objects of known size are perceived to have a constant size regardless of their distance. It is done.

  For example, when the first person walks away from the second person, even though the image of the first person in the retina of the second person's eye actually becomes smaller as the first person moves away from the second person, Two people generally do not have the impression that the first person is smaller. Similarly, when viewed along a railway line that moves away, the lines are parallel even if the distance between the lines of the retinal image near the line is larger than the distance between the lines of the retinal image near the line. Perceived as. In other words, it is not perceived that the distance between the tracks decreases with distance. The basis for the scaling effect according to embodiments of the present invention in the generation and / or support of depth cues is not limited by any particular theory or interpretation, but the effect of scaling is attributed to coexisting with magnitude constancy. Seems to be able to.

  A method according to embodiments of the present invention optionally uses a floppy disk, a CD, a flash memory used to generate and / or display a display image that achieves enhanced depth perception. Or any of a variety of computer-accessible storage media, such as ASICs. The methods and apparatus according to embodiments of the present invention can be used to enhance depth perception in many different different applications and two-dimensional visual representation formats. For example, the method can be used to enhance the depth perception of computer game images, animated movies, visual advertising, and video training.

Accordingly, in an embodiment of the present invention, a method for generating a visual display of a scene having a desired depth illusion, comprising:
Generating a sequence of display images of the scene;
Setting the speed of features in the scene and generating non-perspective distortion of the scene in the display image;
Sequentially displaying display images;
A method comprising:

In an embodiment of the present invention, there is further provided a method for generating a visual display of a scene having an enhanced depth illusion,
Acquiring each image for a different location in relation to at least some features in the scene and generating a sequence of images of the scene;
Sequentially displaying display images;
Including
Generating the sequence comprises:
Systematically setting the position of features in the scene for each image, generating non-perspective distortion of the image, and providing enhanced depth perception to the sequence of images;
A method comprising the steps of:

In an embodiment of the present invention, there is further provided a method for generating a visual display of a scene having an enhanced depth illusion,
Acquiring each image by a virtual camera from a different location in relation to at least some features in the scene and generating a sequence of display images of the scene;
Sequentially displaying display images;
Including
Generating the sequence includes providing at least one feature in the scene with a speed to generate relative motion between features that are nominally stationary relative to each other in the scene; The speed provided for a particular feature in the feature is a function of the depth of the particular feature in the scene relative to the camera.
Provide a method.

  Optionally, the method comprises the step of generating a visual display according to claim 3, wherein the virtual camera position is along a non-planar three-dimensional trajectory. Alternatively, the virtual camera position exists along a planar trajectory. Optionally, the virtual camera position exists along a linear trajectory.

  Optionally, some of the locations are closer to the scene than others. Optionally, at least two positions are displaced relative to each other laterally with respect to the scene.

  In embodiments of the invention, features at different depths relative to the camera are imaged at different focal lengths.

  In an embodiment of the invention, the scene is divided into layers. Optionally, the same speed is set for features in the same layer. Optionally, different speeds are set for different layers.

  In an embodiment of the invention, at least one position of the virtual camera, the camera images different layers from different positions along the camera's optical axis.

  In an embodiment of the present invention, features close to the camera are provided with a velocity in a first direction, features with an intermediate distance are stationary, and features with a greater distance are in a second direction substantially opposite to the first direction. Speed is provided. Alternatively, features close to the camera are provided with a first speed in a first direction, and features with an intermediate distance are provided with a second speed that is less than the first speed in the first direction, for very large distances. The feature is stationary. Alternatively, features near the camera are stationary, features at intermediate distance are provided with a first velocity in a first direction, and features at a far distance have a second velocity greater than the first velocity in the first direction. Provided. Optionally, the first direction is substantially parallel to the direction of motion between the different positions.

  In an embodiment of the invention, the moving feature is provided with a change in position or an additional speed consistent with the change provided to the correspondingly arranged stationary object.

In an embodiment of the present invention, there is further provided a method for generating a visual display of a scene having an enhanced depth illusion,
Acquiring each image by a virtual camera from a different location in relation to at least some scene features to generate a sequence of display images of the scene;
Sequentially displaying display images;
Including
The focal length at which the virtual camera images a particular feature in the scene is a function of the distance of the particular feature from the camera,
Provide a method.

  Optionally, some features far from the camera are imaged at a focal length that is greater than the focal length used to image features that are relatively close to the camera.

  Optionally, the method includes displaying the display image so that it can be viewed by both eyes of the observer.

  Embodiments of the present invention further provide a visual display image generated according to any of the preceding claims.

  The embodiment of the present invention further provides a sequence of visual display images including the visual display image according to the present invention.

  Embodiments of the present invention further provide a sequence of visual images in which the imaging position moves between images and a stationary object moves relative to each other according to the distance between the object and the imaging position. To do.

  Embodiments of the present invention further provide a computer-accessible storage medium that encodes a method or a sequence of images or visual images according to the present invention.

(Simple description of the drawing)
Non-limiting examples of embodiments of the present invention are described below with reference to the figures attached to this document listed after this paragraph. In the figures, identical structures, elements or portions that appear in more than one figure are generally numbered the same in all the figures in which they appear. The dimensions of the components and features shown in the figures have been selected for the convenience and clarity of presentation and are not necessarily to scale.
FIG. 1A schematically illustrates a scene including the ball, girl, and tree discussed in the overview that is imaged to provide a motion sequence of displayed images according to the prior art.
FIG. 1B schematically illustrates the scene shown in FIG. 1A that is imaged to provide a motion sequence in accordance with an embodiment of the present invention.
FIG. 1C schematically shows a sequence of display images generated according to the prior art as shown in FIG. 1A and generated according to the present invention as shown in FIG. 1B.
FIG. 2 schematically illustrates dividing a scene into layers and assigning speeds to the layers according to an embodiment of the present invention.
3A-3E schematically illustrate distortion of a version of a scene used to generate a display image of the scene according to an embodiment of the present invention.
FIG. 4 shows a top view of the distortion shown in FIGS. 3A-3E, according to an embodiment of the present invention.
FIG. 5A schematically illustrates generating a display image by capturing different features of the scene at different focal lengths.
FIG. 5B schematically shows a sequence of display images generated according to the prior art as shown in FIG. 1A and generated according to the present invention as shown in FIG. 5A.

  FIG. 1A schematically illustrates a scene 500 including a tree 501, a girl 502 in front of a tree, and a ball 503 in front of a girl, imaged by a moving camera represented by an hourglass icon 520, according to the prior art. The hourglass icon constriction 521 represents the optical center of the camera 520, and the broken line 522 represents the optical axis of the camera. Optionally, the camera 520 moves at a constant speed along the linear trajectory 530 in the direction indicated by the block arrow 531 and acquires camera images of the scene 500 at regular intervals.

  As an example, the camera 520 is oriented such that its optical axis intersects the girl 502 at each position along the trajectory 530 that captures the image of the scene 500. The camera 520 is schematically shown in five positions P1-P5 where it obtains corresponding display images IM1-IM5 of the scene 500 along a trajectory 530 for use in providing a motion sequence of the images of the scene. Indicated. The scene 500 can be an actual scene, and the camera 520 can be, for example, an actual camera held by a girl's parent in a scene in a car. However, for convenience of presentation, it is assumed that image 500 is a composite image and camera 520 is a computer graphics camera that acquires an image of scene 500.

Images IM <b> 1 to IM <b> 5 are illustrated in the orientation in which they are acquired by the camera 520 from the back, upside down and in a state where the left and right are inverted. The images IM1 ^{* to} IM5 ^* are images IM1 to IM5 that are viewed by the observer and are reversed right and left with respect to the directions of the images IM1 to IM5 and correctly oriented up and down. For a part of the images IM1 to IM5, a light beam 560 from the center point 561 of the tree 501 that passes through the optical center 521 of the camera 520 and enters the image indicates the position of the center point in the image. Similarly, the light ray 570 indicates the position of the image of the center point 571 of the ball 503 with respect to a part of the images IM1 to IM5. The center point 580 of the girl 502 is projected at a position where the image intersects the optical axis 522 in all the images IM1 to IM5.

  For convenience of presentation, at position P 3, the optical axis 522 passes through the center points 561 and 580 of the tree 501 and the girl 502 and is directly above the ball center 571 on the outer periphery of the ball 503. Pass the “top” point 572. Positions P1 and P2 are mirror images of positions P4 and P5, respectively, in a plane perpendicular to the trajectory 530 passing through the optical center 521 of the camera 520 at the position P3. Since the girl, tree, and ball are aligned one after another at the position P3, the image of the ball, girl, and tree is layered on top of each other at the position P3 where the image IM3 is acquired. In the images IM1 to IM5, due to motion parallax, the tree 501 gradually moves from the left side of the girl 502 to the right side of the girl, and the ball 503 gradually moves from the right side to the left side of the girl. The amount that the ball and tree move relative to the girl can provide a small amount of depth perception when the images IM1-IM5 are shown sequentially.

  FIG. 1B schematically illustrates a scene 500 that is imaged to provide an image display for a motion sequence of the scene 500 that generates an enhanced depth sensation in accordance with the present invention.

As in FIG. 1A, in FIG. 1B, the camera 520 moves along the trajectory 530 and acquires an image of the scene 500 at positions P1 to P5 where the optical axis 522 intersects the girl 502, respectively. However, unlike FIG. 1A, the tree 501 and the ball 502 are not stationary during imaging by the camera 520. Instead, the tree 501 optionally moves from left to right as shown by the block arrow 590 during imaging, and optionally the ball 503 moves from right to left during imaging as shown by the block arrow 591. Move to. At positions P1 to P5 of the camera 520, the tree 501 is imaged at corresponding positions T1 to T5, and the ball 503 is imaged at positions B1 to B5, respectively. The figures acquired by the camera 520 at the positions P1 to P5 are displayed as IM11 to IM15, and are displayed as IM11 ^{* to} IM15 ^* in the display orientation state.

  In some embodiments, the tree 501 and the ball 503 move at a constant speed, preferably in a direction parallel to the movement of the camera 520. Optionally, they move at a varying speed. As an example, in FIG. 1B, the tree 501 moves at a constant speed, and the positions T1 to T5 are equally spaced. The ball 503 moves at a constant speed between the positions B2 to B4. However, between positions B1 and B2 and between positions B4 and B5, the ball 503 optionally moves at a higher speed than the speed at which it moves between positions B2 and B4.

  Increasing the velocity of the ball 503 between positions B1 and B2 and between B4 and B5 enhances the relative motion between the girl and the ball in an image acquired at a position relatively far from P3. At a position far from P3, the angle at which the light beam from the ball intersects the optical axis 522 is relatively large, so that when the ball 503 moves at the same given speed, between the ball and the girl in the perspective images IM11 and IM15. The relative movement is substantially reduced compared to the case of the images IM12 to IM14.

Due to the “artificial” movement of the trees 501 and balls 503, in the sequence of images IM11 ^{* to} IM15 ^* , the trees and balls are relative to the girl and to each other than in the images I1 ^{* to} I5 ^* . Move a fairly large distance. For ease of comparison, images IM1 ^{* to} IM5 ^* and images IM11 ^{* to} IM15 ^* acquired in FIG. 1A are shown in FIG. 1C. Image IM118～IM15 ^* results of a large relative movement of the tree and the ball in the exercise sequence including image IM11 ^* ~IM15 ^* according to the embodiment of the present invention, as compared to corresponding movement sequence including the image ^{I1 * ~I5} * Show enhanced depth perception.

  The movement of the camera 520 along the trajectory 530 in FIGS. 1A and 1B is relative to the scene 500, by keeping the camera stationary and moving the scene, not to mention the images acquired at positions P1-P5 It is noted that it can be reproduced. In the case of the “non-moving camera”, the effect of the speed change of the feature in the scene 500 according to the present invention as shown in FIG. 1B can be brought about by the vector addition of speed. It should be noted that while the image according to FIG. 1A can be acquired by an actual camera, the image of FIG. 1B cannot generally be acquired as such. However, the method of the present invention is suitable for both composite images and animations of actual objects. They can also be used in a limited way when manipulating real images.

  In general, a scene such as scene 500 has more features than just a tree, a girl, and a ball, usually grass, stones, shrubs, swings hung on a tree, and perhaps stationary during the creation of this story. Will include dogs that are most certain to be difficult to keep. In order to maintain the realism of the scene 500, it is generally advantageous to adjust the speed of features (not shown) in the scene in addition to adjusting the speed of the trees and balls.

  In an embodiment of the invention, to adjust the speed of additional features in the scene 500, the scene is divided into layers that are each optionally parallel to the trajectory and located at different distances from the trajectory 530. FIG. 2 schematically shows a scene 500 divided into a plurality of layers 600 according to an embodiment of the present invention. Optionally, layer 600 is planar and has the same uniform thickness.

  In an embodiment of the present invention, each layer 600 is provided with a speed so that relatively smooth changes in the speed of features in the scene 500 exist as a function of their respective depths. The layers labeled 601, 602, and 603 include a tree 501, a girl 502, and a ball 503, respectively. An arrow 610 arranged along the line 611 and facing the layer 600 schematically illustrates the rate assigned to the layer in accordance with an embodiment of the present invention. The rate assigned to a given layer 600 is indicated by the arrow 610 facing the layer. The direction of the arrow schematically indicates the direction of the speed, and the length of the arrow schematically indicates the magnitude thereof. Layers 601 and 603, including tree 501 and ball 503, are assigned speeds in opposite directions and have maximum, not necessarily equal speeds in their respective directions. Since the camera 520 is assumed to move along the trajectory 530 while adjusting its orientation so that the optical axis 522 is directed at the girl, the layer 602 that includes the girl 502 is assigned zero velocity.

The speed 610 can be determined in various ways in embodiments of the present invention. For example, if Z is the distance from the trajectory 530 of a given layer 600 and Z ₀ is the distance from the trajectory of girl 502, then the magnitude of velocity 610 is proportional to the power of (Z−Z ₀ ), For example, it can be (Z−Z ₀ ) ^1/2 or (Z−Z ₀ ) ² or the exponential function e ^α | (Z−Z ₀ ) |. Further, for example, as measured by the progress of the camera 200 along the trajectory 530, and when the images IM11 ^* -IM15 ^* (FIGS. 1B and 1C) are displayed in motion sequences, they are converted into a time sensation of display. In addition, the speed 610 is not necessarily constant over time. For example, as indicated by positions B1 to B5 in FIG. 1B and as described in the description of the figure, the ball 503 is not assigned a constant speed. Also in FIG. 2, the speed of features in the scene 500 is determined by the speed 610 of the layer 600 in which they are placed, which is discontinuous at the boundary between layers with essentially different speeds. It is noted that in some embodiments of the invention, the feature speed is a continuous function of depth in the scene. Therefore, it is considered desirable to increase the number of layers.

  It is noted that other speed profiles can be used in various embodiments of the invention. In the case of the embodiment shown in FIG. 2, the child is stationary. In other embodiments of the invention, the stationary surface can be virtually any plane between the camera's motion plane and infinity.

The sense of depth perception caused by the motion sequence of images IM11 ^* -IM15 ^* is enhanced by the increasing relative speed of the ball, girl, tree, and layer 600 features intervening between them, resulting from the selection of speed 610. not only. The inventors have found that the enhancement of depth sensation seems to also be caused by distortion introduced into the sequence of images IM11 ^* -IM15 ^* by the choice of speed.

  For example, assume that the sequence of acquired images of the scene 500 is an “actual” scene where the tree 501 does not move, the girl is stationary as a premise, and the inanimate ball does not move spontaneously. Assume that the tree, ball, and girl are aligned back and forth as shown in FIG. 1A so that the straight line passes through the tree and girl center points 561 and 580 and the vertex 572 of the ball 503. When the camera 520 is placed at position P3, the line optionally coincides with the optical axis 522. For example, in the conventional fluoroscopic imaging method shown in FIG. 1A, this line is mapped to a straight line obtained by the camera 520 at every position of the camera, like every straight line in the scene 500.

However, when imaging a scene 500 in accordance with an embodiment of the present invention, the speed assigned to features and / or layers in the scene introduces distortion that removes the limitations of fluoroscopic imaging, resulting in the mapping of straight lines to curves. The FIGS. 3A-3E show the “perspective” introduced into the images IM11 ^* -IM15 ^* , for example by imaging the scene 500 using a speed selection according to an embodiment of the invention similar to that shown in FIG. 1B. FIG. 5 schematically illustrates a “perspective breaking” strain.

  3A-3E show the position of the tree 501 and ball 503 relative to the girl 502 for each of the positions P1-P5 (FIG. 1B) of the camera 520. FIG. Each figure shows an xyz coordinate system that is stationary with respect to the girl and has its origin at the girl's center point 580 to help visualize the spatial relationship of the tree, girl, and ball. Yes. Lines 621-625 connect the center points 561 and 580 and the vertex 572 in FIGS. 3A-3E, and in all figures, each line 621-625 is each other in the scene 500 where the speed is adjusted, eg, according to FIG. It is estimated that it passes through the same point of the feature. In image 3C, line 623 is a straight line and correctly reflects the assumption that trees, girls, and balls are aligned along the y-axis of the coordinate system in “actual scene 500”. However, in FIGS. 3A, 3B, 3D, and 3E, the speed change according to embodiments of the present invention transforms the actual scene into a distorted scene, and the line 623 is no longer a straight line 621, 622, 624, And 625, respectively. Similarly, almost all other lines that are straight lines in the real image (FIG. 3C) are transformed into non-straight lines.

  FIG. 4 is a plan view of the scenes shown in FIGS. 3A-3E overlaid on top of each other, i.e., along the z-axis, so that the distortion that line 623 experiences at positions P1, P2, P3, and P4 is easily visible. The figure seen from the standpoint. Each position of the tree 501 and the ball 503 is displayed at the position of the camera 520 corresponding to the position of the tree and the ball. Since the position of the girl 502 does not change, no camera position is displayed.

Not only does the straight line not be mapped to the straight line as in the conventional fluoroscopic imaging method, but also, as shown in FIGS. 3A to 3E and FIG. Images IM11 ^* -IM15 ^* (FIGS. 1B, 1C) record distortion and therefore do not provide a sequence of images corresponding to the actual scene. Instead, the image is, in embodiments of the present invention, optionally depth even when slightly visible and is recorded in the images IM11 ^* -IM15 ^* when viewed sequentially. Record scenes characterized by time-varying distortions that result in enhanced sensation.

  In some embodiments of the invention, features in a scene, such as scene 500, are imaged at different focal lengths to control the relative speed of the features in the image sequence and the depth perception of the motion sequence using the image. Is done. For example, the relative speed of a given feature with respect to other features in the scene can be increased or decreased by imaging the feature with a larger focal length than the other features.

  In an embodiment of the invention, cameras with different focal lengths are spaced along the optical axis in a way that we believe to mimic the way a real image is projected by the eye. In particular, a given plane at a certain focal length is imaged with a first lens whose field of view forms an angle α. For other focal lengths intended to image different planes, the cameras are placed at positions along the axis where the same area of a particular plane is visible at their viewing angles.

  Therefore, in the embodiment of the present invention, a set of cameras aiming at the same scene is spaced along the optical axis. By following the provisions of the previous paragraph, the focal length of each camera is adjusted to compensate for the relative scale changes that occur for different distances from the scene of each particular camera. As a result, all cameras see the same scene at the same scale, but at different focal lengths.

  Since all cameras see only a portion of the depth of the scene, a picture made up of all “slices” (seen by different axially spaced cameras) projects the entire scene. Due to the fact that the change in perspective between scenes is enhanced when the camera moves laterally with respect to the optical axis, the resulting image series has a significantly enhanced perception of significantly enhanced depth. To communicate. This type of depth enhancement caused by an increased perspective change between images, using a conventional lens with a constant focal length, either in the real world using known methods or in the computer graphics world. Achieving is difficult if not impossible.

  The above methodology is based on our understanding of why this embodiment works, but we do not want to be bound by an unproven explanation. However, it has been found that this embodiment achieves enhanced perception of depth of field.

  FIG. 5A schematically illustrates the effect of imaging different features of the scene 500 at different focal lengths in accordance with an embodiment of the present invention. As in FIGS. 1A and 1B, in FIG. 5A, camera 520 images scene 500 along trajectory 530 at positions P1-P5. However, at positions P1, P2, P3, P4, and P5, the camera 520 captures the girl and the ball at the focal length F1 in the images IM21, IM22. . . IM25 is imaged, and the tree 501 is imaged at the focal length F2 with images IM31, IM32. . . Each IM35 is imaged. Only images IM23, IM24, IM25, IM33, IM34, and IM35 are shown in FIG. 5A for convenience of presentation and prevention of confusion. As an example, F2 is greater than F1.

As a result of imaging the tree at a higher focal length, the image IM31. . . The movement of the tree at IM35 is the image IM21. . . It increases compared to the movement of the tree when imaged by IM25. The display image according to the embodiment of the present invention is IM21. . . The feature of IM25 is IM31. . . The images IM21.. IM21. So that the distant features overlying the features of IM 35 are properly occluded by the close features. . . IM25 is converted into an image IM31. . . Formed by aligning and bonding with IM35. Composite images CI23-33, CI24-34, and CI25-35 are shown side by side behind the images from which they are formed, IM23 and IM33, IM24 and IM34, and IM35 and IM36, respectively. Composite display images CI21-31 ^* , CI22-32 ^* ,. . . CI25-35 ^* are composite images CI21-31, CI22-32,. . . Corresponds to CI25-35.

FIG. 5B illustrates composite display images CI21-31 ^* , CI22-32 ^* ,... So that composite images according to embodiments of the present invention can be easily compared with prior art images. . . CI25-35 ^* and images IM1 ^* -IM5 ^* acquired by the prior art as shown in FIG. 1A. The image shows that relative motion between the tree and the girl and the ball is substantially increased by imaging the tree with a longer focal length F2. Increasing relative motion results in increased depth perception as compared to the depth perception provided by the prior art images IM1 ^* -IM5 ^* when the composite display image is displayed in a motion sequence.

  In the above description, it has been described that a depth sensation is given to a stationary object, but it will be understood that the present invention is not limited to a scene in which all objects are stationary. In particular, moving objects are typically provided with movement as described above based on their instantaneous layer position in addition to the “real” movement of the object. Similarly, when a scene is imaged using a camera with a variable focal length, the influence of the focal length changes to match the instantaneous position that changes with the actual movement of the object.

  It will further be appreciated that the speed and focal length can be adjusted automatically or in response to user input via a mouse or the like.

  In the description and claims of this application, each of the verbs “comprise”, “include”, “have” and their conjugations is intended to mean that the verb object is not necessarily a verb. Used to indicate that the subject is not a complete list of members, components, elements, or parts.

  Although the present invention has been described using detailed descriptions of embodiments thereof, it has been presented by way of example only and is not necessarily intended to limit the scope of the invention. The described embodiments comprise various features, not all of which are required for all embodiments of the invention. Some embodiments of the invention utilize only some of the features or possible combinations of features. Embodiments of the invention that include variations of the described embodiments of the invention and different combinations of features shown in the described embodiments will be apparent to those skilled in the art. The scope of the present invention is limited only by the appended claims.

FIG. 2 schematically illustrates a scene including a ball, a girl, and a tree discussed in the overview that is imaged to provide a motion sequence of a displayed image according to the prior art. 1B schematically illustrates the scene shown in FIG. 1A imaged to provide a motion sequence in accordance with an embodiment of the present invention. 1A schematically shows a sequence of display images generated according to the prior art as shown in FIG. 1A and generated according to the present invention as shown in FIG. 1B. Fig. 4 schematically illustrates dividing a scene into layers and assigning speeds to the layers according to an embodiment of the present invention. Fig. 4 schematically illustrates distortion of a version of a scene used to generate a display image of a scene according to an embodiment of the present invention. FIG. 3C shows a top view of the distortion shown in FIGS. 3A-3E according to an embodiment of the invention. It schematically shows that a display image is generated by imaging different features of a scene at different focal lengths. 1A schematically illustrates a sequence of display images generated in accordance with the prior art as shown in FIG. 1A and generated in accordance with the present invention as shown in FIG. 5A.

Claims (25)

Generating a sequence of display images of the scene;
Setting the speed of features in the scene to generate non-perspective distortion of the scene in the display image; and sequentially displaying the display image;
Generating a visual representation of a scene having a desired depth illusion. Acquiring each image for a different location relative to at least some features in the scene and generating a sequence of images of the scene; and sequentially displaying the displayed images;
Including
Generating the sequence comprises:
Systematically setting the position of features in the scene for each image, generating non-perspective distortion of the image, and providing enhanced depth perception to the sequence of images;
Generating a visual representation of a scene having an enhanced depth illusion. Acquiring each image by a virtual camera from a different location in relation to at least some features in the scene and generating a sequence of display images of the scene; and sequentially displaying the display images;
Including
Generating the sequence includes providing at least one feature in the scene with a speed to generate relative motion between features that are nominally stationary relative to each other in the scene; A method for generating a visual display of a scene with an enhanced depth illusion, wherein the speed provided to the particular feature in the image is a function of the depth of the particular feature in the scene relative to the camera.   The method of generating a visual display according to claim 3, wherein the virtual camera position is along a non-planar three-dimensional trajectory.   The method of generating a visual display according to claim 3, wherein the virtual camera position is along a planar trajectory.   The method of generating a visual display according to claim 3, wherein the virtual camera position is along a linear trajectory.   The method of generating a visual display according to any of claims 3 to 6, wherein some of the locations are closer to the scene than other locations.   8. A method of generating a visual display according to any of claims 3 to 7, wherein at least two positions are displaced relative to each other laterally with respect to the scene.   The method of generating a visual display according to any of claims 3 to 8, wherein features at different depths relative to the camera are imaged at different focal lengths.   The method of generating a visual display according to any of claims 3 to 9, wherein the scene is divided into layers.   The method of generating a visual display according to claim 10, wherein the same speed is set for features of the same layer.   12. A method for generating a visual display according to claim 10 or 11, wherein different speeds are set for different layers.   The method for generating a visual display according to any of claims 3 to 12, wherein at at least one position of the virtual camera, the camera images different layers from different positions along the optical axis of the camera.   Features near the camera are provided with a velocity in a first direction, features at an intermediate distance are stationary, and features at a greater distance are provided with a velocity in a second direction substantially opposite to the first direction. Item 14. A method for generating a visual display according to any one of Items 3 to 13.   Features close to the camera are provided with a first velocity in the first direction, features with an intermediate distance are provided with a second velocity less than the first velocity in the first direction, and features with very large distances are stationary. 15. A method of generating a visual display according to claim 14.   Features close to the camera are stationary, features at intermediate distances are provided with a first speed in a first direction, and features at far distances are provided with a second speed greater than the first speed in the first direction. 15. A method for generating a visual display according to claim 14.   17. A method for generating a visual display according to any of claims 14 to 16, wherein the first direction is substantially parallel to the direction of motion between the different positions.   18. A visual display as claimed in any one of the preceding claims, wherein the moving feature is provided with a change in position or an additional speed consistent with the change provided to the correspondingly arranged stationary object. Method. Acquiring each image with a virtual camera from a different position in relation to at least some scene features to generate a sequence of display images of the scene; and sequentially displaying the display images;
Including
A method of generating a visual representation of a scene with an enhanced depth illusion, where the focal length at which a virtual camera images a particular feature in the scene is a function of the distance of the particular feature from the camera.   20. The method for generating a visual display of claim 19, wherein at least some features far from the camera are imaged at a focal length that is greater than a focal length used to image features that are relatively close to the camera.   21. A method of generating a visual display according to any of claims 1 to 20, comprising displaying a display image for viewing by both eyes of an observer.   A visual display image generated according to any one of claims 1 to 21.   A sequence of visual display images comprising the visual display image of claim 22.   A sequence of visual images in which an imaging position moves between images and a stationary object moves relative to each other according to the distance between the object and the imaging position.   A computer-accessible storage medium encoded with the method according to any of claims 1 to 21 or the sequence of images or visual images according to any of claims 22 to 24.