JPH08116554A - Pseudo stereoscopic image generating method and pseudo stereoscopic image generating device - Google Patents

Abstract

PURPOSE: To obtain the method in which a stereoscopic image is displayed even when a direction of a motion vector of a moving image is other than the horizontal direction. CONSTITUTION: An average motion vector of plural frames of a moving image is calculated and a 2nd frame is selected based on the magnitude of the average motion vector. An image of the 1st frame at a time (t) and an image of the 2nd frame at a time t' are rotated so that the motion vector is directed in the horizontal direction and the image is expanded so as to avoid a gap with a display image caused as the result of rotation. Each image is displayed as a right side image and a left side image after application of the rotation magnification processing to the image. Thus, even when the motion vector of moving image data is directed other than the horizontal direction, since the image is directed in the horizontal direction by the rotation of the image, a viewer enjoys a stereoscopic image without turning his face.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for displaying a stereoscopic image by extracting image frames having a time difference from moving image data and displaying them as a right side image and a left side image, respectively. In particular, the present invention relates to a method and apparatus capable of stereoscopic viewing even when the motion in the moving image data has a motion other than the horizontal direction.

[0002]

2. Description of the Related Art Conventionally, there has been known a method of continuously and stereoscopically reproducing a subject based on moving image data taken by a single video camera or the like. Such a method is described, for example, in JP-A-54-5615. FIG. 7 shows an explanatory view for explaining the principle of the stereoscopic image reproducing method described in this publication.

According to this method, first, the landscape on the ground is continuously photographed by the video camera 1 attached to an airplane or the like. At this time, as shown in FIG. 7, images from different viewpoints are obtained at time t1 and time t2.
For example, in the explanatory diagram shown in FIG. 7, time t
An image in the range indicated by a in FIG. 7 is obtained at 1 and an image in the range indicated by b is obtained at time t2. The screen a and the screen b thus obtained are continuously recorded on the magnetic tape 2. For example, image a is 3
The image b is recorded at the position a and the image b is recorded at the position 3b. What is recorded on the magnetic tape 2 is a normal moving image, which can be displayed on a general television device by a general video tape reproducing device.

When the image data recorded on the magnetic tape 2 is reproduced by two reproducing heads with a time difference in the feeding direction in this way, image data having two predetermined time differences can be obtained. The images having the time difference thus obtained are reproduced by the two cathode ray tubes 5a and 5b, respectively. Then, when a person looks at these two images with the stereoscope 4, it becomes possible to reproduce and observe the landscape continuously recorded on the magnetic tape 2 as a stereoscopic image.

As can be understood from the method of photographing the image data, the image data displayed on the two cathode ray tubes 5a and 5b are image data when the landscape A is viewed from different points. Therefore, stereoscopic viewing is possible by viewing these image data with the right eye and the left eye of a human, respectively.

[0006]

According to the above prior art, a stereoscopic image can be easily obtained based on moving image data taken by using a single video camera or the like. However, as can be understood from the explanatory diagram shown in FIG. 7, this moving image data is premised on moving in a fixed direction. Then, since the images deviated in the direction of this movement are displayed in the right eye and the left eye, respectively, there is a limitation that the direction of the movement must be horizontal (the direction in which the human eyes of the observer are aligned).

As a result, for an image in which the moving direction of the moving image changes, it is necessary for the viewer to adjust the angle of the face so that the motion vector is parallel to the direction in which the human eyes are lined up. However, even in this case, the images displayed on the right eye and the left eye must be displayed on each eye as they are. Therefore, even if the angle of the face is changed, for example, in order to prevent the image to be displayed on the left eye from being seen by the right eye, a special optical system may be required in the device of the conventional art. is there.

As a method of displaying an image for the right eye (right side image) and an image for the left eye (left side image) in the right and left eyes, respectively, one cathode ray tube is used in addition to the method using two cathode ray tubes. There is also a method in which a person wears glasses with a liquid crystal shutter. This method displays a right-side image and a left-side image on one CRT every 1/30 second, and in synchronization with that, closes the left side or the right side of the liquid crystal shutter incorporated in the eyeglasses, thereby displaying the right side image and the left side image, respectively. It is supplied to the right or left eye. According to this method, it is possible to supply the right-side image and the left-side image to the right eye and the left eye, respectively, even when a person changes the angle of the face without using a special optical system.

[0009] However, it is extremely complicated work that the image observer must change the angle of the face each time the motion of the moving image changes, which is impossible in reality. I have to say. Therefore, it is virtually impossible to perform stereoscopic vision for an image such as a movie in which the direction of movement changes with time.

The present invention has been made in view of the above problems, and an object thereof is to provide a method for stereoscopic viewing without changing the angle of the observer's face even if the direction of movement of the image changes. It is to provide a device.

[0011]

In order to solve the above-mentioned problems, the present invention displays moving image data consisting of a plurality of time-series frame images as either the right-side image or the left-side image, In the method of displaying a pseudo three-dimensional image by displaying the shifted moving image data that is displaced by a predetermined period from the moving image data that is left side image or right side image, from the moving image data A motion vector calculating step of calculating a motion vector of the image; and an image rotating step of rotating the moving image data so that the motion vector is in a horizontal direction, wherein the rotated image is displayed. Is a pseudo-stereo image generation method.

The present invention also provides the above pseudo stereo image generation method, wherein the size of the image data is 1
A pseudo-stereo image generating method, including a scaling step for changing within a range from a multiple to | cos θ | + a · | sin (θ) | times. Here, θ is the rotation angle of the image in the image rotation step, and a is the aspect ratio of the image data.

In order to solve the above-mentioned problems, the present invention further comprises first display means for displaying moving image data composed of a plurality of time-series frame images as either one of a right side image and a left side image, An apparatus for displaying a pseudo three-dimensional image, including second display means for displaying the shifted moving image data which is deviated from the displayed moving image data by a predetermined period as one of the left side image and the right side image. In the above, the first image processing apparatus includes a motion vector calculating unit that calculates a motion vector of an image from the moving image data, and an image rotating unit that rotates the moving image data so that the motion vector becomes horizontal. The second display means is a pseudo-stereo image generating device characterized by displaying a rotated image.

Further, the present invention is the above pseudo-stereo image generating apparatus, wherein the size of the image data is
cos θ | + a · | sin (θ) | times the scale means for changing within the range. Here, θ is the rotation angle of the image in the image rotation step, and a is the aspect ratio of the image data.

[0015]

In the image rotating step of the first aspect of the invention, the image is rotated so that the motion vector of the moving image is in the horizontal direction. The observer can match the moving direction of the moving image with the direction of the human right and left eyes, that is, the horizontal direction without changing the angle of the face of the observer.

Further, in the scale step of the invention of claim 2, the image data is scaled from 1 × | cos θ | + a · | sin
(Θ) | Expands within the range up to. Therefore, it is possible to prevent a gap from being generated in the original image frame even when the image is rotated.

The inventions of claims 3 and 4 relate to an apparatus for carrying out the method of claims 1 and 2.
The operation is the same as in claims 1 and 2.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT A preferred embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is an explanatory view for explaining the principle of the pseudo stereo image generation method according to the present invention. A pseudo stereo image generation method according to the present invention is disclosed in the above-mentioned Japanese Patent Laid-Open No.
It shares the principle with the method described in Japanese Patent No. 5615. That is, it is the same as the above-described conventional example that image data having a predetermined time difference is sequentially extracted from moving image data composed of a plurality of image frames appearing in time series and displayed as a right side image and a left side image, respectively.

A feature of this embodiment is that the right and left images are rotated / enlarged based on the motion vector of the moving image as shown in FIG. Then, the image data after the rotation / enlargement is displayed for the right eye and the left eye, respectively.

First, by rotating, the direction of the motion vector of the moving image data is changed to the alignment of the left and right eyes of the human being,
That is, they can be aligned in the horizontal direction, and stereoscopic viewing is possible without tilting the observer's face. Also, rotating the image creates a gap between the image and the original image frame, which makes it difficult to see visually. However, by enlarging the image, it becomes possible to push out the gap to the outside of the frame, which makes the image visually good. It is possible.

The operation representing the operation of the pseudo-stereo image generating method according to this embodiment is shown in FIG. As shown in FIG. 2, first, in step S2-1, an average motion vector of a plurality of frames included in the moving image data is calculated. An explanatory diagram of the calculation of this average motion vector is shown in FIG. As shown in FIG. 3, for example, in the present embodiment, time t, t + 1, t + 2, t +.
This average motion vector is calculated from the image frames at the five times of 3 and t + 4.

First, before calculating the average motion vector, the motion vector between adjacent image frames is calculated. That is, it is obtained between the image frame at the time t and the image frame at the time t + 1, and in the following, sequentially between the time t + 1 and the time t + 2, between the time t + 2 and the time t + 3, and between the time t + 3 and the time t + 4. Desired. In this way, motion vectors between all four frames are obtained.

Next, the average motion vector is obtained by calculating the average value of the obtained motion vectors. In this embodiment, the average motion vector in a plurality of frames is calculated in this way. In the present embodiment, two image frames having a predetermined time difference are extracted as in the conventional case, but the image frame at time t in FIG. 3 is selected as one of the image frames. The image frame at the time t is called the first frame in this embodiment. That is, the image data of the first frame is obtained by simply sequentially extracting the image frames forming the moving image data. Then, as will be described later, an image frame that allows stereoscopic viewing by combining with the sequentially fetched first frame is called a second frame.

In step S2-2 in FIG. 2, the second motion vector is calculated based on the average motion vector magnitude.
The image frame that is the frame is selected. As described above, the image frame at time t, that is, the first
The selection of the second frame that forms a stereoscopic view when combined with the frame is performed based on the magnitude of the motion vector. In FIG. 4, this step S2-
The explanatory view explaining operation | movement of 2 is shown. As described above, the frame at time t is used as the first frame, and the second frame suitable for forming a stereoscopic view with this frame is used.
The frame is selected in this step S2-2. In this selection, when the average motion vector calculated in step S2-1 is large, a frame close to time t is selected. On the other hand, when the value of the average motion vector is small, the image frame temporally separated from the image frame at time t is selected as the second frame. Thus, when the motion vector is large, the second frame is selected so as to reduce the motion, that is, the difference between the right image and the left image, and conversely, when the motion vector is small, the difference between the right image and the left image is increased. Then, the second frame is selected. In this way, smooth stereoscopic viewing is possible by selecting the second frame so that the difference between the right image and the left image has an appropriate value.

As will be described later, one of the first frame and the second frame is the right image, and the other is used as the left image.

Further, in the present embodiment, the average value of the motion vectors among a plurality of frames is calculated and the second frame is selected based on this magnitude, but as another method, the time t
It is also possible to add the motion vectors of the adjacent frames sequentially from the above frame and select the frame whose total sum is the closest to the predetermined size as the second frame. Furthermore, instead of obtaining the motion vector between adjacent frames, the time t for the frame at time t
Alternatively, the motion vector of each frame of +1, t + 2, t + 3, and t + 4 may be obtained, and the frame having the closest size to the predetermined size may be selected as the second frame.

Next, in step S2-3 in FIG. 2, the rotation angle of the image and the expansion coefficient of the image are calculated based on the direction of the average motion vector obtained as described above. This rotation / expansion is performed by the following formula.

(Xnew, Ynew) ^t = K · R · (X
org, Yorg) ^t where Xorg and Yorg are the coordinate values of the original image data. Then, Xnew and Ynew represent coordinate values after rotation and enlargement, respectively. Here, R is a transformation matrix and controls the rotation of the image. K is an expansion coefficient (scalar) and controls the expansion of the image.
Here, the conversion matrix R is obtained as follows.

[0030]

[Equation 1] Next, the expansion coefficient K is obtained as follows.

K = | cos θ | + a · | sin (θ) | where θ is the rotation angle. That is, it is an angle formed by the motion vector between images and the horizontal axis of the images (usually called the X axis), and the counterclockwise direction is the positive direction. Further, a is the aspect ratio of the display screen. That is, the vertical: horizontal = 1: (a) of the display screen. In this way, the transformation matrix R and the enlargement coefficient K are calculated, and the image is rotated / enlarged by these parameters. An explanatory diagram of this rotation / enlargement is shown in FIG. The shape of the frame when the original image frame is enlarged as shown in FIG. 5 is shown as a “rotated frame”. Here, the rotation angle is represented by θ. When this rotation is performed, a certain gap naturally occurs between the original image frame and the original image frame. In FIG. 5, this gap is indicated by hatching. Since it is generally not preferable that a gap as shown by this hatching appears on the screen, the frame after this rotation is enlarged until the gap disappears in this embodiment. This expansion is performed by the expansion coefficient K described above.

FIG. 5 also shows the frame after the enlargement after the enlargement count K times. As shown in FIG. 5, by enlarging by K times, the enlarged frame completely includes the original image frame, and a gap as shown by hatching is the size of the original image frame. It will not appear on the screen.

However, when it is not preferable to enlarge the image, the enlargement ratio may be fixed to 1. Further, a limit value (K limit) of the enlargement ratio may be provided, and when K obtained above exceeds the limit value, K limit may be used as the enlargement value.

After the rotation angle θ and the enlargement coefficient K are calculated in this way in step S2-3, the rotation / enlargement processing is performed with the rotation angle θ and the enlargement coefficient K in step S2-4 in FIG. is there.

Then, in step S2-5 in FIG. 2, the first and second frames after rotation / enlargement are displayed. Here, which of the first frame and the second frame is the right screen and which is the left screen is determined as follows.

[0036] Here, t is the time when the first frame is obtained, and t ′
Represents the time when the second frame was obtained.

As a result of the above, the motion vector in the rotated / enlarged image is a component only in the horizontal direction (direction parallel to the line connecting the right eye and the left eye of the human), and its magnitude is almost constant. Become. Furthermore, the converted image spreads over the entire display screen, and no extra gaps occur. As described above, according to the present embodiment, even if the direction of movement of the image is not constant, it is possible to enjoy the stereoscopic image while keeping the angle of the human face parallel to the plane of the display screen.

An example of the configuration of an apparatus that realizes the above-described pseudo stereo image generation method will be described with reference to FIG.
The device described below has a plurality of frame memories, and is configured to select a predetermined image frame from the plurality of frame memories. In the pseudo stereo image generation method described above, the second frame must be selected based on the average motion vector, but a configuration using two magnetic heads, for example, can be considered for the selection of the second frame. However, since this is mechanically complicated and its accuracy also becomes a problem, the example of the apparatus shown in FIG. 6 is configured to perform all electronic processing using digital technology.

As shown in FIG. 6, moving image data output from, for example, a video reproducing device, that is, an input image is first supplied to the A / D conversion circuit 20. This A /
The D conversion circuit 20 converts an input image that is an analog signal into a digital signal. The image data converted into the digital signal is sequentially stored in the frame memories # 0 to # 6 via the memory I / O circuit 22.

The memory I / O circuit 22 cyclically writes the digitized output image data in the frame memories # 0 to # 6. That is, when writing is started from # 0 and writing to the frame memory # 6 is completed, the next frame is written again in the frame memory # 0. That is, the latest seven frame images are stored in the frame memories # 0 to # 6. The motion vector detection circuit 24 reads the data stored in the frame memories # 0 to # 6 via the memory I / O circuit 22 and calculates the average value of the motion vectors. In this calculation, basically, the motion vector between frames is first obtained as shown in FIG. 3, and the average motion vector is obtained by averaging the motion vectors between the frames. In order to calculate the motion vector between the frames, it has long been performed to compare the image data included in the two frames while shifting the image data by a predetermined amount. However, in recent years, with the development of image data processing technology, various motion vector detection methods have been proposed. For example, Japanese Unexamined Patent Publication No. 4-290385 discloses an apparatus that corrects camera shake by extracting not a whole image data but only a predetermined representative point to calculate a motion vector. Various other conventionally known motion vector detection methods can be applied to this embodiment.

The motion vector thus obtained is transferred from the motion vector detection circuit 24 to the rotation expansion conversion circuit 2
6. The rotation expansion conversion circuit 26 calculates the image rotation angle θ and the image expansion coefficient K as described above. At the same time, the second frame to be selected is determined based on the magnitude of the average motion vector. Based on this determination, the rotation expansion conversion circuit 26 determines that the first frame and the second frame
The image data of the frame is taken out from the frame memories # 0 to # 6 via the memory I / O circuit 22. Then, predetermined rotation / enlargement processing is performed based on the calculated rotation angle θ and enlargement coefficient K. Then, depending on the range of the value of the rotation angle θ, either the first frame or the second frame is the right side screen, and the other is the left side screen.

After the right side screen and the left side screen are generated in this way, these data are supplied to the D / A conversion circuit 28.

In the D / A conversion circuit 28, the right side screen and the left side screen supplied from the rotation enlargement conversion circuit 26 are respectively D / A converted and output as an analog signal to the outside.

In this way, according to the configuration shown in FIG. 6, the pseudo stereo image can be generated only by electronic processing without using a mechanical configuration.

Although a display device is not shown in FIG. 6, various conventionally known display devices are applicable. For example, as the display device, a parallax barrier, a lenticular (lens), glasses with a liquid crystal shutter,
In the pseudo-stereo image generation method and apparatus as described above, which include the color classification by RB and the method using polarized light, the image frame from time t to time t + n (n is an integer of 1 or more) is stored. However, at times t-n to t +
It is also suitable to store image frames in the range of n.
That is, as the range for selecting the second frame, not only the range of +1 to + n with respect to the time t but also the range of −n to −1 are candidates. In this case, on the same side as the second frame selected immediately before (that is, +1 to + n).
(Or any one of -1 to -n) should be selected first, and the frame having the most suitable motion vector magnitude on the selected side should be selected as the second frame. And if the motion vector is not large enough-
It is preferable to select the second frame from all ranges from n to + n. With this configuration, it is not necessary to decide whether the first frame or the second frame should be the right side or the left side. For example, the first frame is always the right side screen and the second frame is always the left side screen. Is possible.

[0046]

As described above, according to the first aspect of the invention, the image data is rotated so that the motion vector of the moving image becomes horizontal, so that the viewer does not have to change the angle of the face. This has the effect of providing a pseudo-stereo image generation method that allows viewing.

According to the second aspect of the invention, the size of the image data is from 1 × to | cos θ | + a · | sin
(Θ) | It is possible to prevent a stereoscopic image that is difficult to see without a gap being generated on the displayed screen because it is magnified up to twice.

Further, according to the inventions of claims 3 and 4, there is an effect that a pseudo stereo image generating apparatus having the same effect as the above method can be obtained.

[Brief description of drawings]

FIG. 1 is an explanatory diagram showing the principle of operation of the present invention.

FIG. 2 is a flow chart showing the operation of the preferred embodiment of the present invention.

FIG. 3 is an explanatory diagram illustrating calculation of an average motion vector in step S2-1 in FIG.

FIG. 4 is an explanatory diagram illustrating selection of a second frame in step S2-2 in FIG.

FIG. 5 is the rotation in step S2-4 in FIG.
It is an explanatory view explaining an expansion process.

FIG. 6 is a configuration block diagram of a preferred embodiment of a pseudo stereo image generating apparatus according to the present invention.

FIG. 7 is an explanatory diagram showing an operation of a conventional stereoscopic image reproducing method.

[Explanation of symbols]

 20 A / D conversion circuit 22 Memory I / O circuit 24 Motion vector detection circuit 26 Rotation expansion conversion circuit 28 D / A conversion circuit

Claims (4)

[Claims] 1. A shift moving image in which moving image data composed of a plurality of time-series frame images is displayed as either one of a right side image and a left side image, and is shifted from the displayed moving image data by a predetermined period. A method of displaying a pseudo three-dimensional image by displaying data as either one of the left image and the right image, a motion vector calculating step of calculating a motion vector of the image from the moving image data, An image rotating step of rotating the moving image data so that a motion vector is in a horizontal direction, and the rotated image is displayed, and the pseudo stereo image generating method is characterized. 2. The pseudo stereo image generation method according to claim 1, wherein the size of the image data is from 1 times to | cos.
A pseudo-stereo image generating method, including a scaling step of changing within a range of up to θ | + a · | sin (θ) | times. Here, θ is the rotation angle of the image in the image rotation step, and a is the aspect ratio of the image data. 3. A first display means for displaying moving image data composed of a plurality of time-series frame images as one of a right side image and a left side image, and a predetermined period from the displayed moving image data. A second display unit for displaying the shifted moving image data as one of the left image and the right image, and a device for displaying a pseudo three-dimensional image, in which the image movement is changed from the moving image data. Motion vector calculation means for calculating a vector, and image rotation means for rotating the moving image data so that the motion vector is in a horizontal direction, wherein the first and second display means are rotated images. A pseudo-stereo image generating device characterized by displaying. 4. The pseudo-stereo image generating apparatus according to claim 3, further comprising increasing the size of the image data from 1 to | cos θ | + a ·
| Sine (θ) | scale means for changing within a range of up to | Where θ is the rotation angle of the image of the image rotation,
a is the aspect ratio of the image data.