JP2000078611A - Stereoscopic video image receiver and stereoscopic video image system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a stereoscopic video image by decreasing the cost of the stereoscopic receiver and processing the received video information in real time. SOLUTION: A stereoscopic video image transmitter 101 in the processing of generating a stereoscopic video image from a 2-dimensional video image conducts pre-processing that extracts additional information such as each pixel or a depth of each pixel in the 2-dimensional video image required to generate the stereoscopic video image from the 2-dimensional video image, and sends a coded signal consisting of the additional information obtained by the pre- processing and the 2-dimensional video image to a stereoscopic video image receiver 105. The stereoscopic video image receiver 105 receives the transmitted signal to decode respectively the 2-dimensional video image and the additional information and generates a stereoscopic video image by using parallax information based on the decoded 2-dimensional video image and the decoded additional information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional video receiving apparatus and a three-dimensional video system for generating a three-dimensional video based on additional information such as depth information extracted from a two-dimensional video signal and a two-dimensional video signal.

[0002]

2. Description of the Related Art Conventionally, while the development of a consumer stereoscopic video display device (stereoscopic display) has been progressing, the amount of software for consumer stereoscopic video has been small, and a new stereoscopic video has been produced to display the stereoscopic video. There is a need to.

Therefore, as a method of utilizing the assets of the conventional two-dimensional video, a method of converting the conventional two-dimensional video into a stereoscopic video has been proposed. FIG. 10 shows a configuration for converting a two-dimensional image into a three-dimensional image. The transmitting device 1001 encodes and transmits a two-dimensional video signal by the encoding device 1002 in the same manner as transmitting a two-dimensional video signal. Receiving device 1003
Receives a coded signal from the transmitting device 1001 and decodes the two-dimensional video signal with the decoding device 1004, and then generates two parallax video signals with the parallax video generator 1005. That is, the receiving device 1003 handles all the conversion processing to the stereoscopic video.

As a method of generating a stereoscopic image, for example, there is a method of generating a stereoscopic image such that the vicinity of the center of the screen is always in front in the depth direction and the vicinity of the center is always viewed in front.

As another method, it is assumed that the moving area is in the foreground, and when the movement is from left to right, the image given to the right eye is delayed several fields from the left eye. By doing so, the moving area is perceived in the foreground.

[0006] At present, the above-mentioned image processing techniques have been developed, and studies for estimating a three-dimensional shape from a two-dimensional image irrespective of a moving image or a still image have been actively conducted. On the other hand, MPEG-4 has been standardized for video encoding.
The configuration is shown in FIG. First, a two-dimensional video as shown in FIG. As shown in FIGS. 12B to 12D, the arithmetic unit 1012 decomposes the image into constituent elements, encodes each element with the encoding unit 1013, and encodes information on the positional relationship between each element. Become Each of the encoded signals is supplied to a multiplexer 101.
4 and transmitted (transmitted) from the transmitting device. The transmitted signal is received by the receiving device 1015, and is first demultiplexed by the demultiplexing device 1016 for each element. Each separated element is decoded by the decoding device 1017,
At the same time, information on the positional relationship between elements is also decoded. The decoded signal is supplied to the combining unit 101 as shown in FIG.
At 8, the images are synthesized based on the positional relationship information and output as a video similar to the original image.

[0007]

However, the output of the apparatus shown in FIG. 11 is a two-dimensional image. Therefore,
If all of the processing for converting a two-dimensional video into a more realistic three-dimensional video is performed by a three-dimensional video receiver, the three-dimensional video receiver becomes expensive or requires a large amount of processing, making real-time processing difficult. There are drawbacks.

Accordingly, the present invention provides a stereoscopic video receiving apparatus and a stereoscopic video system capable of reducing the cost of the stereoscopic video receiving apparatus and generating a stereoscopic video by processing received video information in real time. The purpose is to do.

[0009]

According to a third aspect of the present invention, there is provided a stereoscopic video receiving apparatus for generating a stereoscopic video based on video information transmitted from a transmitting apparatus. Receiving means for receiving a signal in which two-dimensional video information and additional information for generating a three-dimensional video transmitted from the receiver are coded. From the signal received by the receiving means, the two-dimensional video information, the additional information and Respectively, and generating means for generating a stereoscopic video using disparity information using the two-dimensional video information and the additional information decoded by the decoding means.

[0010] By the above means, the two-dimensional video information and the additional information are transmitted from the transmitting device, so that the receiving device does not need to newly generate information for forming a three-dimensional video using the two-dimensional video information. Since the transmitted additional information can be used immediately, signal processing in the receiving apparatus can be greatly simplified.

[0011]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows a first embodiment of the present invention.
FIG. 2 is a configuration diagram for describing an embodiment. This stereoscopic video system includes a transmitting device (stereoscopic video transmitting device) that transmits video information, and a receiving device (stereoscopic video receiving device) that receives video information transmitted from the transmitting device and displays the video. For example, as a transmitting device,
It is a transmitting device, a video camera, a video tape recorder (VTR) or the like in a television broadcasting station. As a receiving device that receives and displays video information transmitted from such a transmitting device, there is a stereoscopic display device such as a stereoscopic display.

As shown in FIG. 1, a transmitting apparatus 10 on the transmitting side
1 includes an arithmetic device 102, an encoding device 103, and a multiplexing device 104, and a receiving device 105 on the receiving side includes a demultiplexing device 106, a decoding device 107, and a disparity image generating device 108.

First, a two-dimensional video signal to be transmitted is input to an arithmetic unit 102 in the transmitting apparatus 101 on the transmitting side. The arithmetic unit 102 performs preprocessing necessary for generating a stereoscopic image,
Calculate additional information effective for generating a stereoscopic video. Information useful for generating a stereoscopic image includes a three-dimensional shape directly estimated from a two-dimensional image, which is indicated by, for example, a depth value for each pixel of the two-dimensional image.

At present, much research has been conducted in the field of computer vision on how to extract a three-dimensional shape from a two-dimensional image. Since the two-dimensional video signal that changes in the time axis direction is input to the arithmetic unit 102 of the first embodiment, the parameters for estimating the three-dimensional shape include motion parallax, shadow, and texture. Gradient, line perspective,
The two-dimensional shape of the elements, the overlap between the elements, the arrangement and size of the elements, and the like. Furthermore, parameters for estimating the motion parallax, the two-dimensional shape of the elements, and the overlap between the elements include a motion vector and a displacement between the motion vectors, and the above parameters are related to each other. Therefore, all the above-mentioned parameters other than the depth value correspond to the additional information effective for generating the stereoscopic video. That is,
The minimum required three-dimensional element parameters for generating a stereoscopic video are determined by the transmitting device 101 and transmitted to the receiving device 105.

The operation of the additional information related to the estimation of the three-dimensional shape from the two-dimensional image as described above requires many processes. In the first embodiment, these processes are provided on the transmitting side. In many cases, real-time processing is not required to perform the processing, and an arithmetic unit having higher performance than the receiving side is used on the transmitting side. For this reason, it is possible to generate a more accurate stereoscopic video. Therefore, in estimating a three-dimensional shape from a two-dimensional image, it is preferable that the three-dimensional shape is automatically generated by the above-described calculation. In addition, a person manually sets a depth according to the two-dimensional image and sets the depth. It may be additional information.

The additional information generated by the arithmetic unit 102 is encoded by the encoding unit 103 together with the two-dimensional video information. The two encoded signals are multiplexed by the multiplexing device 104 and transmitted from the transmitting device 101 to the receiving device 1.
Send to 05.

The signal transmitted from the transmitting device 101 is received by the receiving device 105. This signal is first separated into a two-dimensional video signal and an additional information signal by the demultiplexing device 106 of the receiving device 105, and each signal is decoded by the decoding device 107. The decoded signal is input to the parallax image generation device 108, and post-processing necessary for generating a stereoscopic image is performed. After two parallax images are generated,
Serve to separate eyes of observer.

FIG. 2 is a conceptual diagram in the case where a depth value for each pixel is used as additional information. As shown in FIGS. 2A to 2D, a two-dimensional video signal 111 input to the transmission device 101 and a depth value (additional information) 112 for each pixel calculated from the two-dimensional video signal 111 are described. Receiving device 10
5 is transmitted. The receiving device 105 outputs the parallax video (left video) 113 and the parallax video (right video) 114 generated from the received information to the outside.

When the depth value of each pixel is known,
The position where each pixel of the two-dimensional image is converted to the position of the parallax image is uniquely determined from the size of the screen, the observation distance of the observer, and the distance between the eyes.

For example, as shown in FIG. 3, the pixel at the original position of the two-dimensional image is moved to the pixel position for the right eye and the pixel position for the left eye in accordance with the depth value, and the geometrical arrangement is performed. Find the position. Also, when converting a 2D image to a parallax image, it is necessary to generate pixels that do not exist in the 2D image. In this case, for example, it is generated by interpolating from adjacent pixels.

As described above, when a stereoscopic video is generated from a two-dimensional video, processing is shared between the transmitting side and the receiving side. Determining a depth value with a high processing load is performed on the transmitting side in non-real time, and a parallax image is generated on the receiving side from a depth value that needs to be changed depending on the shape of the display device and the observation distance.

Thus, it is possible to generate a high-precision stereoscopic video in the receiving apparatus in real time. Further, since the depth value is sent as additional information of a normal two-dimensional video signal, compatibility with a television that can display only two-dimensional video can be maintained.

FIG. 4 is a block diagram for explaining a second embodiment of the present invention. First, a two-dimensional video signal to be transmitted is input to the arithmetic device 202 in the transmitting device 201 on the transmitting side. The arithmetic unit 202 performs pattern recognition on the input two-dimensional video signal, and divides the screen of the video into regions for each element. Various methods of region division are introduced in, for example, a video processing handbook (Showasha, published in 1987). In the arithmetic unit 202,
At the same time, a process of estimating the depth of each element is performed.

The video signal decomposed for each element by the arithmetic unit 202 is encoded by the encoding unit 203 for each element. It is to be noted that, as shown in FIG. 4, a plurality of encoding processing units are arranged inside the encoding device 203 to perform processing in parallel, but a single encoding processing unit May be sequentially processed.

The encoding device 203 encodes each element and also encodes positional relationship information on how to combine the elements in an arrangement. The encoding method is M
Scene description language (BIF) standardized by PEG4
S: Binary Format for Scene
s) may be used.

The encoding device 203 also encodes a depth value for each element as additional information. FIG. 5 shows an example of this encoding format. As shown in FIG. 5, the index also stores a value indicating the length of the signal with an identification code indicating that the signal is depth value information. The name (name) is an identification code for each element. Depth indicates the depth value of the element. Here, the information of the depth value is treated as additional information different from the position information. However, since the arrangement of each element in the three-dimensional space can be specified in the scene description language, the depth value is specified in the scene description language. May be.

Each signal encoded by the encoding device 203 is multiplexed by the multiplexing device 204, converted into a transmission format, and then transmitted (transmitted) from the transmission device 201.

The transmitted signal is received by the demultiplexing device 206 in the receiving device 205, and demultiplexed for each multiplexed element. Each of the separated elements is decoded by the decoding device 207, and at the same time, additional information indicating the positional relationship and the depth value of each element is also decoded.

The decoding device 207, as shown in FIG.
A plurality of decoding processing units are arranged inside and decoding processing is performed in parallel. However, if this processing can be performed in real time, a plurality of elements may be sequentially processed by one decoding processing unit.

The signal decoded by the decoding device 207 is input to a parallax video generator 208, and the parallax video generating device 208 performs post-processing required for stereoscopic video generation.
After one parallax image is generated, it is provided to the observer's separate eyes.

FIG. 6 is a conceptual diagram when a depth value for each element is used as additional information. That is, FIG.
6B, the background image 222, the house image 223, and the bus image 224 in the original image 221 are compared with the original image 221 of the two-dimensional image as shown in FIGS. To divide. The image 222 of the background, the image 223 of the house,
Depth values are determined in the order of the bus image 224. And
The positional relationship information of each element and the depth value of each element are encoded together with the original image and transmitted to the receiving device. The receiving device encodes and transmits the two-dimensional video signal,
The positional relationship information between the elements and the depth value of each element are decoded, and based on the information, the information shown in FIGS.
As shown in FIG. 7, a right image 225 and a left image 226 are generated as parallax images.

Next, FIG. 7 shows a method of interpolating a region having no pixels that occurs when a parallax image is converted. That is, when a three-dimensional image is created by rotating or moving a subject image in order to produce a three-dimensional effect, the original two-dimensional display area does not always match the area for the three-dimensional image. For this reason,
A non-image portion having no pixels may be generated with respect to the background image.

For example, as shown in FIGS. 7 (a) and 7 (b), the transmitting device replaces the background image 2 with the original image 231.
32, the receiving device receives each element decomposed into the house image 233, and generates a parallax image according to the image and depth value of each element. Then, as shown in FIG. 7C, a non-image portion having no pixel may occur. In this case, FIG.
As shown in (d), the house image 233 is interpolated by enlarging it in the spatial axis direction. It is also possible to interpolate an area without pixels from adjacent pixels.

FIG. 8 is a configuration diagram for explaining a third embodiment of the present invention. Note that the same parts as those in FIG. In this embodiment, as shown in FIG. 8, a parallax image generating device 302 in a receiving device 301 based on parallax information input by a user through a parallax adjustment input device 303 adjusted by an observer.
To generate a stereoscopic video. Thus, it is possible to provide a stereoscopic image that matches the viewer's preference. That is, the user can adjust the parallax information and adjust the sense of depth of the stereoscopic image.

FIGS. 9A to 9C are diagrams for explaining an example of parallax adjustment. FIG.
As shown in (a), a depth value of 3 is applied to the screen surface 311 by the preprocessing of the arithmetic unit 102 of the transmitting apparatus 101.
Specify 12.

Then, as shown in FIG. 9B, when the user adjusts the depth reference plane, the user inputs an adjustment value of the depth reference plane with the parallax adjustment input device 303.
Then, the parallax image generating device 302 of the receiving device 301
The reference plane of the depth value is changed from the depth value 312 to the depth value 313.
To generate a parallax image. Thereby, it is possible to change the depth value specified by the transmission device 101 according to the adjustment value of the reference plane of the depth input by the observer, and to change the generated image to an image that is viewed entirely in the back or an image that is viewed in front. It is possible to provide a stereoscopic image according to the viewer's preference.

When the user adjusts the depth,
The user inputs the adjustment amount of the depth amount using the parallax adjustment input device 303. Then, the parallax image generating device 302 of the receiving device 301 generates a parallax image by increasing or decreasing the depth value 312 according to the adjustment amount and changing the depth value 312 to the depth value 314. Thereby, the depth value specified by the transmission device can be changed according to the adjustment amount of the depth amount input by the observer, and the generated image can be changed to a video with a large perspective or a video with a small perspective. 3D images can be provided according to the requirements.

As described in each of the first to third embodiments, in the process of generating a stereoscopic video from a two-dimensional video, the stereoscopic video transmission device is required to generate a stereoscopic video from the two-dimensional video. Pre-processing to extract additional information such as the depth value of each pixel or each element in a simple 2D video,
The additional information and the two-dimensional video obtained in this preprocessing are transmitted to the stereoscopic video receiver as encoded signals. The stereoscopic video receiving device receives the transmitted signal, decodes the two-dimensional video and the additional information, respectively, and generates a stereoscopic video using disparity information based on the decoded two-dimensional video and the additional information. It is supposed to.

By this means, processing for extracting additional information, such as estimation of a depth value, which requires a large amount of calculation, is performed in a non-real-time manner by a transmitting device, so that highly accurate additional information can be obtained. When received, a high-precision stereoscopic video can be generated in real time.

Further, when generating a stereoscopic video based on the additional information such as the depth value obtained on the transmitting side, the observer can change the additional information such as the depth value on the receiving side. It is possible to provide combined stereoscopic images.

[0041]

As described in detail above, according to the present invention, the cost of the stereoscopic video receiving apparatus can be reduced, and the received video information can be processed in real time to generate a stereoscopic video.

[Brief description of the drawings]

FIG. 1 is a system configuration diagram according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram for explaining the concept of the first embodiment of the present invention.

FIG. 3 is a diagram illustrating a geometric relationship between a perceived depth and a generated parallax image.

FIG. 4 is a system configuration diagram according to a second embodiment of the present invention.

FIG. 5 is a diagram showing a transmission format when a depth value is used as additional information for each element.

FIG. 6 is a view for explaining the concept of the second embodiment;

FIG. 7 is a view for explaining interpolation of a non-image portion having no pixels.

FIG. 8 is a diagram showing a configuration of a stereoscopic video system according to a third embodiment.

FIG. 9 is a view for explaining a method of adjusting the depth of an image.

FIG. 10 is a diagram showing a configuration of a conventional system for generating a stereoscopic video from a two-dimensional video.

FIG. 11 is a view for explaining a configuration for transmitting a conventional two-dimensional video.

FIG. 12 is a diagram illustrating a configuration for transmitting a conventional two-dimensional video.

[Explanation of symbols]

 101 (201) ... transmitting device 102 (202) ... arithmetic device 103 (203) ... coding device 104 (204) ... multiplexing device 105 (205, 301) ... receiving device 106 (206) ... demultiplexing device 107 (207) ) Decoding device 108 (208, 302) Parallax image generation device 303 Parallax adjustment input device

Claims (7)

[Claims] 1. A stereoscopic video receiving apparatus for generating a stereoscopic video based on video information transmitted from a transmitting apparatus, comprising: two-dimensional video information transmitted from the transmitting apparatus; and additional information for generating a stereoscopic video. Receiving means for receiving an encoded signal, decoding means for respectively decoding the two-dimensional video information and the additional information from the signal received by the receiving means, and decoding by the decoding means Generating means for generating a stereoscopic video using parallax information using the obtained two-dimensional video information and the additional information. 2. The stereoscopic video receiving device according to claim 1, wherein the additional information is a depth value corresponding to each pixel of the two-dimensional video. 3. The stereoscopic video receiving apparatus according to claim 2, further comprising means for changing the depth value based on a value designated by an observer based on the depth value. 4. A stereoscopic video receiving apparatus for generating a stereoscopic video based on video information transmitted from a transmitting apparatus, comprising: two-dimensional video information divided for each element transmitted from the transmitting apparatus; Receiving means for receiving a signal in which positional information indicating the positional relationship of the elements and a depth value of each element are encoded; and obtaining the two-dimensional video information, positional information, and the depth value from the signal received by the receiving means. Decoding means for decoding the respective elements; synthesizing each element based on the two-dimensional video information for each element decoded by the decoding means, position information between the elements, and a depth value for each element; Generating means for generating a stereoscopic video using the parallax information. 5. The stereoscopic image according to claim 4, further comprising: a unit that changes the depth value based on a value specified by an observer based on the depth value received by the receiving unit. Receiver. 6. A stereoscopic video system comprising: a transmitting device that transmits video information; and a receiving device that receives video information transmitted from the transmitting device and generates a stereoscopic video. A pre-processing unit that determines the three-dimensional shape of the image from the two-dimensional video signal and generates additional information for generating a stereoscopic video; and encodes the two-dimensional video signal and generates the two-dimensional video signal. Encoding means for encoding the additional information; and transmitting means for transmitting a signal obtained by encoding the two-dimensional video signal and the additional information to the receiving apparatus by the encoding means, wherein the receiving apparatus comprises: Receiving means for receiving a signal transmitted from the receiving means; decoding means for respectively decoding the two-dimensional video signal and the additional information from the signal received by the receiving means; Stereoscopic video system based on the two-dimensional video signal decoded and the additional information by stage, characterized by comprising a generating means for generating a stereoscopic image using the disparity information. 7. A stereoscopic video system comprising: a transmitting device that transmits video information; and a receiving device that receives video information transmitted from the transmitting device and generates a stereoscopic video. A preprocessing means for dividing the two-dimensional video signal into elements according to the content of the video and determining a depth value of each element; a two-dimensional video signal for each element divided by the preprocessing means and a Encoding means for encoding position information and encoding a depth value for each element; a two-dimensional video signal for each element by the encoding means;
A transmitting unit for transmitting a signal obtained by encoding position information between each element and a depth value for each element; the receiving device receiving a signal transmitted from the transmitting device; From the signals received by the means
Decoding means for respectively decoding the two-dimensional video signal, the position information between the elements, and the depth value for each element; and the two-dimensional video signal for each of the elements decoded by the decoding means. A stereoscopic video system comprising: generating means for generating a stereoscopic video using disparity information by synthesizing each element based on the position information and the depth value of each element.