JP2001061164A - Transmission method of stereoscopic video signal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To compress transmission value of stereoscopic video signals in consideration of the stereoscopic visual characteristics by coding the depth value so as to preferentially assign the transmission value to the frequency component of high perception sensitivity according to the time and space characteristics defined to the visual depth change, when an image sent from the visual point direction and the depth value corresponding to the image are transmitted. SOLUTION: Information on the left eye images 11 of an input terminal are inputted to a coding part 103 and a depth estimating part 101, and the information on the right eye images 12 are inputted to the part 101 via the input terminal. The part 101 estimates the depth value, corresponding to every pixel of the left eye image information by referring to the right and left image information. A depth information compression part 102 performs processing to compress the depth information in consideration of the visual characteristics. In regard to the configuration of the part 102, a post-quantization step of the DCT processing is controlled to compress the quantity of information. Otherwise, the quantity of information may be compressed after the direct filter processing is applied with respect to the depth value in the space direction.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for transmitting a stereoscopic video signal, and more particularly to a method for transmitting a two-dimensional video from one viewpoint and a depth value or a parallax amount corresponding to each image of the two-dimensional video. And a method for compressing the amount of transmission.

[0002]

2. Description of the Related Art When transmitting stereoscopic video information, it is necessary to transmit video information of two images corresponding to, for example, left and right eyes. The two images at this time are images having parallax such that a stereoscopic image is perceived when the observer observes with the left and right eyes separately. When a stereoscopic image is composed of two images, if it is transmitted as it is, it is 2 times smaller than a conventional 2D image.
Double the amount of transmission is required. Therefore, methods for reducing the amount of transmission are being studied. As one of the methods, there is a transmission method called a disparity compensation method (for example, “Study on disparity estimation method in stereoscopic image coding” NHK Giken R & D
No. 47 Nov 1997).

[0003] The block configuration of the parallax compensation system is as shown in FIG.

[0004] There are two inputs, left eye image information and right eye image information. The left eye image information is transmitted as it is. For the right eye image information, the parallax estimation and compensation unit compresses the transmission amount using the correlation with the left eye image information. For example, the right-eye image is divided into sub-blocks, the closest sub-block is found in the left-eye image for each sub-block, and the positional shift amount (disparity vector) is obtained. Next, a right-eye image is predicted using the obtained disparity vector and the left-eye image, and a residual (compensation error) between the predicted image and the original right-eye image is obtained. The left-eye image, the disparity vector, and the compensation error are compressed and transmitted by an appropriate compression method.

In the parallax compensation system, a parallax vector corresponds to one parallax representing a sub-block. The compensation error may be considered to include disparity that cannot be expressed only by the disparity vector, and information of an image that does not exist as a blocking area in the left-eye image but exists only in the right-eye image.

As another method of transmitting a stereoscopic video, a method of transmitting a video from one viewpoint and a distance (depth value or parallax) corresponding to each pixel of the video has been considered.
When this method is replaced with a parallax compensation method, it can be seen that a parallax vector is obtained and transmitted for each pixel.

In the above two methods, the amount of transmission is compressed by utilizing the correlation between images between viewpoints (for example, between left and right). However, it is not necessarily considered that human visual characteristics in stereoscopic vision are considered. I can not say. There are the following transmission methods in consideration of human visual characteristics in stereoscopic vision. The left eye image information is transmitted as it is. Then, regarding the right eye image information, a difference from the left eye image information is obtained, and the difference information is subsampled and compressed. As a visual characteristic of stereoscopic vision, when the depth changes rapidly in the time axis direction, there is a characteristic that the change cannot be perceived,
The above method utilizes this characteristic, but in this method, since the difference information of the video is sub-sampled, a phenomenon that the horizontal resolution is reduced in a portion where the depth is changed appears.

[0008]

As described above, in the conventional method of transmitting stereoscopic video information, there is no technique for compressing the amount of transmission that makes full use of human visual characteristics in stereoscopic vision.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a stereoscopic video signal transmission method capable of compressing a transmission amount in consideration of visual characteristics in stereoscopic viewing when transmitting stereoscopic video information.

[0010]

SUMMARY OF THE INVENTION In order to achieve the above object, the present invention provides a method for transmitting an image from one viewpoint direction and a depth value corresponding to the image in a time and space with respect to a change in visual depth. According to the frequency characteristic, the depth value is encoded such that the transmission amount is preferentially allocated to the frequency component having a high perceptual sensitivity.

That is, according to the present invention, when transmitting a stereoscopic video, the transmission amount of the stereoscopic video is reduced by using one or more of the following visual characteristics in stereoscopic vision.

(A) The depth cannot be perceived below a certain amount of depth (this value is called a depth discrimination threshold). Further, the depth discrimination threshold changes with the spatial frequency component of the image, and the higher the spatial frequency of the image, the smaller the depth discrimination threshold.

(B) When the depth is changed periodically on the time axis, the depth cannot be perceived when the time frequency exceeds a certain amount (this value is called a critical frequency of the depth change). Also, the critical frequency of the depth change changes with the spatial frequency component of the image,
The higher the spatial frequency of the image, the lower the critical frequency of the depth change.

(C) The depth discrimination threshold changes depending on the contrast of the image. When the contrast of the image is low, the depth discrimination threshold increases.

(D) The depth discrimination threshold between a case where the image is composed of a luminance component and a case where the image is composed of a color component is smaller when the image is composed of a luminance component.

[0016]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 shows an embodiment of the present invention. Here, for example, two left and right images are assumed as stereoscopic images. The left-eye image information is transmitted as it is, or is encoded and transmitted according to the MPEG standard. In the figure, the left-eye image information of the input terminal 11 is input to the encoding unit 103 and also to the depth estimation unit 101.

The right eye image information is input to the depth estimating unit 101 via the input terminal 12. The depth estimating unit 101 estimates a depth value (parallax) corresponding to each pixel of the left eye image information with reference to the left and right image information.

As a method of estimating the depth value, for example, the movement amount of each corresponding pixel between the left and right image information is obtained using block matching, and the value is used as the depth value. Next, the depth information compression unit 102 performs processing on the determined depth value in consideration of human visual characteristics to compress the information amount.

An example of visual characteristics in stereoscopic vision is disclosed in Reference 1 ".
Basics of 3D Video "NHK Broadcasting Technology Research, Ohmsha 1995) and Reference 2" Binocular Vision and Ster
eopsis "Ian P. Howard et al. Oxford University
Press 1995). Reference 1, 28
FIG. 2.16 on the page shows the depth discrimination threshold when the depth changes in the horizontal direction. The stimulus used for the experiment is shown in FIG. 2.15 on page 27 of this document. . FIG. 5.13 on page 164 of document 2 shows the depth discrimination threshold when the depth changes in the vertical direction. The stimuli used in the experiment are shown in (a) and the results are shown in (b). This indicates that stereoscopic vision cannot be performed when the parallax is equal to or less than a certain amount, and that this value (depth discrimination threshold) shows bandpass characteristics with respect to the spatial frequency of the depth change, indicating that stereoscopic vision cannot be performed at high spatial frequencies. .

FIG. 2.21 on page 32 of Document 1 shows a depth discrimination threshold when the depth periodically changes in the time axis direction. The depth discrimination threshold shows a band-pass characteristic with respect to the time frequency, and the high time frequency indicates that stereoscopic viewing is not possible.

The depth compression section 102 performs processing so that depth information is compressed in consideration of visual characteristics. FIG. 2 shows a configuration example of the depth information compression processing unit 102.

For example, a discrete cosine transform (DCT) unit 20
In step 2, the screen is divided into blocks of a fixed area with respect to the input depth value, and DCT in the spatial axis direction is performed for each block.
Perform processing.

The high spatial frequency components for which the stereoscopic vision cannot be perceived for each frequency component after the DCT processing are removed. For other frequency components, the quantization step is controlled so that the quantization step becomes finer for a frequency having a small depth discrimination threshold. As this method, for example, a quantization table in consideration of visual characteristics may be created in advance and stored in the quantization table 204. The quantization table 204 is created in accordance with, for example, a characteristic graph indicating the depth resolution shown in FIG.

In the quantization section 203, the quantization table 20
With reference to the value of 4, quantization is performed. As a result, removal and quantization of high spatial frequency components in consideration of visual characteristics are realized. When the DCT process is performed in the time axis direction, removal of a high time frequency component and control of the quantization step are performed in the same manner as the process performed on the spatial frequency.

With respect to the configuration of the depth information compression unit 102, in the above example, after the DCT processing, the quantization step is controlled to compress the amount of information, but the depth value is directly filtered in the spatio-temporal direction. May be performed to compress the amount of information. The compressed depth value is encoded and transmitted by the encoder 103 together with the left eye image information.

In the first embodiment, the three-dimensional image is composed of two left and right images. However, the concept of this embodiment can be applied to a plurality of three-dimensional images.

FIG. 3 shows a second embodiment of the present invention.

In the first embodiment, only the left eye image information and the depth value are transmitted. However, for example, there is an image that is not shielded in the left-eye image and exists only in the right-eye image.
This is shown in FIG. In the second embodiment, this video is transmitted as compensation error information. The depth estimating and compensating unit 301 estimates the depth value of each pixel as in the first embodiment, and converts the depth value into the depth information compressing unit 1.
Give to 02.

After estimating the depth value of each pixel, the depth and compensation unit 301 predicts the right eye image from the left eye image information and the depth value of each pixel. The difference between the predicted right-eye image information and the original right-eye video information is provided to the compensation error compression unit 302 as compensation error information.

The compensation error compression section 302 compresses the compensation error based on the compressed depth value. As can be seen from FIG. 4, when the depth changes in the horizontal direction, it can be considered that the compensation error information is generated particularly in the vicinity where there is a spatial frequency component having a high depth change. This area is hereinafter referred to as a shielding area. Therefore, the depth information compression unit 102
In accordance with the depth value input from, an area having a high spatial frequency component of the depth change in the horizontal direction is extracted as an occluded area, and if a compensation error exists in an area other than the occluded area, it is removed as noise.

FIG. 5 shows a configuration example of the compensation error compression section 302.

The compensation error compression unit is provided with a compensation error from the depth estimation and compensation unit 301, and
From 02, a depth value is input. It is assumed that the depth value is input in the form of a frequency component.

The frequency extracting section 502 extracts only a component having a high spatial frequency in the horizontal direction. This component is inversely quantized by an inverse quantization unit 503, and then inversely inversed by an inverse DCT unit 504.
Perform CT processing. The output subjected to the inverse DCT processing is compared in a binarization processing section 505 as to whether or not the absolute value is equal to or more than a predetermined value. If the output is equal to or more than the predetermined value, the area is determined to be a shielding area and extracted. This result is obtained by the noise elimination unit 5
06.

The noise removal unit 506 removes compensation errors other than the shielding area from the inputted compensation error information as noise. Since the depth value input from the depth information compression unit 102 has a depth value less than the depth discrimination threshold removed, a compensation error generated only by the depth less than the depth discrimination threshold is also removed as noise. The compensation error information from which noise has been removed is input to the encoding unit 303 through DCT processing and quantization processing as in the case of the conventional parallax compensation method, and is encoded and transmitted together with the depth value and left-eye image information.

Next, a third embodiment of the present invention will be described.

In the third embodiment, the depth value is compressed according to the spatial frequency component of the left eye image information. The relationship between the spatial frequency of an image and the depth discrimination threshold is described in reference
This is shown in Figure 5.7 of the page. This characteristic indicates that the depth discrimination threshold increases as the spatial frequency component of the image decreases. As for the relationship between visual characteristics and spatial frequency and temporal frequency of video, it is known that sensitivity is high at high spatial frequencies at low spatial frequencies and high at low spatial frequencies at high spatial frequencies.

FIG. 6 shows a third embodiment. 1 are given the same reference numerals. Left eye image information
The data is input to the depth estimating unit 101, and is also input to the spatial frequency analyzing unit 601 and the encoding unit 103. The spatial frequency analysis unit 601 analyzes a spatial frequency component of the left eye image. The analysis of the spatial frequency is performed, for example, by a convolution operation on the left-eye image using Gabor filters having different sizes. FIG. 7 shows the characteristic shape of this filter. FIG. 7A shows the relationship between the space axis and the level, and FIG. 7B shows the relationship between the frequency axis and the level. The filter characteristic is expressed as S (x) = exp (−
(X−2) / (xσ−2)) * cos (2πFx).

FIG. 8 shows a configuration example of the spatial frequency analysis unit 601 described above. The left eye image information is input to the filter operation units 802, 803, and 804, and the output of each operation unit is input to the spatial frequency determination unit 805. The filter calculation units 802, 803, and 804 are set so that, for example, the filter center frequencies are f, 2f, and 4f.
The spatial frequency determination unit 805 selects one of the three types of filter results that gives the maximum output, and outputs the result to the depth information compression unit 602 as a quantization parameter.

The depth information compression section 602 receives the depth value from the depth estimation section 101. The depth information compression unit 602 has a configuration as shown in FIG. 9A, for example. It includes a DCT processing unit 202, a quantization unit 903, and a quantization table 902. In the quantization table 902, a plurality of types of quantization tables according to the spatial frequency components of the left-eye video information are created and stored in advance.

FIG. 9B shows the quantization table 902 described above.
Shows an example of the contents. The quantization table 902 selects a quantization parameter to be used according to an input quantization parameter. The quantization unit 903 can compress depth information in consideration of the spatial frequency and visual characteristics of an image by performing quantization with reference to the value of the quantization table.

FIG. 10 shows a fourth embodiment, and shows an example in which the second embodiment is applied to the third embodiment.

Next, a fifth embodiment will be described.
In this embodiment, the depth value is compressed according to the contrast of the left eye image information. The relationship between the contrast and the depth discrimination threshold is shown in FIG.
This relationship is such that the depth discrimination threshold increases as the contrast decreases.

FIG. 11A shows a fifth embodiment.
The left eye image information is input to the depth estimation unit 101, the contrast analysis unit 1101, and the encoding unit 103. The depth estimating unit 101 calculates a depth value from left-eye image information and right-eye image information. The contrast analysis unit 1101
Calculate the contrast of the left eye image information. The calculated result is output to the depth information compression unit 1102 as a quantization parameter. In the quantization table 902, a plurality of types of quantization tables according to the contrast of the left-eye video information are created and stored in advance. For example, it is created so that the quantization step becomes coarser as the contrast becomes lower (see FIG. 11B).

The quantization table 902 selects a quantization table according to an input quantization parameter. Subsequent operations are the same as in the third embodiment. As a result, it is possible to compress the depth information in consideration of the contrast and the visual characteristics of the video.

FIG. 12 shows a sixth embodiment in which the second embodiment is applied to the fifth embodiment. The function and operation of each unit are as described above.

FIG. 13 shows a seventh embodiment.

In this embodiment, the depth value is compressed according to whether the left-eye image information consists of luminance contrast or color contrast. The relationship between luminance, color and depth discrimination threshold is shown in FIG. FIG. 6.12 shows that the depth discrimination threshold composed of the color contrast is larger than the depth discrimination threshold of the video composed of the luminance contrast. Further, the luminance contrast has a peak in the depth discrimination threshold with respect to the time frequency shifted to the higher frequency side with respect to the color contrast. Further, in the description on pages 207 to 208 of the document, it is stated that the depth discrimination threshold composed of blue-yellow is larger when the color contrast is composed of red-green and when the color contrast is composed of blue-yellow. I have.

In FIG. 13, left eye image information is input to a depth estimating unit 101, a luminance and color component analyzing unit 1301, and an encoding unit 103. The depth estimating unit 101 calculates a depth value from left-eye image information and right-eye image information.
The luminance and color component analysis unit 1301 has, for example, a configuration as shown in FIG.

In FIG. 14A, for example, when the left eye image information is input in RGB format, the luminance (L), red-green (r-g), yellow-blue (y-b) ) Format. Component extraction unit 14 for each component
From 03 to 1405, for example, a contrast is extracted. The contrast of each component is input to the quantization parameter determination unit 1406. Quantization parameter determination unit 140
At 5, the contrasts are compared, and 0 is output to the depth compression unit 1302 if the luminance contrast is equal to or more than a certain value, and 1 if the luminance contrast is smaller than the certain value and the red-green contrast is equal to or more than a certain value. If the luminance contrast is equal to or less than a certain value, the red-green contrast is equal to or less than a certain value, and the yellow-green contrast is equal to or more than a certain value, 2 is output to the depth compression unit 1302. If all the contrasts are equal to or less than a certain value, 3 is used as a quantization parameter and the depth compression unit 1
Output to 302.

In the depth compression section 1302, as the quantization table 902, a plurality of types of quantization tables according to the quantization parameters are created and stored in advance. FIG. 14B shows an example of the quantization table. The quantization table 902 selects a quantization table according to an input quantization parameter. Subsequent operations are the same as those of the third embodiment. This makes it possible to compress the depth information in consideration of the luminance component and color component of the video and the visual characteristics.

FIG. 15 shows a second embodiment of the seventh embodiment.
An eighth embodiment to which the eighth embodiment is applied is shown. The function and operation of each unit are as described above.

FIG. 16 shows a ninth embodiment. This embodiment is an example in which the first, second, third, fourth, and fifth embodiments are combined. Blocks corresponding to those in the previous embodiment are denoted by the same reference numerals.
The left eye image information is input to the depth estimating unit and compensating unit 301, the image analyzing unit 1601, and the encoding unit 303. The depth estimation unit 301 outputs a depth value and a compensation error. The compensation error is input to compensation error compression section 302.
The depth value is input to the depth information compression unit 1602.
An analysis result from the image analysis unit 1601 is input to the depth information compression unit 1602.

The image analyzer 1601 is configured as shown in FIG. 17, for example. The left-eye image information input to the image analysis unit 1607 is converted from RGB into luminance (L), red-green (r-g), and yellow-blue (y-b) by the color conversion unit 1702. Thereafter, the spatial frequency components are obtained for the respective inputs by the filter operation units 1703 to 1705 having the characteristics described in the above embodiment (FIG. 8). This result is input to the quantization parameter determination unit 1706. The quantization parameter determination unit 1706 determines a quantization parameter from each frequency component of luminance and color components. The determination of the quantization parameter is performed as follows. For example, when determining the quantization step width, first, it is searched whether or not a certain amount of frequency components necessary for perceiving a small depth is included. If it is included, the quantization step is set to “fine”. If it does not contain a certain amount of frequency components, it is necessary to perceive a greater depth. If included, set the quantization step to “medium”
If not, set "Coarse".

FIG. 18 is a flowchart showing the above setting. Next, a reproduction range of the time value component of the depth value is determined. First, a search is performed to determine whether or not a certain amount of frequency components necessary for perceiving a high time frequency in the depth is included. If it is included, the reproduction range of the time frequency is set to “low, middle, and high”. Subsequently, a search is performed to determine whether or not a certain amount or more of a frequency component necessary for perceiving a lower time frequency is included.

In the case of including, the reproduction range of the time frequency is set to "
Set to “low / medium.” If not included, set to “low.”

FIG. 19 is a flowchart showing the above setting. The determined quantization parameter is output to depth information compression section 1602. Depth compression unit 160
2, in the quantization table 1202, a plurality of types of quantization tables according to the quantization parameters are created and stored in advance. For example, the quantization table is created based on the visual characteristics of the first embodiment so as to be corrected by the quantization parameter.

The subsequent operation is the same as in the above embodiments. As a result, it is possible to compress the depth information in consideration of the spatial frequency component, the contrast, the luminance component, the color component, and the visual characteristics of the video.

[0059]

As described above, according to the present invention,
When transmitting a stereoscopic video signal, it is possible to compress the transmission amount of the stereoscopic video signal by compressing depth information that cannot be perceived by a human in consideration of human visual characteristics in stereoscopic vision.

[Brief description of the drawings]

FIG. 1 is a diagram showing a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a configuration of a depth information compression unit in FIG. 1 and a relation between a quantization step and a spatial frequency and a time frequency with respect to a change in depth.

FIG. 3 is a diagram showing a second embodiment of the present invention.

FIG. 4 is an explanatory diagram showing the relationship between depth and video.

FIG. 5 is a diagram showing a specific configuration example of a compensation error compression unit in FIG. 3;

FIG. 6 is a diagram showing a third embodiment of the present invention.

FIG. 7 is a diagram illustrating a characteristic example of a filter used in the third embodiment.

FIG. 8 is a diagram showing a configuration example of a spatial frequency analysis unit in FIG. 6;

FIG. 9 is an explanatory diagram showing a configuration of a depth information compression unit and an example of a quantization table used therein.

FIG. 10 is a diagram showing a fourth embodiment of the present invention.

FIG. 11 is an explanatory diagram showing a configuration of a fifth embodiment of the present invention and an example of a quantization table used therein.

FIG. 12 is a diagram showing a sixth embodiment of the present invention.

FIG. 13 is a diagram showing a seventh embodiment of the present invention.

FIG. 14 is an explanatory diagram showing a configuration of a luminance and color component analysis unit in FIG. 13 and an example of a quantization table used therein.

FIG. 15 is a diagram showing an eighth embodiment of the present invention.

FIG. 16 is a diagram showing a ninth embodiment of the present invention.

FIG. 17 is a diagram illustrating a configuration example of an image analysis unit in FIG. 16;

FIG. 18 is a flowchart illustrating an example of a procedure for determining a quantization parameter in the embodiment of FIG. 16;

FIG. 19 is a flowchart showing another example of the procedure for determining a quantization parameter in the embodiment of FIG. 16;

FIG. 20 is an explanatory diagram of a conventional stereoscopic video compression method.

[Explanation of symbols]

101: depth estimation unit, 102: depth information compression unit, 1
03: coding unit, 202: DCT unit, 203: quantization unit, 204: quantization table, 301: depth estimation and compensation unit, 302: compensation error compression unit, 601: spatial frequency analysis unit, 602: depth information compression Section, 1101... Contrast analysis section, 1102... Depth information compression section, 1301
... luminance and color component analysis unit, 1601 ... image analysis unit, 16
02: Depth information compression unit.

Claims (4)

[Claims] When transmitting an image from one viewpoint direction and a depth value corresponding to the image, priority is given to a frequency component having a high depth perception sensitivity in accordance with time and space frequency characteristics for visual depth conversion. A stereoscopic video signal transmission method, which encodes a depth value to which a transmission amount is assigned. 2. An image from one viewpoint direction, a depth value corresponding to the image, an image from one viewpoint direction and an image other than one viewpoint predicted from the depth value corresponding to the image, and one actual viewpoint. A stereoscopic video transmission method comprising: encoding a compensation error having a high degree of depth perception in accordance with temporal and spatial frequency characteristics with respect to a change in visual depth when transmitting a compensation error with a video other than the above. 3. The method according to claim 1, wherein a characteristic amount of the image is obtained, and a transmission amount is preferentially allocated to a frequency component having a high depth perception sensitivity in accordance with a relationship between the characteristic amount of the image and a temporal and spatial frequency characteristic with respect to a depth change. The stereoscopic video transmission method according to claim 1. 4. The three-dimensional image according to claim 3, wherein the feature amount is at least one of a spatial frequency component of luminance of the image, a spatial frequency component of color, and a contrast of the image. Transmission method.