JP2001238230A - Device for extracting three-dimensional structural information in multiple-lens stereoscopic television system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem of a conventional device for extracting three- dimensional structural information in a multi-lens stereoscopic television system that has had deteriorated estimation accuracy of a depth because of an occlusion area that is seen by one camera but not seen from other camera. SOLUTION: The device for extracting three-dimensional structural information is provided with a depth quantity calculation means 4 that calculates the depth quantity by using images photographed by any of comparison cameras and a reference camera, a depth quantity reliability calculation means 5 that sequentially calculates the reliability of the depth quantity calculated by the depth quantity calculation means and comparing the highest reliability precedingly calculated with the reliability newest calculated to provide an output of the depth quantity corresponding to the highest reliability among the calculated reliabilities, and a camera drive means 7 that moves a position of any camera among the comparison cameras every time the depth quantity reliability calculation means completes the reliability calculation.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for extracting three-dimensional structure information in a multi-view stereoscopic television system.

[0002]

2. Description of the Related Art In recent years, multi-view stereoscopic television systems have been actively studied. For example, a method utilizing three-dimensional structure information in encoding a multi-view stereoscopic television image signal (Belle L.
Tsenget al., “Multiviewpoint Video Coding with MP
EG-2 Compatibility ", IEEE Trans.on CIRCUITS AND SYS
TEMS FOR VIDEO TECHNOLOGY, VOL.6, No.4, pp.414-419 (19
96)) has been proposed. Since multi-view stereoscopic television images have a high degree of spatial redundancy, it is effective to use three-dimensional structure information for compression coding. In particular, when transmitting a multi-view stereoscopic television signal, a small number of viewpoint images and three-dimensional structure information are transmitted, and the image between the viewpoints that has not been transmitted using the three-dimensional structure information on the receiving side is internally stored. Since it is inserted, high accuracy is required for extracting the three-dimensional structure information.

Hitherto, many experimental results have been reported on extracting three-dimensional structural information from a multi-view stereoscopic television image signal (for example, see Okutomi et al., “A Mult”).
ipleBaseline Stereo ”, IEEE Trans on PATTERN ANALY
SIS AND MACHINE INTELLIGENCE, VOL.15, No.4, pp.353-36
3 (1993)), the accuracy is better than the extraction accuracy when extracting from a two-lens stereoscopic TV image. The reason why the extraction accuracy in the case of extracting from a multi-view stereoscopic television image is good is that, for example, in the case of a three-view type or more, the area invisible from each camera (occlusion area) is reduced as compared with the case of a binocular type. This is because the accuracy of the depth amount increases. Using this property, a plurality of extraction systems (distance measurement systems) for extracting three-dimensional structure information from a multi-view stereoscopic television image signal have been developed.

[0004]

In order to accurately extract three-dimensional structure information, a large number of cameras are arranged so as to reduce the occlusion area, which is an area that cannot be seen by the camera, without photographing a subject. Not be. Therefore,
Kiyohide Satoh et al., “Passive Depth Acquisition
for 3D Image Displays ", IEICE Trans.Inf. & Syst., Vo
l.E77-D, No. 9, pp. 949-957 (1994)), and Jong-Il Park e
t al., “Image-Based Rendering from Multi-View Ima
ges ", Journal of the Institute of Image Information and Television Engineers Vol.52, No.3, pp.371-376
(1998)), in order to solve the problem of the occlusion area, the number of cameras was increased, and the cameras were arranged so that the occlusion area did not occur. However, in order to arrange cameras so as not to generate an occlusion area over a wide area, it is necessary to arrange dozens or more cameras, which not only increases the size of the apparatus but also the number of cameras There is a problem to be solved that it is difficult to completely eliminate the occlusion region even if the number of occlusions is increased.

In the above-described conventional three-dimensional structure information extraction system (distance measurement system) for extracting three-dimensional structure information from a multi-view stereoscopic television image signal, a reference camera (hereinafter referred to as a reference camera) is used. Multiple cameras (hereafter,
(Referred to as a reference camera) are arranged close to each other, the amount of parallax between the images of the reference camera and the reference camera is obtained, and the obtained amount of parallax is converted into the amount of depth. Although the requirements for the estimation accuracy of the depth amount differ depending on the application, the extraction of the three-dimensional structure information for the purpose of encoding the multi-view stereoscopic television image signal requires a very high estimation accuracy,
The accuracy of estimating the depth by the conventional device is still insufficient.

[0006] The accuracy of estimating the depth obtained from the multi-view stereoscopic television image by the distance measurement system is degraded by an area that is visible from a certain camera but is invisible from other cameras, a so-called occlusion area. Since it is impossible to estimate the depth in the occlusion area, it is necessary to devise shooting conditions. Further, it is possible to increase the number of cameras to be photographed, reduce the occlusion area, and improve the accuracy of estimating the depth amount. But in this,
As described above, the size of the apparatus is increased, and the camera calibration work becomes complicated as the number of cameras increases. Therefore, it is desirable to use as few cameras as possible.

An object of the present invention is to provide an apparatus for extracting three-dimensional structure information in a multi-view stereoscopic television system in which the occlusion area is reduced without increasing the number of cameras, and thus the accuracy of depth estimation is improved. To provide.

[0008]

SUMMARY OF THE INVENTION In order to achieve the above object, a multi-view stereoscopic television system according to the present invention is provided.
In a conventional three-dimensional structure information extraction system that extracts three-dimensional structure information from a multi-view stereoscopic television image signal, a device that extracts three-dimensional structure information has a fixed positional relationship between all cameras (the position of each camera). Is fixed), it is difficult to take measures against the above-mentioned problems related to the occlusion area. However, originally, the occlusion area is generated by the relationship between the depth of the subject and the position of the camera, so based on the idea that the occlusion area can be reduced by moving the position of the camera. It's something I've done.

That is, an apparatus for extracting three-dimensional structure information in a multi-view stereoscopic television system according to the present invention uses each image taken by one of a plurality of reference cameras and a reference camera. Depth amount calculation means for calculating the depth amount by sequentially calculating the reliability of the depth amount calculated by the depth amount calculation means, by comparing the previously calculated maximum reliability with the latest calculated reliability A depth amount reliability calculating means for outputting a depth amount corresponding to the maximum reliability among the calculated reliability,
And the depth amount reliability calculating means includes a camera driving means for moving the position of any one of the plurality of reference cameras every time the calculation of the reliability is completed. Is what you do.

Further, in the apparatus for extracting three-dimensional structure information in a multi-view stereoscopic television system according to the present invention, the depth amount reliability calculating means may include any one of the plurality of reference cameras and a reference camera. Calculating a linear sum of the evaluation function value of the parallax amount estimated using each image captured by each of the images and the continuity of the depth amount, and using the linear sum as an index representing the reliability of the depth amount. It is characterized by the following.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below based on embodiments of the present invention with reference to the accompanying drawings. FIG.
Represents 3 in the multi-view stereoscopic television system according to the present invention.
One embodiment of an apparatus for extracting dimensional structure information is shown in a block diagram. In the present embodiment, three cameras are used. In FIG. 1, 1 is a reference camera, 2-1 is a reference camera A, 2-2 is a reference camera B, 3
Is a parallax amount estimating unit, 4 is a depth amount calculating unit, 5 is a depth amount reliability calculating unit, 6 is an evaluation function calculating unit, and 7 is a camera driving unit.

FIG. 2 shows a position where each camera is arranged. In FIG. 1, reference cameras A and B shown by reference numerals 2-1 and 2-2 are shown by reference numeral 1 in FIG. This indicates that the camera is at an initial position (that is, a position before moving) indicated by b ₁ and b ₂ from the reference camera. Here, the amount of parallax is estimated with integer pixel precision, and only the amount of parallax in the horizontal direction is to be processed.
The optical axes of the three cameras (reference camera, reference camera A and reference camera B) are all parallel, and the plane including the lens centers of all the cameras is orthogonal to the optical axis.

The operation will be described with reference to FIGS. First, in the parallax estimating unit 3 (see FIG. 1), each pixel of the image of the reference camera (indicated by reference numeral 1 in FIGS. 1 and 2) is referred to by the reference camera A (indicated by reference numeral 2-1 in FIGS. 1 and 2). ) Is estimated at the initial position from the reference camera to the reference camera. The block matching method is used to estimate the amount of parallax d. The block matching method is a known method that is widely used to obtain a disparity vector and a motion vector, and a vector amount is detected for each block.

FIGS. 3A and 3B illustrate the block matching method, in which FIG. 3A shows an image one frame before, and FIG. 3B shows an image at the present time, respectively, from the image shown in FIG. The movement amount including the direction to is the motion vector, and corresponds to the disparity vector in this case. In FIG. 3, it now has a function as a matching window,
The template shown in FIG. 3A is a template divided into L pixels as one block, and this template is moved within an appropriate search range (see FIG. 3A), and a similar block is obtained as shown in FIG. Is searched for in the image shown in. As a result, a shift when the error is minimized between the two images is obtained by Expression (1), and the obtained shift is calculated by a parallax d from the reference camera A to the reference camera (a parallax between the two cameras). It is estimated that there is. Note that the appropriate search range is a range in which pixel values of two camera images of the reference camera A and the reference camera can be used.

(Equation 1) Where, f (n): pixel value of reference camera image g (n): pixel value of reference camera image L: number of pixels in matching window

In the same manner, an estimated amount of parallax from the reference camera B indicated by reference numeral 2-2 in FIGS. 1 and 2 to the reference camera indicated by reference numeral 1 in FIGS. As described above, the parallax amounts d (here, both represented by d) from the reference camera A to the reference camera and from the reference camera B to the reference camera are estimated.
These estimated parallax amounts d are shown in FIGS.
And 2-2 are the values when both reference camera A and reference camera B are at the initial position.

The obtained estimated amount of parallax (referred to as the initial estimated amount of parallax corresponding to the initial position) d is supplied to the depth amount calculator 4 shown in FIG. 1, and the estimated amount of parallax is calculated by the following equation (2). the equation (2) is a formula for the depth amount with respect to the reference camera a, with respect to the reference camera B, comprising a camera position indicated by b ₁ (see FIG. 2) to that replaced the camera position b _2. ) Is converted to the depth amount Z. This converted depth is referred to as an initial depth.

(Equation 2) Where: f: focal length of the lens

FIG. 4 is a flowchart showing a process of signal processing by a device for extracting three-dimensional structure information in a multiview stereoscopic television system according to the present invention. In FIG. 4, in step 1 (S1), images picked up by one selected reference camera and reference camera are input, and in step 2 (S2), the amount of parallax d is estimated, and then in step 3 (S3). ), The amount of parallax d is converted (calculated) into the amount of depth Z. This is what has been described above. Up to this point, reference camera A and reference camera B
Are in the initial position (not moving), and are in a stage where the initial parallax amount d and the initial depth amount Z are obtained.

Next, the initial depth Z calculated as described above
Is supplied to the depth amount reliability calculation unit 5 shown in FIG. 1 to calculate the reliability of the initial depth amount Z (an index indicating the depth amount reliability shown in step 4 (S4) in FIG. 4). R calculation). The depth amount reliability calculator 5 first verifies the continuity of the depth amount Z because it is known that the reliability of the depth amount is high if the continuity of the depth amount is good. With respect to the depth Z of each pixel, the sum of the absolute values of the differences from the eight peripheral pixels is calculated by equation (3), and this is defined as an index D representing the continuity of the depth. The greater the value of the index D representing the continuity of the depth amount, the lower the continuity of the depth amount, which means that the reliability of the depth amount is lower. For this reason, pixels for which the index D indicating the continuity of the depth amount does not satisfy the predetermined threshold ε are set as the targets of the re-estimation of the parallax.

(Equation 3) Here, Z (n): Depth amount at pixel n i: Number of peripheral pixels W: Size of continuity verification window

Further, the depth amount reliability calculating section 5 calculates an index R indicating the reliability of the depth amount by using the index D indicating the continuity of the depth amount according to equation (4). R = E + αD (4) where, E: evaluation function α: constant

In the above equation (4), the evaluation function E is, for each of the reference camera A and the reference camera B, a luminance value of a pixel in a matching window (template shown in FIG. 3) of an image taken by each camera. The evaluation function E, which is a value obtained by adding the sum of the square errors, is calculated by the equation (5). Evaluation function calculator 6 shown in FIG.
Is for performing this operation (operation of the expression (5)).

(Equation 4) Here, k: number of the reference camera (two in this embodiment) d: amount of parallax corresponding to the position b ₁ of the reference camera

The larger the value of the evaluation function E, the larger the difference between the luminance values of the reference camera A or B and the reference camera, and the lower the correlation between the corresponding points obtained from the luminance, the lower the correlation. The reliability of the pixel value is low. In any case, the greater the value of the index R representing the reliability of the depth amount, the lower the reliability.

The above is a description of a plurality of reference cameras (in this embodiment,
Are the reference cards indicated by reference numerals 2-1 and 2-2 in FIGS.
Camera A and reference camera B) are all in their initial positions,
Chib ₁ And b_Two (See Figure 2)
Was. Next, the positions of a plurality of reference cameras are moved (this embodiment
As for b, shown in FIG.₁ And b_Two Change the length of
Each camera output image
For images, do the same as above.

Now, the reference camera A is b ₁ and the reference camera B is
For but b ₂ (see FIG. 2) in which initial positions of the cameras is calculated by the initial depth amount (2), also, the index R which represents the depth of reliability, the depth amount reliability calculation unit 5
Suppose that it was calculated by the equation (4) in (see FIG. 1). The depth amount reliability calculation unit 5 stores the calculated value of the depth amount reliability in a built-in memory, and sends one of the reference camera A and the reference camera B (for calculating the depth amount) to the camera driving unit 7 shown in FIG. An instruction to change the camera position is given, such as making the position of the related reference camera smaller or larger in a three-dimensional space where the reference camera has an imaging area common to the imaging area of the reference camera (see FIG. 4). Step 7 (S7) of the flowchart.

The value of the index R, which is driven by the camera driving unit 7 based on the instruction and indicates the reliability of the depth of the obtained new position of the camera, is obtained. Step 5 (S
In 5), the initial depth amount is compared with the value of the index R representing the depth amount reliability held in the built-in memory.

As described above, when the position of the camera is sequentially changed, the index R indicating the highest depth amount reliability before the change is obtained.
Is compared with the value of the index R representing the depth amount reliability after the change, and when the value of the index R representing the depth amount reliability finally takes the lowest value (at this time, the depth amount reliability Is determined to be a high-precision depth amount to be extracted by the present invention, that is, three-dimensional structure information, the depth amount update is completed, and the depth amount is output from the depth amount reliability calculation unit 5 shown in FIG. (Step 6 (S
6)).

[0026]

According to the present invention, by making the reference camera movable in the multi-view stereoscopic television system, the error of the parallax estimation caused by the occlusion area when the reference camera is fixed is reduced by the number of cameras. It is reduced without increasing, and thereby, the extracted three-dimensional structure information can be made more accurate.

According to the present invention, a device for extracting high-precision three-dimensional structure information can be realized with an extremely small number of cameras, and the scale of the device can be reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of an apparatus for extracting three-dimensional structure information in a multi-view stereoscopic television system according to the present invention.

FIG. 2 shows a position where each camera (reference camera, reference camera A and reference camera B) is arranged.

FIG. 3 illustrates a block matching method.

FIG. 4 is a flowchart illustrating a signal processing process performed by a device for extracting three-dimensional structure information in a multiview stereoscopic television system according to the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 reference camera 2-1 reference camera A 2-2 reference camera B 3 parallax amount estimating unit 4 depth amount calculating unit 5 depth amount reliability calculating unit 6 evaluation function value calculating unit 7 camera driving unit

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Yasuaki Kinji 1-10-11 Kinuta, Setagaya-ku, Tokyo Japan Broadcasting Corporation Broadcasting Research Institute F-term (reference) 2F065 AA04 AA53 DD02 FF05 JJ03 JJ05 JJ26 MM23 QQ13 QQ27 QQ29 QQ36 QQ38 QQ41 2H059 AA10 5B057 BA11 DA20 DB02 DC30 5C061 AB04 AB08

Claims (2)

[Claims] 1. An apparatus for extracting three-dimensional structure information in a multi-view stereoscopic television system, wherein each of the three-dimensional structure information is extracted by using an image taken by one of a plurality of reference cameras and a reference camera. Depth amount calculation means for calculating the depth amount, by sequentially calculating the reliability of the depth amount calculated by the depth amount calculation means, by comparing the previously calculated maximum reliability with the latest calculated reliability, Depth amount reliability calculating means for outputting a depth amount corresponding to the maximum reliability among the calculated reliability, and in the depth amount reliability calculating means, each time the calculation of the reliability is completed, the plurality of references Extracting three-dimensional structure information in a multi-view stereoscopic television system, comprising camera driving means for moving the position of any one of the cameras That equipment. 2. The extracting device according to claim 1, wherein
The depth amount reliability calculating means includes an evaluation function value of a parallax amount estimated using each image captured by each of the plurality of reference cameras and a reference camera, and a depth amount An apparatus for extracting three-dimensional structure information in a multi-view stereoscopic television system, wherein a linear sum with continuity is obtained and the linear sum is used as an index indicating the reliability of the depth amount.