JP3525712B2 - Three-dimensional image capturing method and three-dimensional image capturing device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image capturing method and apparatus for acquiring a three-dimensional image composed of luminance information and distance information, and more particularly, to a measuring object ( The present invention relates to a three-dimensional image capturing method and a three-dimensional image capturing apparatus capable of taking measures against occlusion and improving the accuracy of distance information to be measured even when an object to be measured) has a relatively complicated shape.

[0002]

2. Description of the Related Art As a method for measuring the shape of a target object, a passive method typified by stereo vision and an active method by irradiating the target object with light and detecting the reflected light
(Laser radar, active stereo, moire, interference) exist. In general, the active method has higher measurement accuracy, but the limit of the light projecting means that irradiates light determines the measurable range within the range of the power of the laser, for example. Often cannot be measured. On the other hand, the passive method is versatile and has the advantage that there are few restrictions on the target object.

Among passive methods, shape measurement by stereoscopic vision is known as a method with relatively high accuracy.
Shape measurement by stereo vision applies the principle of triangulation,
For example, as shown in FIG. 4, one parallax image (luminance image) serving as luminance information from two parallax image pairs at different angles of view of observation systems A and B obtained by imaging a measured object from different positions. The distance value from the image pickup point at each pixel is calculated.

According to the above-described stereoscopic shape measurement, the occlusion problem occurs because the optical axis of the observation system for obtaining each parallax image is different. This is especially troublesome when the brightness value and the distance value are in one-to-one correspondence. That is, as shown in FIG. 4, a phenomenon (occlusion) occurs in which a region visible at the angle of view of the observation system A that obtains the brightness information of the measured object 40 cannot be seen at the angle of view of another observation system B. However, the brightness value and the distance value cannot be made to correspond to each pixel in the brightness information in a one-to-one relationship, and as a result, some pixels do not have the distance information.

In order to deal with such an occlusion problem, as a method of measuring distance information when an object to be measured having an occlusion area has a relatively complicated shape, the object to be measured is measured from a plurality of different positions. There has been proposed a method of capturing a plurality of parallax images and repeating a distance calculation process to calculate the distance of an occlusion part.

That is, according to this method, a continuous stereo image of the object to be measured is created from a plurality of parallax images obtained by imaging the object to be measured from a plurality of different positions. Then, the continuous stereo image is decomposed into directional components to calculate the depth of a portion without occlusion, and the depth is calculated again from the continuous stereo image excluding the depth determined portion. By repeating the processing, the depth of the occlusion portion is calculated. Depth is calculated by scanning a continuous stereo image with straight lines of various inclinations, obtaining the variance on that straight line, and if this variance is less than a certain reference value, the depth of the object corresponds to the direction (tilt) of that straight line. Is considered to exist (Japanese Patent Laid-Open No. 6-74).
723).

[0007]

However, in the above method, when calculating the depth, it is possible to create a continuous stereo image from all the plurality of picked-up parallax images without selecting the parallax images in the region where occlusion does not occur. It is adopted. In this case, depending on the adopted parallax image pair, there is a possibility that the distance measurement accuracy may be degraded. The reason for this will be described below.

That is, as shown in FIG.
In stereo vision (stereo image pair) composed of a pair of parallax images obtained by imaging from different positions via
The length of one side of the pixel size of the captured image is λ, the baseline length that is the interval between the lens center positions in each parallax image is B,
When the focal length of the lens is F, the parallax d which is the pixel difference between the corresponding pixels in the stereo image pair and the measured object 4
The relationship with the distance value z from the surface where 0 is located to the center of the lens 50 can be expressed by Expression 1.

[0009]

[Formula 1] d = B · F / λ · z

The parallax d (pixel difference) is directly measured from the stereo image pair, and when the base line length B is smaller than the formula 1, the parallax d itself becomes small and the error generated in the parallax becomes large. Therefore, in the above-described continuous stereo image creation, it is expected that the distance information (depth) by the straight line calculated using the parallax that may have a large error will naturally have a large error. In FIG. 5, the image pickup image plane is positioned in front of the lens 50, but in the actual optical system, the image plane (image pickup plane) is formed on the rear side on the image formation center which is symmetrical with respect to the lens 50. Surface) exists.

The present invention has been made in view of the above circumstances, and provides a three-dimensional image capturing method and a three-dimensional image capturing apparatus capable of avoiding the occurrence of occlusion as much as possible and improving the measurement accuracy of distance information. The purpose is to

[0012]

In order to achieve the above object, the invention of claim 1 is characterized in that in a three-dimensional image capturing method for measuring luminance information and distance information for an object to be measured, the three-dimensional image capturing method is performed in the following procedure. There is. First, a plurality of parallax images are acquired by photographing the measured object from a plurality of image pickup positions. Next, one of the parallax images is used as luminance information, and it is necessary for stereoscopic viewing from the image pair of the reference parallax image that is the luminance information and the first parallax image at the imaging position closest to the imaging position. The first corresponding pixel search is performed, and the first
In the range where occlusion does not occur, each distance from the image pickup position for each pixel of the luminance information is calculated from the parallax of the image pair and stored as distance information. Corresponding pixel search is performed again from the image pair of the reference parallax image and the second parallax image at the image pickup position closest to the image pickup position to obtain the second pixel.
The respective distances corresponding to the respective pixels are calculated in the range where the occlusion does not occur, and the distance information rewriting process for rewriting the previously stored distance information is performed for the pixels for which the distance can be calculated. Regarding this distance information rewriting process, the process is sequentially repeated for the image pairs of the reference parallax image and each parallax image having the third closest image pickup position and thereafter. And the first
Pixels in the range where occlusion occurs are stored as occlusion pixels without distance information.

According to a second aspect of the present invention, in the three-dimensional image capturing method according to the first aspect, the reference parallax image is a parallax image at an intermediate position with respect to the object to be measured among a plurality of parallax images. Is characterized by.

According to a third aspect of the present invention, in the three-dimensional image capturing method according to the first aspect, the corresponding pixel search in the distance information rewriting process is based on the distance information stored by the distance information rewriting process up to immediately before. It is characterized by doing.

According to a fourth aspect of the present invention, in the three-dimensional image capturing method according to the first aspect, for a pixel in the range in which the first occlusion occurs, distance information of the nearest pixel to this pixel is obtained. It is characterized by allocating.

According to a fifth aspect of the present invention, there is provided a three-dimensional image pickup device for measuring luminance information and distance information with respect to an object to be measured, which is characterized by having the following respective configurations. Imaging unit. The image capturing section captures the object to be measured from a plurality of image capturing positions, and can acquire a plurality of parallax images. Corresponding pixel searching means. The corresponding pixel search means uses one of the plurality of parallax images as luminance information, sequentially selects image pairs of the reference parallax image and each of the other parallax images to be luminance information, and searches for a corresponding pixel required for stereoscopic vision. Is to do. Distance information calculation means. The distance information calculation means calculates the distance from the image pickup position in the pixel in the reference parallax image from the parallax of each image pair. Distance value memory. The distance value memory stores the distance information about the pixels whose distances can be calculated by the distance information calculating means. Occlusion area processing section. The occlusion area processing section stores pixels whose distances cannot be calculated by the distance information calculating means as occlusion pixels. Three-dimensional image memory. This three-dimensional image memory stores three-dimensional information by associating distance information or occlusion pixels with each pixel of the reference parallax image. Distance information rewriting means. This distance information rewriting means is based on the distance information of each pixel of the three-dimensional image memory, based on the pixel pair having the maximum baseline length in image pickup of each image pair which is the basis of distance calculation by the distance information calculating means. It is rewritten into distance information.

According to a sixth aspect of the present invention, in the three-dimensional image pickup device according to the fifth aspect, a distance value assignment processing means is provided for assigning the distance value of the nearest pixel to the occlusion pixel. I am trying.

According to the present invention, the first corresponding pixel search necessary for stereoscopic viewing is performed from the image pair of the reference parallax image and the first parallax image at the image pickup position closest to the image pickup position. Occurrence of occlusion can be avoided as much as possible by stereo vision with a short reference length.

Further, since the corresponding pixel search is performed a plurality of times from the image pair of the reference parallax image and the parallax image, and the distance information is rewritten, the distance information is rewritten to the stereoscopic distance information having a long reference length. It enables improvement of information measurement accuracy.

By making the reference parallax image a parallax image at an intermediate position with respect to the object to be measured in the plurality of parallax images, the range in which occlusion occurs can be reduced.

By performing the corresponding pixel search in the distance information rewriting process on the basis of the distance information stored by the distance information rewriting process up to immediately before, the processing speed can be increased.

For the pixels in the range in which the first occlusion occurs in the first corresponding pixel search, by assigning the distance information of the nearest pixel to this pixel, the rounding of the edge is avoided and the edge is rounded. You can save.

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION An example of an embodiment of a three-dimensional image pickup device and a three-dimensional image pickup method of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an OA camera as a three-dimensional image capturing device, and FIG. 2 is a flow chart for explaining the operation of the three-dimensional image capturing device.

The three-dimensional image pickup device is a device for measuring luminance information and distance information for an object to be measured, and an image pickup section 1 for picking up an image of the object to be measured to obtain an image, and the image pickup for obtaining a plurality of parallax images. Imaging unit moving mechanism unit 2 for moving the unit 1 in parallel within a certain distance range, and imaging unit moving mechanism unit 2
Image pickup position memory 3 that stores each image pickup position, a luminance value memory 4 that stores each parallax image, an output luminance value memory 5 that selectively stores one of the parallax images as a reference parallax image, and a reference parallax image. Corresponding pixel searching unit (corresponding pixel searching means) searching for a corresponding pixel in the image pair of the image and the parallax image.
6, a distance information calculation unit (distance information calculation means) 7 that calculates distance information of each pixel of the reference parallax image from the image pair, a distance value memory 8 that stores the distance information, and occlusion in the reference parallax image from the image pair. An occlusion area processing unit 9 that detects and processes pixels in the area; a three-dimensional image memory 10 that stores luminance information and distance information of the measured object;
It is configured to include.

The image pickup unit 1 linearly step-moves within a fixed distance range by the image pickup unit moving mechanism unit 2 and picks up an image of an object to be measured from different positions for each step, thereby obtaining a plurality of parallax images. I'm supposed to get it. In the corresponding pixel search unit 6, the output brightness value memory 5
The image pairs of the reference parallax image and the other parallax images selected as the luminance information in 1 are sequentially selected, and a corresponding pixel required for stereoscopic viewing is searched.

A combination of this reference parallax image and each of the other parallax images performs corresponding pixel search for various image pairs having a short baseline length and a long baseline length, so that an occlusion occurrence area is small. Distance information (from an image pair with a short baseline length) to one with a high distance value measurement accuracy (from an image pair with a long baseline length) can be obtained. That is, according to an image pair with a short baseline length (close to the optical axis), the angle of view in imaging is almost the same, so most of the pixels in the captured images (reference parallax image and parallax image) exist in both images, and Occurrence of occlusion in which only pixels are obtained can be avoided. Further, according to the image pair having a long base line length, the value of the parallax in Expression 1 described above becomes large, and it is possible to reduce the error due to the parallax and improve the measurement accuracy of the distance value.

In the output luminance value memory 5, the distance information can be added to each pixel of the luminance information, in other words, the occlusion area is not generated as much as possible, and the intermediate position is set with respect to the measured object. The image at the imaging position is selected as the reference parallax image (luminance information). That is, if the reference parallax image is an image obtained at an imaging position that is an intermediate position with respect to the object to be measured, and if a stereo image pair consisting of a parallax image with a slightly different angle of view from this image is taken, the object to be measured is Can be stereoscopically imaged from almost the front, and the occurrence of an occlusion area can be minimized.

The distance information calculation unit 7 calculates the distance from the image pickup position in the pixel of the luminance image information from the image pair of the reference parallax image and each parallax image. In this case, in the image pair of the reference parallax image and each parallax image, there is no corresponding pixel in the portion in which the occlusion area is generated, and thus the distance cannot be calculated. Then, in the distance value memory 8, the distance information is stored for the pixels whose distances can be calculated by the distance information calculation unit 7, and the pixels whose distances cannot be calculated by the distance information calculation unit 7 are stored as occlusion pixels. Further, the occlusion area processing unit 9 is provided with distance value allocation processing means, and the distance value memory 8
The distance value of the nearest pixel is assigned to each of the pixels that have been determined to be occlusion pixels.

The three-dimensional image memory 10 stores three-dimensional information in which distance information or occlusion pixels are associated with each pixel of the reference parallax image as the luminance image information. The distance information of each pixel of the three-dimensional image memory 10 can be rewritten into the distance information of the pixel pair that maximizes the baseline length in the image pickup of each image pair that is the basis of the distance calculation in the distance information calculation unit 7. Is configured.

Next, an image pickup method using the above three-dimensional image pickup device will be described with reference to FIGS. When the start of image pickup is selected by the switch (not shown) of the three-dimensional image pickup device, the image pickup unit moving mechanism unit 2 makes a step movement in which the image pickup unit 1 is repeatedly stopped and moved (step 101). Image pickup is performed at each position where the image pickup unit 1 is stopped (step 102), a plurality of parallax images are acquired, and the obtained luminance images are stored in the luminance value memory 4 (step 103) repeatedly. (Step 10
4). The positions at which the image pickup unit 1 stops are set at several tens of places at equal intervals within a fixed straight line distance in which the image pickup unit 1 can move in parallel. Therefore, the brightness value memory 4 stores the brightness information of each pixel of each parallax image.

Next, an image used as luminance image information is selected as the reference parallax image from the plurality of parallax images (step 105). As described above, this reference parallax image is captured at the center position with respect to the measured object, that is, the parallax image captured at the intermediate stop position of the moving imaging unit 1 so that the occlusion area is not generated as much as possible. Use the image as a reference parallax image. The brightness value information of each pixel of the selected reference parallax image is stored in the output brightness value memory 5 (step 106).

Then, in the corresponding pixel searching unit 6, the parallax image taken at the position closest to the image pickup position where the reference parallax image is picked up is selected based on the data of the image pickup position memory 3, and the parallax image and the luminance image information are selected. Forming a first stereo image pair with the reference parallax image selected as (step 1
07). In this case, since the optical axes of the reference parallax image and the parallax image are located at the closest positions, the occurrence of occlusion regions can be minimized.

Then, in this stereo image pair, a corresponding pixel search for searching a corresponding pixel is performed (step 1).
08). When searching for the corresponding pixel, since the optical axes of the images are close to each other, the corresponding pixel of the parallax image exists at a position close to the pixel position of the reference parallax image, and thus the search area for searching the corresponding pixel can be set small. . As a corresponding pixel search method, a window of a certain size is provided in the vicinity of the target pixel, and the pixel having the smallest difference in the luminance value of the pixel in this window, that is, the pixel that minimizes Dw shown in Expression 2 is selected. It can be considered as a corresponding pixel.

[0034]

[Formula 2]

In equation (2), W is the set correlation window size, I
ref is the brightness value of the pixel in the reference parallax image of the stereo image pair, and Is is the brightness value of the pixel in the parallax image forming the stereo image pair. When the corresponding pixel can be searched, the distance information (distance value z) is calculated by Equation 3 (step 109). Expression 3 is a modification of Expression 1 and is the same as Expression 1 regarding notation. This operation is performed for all the pixels of the reference parallax image.

[0036]

[Formula 3] z = B · F / λ · d

The corresponding pixel in the stereo image pair can be searched for, and the distance value of each pixel for which the distance value is calculated by the equation 3 is
It is stored in the distance value memory 8 (step 110). When the distance value cannot be calculated by the occlusion, the information that the pixel is an occlusion pixel is stored in the distance value memory 8.

Next, except for the parallax image used as one of the stereo image pairs in the previous step, the parallax image taken at the position closest to the image pickup position at which the reference parallax image is taken is taken as the parallax image of the second stereo image pair. (Step 11
1). Since the distance value (the distance value overwritten in step 115 described later) stored in advance in the distance value memory 8 has been calculated in the previous step, the expected value of the parallax d can be calculated by the equation 1 (step 11
2). By using the parallax d to search for the corresponding pixel (step 113), the search time can be shortened (the processing time can be shortened).

Distance information of each pixel of the reference parallax image is calculated from the corresponding pixels in the stereo image pair (step 114).
The procedure is the same as in the previous step, but the baseline length of the second stereo image pair is longer than the baseline length of the first stereo image pair in the previous step, so it is possible to obtain distance information with higher measurement accuracy. Becomes On the other hand, the number of pixels whose distance value cannot be calculated increases due to the occurrence of occlusion. Then, regarding the pixels for which the distance value can be calculated by the second stereo image pair, regarding the pixels already calculated by the first stereo image pair and written in the distance value memory 8,
Data is overwritten with the newly calculated distance value (step 115).

Thereafter, by the same procedure, each stereo image pair is set for all the picked-up parallax images, and the overwriting operation is performed by the distance value information calculated in each step, whereby the first stereo image pair is set. For each pixel of the reference parallax image in the region where the occlusion due to No. occurs, the distance information by the stereo image pair having the longest baseline length can be obtained (step 116).

Even after forming a stereo image pair using all parallax images and calculating distance information at each pixel,
Occlusion may occur depending on the imaging target, that is, occlusion pixels may be stored in the distance value memory 8 as occlusion pixels. Such a region is considered to be an edge region (see the occlusion region in FIG. 1) that can be seen from the position where the reference parallax image (luminance image information) is captured.

A method of processing these occlusion pixels will be described. As shown in FIG. 4, the background object 41
If the measured object 40 exists in front of, the actual distance value (depth information) in a horizontal plane cross section of the measured object
Is as shown in FIG. Here, the shaded area corresponds to the occlusion area. When the distance value obtained by taking the weighted average from the surrounding pixels is made to correspond to the pixel in the occlusion area, unnecessary "rounding" of the edge occurs as shown in FIG. 5B. When such a “blurring” occurs, the edge of the boundary between the background image and the object image can be detected when trying to separate the image of the measured object 40 from the image of the background object 41 using the distance information. The separation becomes difficult.

Therefore, in the occlusion area processing unit 9 of the three-dimensional image pickup apparatus of the present invention, the distance value of the nearest pixel is assigned to the pixels in the occlusion area (step 118) (step 119). As a result, the profile of the distance information becomes as shown in FIG. In this case, the distance information of each pixel itself may have a large error, but the edge portion is preserved, and the image of the measured object can be easily separated from the background image.

When the processing of the occlusion area is completed,
A distance value is obtained for each pixel of the reference parallax image (luminance information image) (step 117), and the information of the luminance value and the distance value is stored in the three-dimensional image memory as a three-dimensional image having each pixel (step 120). Then, the operation ends.

In the above-described three-dimensional image pickup device, the image pickup unit 1 is stepwise moved in parallel by the moving mechanism as a means for obtaining a plurality of parallax images to obtain parallax images at different image pickup positions. A plurality of parallax images may be acquired by installing the optical system and capturing an image.

As the brightness information used when searching for the corresponding pixel, infrared reflected light obtained by an image pickup device having sensitivity in the infrared region other than the normal R, G, B signals used for the brightness information image. You may use the information. Since different materials generally have different reflection spectra, it is considered that the corresponding pixel search can be performed by using a plurality of spectra even when the corresponding point search is impossible with normal visible light.

Further, when the corresponding pixel is searched, the corresponding pixel is detected by minimizing the difference between the brightness values in the correlation window. However, the characteristic amount is calculated by detecting the straight line component by Hough transform or the like. Pixels may be detected.

According to the above-described three-dimensional image capturing method, a plurality of parallax images are collectively acquired and the corresponding pixel search and distance value calculation are first performed. It is also possible to reduce the capacity of the brightness value memory by performing image pickup from different viewpoints after performing the image pickup in step 1 to obtain the reference parallax image, and performing the corresponding pixel search and the distance value calculation each time.

[0049]

According to the present invention, a plurality of parallax images are acquired for an object to be measured, and a distance value at each pixel of the reference parallax image is calculated by a stereo image pair of the reference parallax image and each parallax image. By doing so, the distance value can be calculated from the stereo image pair having the maximum base line length in the range where occlusion does not occur, so that it is possible to obtain distance information with high measurement accuracy while avoiding occlusion. For pixels in the area where occlusion cannot be avoided,
By assigning the distance information of the nearest pixel to the pixel, the edge can be preserved and the object image can be easily separated from the background image using the distance information.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an OA camera as an example of implementation of a three-dimensional image capturing apparatus of the present invention.

FIG. 2 is a flow chart diagram for explaining the operation of the three-dimensional image pickup device.

3 (a), (b) and (c) are distance information explanatory diagrams in a horizontal section with respect to the measured object for explaining a method of adding a distance value to a pixel in which occlusion cannot be avoided.

FIG. 4 is an imaging plane explanatory diagram for explaining occlusion that occurs in an object to be measured in stereoscopic view.

FIG. 5 is an optical relationship explanatory diagram showing a relationship between a parallax and a base line length and distance in stereoscopic vision.

[Explanation of symbols]

1 ... Image pickup unit, 2 ... Image pickup unit moving mechanism unit, 3 ... Image pickup position memory, 4 ... Luminance value memory, 5 ... Output luminance value memory, 6 ... Corresponding pixel searching unit (corresponding pixel searching unit), 7
... distance information calculation unit (distance information calculation means), 8 ... distance value memory, 9 ... occlusion region processing unit, 10 ... three-dimensional image memory

Front page continuation (58) Fields surveyed (Int.Cl. ⁷ , DB name) G01B 11/00-11/30 G01C 3/00-3/32 G06T 1/00 G06T 7/00

Claims (6)

(57) [Claims] 1. A three-dimensional image capturing method for measuring luminance information and distance information for an object to be measured, wherein the object to be measured is imaged from a plurality of imaging positions to obtain a plurality of parallax images, and one of the parallax images is acquired. As the luminance information, and the first corresponding pixel search required for stereoscopic vision from the image pair of the reference parallax image serving as the luminance information and the first parallax image at the imaging position closest to the imaging position. In the range where the first occlusion does not occur, each distance from the image pickup position for each pixel of the luminance information is calculated from the parallax of the image pair and stored as distance information. Corresponding pixel search is performed again from the image pair with the second parallax image at the nearest imaging position, and each distance corresponding to each pixel is calculated in the range in which the second occlusion does not occur, For the pixels for which the distance has been calculated, the distance information rewriting process for rewriting the already stored distance information is performed, and for this distance information rewriting process, the parallax images of which the proximity of the reference parallax image and the imaging position is the third or later The three-dimensional image capturing method is characterized in that the processing is sequentially repeated for the image pair of and the pixels in the range in which the first occlusion occurs are stored as occlusion pixels without distance information. 2. The three-dimensional image capturing method according to claim 1, wherein the reference parallax image is a parallax image at an intermediate position with respect to the measured object among the plurality of parallax images. 3. The three-dimensional image capturing method according to claim 1, wherein the corresponding pixel search in the distance information rewriting process is performed based on the distance information stored by the distance information rewriting process up to immediately before. 4. The three-dimensional image capturing method according to claim 1, wherein, for a pixel in a range in which the first occlusion is occurring, distance information of a pixel closest to the pixel is assigned. 5. An apparatus for measuring luminance information and distance information for an object to be measured, wherein the object to be measured is photographed from a plurality of image pickup positions, respectively.
An imaging unit capable of acquiring a plurality of parallax images, one of the plurality of parallax images as luminance information, and an image pair of the reference parallax image and each of the other parallax images to be the luminance information are sequentially selected for stereoscopic viewing. Corresponding pixel searching means for searching for a required corresponding pixel, distance information calculating means for calculating a distance from an image pickup position in a pixel in the reference parallax image from parallax of each image pair, and distance in the distance information calculating means A distance value memory that stores distance information about pixels that can be calculated, an occlusion area processing unit that stores pixels whose distances could not be calculated by the distance information calculation means as occlusion pixels, and for each pixel of the reference parallax image. A three-dimensional image memory for storing three-dimensional information in association with distance information or occlusion pixels, and distance information for each pixel of the three-dimensional image memory. And a distance information rewriting unit that rewrites the distance information by the pixel pair that maximizes the baseline length in the image pickup of each image pair, which is the basis of the distance calculation by the distance information calculation unit. Image capturing device. 6. The three-dimensional image pickup device according to claim 5, further comprising a distance value assignment processing means for assigning a distance value of the nearest pixel to the occlusion pixel.