JP2021051347A - Distance image generation apparatus and distance image generation method - Google Patents

Abstract

To generate a distance image having an advantage of a distance image based on an image captured by a stereo camera and an advantage of a distance image based on an image captured by a monocular camera.SOLUTION: A distance image generation apparatus 1 includes: an image acquisition unit 121 that acquires a first distance image generated based on two captured images captured by an imaging apparatus 2 which captures a subject from at least two direction at the same time and a second distance image generated based on at least one of the two images; a reliability degree identification unit 122 that identifies reliability of distances corresponding to pixels regarding each of a plurality of first pixels constituting the first distance image and each of a plurality of second pixels constituting the second distance image; and a distance image generation unit 123 that generates a distance image by synthesizing pixel values based on the reliability of the first pixel and the reliability of the second pixel regarding the plurality of first pixels and the second pixels whose positions correspond to each other in the distance image.SELECTED DRAWING: Figure 2

Description

The present invention relates to a distance image generator and a distance image generation method.

Conventionally, based on an image captured by an imaging device, a plurality of pixels constituting the captured image generate a distance image indicating a distance between a subject corresponding to the pixel and the imaging device. For example, Patent Document 1 discloses that a distance image is generated based on a left and right image acquired by a stereo camera. Further, Patent Document 2 discloses that a distance image is generated based on an image captured by a monocular camera.

Japanese Unexamined Patent Publication No. 2018-139120 Japanese Unexamined Patent Publication No. 2019-0081460

The distance image generated by a rigorously calibrated stereo camera has the advantage of high accuracy of the estimated distance in the area where the distance could be estimated, but it is noisy and the left and right images are matched. There is a disadvantage that there is a region where the distance cannot be measured. On the other hand, the distance image generated by the monocular camera has the advantage of low noise and the distance is estimated for all pixels, but the accuracy of the estimated distance is lower than the distance estimated by the stereo camera. There is a disadvantage.

Therefore, the present invention has been made in view of these points, and generates a distance image having the advantages of a distance image based on an image captured by a stereo camera and the advantages of a distance image based on an image captured by a monocular camera. It is an object of the present invention to provide a distance image generation device and a distance image generation method capable of generating a distance image.

The distance image generator according to the first aspect of the present invention is a plurality of pixels constituting the image, each of which is generated based on two images captured by an image pickup device that simultaneously captures a subject from at least two directions. The first distance image, which is a distance image in which the distance between the subject corresponding to the above and the image pickup device is estimated, is acquired, and the second distance image, which is the distance image generated based on one of the two images. An image acquisition unit that acquires a distance image, each of a plurality of first pixels that are a plurality of pixels constituting the first distance image, and a plurality of second pixels that are a plurality of pixels constituting the second distance image. For each of the first and second pixels, the reliability specifying unit that specifies the reliability of the distance corresponding to the pixel, and the plurality of first pixels and the second pixels whose positions in the distance image correspond to each other are the first. It includes a distance image generation unit that generates a distance image by synthesizing pixel values based on the reliability of the pixel and the reliability of the second pixel.

The image acquisition unit inputs one of the two images to the deep learning model that outputs the second distance image in response to the input of the image captured by the imaging device, thereby inputting the deep learning model. The second distance image output from may be acquired.

The reliability specifying unit has the first pixel and the second pixel based on the difference between the distance corresponding to the first pixel corresponding to the position in the distance image and the distance corresponding to the second pixel. The reliability of each of the pixels may be specified.

The maximum value and the minimum value are set for the reliability, and the sum of the reliability of the first pixel and the reliability of the second pixel corresponding to the positions in the distance image is constant. The reliability specifying unit specifies the reliability of each of the first pixel and the second pixel so that the larger the absolute value of the difference is, the higher the reliability of the second pixel is. You may.

The first distance image includes pixels whose distance could not be estimated, and the reliability is set to a maximum value and a minimum value, and the positions in the distance image correspond to each other. The sum of the reliability of the first pixel and the reliability of the second pixel is constant, and the reliability specifying unit cannot estimate the distance of the first pixel in the first distance image. In the case of a pixel, the reliability of the first pixel may be the minimum value, and the reliability of the second pixel at a position corresponding to the first pixel may be the maximum value.

The distance image generation method according to the second aspect of the present invention constitutes the image generated based on two images captured by an imaging device that simultaneously captures a subject from at least two directions, which is executed by a computer. A first distance image, which is a distance image in which the distance between the subject corresponding to each of the plurality of pixels and the image pickup device is estimated, is acquired, and the distance generated based on one of the two images. A step of acquiring a second distance image which is an image, each of a plurality of first pixels which are a plurality of pixels constituting the first distance image, and a plurality of pixels which are a plurality of pixels constituting the second distance image. For each of the second pixels, the step of specifying the reliability of the distance corresponding to the pixel, and for each of the plurality of first pixels and the second pixel whose positions in the distance image correspond to each other, the first It includes a step of generating a distance image by synthesizing a pixel value based on the reliability of a pixel and the reliability of the second pixel.

According to the present invention, it is possible to generate a distance image having the advantages of a distance image based on an image captured by a stereo camera and the advantages of a distance image based on an image captured by a monocular camera.

It is a figure explaining the outline of the distance image generation apparatus which concerns on this embodiment. It is a figure which shows the structure of the distance image generation apparatus which concerns on this embodiment. It is a flowchart which shows the flow of processing in the distance image generation apparatus which concerns on this embodiment.

FIG. 1 is a diagram illustrating an outline of a distance image generation device according to the present embodiment. The distance image generator is, for example, a computer mounted on a vehicle such as a passenger car, a bus, or a truck. The distance image generation device is a device that is also mounted on the vehicle and generates a distance image based on the captured image acquired from the image capturing device that images the front of the vehicle.

Here, the image pickup apparatus is a stereo camera provided with a plurality of lenses and a plurality of imaging means, and simultaneously captures a subject from at least two directions. Further, the distance image is an image showing the distance between the subject corresponding to each of the plurality of pixels constituting the captured image and the imaging device. The pixel value of each of the plurality of pixels constituting the distance image indicates the estimated distance.

Distance image generating apparatus, as shown in FIG. 1, one of the images of the first distance image I ^S and, the two imaging imaging device is a distance image generated based on two captured image captured acquiring a second distance image I ^M is generated distance image based on. Distance image generating apparatus, respectively of a plurality of pixels constituting the first distance image I ^S, for the each of the plurality of pixels constituting the second distance image I ^M, specifying the reliability of the distance corresponding to the pixel ..

Distance image generating device includes a pixel of the first distance image I ^S a position in the range image is compatible, and the pixel of the second distance image I ^M, by synthesizing based on the specified reliability, range image to generate the I ^C. Distance image generating apparatus, the pixel value of the reliability is high pixel, and generates a range image I ^C, which is reflected strongly than the pixel value of unreliable pixels. Thus, the distance image generation unit may generate from the first distance image I ^S and the second distance image I ^M, range image I ^C to distance estimation accuracy is strongly reflects the pixel values of the high pixel.

Thus, the distance image generating apparatus, the first distance image I ^S, can be distances to reflect the pixel values of the pixels can be estimated to range image I ^C. The distance image generating apparatus, the first distance image I ^S, distance that could not be located and noise estimation does not reflect the pixel values of pixels corresponding to the position that occurred range image I ^C, corresponding to the position pixel value of the pixel in the second range image I ^M which can be reflected in the range image I ^C a. Therefore, the distance image generator can generate a distance image having the advantages of a distance image based on an image captured by a stereo camera and the advantages of a distance image based on an image captured by a monocular camera.
Subsequently, a detailed description of the distance image generator will be given.

[Configuration of distance image generator 1]
FIG. 2 is a diagram showing a configuration of a distance image generation device 1 according to the present embodiment. The distance image generation device 1 includes a storage unit 11 and a control unit 12.

The storage unit 11 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 11 stores various programs for operating the control unit 12. The storage unit 11 stores a distance image generation program that causes the control unit 12 to function as an image acquisition unit 121, a reliability specifying unit 122, a distance image generation unit 123, and an output unit 124.

The control unit 12 is a computing resource including a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) (not shown). The control unit 12 functions as an image acquisition unit 121, a reliability identification unit 122, a distance image generation unit 123, and an output unit 124 by executing a distance image generation program stored in the storage unit 11.

The image acquisition unit 121 captures a subject corresponding to each of a plurality of pixels constituting the captured image, which is generated based on the two captured images captured by the imaging device 2 that simultaneously captures the subject from at least two directions. acquiring a first distance image I ^S is a distance image obtained by estimating the distance between the device 2. The image acquisition unit 121 acquires a second distance image I ^M imaging apparatus 2 is a distance image generated on the basis of one image of the two captured image captured.

For example, the image acquisition unit 121 acquires the first captured image and the second captured image captured at the same time from the imaging device 2, and performs stereo matching of the first captured image and the second captured image. The first distance image ^IS is acquired. In stereo matching, for example, two pixels in which the same subject appears are specified by comparing the similarity between the periphery of each pixel included in the first captured image acquired at the same time and the periphery of each pixel included in the second captured image. Then, it is a method of estimating the distance to the subject corresponding to each pixel based on the magnitude of the parallax specified based on the positions of the two pixels. Specific methods of stereo matching include block matching and semi-global matching.

The generation of a distance image by stereo matching of a stereo camera calibrated with high accuracy has an advantage that the estimated distance accuracy is high for pixels whose parallax can be specified. On the other hand, of the two captured images captured by the imaging device 2, the subject reflected near the left end of the image captured by the imaging means located on the left side is not reflected in the captured image captured by the imaging means located on the right side. Therefore, the parallax cannot be specified for the pixel corresponding to the vicinity of the left end, and the distance cannot be estimated. Similarly, of the two captured images captured by the imaging device 2, the subject reflected near the right end of the image captured by the imaging means located on the right side is not reflected in the captured image captured by the imaging means on the left side. Therefore, the parallax cannot be specified near the right end, and the distance cannot be estimated.

Thus, pixels in the vicinity of the right end or near the left end of the first distance image I ^S, the distance is a pixel that can not be estimated. Further, even if the pixels are near the center, the parallax in the area not shown in one of the images cannot be specified. Similarly, the parallax can be specified by erroneously matching two pixels in which the same subject appears and the distance cannot be estimated from the two captured images or two pixels in which different subjects appear. An error may occur and the distance estimation may be incorrect. Therefore, the generation of a distance image by stereo matching has a disadvantage that there is a region where the distance cannot be estimated and a disadvantage that there is a lot of noise. Further, the stereo camera has a disadvantage that the distance estimation accuracy is significantly lowered when the epipolar line is distorted due to vibration or deterioration over time.

The image acquisition unit 121, the deep learning model imaging device 2 outputs a second range image I ^M to the input of one of the captured image captured, one of the two captured images the imaging device 2 is imaged simultaneously by entering one of the images, to acquire a second distance image I ^M which is outputted from the deep learning model. The generation of the distance image by the deep learning model has the advantage that the distance is estimated for all the pixels and the noise is low, but the error from the correct answer at the time of learning cannot be made zero, and the distance is estimated. The disadvantage is that the value contains a certain amount of error.

Note that the image acquisition unit 121 is itself was decided to acquire the first distance image I ^S by performing a stereo matching is not limited thereto, the first distance image I generated by an external device (not shown) ^S may be acquired. The image acquisition unit 121 is itself was decided to obtain a second range image I ^M by inputting the captured image to deep learning model is not limited thereto. Image acquisition unit 121 may acquire the second distance image I ^M generated using deep learning model in an external device.

Reliability specification unit 122, respectively of the plurality of first pixel of a plurality of pixels constituting the first distance image I ^S, a plurality of the second pixel is a plurality of pixels constituting the second distance image I ^M For each, the reliability of the distance corresponding to the pixel is specified.

Here, the maximum value and the minimum value are set for the reliability, and the sum of the reliability of the first pixel corresponding to the position in the distance image and the reliability of the second pixel is constant. .. In this embodiment, it is assumed that the minimum value of the reliability is 0 and the maximum value is 1. Reliability ^C _{S ij} of the first pixel, if the reliability ^C _{M ij} of the second pixel, and the reliability ^C _{S ij} of the first pixel, the relationship between the reliability ^C _{M ij} of the second pixel has the following formula It is shown by (1). Here, i is the index value of the pixel in the horizontal direction in the distance image, and j is the index value of the pixel in the vertical direction in the distance image. Assuming that the size of the image is W × H, the minimum value of i is 0, the maximum value is W-1, the minimum value of j is 0, and the maximum value is H-1.

The reliability specifying unit 122 of the first pixel and the second pixel, respectively, is based on the difference between the distance corresponding to the first pixel corresponding to the position in the distance image and the distance corresponding to the second pixel. Confidence of C ^S _ij and C ^M _ij are specified. Reliability specification unit 122, the greater the absolute value of the difference is, as reliability C ^M _ij of the second pixel becomes high, each of the reliability C ^S _ij of the first pixel and second _pixel, C to identify the ^M _ij.

Further, in the first distance image ^IS , for the pixel whose distance could not be estimated, the pixel value of the pixel is set to a value that can identify that the distance could not be estimated. Reliability specification unit 122, if the first pixel in the first distance image I ^S is the distance a pixel that could not estimate the reliability C ^S _ij of the first pixel and the minimum value, the first a maximum value of the reliability C ^M _ij of the second pixel position corresponding to the pixel.

Specifically, the reliability specification unit 122 specifies the reliability ^C _{M ij} of the second pixel by the following equation (2), then the reliability ^C _{M ij} of the second pixel identified formula (1) substituting the result, specifying the reliability C ^S _ij of the first pixel corresponding to the second pixel. In the formula ^(2), _{I S ij} is the pixel value of each pixel (first pixel) of the first range image ^{I S,} _{I M ij,} each pixel in the second range image ^{I M} (second pixel) Is the pixel value of.

The sigmoid (x) represented by the formula (2) is a sigmoid function and is represented by the following formula (3).

Further, d shown in the equation (2) is an offset value. Value input to the sigmoid function, because the distance to the pixel value I ^S _ij of the first pixel shows the absolute value of the difference between the distance that the pixel value I ^M _ij of the second pixel is shown, not to set the offset value Then, the value output from the sigmoid function is 0.5 or more. On the other hand, by setting the offset value, a number less than 0.5, that is, a value close to the minimum value can be obtained from the sigmoid function. Further, a shown in the equation (3) is a parameter for changing the slope in the sigmoid function.

Distance image generating unit 123, for each position in the range image of a plurality of first and second pixels correspond, the pixel value I ^S _ij of the first _pixel, the reliability C ^S _ij of the first _pixel, the pixel value ^I _{M ij} of 2 pixels, and by combining the pixel values based on the reliability ^C _{M ij} of the second pixel, to produce a range image ^{I C.} Distance image generating unit 123, for each position in the range image of a plurality of first and second pixels correspond, the pixel value of the first pixel, which is weighted by the reliability C ^S _ij of the first pixel I and ^S _ij, by combining the pixel values ^I _{M ij} of the second pixels that are weighted by the reliability ^C _{M ij} of the second pixel, to produce a range image ^{I C.}

Specifically, the distance image generation unit 123, the following equation (4), the pixel value ^I _{S ij} of the first pixel, the pixel value ^I _{M ij} of the second pixel, the identified confidence ^C _{S ij} and C enter the ^M _ij, and calculates the pixel value ^I _{C ij} of the pixels constituting the range image ^{I C.} Distance image generating unit 123, by calculating the pixel value I ^C _ij of all the pixels constituting the range image I ^C, and generates a range image I ^C.

The output unit 124, the distance image generation unit 123 outputs a distance image ^{I C} generated. For example, the output unit 124, the distance on the basis of the image, the vehicle or running ahead of the vehicle, and outputs a distance image I ^C to an apparatus for detecting an object of the detection target such as an obstacle present ahead of the vehicle.

[Process flow in the distance image generator 1]
Subsequently, the flow of processing in the distance image generation device 1 will be described. FIG. 3 is a flowchart showing a processing flow in the distance image generation device 1 according to the present embodiment. The distance image generation device 1 shall execute the process related to this flowchart at each time step.

First, the image acquisition unit 121 acquires the first distance image ^{I S,} and acquires a second distance image ^{I M} (S1).
Subsequently, the reliability specification unit 122, the first distance image I ^S and the second distance image I ^M, the position to select the first and second pixels of the corresponding set (S2).

Subsequently, the reliability specifying unit 122, for each of the first and second pixels selected in S2, the reliability ^C _S ^ij, to identify the _{C M ij} based on the equation (1) and (2) ( S3).

Then, distance image generation unit 123, the pixel value ^I _{S ij} and the pixel value ^I _{M ij} of the second pixel of the first pixel selected in S2, the specified reliability ^C _S ^ij, based on _{C M ij} Synthesize (S4). Specifically, the distance image generation unit 123, the equation (4), the pixel value ^I _{S ij} of the first pixel, the pixel value ^I _{M ij} of the second pixel, the identified confidence ^C _S ^ij, the _{C M ij} by entering, to calculate a pixel value ^I _{C ij} of the pixel of the range image ^{I C.}

Subsequently, the distance image generation unit 123 determines whether or not all the first pixels and the second pixels have been selected (S5). When the distance image generation unit 123 determines that all the first pixels and the second pixels are selected, the process is transferred to S6, and when it is determined that all the first pixels and the second pixels are not selected, the process is processed in S2. To move.

If it is determined as YES in S5, the distance image generation unit 123, a distance image is the pixel value I ^C _ij of all pixel calculation constituting the I ^C, range image I ^C becomes a state of being generated. In S6, the output section 124, the distance image generation unit 123 outputs a distance image ^{I C} generated.

[Effect of this embodiment]
As described above, the distance image generating apparatus 1 according to this embodiment, a plurality of a plurality of pixels constituting the first distance image I ^S the imaging apparatus 2 is generated based on two captured image captured respectively of the first pixel, for the respective plurality of second pixels of a plurality of pixels constituting the second distance image I ^M generated based on one image of the two captured images, corresponding to the pixels Identify the reliability of the distance to be done. Then, the distance image generation device 1 is based on the reliability of the first pixel and the reliability of the second pixel for each of the plurality of first pixels and the second pixels whose positions in the distance image correspond to each other. by combining the pixel value to generate a range image I ^C. By doing so, the distance image generation device 1 can generate a distance image having the advantages of the distance image based on the image captured by the stereo camera and the advantages of the distance image based on the image captured by the monocular camera. Further, when the calibration of the stereo camera becomes not strict due to vibration or deterioration over time, the reliability of the distance image based on the image captured by the stereo camera decreases, so that the distance image generator 1 automatically performs. It becomes possible to generate a distance image based on an image captured by a monocular camera.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist thereof. is there. For example, all or a part of the device can be functionally or physically distributed / integrated in any unit. Further, in the above embodiment, the distance image generating apparatus 1 is a passenger car, a bus, as the imaging apparatus of stereotypes provided in a vehicle such as a truck to generate a range image I ^C on the basis of the captured image captured However, it is not limited to this. The distance image generation device 1 is based on an image captured by a stereotype image pickup device provided not only for these vehicles but also for a moving body such as a two-wheeled vehicle or a train that needs to estimate the distance to an obstacle. Te, the range image I ^C may be generated.

The distance image generating apparatus 1 includes a first range image I ^S, it is assumed that generates a range image I ^C by combining the second distance image I ^M, is not limited thereto. Distance image generating apparatus 1, instead of the first distance image I ^S, acquired LiDAR provided in the vehicle (Light Detection and Ranging) is generated, and the object around the vehicle, the distance information indicating a distance between the LiDAR You may. The distance image generating apparatus 1 includes a corresponding distance information may generate a range image I ^C by combining the second distance image I ^M. In the distance information generated by LiDAR, although information can be obtained densely for a subject having a short distance from LiDAR, there is a problem that the obtained information is sparse for a subject having a long distance from LiDAR. In contrast, the distance image generation apparatus 1, the distance information LiDAR-generated by combining the second distance image I ^M, by generating a distance image I ^C, subject distance between LiDAR is far Can be supplemented with information about.

Also included in the embodiments of the present invention are new embodiments resulting from any combination of the plurality of embodiments. The effect of the new embodiment produced by the combination also has the effect of the original embodiment.

1 Distance image generation device 11 Storage unit 12 Control unit 121 Image acquisition unit 122 Reliability specification unit 123 Distance image generation unit 124 Output unit 2 Imaging device

Claims (6)

The distance between the image pickup device and the subject corresponding to each of the plurality of pixels constituting the image, which was generated based on the two images captured by the image pickup device that simultaneously images the subject from at least two directions, was estimated. An image acquisition unit that acquires a first distance image that is a distance image and also acquires a second distance image that is the distance image generated based on one of the two images.
Distances corresponding to pixels for each of the plurality of first pixels, which are the plurality of pixels constituting the first distance image, and each of the plurality of second pixels, which are the plurality of pixels constituting the second distance image. The reliability identification part that specifies the reliability of
For each of the plurality of first pixels and the second pixels whose positions correspond to each other in the distance image, pixel values are synthesized based on the reliability of the first pixel and the reliability of the second pixel. With a distance image generator that generates a distance image,
A distance image generator comprising. The image acquisition unit inputs one of the two images to the deep learning model that outputs the second distance image in response to the input of the image captured by the imaging device, thereby inputting the deep learning model. Acquires the second distance image output from
The distance image generator according to claim 1. The reliability specifying unit has the first pixel and the second pixel based on the difference between the distance corresponding to the first pixel corresponding to the position in the distance image and the distance corresponding to the second pixel. Identifying the reliability of each of the pixels,
The distance image generator according to claim 1 or 2. The maximum value and the minimum value are set for the reliability, and the sum of the reliability of the first pixel corresponding to the position in the distance image and the reliability of the second pixel is constant. Yes,
The reliability specifying unit specifies the reliability of each of the first pixel and the second pixel so that the larger the absolute value of the difference is, the higher the reliability of the second pixel is.
The distance image generator according to claim 3. The first distance image contains pixels for which the distance could not be estimated.
The maximum value and the minimum value are set for the reliability, and the sum of the reliability of the first pixel corresponding to the position in the distance image and the reliability of the second pixel is constant. Yes,
When the first pixel in the first distance image is a pixel whose distance cannot be estimated, the reliability specifying unit sets the reliability of the first pixel as the minimum value and sets the first pixel as the minimum value. The reliability of the second pixel at the position corresponding to the pixel is set to the maximum value.
The distance image generator according to any one of claims 1 to 4. Computer runs,
The distance between the imaging device and the subject corresponding to each of the plurality of pixels constituting the image, which was generated based on the two images captured by the imaging device that simultaneously images the subject from at least two directions, was estimated. A step of acquiring a first distance image which is a distance image and a step of acquiring a second distance image which is the distance image generated based on one of the two images.
Distances corresponding to pixels for each of the plurality of first pixels, which are the plurality of pixels constituting the first distance image, and each of the plurality of second pixels, which are the plurality of pixels constituting the second distance image. Steps to identify the reliability of
For each of the plurality of first pixels and the second pixels whose positions correspond to each other in the distance image, pixel values are synthesized based on the reliability of the first pixel and the reliability of the second pixel. With the steps to generate a distance image,
A distance image generation method comprising.

Images