JP2017084259A - Corresponding point search method and distance measuring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow a processing load to be reduced when a corresponding point is searched for in a corresponding point search method for searching for a corresponding point among a plurality of images with a parallax error.SOLUTION: In a corresponding point search method, at a particular area extraction step, an area indicating at least one of a road area indicating a road and an empty area indicating emptiness in pixels included in a plurality of images is extracted as a particular area (S40 and S50). At a normal search step, regarding a normal area excluding the particular area from the plurality of images, a corresponding point among the plurality of images for each pixel in a reference image of the plurality of images is searched for (S60). In addition, at a particular search step, regarding the particular area, a corresponding point among the plurality of images is searched for by narrowing down a search range in comparison with when a corresponding point is searched for in the normal area (S60).SELECTED DRAWING: Figure 2

Description

  The present invention relates to a corresponding point search method and a distance measuring device for searching for corresponding points between a plurality of images having parallax.

  As said corresponding point search method, what searches for a corresponding point is known by the same process about all the pixels of the image used as a reference | standard (for example, refer patent document 1).

JP 2015-114269 A

  However, in the corresponding point search method, similar processing is performed for all pixels, but there is a demand for reducing the processing load when searching for corresponding points. Accordingly, it is an object of the present invention to reduce the processing load when searching for corresponding points in a corresponding point searching method for searching for corresponding points between a plurality of images having parallax.

  In the corresponding point search method according to one aspect of the present invention, in the specific area extraction step, an area indicating at least one of a road area indicating a road and an empty area indicating sky is extracted as a specific area in pixels included in the plurality of images. To do. Further, in the normal search step, corresponding points between the plurality of images are searched for each pixel in the reference image of the plurality of images with respect to the normal region excluding the specific region from the plurality of images. In the specific search step, for a specific area, the search range is narrower than when searching for corresponding points in the normal area, and corresponding points between a plurality of images are searched.

  According to such a corresponding point search method, when searching for a corresponding point in a specific area indicating a road or the sky, the search range is narrower than the normal area, so that the processing load when searching for the corresponding point is reduced. Can do.

  In addition, description of each claim can be arbitrarily combined as much as possible. At this time, a part of the configuration may be excluded.

It is a block diagram which shows schematic structure of the distance detection apparatus 1 to which this invention was applied. It is a flowchart which shows the distance calculation process which the process part 10 (CPU11) performs. It is an image figure which shows an example of an empty area | region and a road area | region. It is a flowchart which shows the empty area | region extraction process among distance calculation processes. It is a side view which shows the concept of a road area | region. It is a flowchart which shows the road surface extraction process among distance calculation processes. It is a flowchart which shows the corresponding point search process among distance calculation processes. It is explanatory drawing which shows the outline of the cost calculation in the left-right direction. It is explanatory drawing which shows an example of the process which adds a restriction | limiting to a search range in the case of cost calculation. It is a distance image which shows the effect by embodiment.

Embodiments according to the present invention will be described below with reference to the drawings.
[Configuration of this embodiment]
The distance detection apparatus 1 to which the present invention is applied is an apparatus that detects a distance to each point (object) in a captured image by detecting parallax of a plurality of captured images. Note that the parallax indicates that the viewing direction differs depending on where the same object is viewed. In particular, in the present embodiment, the positions at which the objects imaged at different positions are captured in the captured image, and the position shift at this time is referred to as parallax.

  In the distance detection apparatus 1 according to the present embodiment, the correspondence (that is, parallax) between pixels constituting a plurality of captured images is applied with high accuracy and low load by applying the Viterbi algorithm, which is a dynamic programming method, in a plurality of directions. Considered to be able to ask. In particular, the present embodiment is configured to further reduce the processing load by limiting the range of searching for corresponding points in the sky and road regions.

  Specifically, the distance detection device 1 is mounted on a vehicle such as a passenger car, and includes a processing unit 10, two imaging units 21 and 22, and a vehicle control unit 30, as shown in FIG. . Note that the number of imaging units is not limited to two, and three or more imaging units may be provided.

  The imaging units 21 and 22 are each configured as a known camera in which the traveling direction of the vehicle is within the imaging range. The imaging units 21 and 22 constitute a stereo camera in which the central axes are parallel and arranged at a predetermined distance in the horizontal direction. These image capturing units 21 and 22 are set to perform simultaneous image capturing periodically at a predetermined time.

  The processing unit 10 is configured as a known computer including a CPU 11 and a memory 12 such as a ROM or a RAM. The processing unit 10 (CPU 11) acquires captured images captured by the imaging units 21 and 22. Then, various processes such as a distance calculation process, which will be described later, are executed based on a program stored in the memory 12.

  The vehicle control unit 30 performs processing for controlling the vehicle using the processing result of the processing unit 10. For example, the vehicle control unit 30 acquires distance information at each point (each pixel) within the imaging range by the imaging units 21 and 22 from the processing unit 10 and repeatedly uses the distance information to thereby determine the position of the object and Recognize relative speed. Then, when there is a possibility that the object may interfere with traveling, vehicle control for changing the traveling track is performed.

[Process of this embodiment]
In the distance detection device 1 configured as described above, the processing unit 10 (CPU 11) performs a distance calculation process shown in FIG. The distance calculation process is a process for calculating the distance to each point indicated by each pixel in the captured image. The distance calculation process is started, for example, when the power of the distance detection device 1 is turned on, and then repeatedly executed at regular intervals.

  In this process, first, captured images captured by the imaging units 21 and 22 are acquired (S10). Subsequently, the captured image is collimated (S20). Here, the parallelization of the image indicates a process for correcting image distortion or posture deviation by the lens of the captured image.

  Subsequently, a multi-resolution image is generated (S30). Here, the multi-resolution image indicates a plurality of images having different resolutions generated from one captured image. For example, there are three types of captured images: a high-resolution image with an image enlargement ratio η of 1, a medium-resolution image with an image enlargement ratio η of 1/2, and a low-resolution image with an enlargement ratio η of 1/4. These images are generated and recorded in the memory 12 as multi-resolution images.

It should be noted that the corresponding point search process described later is performed on each image constituting the multi-resolution image in the order of decreasing resolution (that is, in the order of decreasing image enlargement ratio η).
Subsequently, an empty area extraction process is performed (S40). The sky region extraction processing is processing for extracting a sky region that is a set of pixels indicating the sky that appears in the captured image. Here, the sky region is detected above the captured image, for example, between buildings as shown in FIG. Moreover, in the road area | region mentioned later, it detects below a captured image.

  In the sky area extraction process and the road surface extraction process described later, by setting the search range to be used in the corresponding point search process described later in these sky areas and road areas, Reduce processing load. Note that in the sky region extraction process and the road surface extraction process, processing is performed using a low-resolution image.

  In the sky region extraction process, as shown in FIG. 4, in the reference image, the first pixel (that is, a pixel) among the pixels constituting this image is selected (S210). Here, the reference image represents a captured image selected in advance among a plurality of captured images such as a captured image by the imaging unit 21.

  Subsequently, an average luminance μ around the selected pixel is calculated (S220). As the “periphery of the selected pixel”, for example, nine pixels of the selected pixel and the neighboring eight pixels are selected. Then, the luminance variance σ around the selected pixel is calculated (S230).

  Subsequently, it is determined whether or not the average luminance μ and the variance σ are within a preset range (S240). Specifically, it is determined whether or not the average luminance μ is between the minimum threshold value and the maximum threshold value prepared in advance with respect to the average luminance μ, and the variance σ is the minimum value prepared in advance with respect to the variance σ. It is determined whether or not it falls within the threshold value and the maximum threshold value.

  If both the average luminance μ and the variance σ are within a preset range (S240: YES), it is determined that the selected pixel is a pixel constituting a part of the sky region (S250). At this time, for example, information for specifying the pixel, such as information on the position of the pixel, is associated, and the fact that the pixel is a part of the empty area is recorded in the memory 12. If either of the average luminance μ and the variance σ is not within the preset range (S240: NO), the process proceeds to S260.

  Subsequently, it is determined whether or not the determination is completed within the calculation area, that is, whether or not all the pixels of the reference image have been selected (S260). If the determination in the calculation area is not completed (S260: NO), the next unselected pixel is selected (S270), and the process returns to S220.

If the determination in the calculation area is completed (S260: YES), the empty area extraction process is terminated.
Subsequently, returning to FIG. 2, road surface extraction processing is performed (S50). The road surface extraction process is a process of extracting a road region that is a set of pixels indicating a road shown in a captured image. At this time, in the road surface extraction process, as shown in FIG. 5, an area located on the ground side of the road surface, that is, the lower side in the vertical direction is not searched (calculated) in the corresponding point search process described later. Set to a good area.

  In the road surface extraction process, as shown in FIG. 6, first, in the reference image, the first pixel (that is, pixel) is selected from the pixels constituting this image (S310). Subsequently, a parallax dr indicating the road surface is calculated based on the y-coordinate value which is the vertical coordinate value of this pixel (S320).

  Here, the road surface is a plane set in advance with reference to a plane on which the ground should exist when there is no inclination of the host vehicle or no road gradient. The road surface may be a lower plane of the tire of the host vehicle or a horizontal plane that passes through the ground side by a predetermined distance from the lower end of the tire, and is set on the assumption that a downward slope of several percent is set with respect to the horizontal plane. May be.

  Further, the road surface is uniquely determined for the distance to the road surface with respect to the y coordinate value of the pixel based on the installation position and the installation angle of the camera. The parallax dr indicates the distance to the road surface, and indicates that the distance is closer as the parallax increases, and indicates that the distance increases as the parallax decreases.

  Depending on the y coordinate value, there is no road surface. In this case, the distance is infinite, that is, the parallax dr is zero. Further, even when a road surface exists in calculation, the parallax dr may be set to 0 when it is not necessary to specify the road surface. For example, in the example illustrated in FIG. 3, the parallax dr is obtained only in an area where the probability that the host vehicle travels is high, and the parallax dr of other areas is set to zero.

Subsequently, it is determined whether or not the parallax dr is 0 for the selected pixel (S340). If the parallax dr is 0 (S340: YES), the process proceeds to S360 described later.
If the parallax dr is not 0 (S340: NO), the parallax range that is in the ground from the road surface is set as the parallax of the area outside the road (S350). At this time, for example, information for specifying the pixel, such as information on the position of the pixel, is associated, and the parallax of the area outside the road for the pixel is recorded in the memory 12.

  Subsequently, it is determined whether or not the determination is completed within the calculation area (S360). In the calculation area, for example, all the pixels of the reference image are shown. If the determination in the calculation area is not completed (S360: NO), the next unselected pixel is selected (S370), and the process returns to S320.

If the determination in the calculation area is completed (S360: YES), the road area extraction process is terminated.
Subsequently, returning to FIG. 2, a corresponding point search process is performed (S60). Corresponding point search processing is processing for associating corresponding points. Here, with respect to each pixel constituting the reference image, the same point is reflected in each pixel constituting the comparison image (for example, a captured image by the imaging unit 22) that is a captured image excluding the reference image. Indicates a pixel.

  As shown in FIG. 7 for details of the corresponding point search process, first, an image excluding an image with the highest resolution among the multiple resolution images (a low-resolution image with the lowest resolution in the present embodiment) is selected (S410). Then, the range corresponding to the pixel corresponding to the sky region and the parallax of the region outside the road is read from the memory 12 and the search is not performed (S415). In the pixel corresponding to the sky region, for example, the search range of the corresponding point is set to 0 or 1 pixel. That is, only pixels whose distance is infinite or very close to infinity are set as the corresponding point search range, and other pixels are outside the corresponding point search range. For the road region, the range recognized as the parallax of the region outside the road is outside the corresponding point search range, and the other pixels are set as the corresponding point search range.

  Subsequently, the cost of the node is calculated (S420). Here, as shown in FIG. 8A, the node is a matrix in which the horizontal axis is the pixel position of the reference image and the vertical axis is the parallax between the pixel position of the reference image and the pixel position of the comparison image. , One by one the elements that make up this matrix are shown. Here, each element represents the relationship between the pixel position of the reference image and the parallax.

  Here, the cost of the node is obtained using the pixel information. The pixel information represents pixel characteristics such as luminance and chromaticity. Moreover, the process which calculates | requires the cost of a node performs the process similar to the process shown by the paragraphs [0019]-[0021] of patent document 1, for example.

  Next, the cost for each direction according to the Viterbi algorithm is obtained (S430). At this time, as shown in FIG. 8B, the parallax cost is obtained using the Viterbi algorithm while changing the coordinates of the pixel selected in the horizontal direction (X direction) for each of the reference image and the comparison image. This process is performed in each direction.

  In addition, about this process, the process similar to the process shown to paragraphs [0022]-[0052] of patent document 1 is performed, for example. Here, in the process of obtaining the cost E for each direction by the Viterbi algorithm, when corresponding to the sky region and the road region in the reference image, the corresponding points are searched for in consideration of the range where the search is not performed. For example, in the example illustrated in FIG. 9, a mark X is attached to a pixel that is outside the corresponding point search range. That is, in this process, it is understood that the search range is limited when searching for corresponding points. As a result, only pixels having a high correlation value are selected as the search range.

  Subsequently, returning to FIG. 7, the minimum parallax is selected according to the cost E (S440), and the correspondence between each pixel in the reference image and each pixel in the comparison image is recorded in the memory 12 (S450). Subsequently, it is determined whether or not a high-resolution image is currently selected (S480).

  If no high-resolution image is selected (S480: NO), a multi-resolution image having the next highest resolution after the currently selected image is selected (S490). Then, the search area is reduced and set (S500). In the process of reducing the search area, when searching for corresponding points in a multi-resolution image with a higher resolution, the search area is limited using the parallax indicated by the corresponding points already obtained in the low-resolution image. To do. For example, about 3 pixels before and after the parallax obtained in a certain pixel are set as the search area. Specifically, if the parallax of the corresponding point obtained in a certain pixel is 5 pixels, a range of 2 to 8 parallax is set in the search area based on a higher resolution image.

In addition, about the correlation of each pixel between images with different resolutions, pixels indicating substantially similar positions are associated with each other.
When such a process is completed, the process returns to S415. If a high resolution image has been selected in the process of S480 (S480: YES), the corresponding point search process is terminated.

  When such corresponding point search processing is completed, the process returns to FIG. 2, and the distance to the object shown in each pixel is calculated according to the parallax of the corresponding point in each pixel (S70). Then, correspondence data between these pixels and distances is output to the vehicle control unit 30 (S80).

  Here, the correspondence data output by this processing is as shown in FIG. That is, when a captured image (scene image) as shown in FIG. 10A is processed using the above process, a distance image as shown in FIG. 10C is obtained. In the distance image shown in FIG. 10C, the distance is expressed by shading, indicating that the distance increases as the density increases. On the other hand, FIG. 10B is a distance image obtained by simply applying the Viterbi algorithm without applying this processing. In the distance image shown in FIG. 10C, the processing load is reduced as compared with the case of obtaining the distance image shown in FIG. 10B. However, the distance image shown in FIG. It can be seen that the distance is detected.

  Subsequently, the pixel group indicating the same distance is recognized as one object based on the distance image, and the position of the object (distance to the object) thus obtained is output to the vehicle control unit 30 (S90). When such processing ends, the distance calculation processing ends.

[Effects of this embodiment]
In the distance detection device 1 described above, the processing unit 10 extracts, as the specific region, a region indicating at least one of a road region indicating a road and a sky region indicating sky in pixels included in a plurality of images. Further, in the normal search step, corresponding points between the plurality of images are searched for each pixel in the reference image of the plurality of images with respect to the normal region excluding the specific region from the plurality of images. In the specific search step, for a specific area, the search range is narrower than when searching for corresponding points in the normal area, and corresponding points between a plurality of images are searched.

  According to such a distance detection device 1, when searching for a corresponding point in a specific area indicating a road or the sky, the search range is narrower than the normal area, so that the processing load when searching for the corresponding point is reduced. Can do.

Moreover, according to such a distance detection apparatus 1, since the corresponding point search method is used, the distance to the object can be detected while reducing the processing load.
In the distance detection apparatus 1 described above, the processing unit 10 extracts, as an empty area, an area in which the average luminance and the luminance dispersion are within a predetermined determination range in the relationship between the pixels in the reference image and the surrounding pixels.

According to such a distance detection device 1, it is possible to satisfactorily extract the sky region using the average luminance and the luminance dispersion.
In the distance detection apparatus 1 described above, the processing unit 10 sets a pixel set in advance for a pixel indicating a sky region as a corresponding point.

  According to such a distance detection device 1, since a pixel that is set in advance as a corresponding point is assumed to have no parallax in the sky region, the processing load when searching for the corresponding point can be further reduced. .

In the distance detection apparatus 1 described above, the processing unit 10 extracts, as a road region, a pixel that may include a portion on the ground side of the road surface in the reference image.
According to such a distance detection apparatus 1, a road area can be extracted by a simple process.

  It should be noted that “a pixel that may include a portion on the ground side from the road surface” can be specified by the orientation of the camera, the lens specifications, and the like when obtaining a captured image. For example, a pixel that is imaged below the horizontal can be used as a road region.

In the distance detection apparatus 1 described above, the processing unit 10 omits searching for corresponding points in a range that is in the ground side of the road surface in pixels indicating the road region.
According to such a distance detection device 1, the search for corresponding points is omitted for the range on the ground side of the road surface, so that the processing load when searching for the corresponding points can be further reduced.

  In the distance detection device 1, the processing unit 10 generates a plurality of sets of arbitrary resolution images having different resolutions from each of the plurality of captured images, and the other than the arbitrary resolution image having the highest resolution among the plurality of sets of arbitrary resolution images. A specific area is extracted from a specific resolution image representing one of them.

  According to such a distance detection device 1, the specific area is extracted by performing image processing on a specific resolution image having a low resolution and a low processing load, so that the processing load when extracting the specific area can be reduced. Can do.

  In the distance detection device 1, the processing unit 10 includes pixel information that represents the characteristics of the pixel in the reference pixel that represents a pixel in the reference image, and pixel information in the comparison pixel that represents a pixel in another image other than the reference image. The pixel cost based on the difference is calculated for each reference pixel while changing the reference pixel and the comparison pixel. Further, the processing unit 10 calculates, for each reference pixel, a parallax cost that represents a cost with respect to a change amount of parallax that is a coordinate difference between the reference pixel and the comparison pixel when changing the reference pixel. The combination with the comparison pixel when the total cost representing the sum of the pixel cost and the parallax cost becomes the minimum value is calculated. Then, the processing unit 10 sets a comparison pixel corresponding to each reference pixel as a corresponding point corresponding to each reference pixel.

  According to such a distance detection device 1, since the corresponding point is obtained by cost calculation, the object can be recognized by a simple process. Therefore, it is possible to reduce the processing load when obtaining the distance to the object.

[Other Embodiments]
The present invention is not construed as being limited by the above embodiment. Further, the reference numerals used in the description of the above embodiments are also used in the claims as appropriate, but they are used for the purpose of facilitating the understanding of the invention according to each claim, and the invention according to each claim. It is not intended to limit the technical scope of The functions of one component in the above embodiment may be distributed as a plurality of components, or the functions of a plurality of components may be integrated into one component. Moreover, you may abbreviate | omit a part of structure of the said embodiment as long as a subject can be solved. In addition, at least a part of the configuration of the above embodiment may be added to or replaced with the configuration of the other embodiment. In addition, all the aspects included in the technical idea specified only by the wording described in the claim are embodiment of this invention.

  In addition to the distance detection device 1 described above, various forms such as a system including the distance detection device 1 as a constituent element, a program for causing a computer to function as the distance detection device 1, a medium storing the program, a distance detection method, and the like Thus, the present invention can be realized.

[Correspondence between Configuration of Embodiment and Means of Present Invention]
In the above embodiment, the processing unit 10 corresponds to the image processing apparatus referred to in the present invention. Of the processes executed by the processing unit 10, the processes of S10 to S100 correspond to the corresponding point search method referred to in the present invention. In the above embodiment, the process of S30 corresponds to the multi-resolution image generation process referred to in the present invention. .

  Moreover, in the said embodiment, the process of S40 and S50 is corresponded to the specific area extraction process and specific area extraction part which are said by this invention, and the process of S60 in the said embodiment is a normal search process, a normal search part, specific, and the said invention. This corresponds to a search step and a specific search unit. Moreover, in the said embodiment, the process of S70 is equivalent to the distance calculating part said by this invention, and the process of S420 is equivalent to the pixel cost calculating process said to this invention in the said embodiment.

  In the above embodiment, the processes in S430 and S440 correspond to the minimum cost pixel calculation process in the present invention, and in the above embodiment, the process in S450 corresponds to the corresponding point setting process in the present invention.

  DESCRIPTION OF SYMBOLS 1 ... Distance detection apparatus, 10 ... Processing part, 11 ... CPU, 12 ... Memory, 21, 22 ... Imaging part, 30 ... Vehicle control part.

Claims (8)

A corresponding point search method (S10 to S100) that is executed by an image processing apparatus (10) that performs image processing on a plurality of images having parallax and searches for corresponding points between the plurality of images,
A specific area extraction step (S40, S50) for extracting, as a specific area, an area indicating at least one of a road area indicating a road and an empty area indicating sky in pixels included in the plurality of images;
A normal search step (S60) for searching for a corresponding point between the plurality of images for each pixel in the reference image of the plurality of images for the normal region excluding the specific region from the plurality of images;
For the specific area, a specific search step (S60) for searching for corresponding points between the plurality of images by narrowing a search range than when searching for corresponding points in the normal area;
Corresponding point search method. In the corresponding point search method according to claim 1,
In the specific area extracting step, pixels whose average luminance and luminance distribution are within a predetermined determination range in relation to the pixels in the reference image and surrounding pixels are extracted as pixels constituting the sky area. Corresponding point search method. In the corresponding point search method according to claim 2,
In the specific search step, a corresponding point search method in which a pixel set in advance for the pixel indicating the sky region is used as the corresponding point. In the corresponding point search method according to any one of claims 1 to 3,
In the specific area extracting step, a corresponding point search method for extracting, as the road area, a pixel that may include a part on the ground side of the road surface in the reference image. In the corresponding point search method according to claim 4,
In the specific search step, a corresponding point search method that omits searching for corresponding points in a range that is in the ground side of the road surface in pixels that indicate the road region. In the corresponding point search method according to any one of claims 1 to 5,
Performing a multi-resolution image generation step (S30) for generating a plurality of sets of arbitrary resolution images having different resolutions from each of the plurality of captured images;
In the specific area extracting step, the specific area is extracted from a specific resolution image representing any one of the plurality of sets of arbitrary resolution images other than the highest resolution arbitrary resolution image. In the corresponding point search method according to any one of claims 1 to 6,
In the normal search step and the specific search step,
A pixel cost based on a difference between pixel information representing a characteristic of the pixel in a reference pixel representing a pixel in the reference image and pixel information in a comparison pixel representing a pixel in another image excluding the reference image, A pixel cost calculation step (S420) for calculating each reference pixel while changing the reference pixel and the comparison pixel;
When changing the reference pixel, a parallax cost representing a cost for the amount of change in parallax, which is a coordinate difference between the reference pixel and the comparison pixel, is calculated for each reference pixel, and for each reference pixel, A minimum cost pixel calculation step (S430, S440) for calculating a combination with a comparison pixel when the total cost representing the sum of the pixel cost and the parallax cost is a minimum value;
A corresponding point setting step (S450) for setting a comparison pixel corresponding to each reference pixel as a corresponding point corresponding to each reference pixel;
Corresponding point search method. A distance detection device (1) for detecting a distance to an object in a captured image based on a plurality of captured images,
A specific area extraction unit (S40, S50) for extracting, as a specific area, an area indicating at least one of a road area indicating a road and an empty area indicating sky in pixels included in the plurality of images;
A normal search unit (S60) for searching for a corresponding point between the plurality of images for each pixel in the reference image of the plurality of images with respect to the normal region excluding the specific region from the plurality of images;
A specific search unit (S60) for searching for corresponding points between the plurality of images with a narrower search range than when searching for corresponding points in the normal region for the specific region;
A distance calculation unit (S70) that calculates the distance to the object indicated by each reference pixel based on the relationship between each reference pixel and the corresponding point;
A distance detection device comprising: