JP4946896B2 - Distance measuring device and distance measuring method - Google Patents

Description

  The present invention relates to a technique for measuring a distance to an object using images photographed using a plurality of photographing means.

  In recent years, a pre-crash safety technique has been adopted as one of safety measures when a car is running. This technology measures the distance to the preceding vehicle to determine whether the host vehicle maintains a safe inter-vehicle distance with respect to the preceding vehicle. This is a technology to prevent accidents by issuing a warning that the vehicle is approaching. One of the problems in this pre-crash safety is that erroneous warnings are frequently issued because it is determined that the actual distance between the host vehicle and the preceding vehicle is large, although it is small.

  Therefore, what is desired for the distance measuring device applied to this pre-crash safety is that when there is no change in the actual distance to the preceding vehicle, the measured distance to the preceding vehicle will not change as much as possible. The variation of the distance to the preceding vehicle is made as small as possible.

  By the way, as a method for measuring the distance to the preceding vehicle, a method using a stereo camera is known. In this method, coordinate points of two images taken from different viewpoints are associated with each other, and a distance is calculated based on the principle of triangulation.

  Here, as a method for associating the coordinates of two images with each other, a phase-only correlation method capable of making a robust and highly accurate association has recently attracted attention (Non-Patent Document 1). This technique has a feature that it is not easily affected by luminance fluctuations and noise because the correlation calculation is performed using only the phase component by removing the amplitude component of the image. However, this method has a problem that the calculation load is high and processing time is required.

Therefore, in Patent Document 1, a vehicle contour is detected, a corresponding point search is performed only on the vehicle contour to calculate a parallax, the calculated parallax is histogrammed, and the maximum parallax whose frequency exceeds a threshold value. A method for calculating the distance to the vehicle using the above parallax as the representative parallax of the vehicle is disclosed.
K.Takita, T.Aoki, Y.Sasaki, T.Higuchi, and K.Kobayashi, "High-Accuracy Subpixel Image Registration Based on Phase-Only Correlation", IEICE Transactions. Fundamentals, E86-A, no8, pp. 1925 -1934, Aug. 2003. JP 7-334800 A

  However, in the method of Patent Document 1, the size of the vehicle is not taken into account at all, and when the vehicle becomes smaller, the number of pixels on the vehicle contour that is the object of the parallax calculation also decreases. And the accuracy of the representative parallax varies depending on the size of the vehicle, and the distance to the vehicle cannot be calculated stably.

  An object of the present invention is to provide a distance measuring device and a distance measuring method capable of stably calculating a distance to an object regardless of the size of the object to be measured for distance.

  A distance measuring device according to the present invention is a distance measuring device that measures a distance to an object to be measured using images photographed using a plurality of photographing means, and is photographed at the same timing by the photographing means. Extraction means for extracting one of the plurality of input images as a standard image and the other input image as a reference image, and extracting an object region in which the object is photographed from the standard image; Area calculating means for calculating the area of the object region extracted by the means, and search means for sequentially setting the attention points in the object region extracted by the extraction means and searching for corresponding points of each attention point from each reference image And a distance calculation unit that calculates a distance to the object based on a plurality of parallaxes between the attention point and the corresponding point, and the search unit calculates the object area calculated by the area calculation unit. surface And changes a setting density point of interest in accordance with the.

  A distance measuring method according to the present invention is a distance measuring method for measuring a distance to an object to be measured using images photographed using a plurality of photographing means, and is photographed at the same timing by the photographing means. An extraction step of extracting any one of the plurality of input images as a standard image and the other input image as a reference image, and extracting an object region in which the object is photographed from the standard image; An area calculating step for calculating the area of the object region extracted in the step, and a search step for sequentially setting a point of interest in the object region extracted in the extraction step and searching for a corresponding point of each point of interest from each reference image And a distance calculation step for obtaining a distance to the object based on a plurality of parallaxes between the attention point and the corresponding point, and the search step includes the area calculation step. And changes a setting density of the target point according to the area of the calculated object region by.

  According to these configurations, out of a plurality of input images captured at the same timing by the imaging unit, any one input image is set as a reference image, and the other input images are set as reference images. An object region that captures the object is extracted, attention points are sequentially set in the extracted object region, corresponding points of each attention point are searched from each reference image, and based on the parallax between the attention point and the corresponding point, The distance to the object is calculated. Here, since the setting density of the attention point is changed according to the area of the object region, the area of the object region can be increased by increasing the setting density of the attention point set in the object region as the area of the object region decreases. The variation in the number of attention points set due to the size of the object can be reduced, the distance measurement accuracy can be kept constant regardless of the area of the object region, and the distance measurement accuracy can be stabilized.

  Moreover, it is preferable that the search means increases the setting density of the attention point as the area of the object region decreases.

  According to this configuration, as the area of the object region becomes smaller, the setting density of the attention point is increased, so that the variation in the number of attention points set due to the size of the object region is reduced, and the distance measurement accuracy is stabilized. Can be achieved.

  Further, it is preferable that the search means increase the setting density of the attention points by reducing the thinning width when setting the attention points as the area of the object region decreases.

  According to this configuration, since the setting density of the attention point is increased by reducing the thinning width when setting the attention point, the setting density can be increased by a simple method.

  Further, the search means hierarchically generates a plurality of the reference images and the reference images having different resolutions so that the resolution becomes higher from the lower hierarchy toward the upper hierarchy, and corresponding points in the lower hierarchy Based on the search result, a search range smaller than the search range of the hierarchy is set in the reference image one level higher, and a search process for searching for a corresponding point within the search range is directed to a higher hierarchy. It is preferable to search for corresponding points by executing automatically.

  According to this configuration, since the search range is narrowed from the lower layer with lower resolution toward the upper layer, the approximate range of the corresponding point is searched by setting the search range over a wide range with the low-resolution reference image. In addition, it is possible to narrow down the search range of the reference image at a high resolution to a narrow range using the search result, and it is possible to search for corresponding points at high speed and with high accuracy.

  In the search process, a reference window is set around the point of interest in the reference image, a reference window having the same size as the reference window is set in the reference image, and within the search range set in the reference image. It is preferable to search for corresponding points by repeatedly executing a similarity calculation process for obtaining a similarity between an image in the reference window and an image in the reference window while shifting the reference window.

  According to this configuration, a reference window is set around the target point in the reference image, a reference window having the same size as the reference window is set in the reference image, and a search range of corresponding points is set and set in the reference image. Since the similarity calculation process in which the similarity between the image in the reference window and the image in the reference window is obtained while the reference window is shifted within the search range is repeatedly executed, corresponding points are searched. Can be searched with high accuracy.

  The similarity calculation process is preferably a process of calculating the similarity based on a phase component obtained by frequency-decomposing an image in the reference window and an image in the reference window.

  According to this configuration, since the similarity is calculated based on the phase component obtained by frequency-decomposing the image in the standard window and the image in the reference window, the similarity can be calculated with high accuracy. Thus, the search accuracy for corresponding points can be improved.

  The similarity calculation process is preferably frequency-resolved using any of fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform.

  According to this configuration, since frequency decomposition is performed using any of fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform, it is possible to perform frequency decomposition with high accuracy. it can.

  Moreover, it is preferable that the said similarity calculation process calculates the said similarity using a phase only correlation method.

  According to this configuration, since the degree of similarity is calculated using the phase-only correlation method, it is possible to search for corresponding points with robustness and high accuracy.

  Moreover, it is preferable that the said similarity calculation process calculates the said similarity using SAD (Sum of Absolute Difference).

  According to this configuration, in the similarity calculation process, since the similarity is calculated using SAD (Sum of Absolute Difference), it is possible to search for corresponding points at high speed.

  In addition, the search means sets a target point in a non-object region other than the object region for a period from the end of the search for the corresponding point of each target point set in the object region to the start of processing for the next frame. The corresponding points of each attention point are sequentially set and searched from each reference image, and the distance calculation means calculates the distance of the attention point from the attention point set in the non-object region and the corresponding point of the attention point. It is preferable.

  According to this configuration, it becomes possible to calculate the distance of the non-object region by using the free time of the calculation time in each frame, for example, information useful for grasping the surrounding environment such as a guardrail and a road shoulder. Thus, it is possible to notify the user of the approach of an obstacle other than the target object that is dangerous.

  Moreover, it is preferable that the search means sequentially sets a point of interest from the lower side to the upper side of the non-object region. According to this configuration, since the distance is calculated with priority from the attention point below the non-object region where there is a high possibility that an obstacle close to the user is present, the approach of the obstacle with a high degree of danger is performed by the user. It becomes possible to notify reliably.

  In addition, it is preferable that the object includes at least one of an automobile, a motorcycle, a bicycle, and a person. According to this configuration, the distance between a car, a motorcycle, a bicycle, and a person can be calculated. In particular, by calculating the distance of a person riding a motorcycle and a bicycle and the distance of a person not riding a car and a bicycle, the distance between the two can be separated, and the movement of the person can be grasped.

  According to the present invention, it is possible to stably calculate the distance to the object regardless of the size of the object to be measured for distance.

(Embodiment 1)
Hereinafter, a distance measuring apparatus according to Embodiment 1 of the present invention will be described with reference to the drawings. FIG. 1 shows a block diagram of a distance measuring apparatus 1 according to Embodiment 1 of the present invention. The distance measuring device 1 is mounted on the body of a vehicle such as an automobile, and measures the distance to an object to be measured using images photographed using a plurality of photographing means. For example, in the present embodiment, two cameras 11 and 12 (an example of a photographing unit), an arithmetic processing device 13 and a display device 14 are provided.

  The cameras 11 and 12 are arranged at the same height so as to be symmetrical with respect to the traveling direction with a predetermined interval in the left and right direction that is orthogonal to the traveling direction of the vehicle body and parallel to the horizontal direction. Then, the front of the vehicle body is photographed at a predetermined frame rate. Here, the cameras 11 and 12 are synchronized in photographing timing so that frames are photographed at the same time, and are objects to be measured for distance. Take a picture. Here, the front vehicle body includes an automobile, a motorcycle, and a bicycle.

  The arithmetic processing device 13 performs image processing, which will be described later, on the two images taken by the cameras 11 and 12, and calculates the distance to the front vehicle body. The display device 14 includes a display device such as a liquid crystal display or an organic EL display, and displays information such as the distance to the front vehicle body calculated by the arithmetic processing device 13. Here, when the vehicle body is provided with a car navigation system, the display device 14 may be configured by a display device of the car navigation system or may be configured by a display device different from the car navigation system. Good.

  FIG. 2 is a block diagram showing a detailed configuration of the distance measuring apparatus 1. As shown in FIG. 2, the distance measuring device 1 includes a base camera 110, a reference camera 120, an extraction unit 20 (an example of an extraction unit), an area calculation unit 30 (an example of an area calculation unit), and a density setting unit 40 (a search unit). An example), a corresponding point search unit 50 (an example of a search unit), a distance calculation unit 60 (an example of a distance calculation unit), and a display unit 70.

  The reference camera 110 is composed of, for example, the camera 11 which is one of the camera 11 and the camera 12 shown in FIG. 1, and images the front of the vehicle body at a predetermined frame rate of, for example, 30 frames per second. An input image including the front vehicle body is acquired. The reference camera 120 includes, for example, the camera 12 which is one of the camera 11 and the camera 12 illustrated in FIG. 1, and photographs the front of the vehicle body at the same frame rate as that of the reference camera 110 and is synchronized with the reference camera 110. An input image including the vehicle body is acquired. Here, as the input image, for example, digital image data in which a plurality of pixels having 256 gradation values from 0 (black) to 255 (white) are arranged in a matrix can be employed.

  The extraction unit 20 uses an input image sequentially input from the reference camera 110 at a predetermined frame rate as a reference image, and an input image sequentially input from the reference camera 120 at a predetermined frame rate as a reference image. An object region that is a region where is photographed is extracted. Here, for example, the extraction unit 20 may adopt a method of extracting a front vehicle body using an edge distribution and left-right symmetry in an image described in JP-A-7-334800, or an SVM (Support Vector). Machine), adaBoost, etc. may be used to learn vehicle body / non-vehicle body data, and a method of extracting the front vehicle body using pattern recognition may be employed. A technique for extracting In addition, when the extraction unit 20 extracts a plurality of regions indicating the front vehicle body, the extraction unit 20 extracts one or a plurality of regions indicating the front vehicle body that is predicted to be close to the own vehicle body as an object region. Also good. Note that, for example, a region having an area of a certain size or more may be employed as the region indicating the front vehicle body that is predicted to be close to the vehicle body.

  The area calculation unit 30 calculates the area of the object region extracted by the extraction unit 20. Here, for example, the area calculation unit 30 may count the number of pixels in the object region, and may employ the counted number of pixels as the area of the object region, or multiply the area per pixel by the counted number of pixels. The value may be adopted as the area of the object region.

  The density setting unit 40 sets the target point setting density according to the area of the object region calculated by the area calculation unit 30. Here, the attention point is a pixel that is sequentially set in the object region by the corresponding point search unit 50, and the density setting unit 40 increases the setting density of the attention point as the area of the object region decreases. Specifically, the setting density of the attention point is increased by reducing the thinning width when the attention point is set as the area of the object region is reduced.

  FIG. 3 shows two images from which the object region has been extracted by the extraction unit 20, (a) shows a case where the area of the object region D1 is large, and (b) shows a case where the area of the object region D1 is small. ing. 4 shows the target point CP set in the object region D1 when the set density of the target point CP is constant regardless of the area of the object region, and FIG. 4 (a) shows the object of FIG. 3 (a). The attention point CP set for the region D1 is shown, and FIG. 4B shows the attention point CP set for the object region D1 of FIG. 3B. In FIG. 3 and FIG. 4, for convenience of explanation, the object region is shown as a rectangle, but actually has a more complicated shape. 3 and 4, in (a), the object region D1 is composed of 16 × 16 pixels, and in (b), the object region D1 is composed of 8 × 8 pixels, and the attention point CP The thinning width is set to skip 3 pixels in the horizontal and vertical directions.

  As shown in FIG. 4A, since the thinning width of the attention point is skipped by 3 pixels in the horizontal and vertical directions, 16 pixels are included in the object region D1 in FIG. However, as shown in FIG. 4B, it can be seen that only four attention points CP are set in the object region D1 in FIG. 3B.

  Therefore, in the object region D1 shown in FIG. 3A, the number of processing points is 16, but in the object region D1 shown in FIG. 3B, the number of processing points is 4, which is shown in FIG. Since the number of processing points is larger than that in FIG. 3B, it can be seen that the distance measurement accuracy increases. In other words, if the thinning width of the point of interest is constant, the number of processing points decreases as the area decreases, so the distance measurement accuracy decreases. On the other hand, when the area increases, the number of processing points increases and the distance measurement accuracy increases. Therefore, it can be seen that the distance measurement accuracy is not stable according to the size of the area of the object region D1.

  Therefore, the density setting unit 40 keeps the distance measurement accuracy constant regardless of the area of the object region D1 by decreasing the thinning width and increasing the setting density of the attention point as the area of the object region D1 decreases. Maintaining and stabilizing the distance measurement accuracy.

  Here, the density setting unit 40 may set the thinning width so that the number of points of interest set in the object area is a constant number regardless of the area of the object area, or the area of the object area may be set in stages. A preferable thinning width at each stage in the case of dividing into two areas may be determined in advance, and the thinning width corresponding to the stage to which the area of the object area to be processed belongs may be set as the thinning width of the object area.

  FIG. 5 is a diagram showing a point of interest CP set in the object region D1 when the thinning width is changed according to the area of the object region D1 shown in FIG. In FIG. 5, the density setting unit 40 sets the thinning width by skipping one piece in the horizontal and vertical directions with respect to the object region D1 shown in FIG. It can be seen that the number of attention points CP shown in FIG. Accordingly, it is understood that the number of processing points in the object region D1 shown in FIGS. 3A and 3B is the same, and the distance measurement accuracy is stable regardless of the area of the object region D1.

  Returning to FIG. 2, the corresponding point search unit 50 sequentially sets the attention points in the object area extracted by the extraction unit 20 according to the setting density of the attention points changed by the density setting unit 40, and corresponds to each attention point. The point is searched from the reference image acquired at the same time as the base image in which the point of interest is set.

  Specifically, the corresponding point search unit 50 hierarchically generates a plurality of standard images and reference images having different resolutions so that the resolution becomes higher from the lower hierarchy toward the upper hierarchy. Based on the corresponding point search result, a search range smaller than the search range of the hierarchy is set as a reference image one level higher, and search processing for searching for a corresponding point within the search range is performed in the upper hierarchy. The corresponding points are searched by executing in a hierarchical manner.

  Here, as the search process, a reference window is set around the target point in the reference image, a reference window having the same size as the reference window is set as the reference image, and the reference window is set within the search range set in the reference image. While shifting, it is possible to employ a process of searching for corresponding points by repeatedly executing a similarity calculation process for obtaining a similarity between the image in the reference window and the image in the reference window.

  Further, as the similarity calculation process, a process for calculating the similarity based on a phase component obtained by frequency decomposition of the image in the reference window and the image in the reference window may be employed. You may employ | adopt the process which calculates similarity, without doing.

  Here, as frequency decomposition, methods such as fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform can be adopted. From the viewpoint of speeding up the processing, fast Fourier transform It is preferable to employ conversion.

  As processing for calculating the similarity based on the phase component, a phase only correlation (POC) method or a DCT code only correlation method can be adopted. However, since the POC is more robust, this In the embodiment, POC is adopted. Further, as a method for calculating the similarity without using the phase component, SAD (Sum of Absolute Difference), SSD (Sum of the squares of the density difference), NCC (Normalized Cross Correlation), or the like can be adopted. In this embodiment, SAD is adopted.

  Examples of the DCT code limiting method include, for example, “Fusion of image signal processing and image pattern recognition—DCT code limited correlation and its application, Hitoshi Kiya, Tokyo Metropolitan University, Faculty of System Design, Dynamic Image Processing Realization Workshop 2007 ′. The method described in 2007.7.7.3.8-9 ″ can be adopted.

  The POC is less susceptible to differences in imaging conditions of the cameras 11 and 12 and noise than the SAD and is robust. However, the POC is costly because of the frequency decomposition. Therefore, it is preferable to adopt POC when importance is attached to robustness, and it is preferable to adopt SAD when speeding up the processing.

  Further, when the extraction unit 20 extracts a plurality of object regions, POC is adopted for the object region indicating the front vehicle body with high importance, and SAD is adopted for the object region indicating the front vehicle body with low importance. Also good. Note that the front vehicle body that is important in avoiding danger may be adopted as the front vehicle body having higher importance. For example, the front vehicle body having a larger object area may have higher importance.

  Next, details of the POC will be described. FIG. 6 is a flowchart showing the flow of POC processing. In addition, in FIG. 6, the case where a Fourier transform is employ | adopted as frequency decomposition is illustrated. First, in steps S1 and S2, Fourier transformation is performed on the image in the standard window, and Fourier transformation is performed on the image in the reference window. Here, assuming that the image in the standard window is represented by equation (1) and the image in the reference window is represented by equation (2), the image in the standard window is Fourier transformed by equation (3), The image in the reference window is Fourier-transformed by equation (4).

  Here, n1 and n2 indicate vertical and horizontal coordinates, N1 and N2 indicate vertical and horizontal sizes of the reference window and reference window, and M1 indicates the vertical direction from the center point of the reference window and reference window. The distance from the center point of the reference window and the reference window to the edge in the horizontal direction is indicated by M2.

  Next, in steps S3 and S4, both images Fourier-transformed in steps S1 and S2 are normalized using equation (5). Next, in step S5, both images normalized in steps S3 and S4 are synthesized. Here, both normalized images are synthesized by multiplying F ′ (k1, k2) and the complex conjugate of G ′ (k1, k2) as shown in Expression (6). Next, in step S6, the image synthesized in step S5 is subjected to inverse Fourier transform using equation (7) to calculate a POC value (r (k1, k2)).

  FIG. 7 is a graph showing the POC value. In this graph, the x-axis and the y-axis represent coordinates in the standard window and the reference window, and the z-axis represents the POC value. As shown in FIG. 7, it is known that the POC value has a steep correlation peak, and has high robustness and estimation accuracy in image matching. The height of the correlation peak increases as the correlation between the standard window and the reference window increases. Therefore, by using, for example, the height of the correlation peak as the similarity between the standard window and the reference window, it is possible to indicate how close the images in both windows are. In addition, the position of the correlation peak indicates the amount of deviation between the standard window and the reference window.

  Then, corresponding points are searched for while shifting the reference window. In this case, since the position of the correlation peak indicates the amount of deviation between the two windows, it is not necessary to shift the reference window one pixel at a time. For example, half the window size may be set as the amount of shift.

  Finally, the position of the corresponding point is obtained from the position of the reference window and the position of the correlation peak of the POC value. Although the position of the reference window is a pixel level value, the position of the correlation peak indicates the amount of deviation between the windows and does not necessarily exist at the pixel level position. Therefore, the amount of deviation can be obtained at the sub-pixel level by interpolating and estimating the position of the correlation peak at the sub-pixel level. Here, as the interpolation estimation method of the position of the correlation peak, a method of fitting a function such as a parabola can be employed. Then, by adding the position of the correlation peak interpolated and estimated at the subpixel level to the position of the reference window, the corresponding point can be obtained at the subpixel level.

Next, details of SAD will be described. FIG. 8 is a schematic diagram for explaining SAD. As shown in FIG. 8, the attention point CP is set in the object region D1 set in the reference image, and the reference window W1 is set around the attention point CP. Here, the attention point CP is sequentially set so as to perform raster scanning in the object region D1 in accordance with the thinning width set by the density setting unit 40. Next, a predetermined pixel within the search range set in the reference image is set as the center point O1 of the reference window W2, and the reference window W2 is set around the center point O1. Here, the center point O1 has the same height in the vertical direction as the point of interest CP, and is set on a straight line of the reference image parallel to the horizontal direction. Next, a correlation value COR _p between the image Img1 in the standard window W1 and the image Img2 in the reference window W2 is calculated using equations (8) and (9).

  Where W is the window size of the standard window W1 and the reference window W2, i is the vertical coordinate, j is the horizontal coordinate, p is a variable that satisfies 0 ≦ p ≦ max_disp, and max_disp is Indicates the final search point.

When the correlation value COR _p is obtained, p is increased by 1, the reference window W2 is shifted in the horizontal direction, and the next correlation value COR _p is obtained. In SAD, since the reference window W2 is shifted by controlling p, the vertical position is the same as the point of interest, and the center point O1 is positioned on a straight line parallel to the horizontal direction. To be displaced.

Then, the correlation value COR _p is adopted as the similarity of the center point O1 of the image in the reference window W2, the position where the similarity is the highest in the reference image is searched, and the point of interest set in the reference image as the searched position It is specified as a corresponding point of CP. As shown in the graph of FIG. 8, the correlation value COR _p has a higher correlation between the image Img1 and the image Img2, and the higher the similarity, the higher the correlation value COR _p .

  Here, since the reference window is discretely shifted in units of pixels, the similarity is calculated with pixel level resolution. However, similar to POC, the similarity may be calculated at the sub-pixel level. Good.

  Returning to FIG. 2, the distance calculation unit 60 obtains the distance to the front vehicle body and the three-dimensional position indicated by the object region D <b> 1 using a stereo distance measurement method based on the attention point and the corresponding point. FIG. 9 is a schematic diagram illustrating a distance measuring method using a stereo method.

  In FIG. 9, f indicates the focal length of the cameras 11 and 12, and the number of pixels on the imaging surface (CCD) of the cameras 11 and 12 and the size (μ) of one pixel are the same. Further, the cameras 11 and 12 are arranged so as to be separated from each other in the left-right direction by a predetermined base line length L and to have parallel optical axes. In this case, if the parallax (the number of displaced pixels) on the imaging surface is d (= d1 + d2), the distance (Z) to the front vehicle body can be obtained by Z = (L × f) / (μ × d). . As the three-dimensional position, a relative position that is the position of the front vehicle body when a three-dimensional coordinate space is set with respect to the own vehicle body may be adopted, or if the vehicle body is equipped with GPS, GPS The absolute position of the front vehicle body may be obtained from the absolute position of the vehicle body obtained by the above, and this absolute position may be adopted.

  Here, for example, the distance calculation unit 60 obtains the representative parallax from the parallax of each pair of the target point set in the object region and the corresponding point of each target point, and the front vehicle body that represents the distance of the representative parallax is represented by the object region Or the distance of each pair of the target point and the corresponding point may be obtained, and the representative distance may be obtained from the obtained distance. As a method for obtaining the representative parallax or the representative distance, a method using a histogram described in Japanese Patent Laid-Open No. 7-334800 may be employed, and the distance or parallax average value, median value, A method of calculating an average value of remaining values obtained by removing the maximum value and the minimum value from the pair distance or parallax as the representative distance or the representative parallax may be adopted.

  Returning to FIG. 2, the display unit 70 includes the display device 14 illustrated in FIG. 1, and displays the distance to the front vehicle body calculated by the distance calculation unit 60, the three-dimensional position, and the like. Here, when the distance calculation unit 60 calculates the distances of the plurality of front vehicle bodies, the display unit 70 may notify each distance to the driver.

  Next, the operation of the distance measuring device 1 will be described. FIG. 10 is a flowchart showing the operation of the distance measuring apparatus 1. In the following flowchart, for convenience of explanation, the number of object regions D1 included in the base image and the reference image is one.

  First, in step S <b> 11, the extraction unit 20 acquires an input image input from the reference camera 110 as a reference image, and acquires an input image input from the reference camera 120 as a reference image.

  In step S12, the extraction unit 20 extracts the object region D1 from the reference image acquired in step S11 as shown in FIG. In step S13, the area calculation unit 30 calculates the area of the object region D1 extracted in step S12.

  In step S14, the density setting unit 40 sets the thinning width so that the thinning width of the target point CP decreases as the area of the object region D1 decreases as illustrated in FIG.

  In step S15, the corresponding point search unit 50 performs a search process. FIG. 11 is a flowchart showing search processing. First, in step S41, the corresponding point search unit 50 executes a process of reducing the resolution of the standard image and the reference image at a resolution determined in advance for each layer, and hierarchically selects a plurality of standard images and reference images having different resolutions. To generate. Here, the number of layers and the resolution in each layer are determined in advance, and the resolution increases from the lower layer to the upper layer, and the base image and the reference image that have not undergone resolution conversion are the highest layer. Is done.

  In step S42, the corresponding point search unit 50 sets a point of interest in the reference image of the highest hierarchy. Here, the attention points are sequentially set so as to perform raster scanning in the object region D1 according to the thinning width set in step S14.

  In step S43, the corresponding point search unit 50 sets a search range for the reference image of the lowest hierarchy. FIG. 13 is a diagram showing a standard window and a reference window set in the standard image and the reference image. As shown in FIG. 13, the search range DR is set to the entire area on the straight line of the reference image that has the same height in the vertical direction as the point of interest CP set in the base image in step S41 and is parallel to the horizontal direction. Has been.

  In step S44, the corresponding point search unit 50 executes similarity calculation processing. FIG. 12 is a flowchart showing similarity calculation processing. First, in step S61, the corresponding point search unit 50 sets the reference window W1 to the reference image in the same hierarchy as the reference image for which the search range DR is set in step S43. Here, the point on the reference image of the hierarchy whose processing target corresponds to the position of the attention point set in the reference image of the highest hierarchy in step S42 is set as the attention point CP of the reference image of the hierarchy. Is done. Then, as shown in FIG. 13, the reference window W1 is set so that the center is located at the attention point CP.

  In step S62, the reference window W2 is set to the reference image of the hierarchy to be processed. Here, the reference window W2 is set so that, for example, the left end position of the search range DR shown in FIG. 13 is the initial position of the center point O1.

  In step S63, the corresponding point search unit 50 calculates the similarity between the image in the standard window W1 and the image in the reference window W2 using the above-described POC or SAD.

  In step S64, when the center point O1 of the reference window W2 has not reached the final position of the search range DR (NO in step S64), the corresponding point search unit 50 shifts the reference window W2 (step S65) and performs processing. Returning to step S63, if the center point of the reference window W2 has reached the final position of the search range DR (YES in step S64), the similarity calculation process is terminated, and the process proceeds to step S45 shown in FIG. Here, as the final position, for example, the right end of the search range DR can be employed. Further, as shown in FIG. 13, the reference window W2 is set such that the center point O1 is sequentially set from the left end point to the right end point of the search range DR, and is shifted from the left side to the right side along the search range DR. To go. Then, the corresponding point search unit 50 calculates the similarity every time the reference window W2 is shifted, and sets the calculated similarity as the similarity at the center point O1. As a result, a distribution of similarity on the search range DR is obtained.

  Returning to FIG. 11, in step S <b> 45, the corresponding point search unit 50 refers to the reference image of the hierarchy whose processing target is the point having the maximum similarity among the similarities on the search range DR obtained by the similarity calculation processing. It is obtained as a corresponding point.

  In step S46, when the corresponding point search unit 50 does not set the highest hierarchy as the processing target (NO in step S46), the corresponding hierarchy search unit 50 sets the upper hierarchy as the processing target and searches for a reference image in the hierarchy. The range DR is set (step S48), the process returns to step S44, and if the highest hierarchy has already been set as the process target (YES in step S46), the process proceeds to step S47.

  FIG. 14 is a diagram for explaining the search range DR, where (a) shows a standard image and a reference image in a certain hierarchy, and (b) is a standard image in a hierarchy one level higher than the hierarchy shown in (a). And a reference image. The corresponding point search unit 50 includes the corresponding points TP obtained in the reference image shown in FIG. 14A, and searches for a reference image that is one level higher than the search range DR shown in FIG. 14A. The range DR is set (FIG. 14B). That is, the corresponding point search unit 50 sets the search range DR on any straight line that has the same vertical height as the point of interest CP and is parallel to the horizontal direction. The range is narrowed toward the higher levels. Here, the resolution of the standard image and the reference image becomes higher as the hierarchy becomes higher. Therefore, the search range DR is set in a wide range with the low-resolution reference image to search the approximate position of the corresponding point TP, and the search result DR of the high-resolution reference image is narrowed using the search result. This makes it possible to narrow down and search for corresponding points at high speed and with high accuracy. Note that the search range DR set in each layer may be a range having a certain width predetermined for each layer, with the corresponding point TP obtained in the lower layer as the center.

  Returning to FIG. 11, if the target point has not reached the final position in step S47 (NO in step S47), the corresponding point search unit 50 performs the process to set the next target point in the reference image in step S42. If the target point is set at the final position (YES in step S47), the search process is terminated and the process proceeds to step S16 shown in FIG.

  Returning to FIG. 10, in step S <b> 16, the distance calculation unit 60 obtains the distance to the front vehicle body shown in the object region D <b> 1, and in step S <b> 17, the display unit 70 displays the distance calculated by the distance calculation unit 60. In step S18, for example, when an operation command for ending the process is input by the user, the process is terminated (YES in step S18), the process is terminated, and the process is not terminated (in step S18). NO), the extraction unit 20 acquires the input image of the next frame acquired by the standard camera 110 as a standard image, and acquires the input image of the next frame acquired by the reference camera 120 as a reference image (step S19). The process returns to step S12.

  As described above, according to the distance measuring apparatus 1, the input images captured at the same timing by the cameras 11 and 12 are acquired as the standard image and the reference image, and the object region capturing the front vehicle body is extracted and extracted from the standard image. Attention points are sequentially set in the object region, the corresponding points TP of each attention point are searched from each reference image, and the distance to the object is calculated based on the parallax between the attention point and the corresponding points. Here, since the setting density of the attention point is increased as the area of the object region is reduced, variation in the number of attention points set due to the size of the area of the object region is reduced. Therefore, the distance measurement accuracy can be kept constant, and the distance measurement accuracy can be stabilized.

(Embodiment 2)
Next, a distance measuring device 1a according to Embodiment 2 of the present invention will be described. In the present embodiment, since the overall configuration and detailed configuration are the same as those in Embodiment 1, FIGS. 1 and 2 are used. In the present embodiment, the same elements as those in the first embodiment are not described, and only differences are described.

  The extraction unit 20 illustrated in FIG. 2 extracts candidate areas that are candidate object areas from the reference image, and extracts object areas from the extracted candidate areas. Here, the extraction unit 20 extracts candidate regions using the same method as the method for extracting the object region according to the first embodiment, and selects one or more candidate regions having high importance from the extracted candidate regions. What is necessary is just to extract as an area | region. Here, as the importance, for example, a value that is higher for a candidate area with a larger area and higher for a candidate area located below the reference image may be employed. Then, the extraction unit 20 may extract candidate areas having importance values greater than a predetermined value as object areas, or extract a predetermined number of candidate areas in descending order of importance as object areas. May be. Here, the reason why the importance level is set higher for a candidate area located on the lower side and having a larger area is that the object indicated by such a candidate area is more likely to be closer to the vehicle body. .

  The correspondence point search unit 50 performs a period on the reference image other than the object region extracted by the extraction unit 20 until the processing for the next frame is started after the search for the corresponding point of each target point set in the object region is completed. A third search process is executed in which attention points are sequentially set in a non-object region, which is a region, and corresponding points of each attention point are searched from each reference image.

  In other words, the third search process uses the free time until the input image of the next frame is acquired after the search for the corresponding point of each target point set in the object area in the current frame. In this process, attention points are sequentially set in the non-object region to search for corresponding points, and when the input image of the next frame is acquired, the search processing in the non-object region is terminated.

  Here, the third search process may be performed by searching for corresponding points using the above-described SAD or POC. In addition, when executing the third search process, the corresponding point search unit 50 sequentially sets attention points from, for example, the leftmost pixel to the rightmost pixel in the first line from the bottom of the non-object region. When all the line processing is completed, the attention points may be sequentially set from the lower side to the upper side of the non-object region so that attention points are sequentially set for the line immediately above. At this time, the corresponding point search unit 50 may set all the pixels in the non-object region as a point of interest, or set a pixel obtained by thinning the non-object region in the horizontal and vertical directions with a certain thinning width as the point of interest. May be. Further, the corresponding point search unit 50 may cause the area calculation unit 30 to calculate the area of the non-object region, and may set the attention point so that the thinning width increases as the area of the non-object region increases.

  The distance calculation unit 60 calculates the distance of the front vehicle body indicated by the object area in the same manner as in the first embodiment, and also uses the stereo method described above from the attention point and the corresponding point in the non-object area. Is used to find the distance between each point of interest. Here, the distance calculation unit 60 displays the display unit when the target point set in the non-object region includes a target point with a shorter distance than a predetermined distance assumed to be close to the vehicle body. Information indicating that an obstacle is approaching may be displayed on 70 to notify the user of the danger.

  Next, the operation of the distance measuring device 1a will be described. FIG. 15 is a flowchart showing the operation of the distance measuring device 1a. Step S81 is the same as step S11 shown in FIG. In step S82, the extraction unit 20 extracts candidate areas from the reference image, obtains importance levels of the candidate areas, and extracts candidate areas having importance levels greater than a predetermined value as object areas. Here, the extraction unit 20 obtains, for example, the center of gravity of each candidate region, assigns a higher importance to a candidate region whose center of gravity is located below the reference image, and multiplies the assigned importance by the area of each candidate region. A value may be set as the importance of each candidate area. In the following description, it is assumed that one object region is extracted for convenience of description.

  The processing in steps S83 to S85 is the same as the processing in steps S13 to S15 shown in FIG.

  In step S86, the corresponding point search unit 50 performs a third search process. FIG. 16 is a flowchart showing details of the third search process. First, in step S101, the corresponding point search unit 50 determines whether or not there is a free time for executing the third search process until the next frame is acquired. In step S101, a non-object area is specified from the reference image. On the other hand, if it is determined in step S101 that there is no free time (NO in step S101), the third search process is terminated and the process returns to step S87 shown in FIG.

  Next, the corresponding point search unit 50 sets one attention point in the non-object region (step S103), searches for a corresponding point with respect to the set attention point (step S104), and returns the process to step S101. Here, the corresponding point search unit 50 sequentially sets the attention points according to the setting pattern 1 for setting each pixel of the non-object region as the attention point or the setting pattern 2 for setting the pixels obtained by thinning out the non-object region as the attention point. .

  FIG. 17 is a diagram showing the points of interest set in the non-object region using the setting pattern 1 over time, and time elapses toward (a) to (c). In FIG. 17, a triangular mark indicates a target point set in the object region D1, a white circle indicates a pixel in the non-object region D2 that is not set as the target point, and a black circle indicates a non-set point that has been set as the target point. The pixel of the object area is shown. Further, the non-object region D2 is a region obtained by removing the object region D1 from the reference image.

  As shown in FIG. 17A, attention points are sequentially set from the leftmost pixel to the rightmost pixel of the first line from the bottom of the non-object region D2, and corresponding points are sequentially searched, and search for the rightmost pixel is performed. When the processing is completed, as shown in FIG. 17B, attention points are sequentially set from the leftmost pixel to the rightmost pixel of the second line from the bottom, and corresponding points are sequentially searched, and FIG. As shown in FIG. 5, when the pixel at the left end of the seventh line from the bottom is set as a point of interest and the search process is completed, the search process is terminated because there is no free time.

  FIG. 18 is a diagram showing the points of interest set in the non-object region using the setting pattern 2 over time, and the time elapses toward (a) to (b).

  In setting pattern 2, since the thinning width is set by skipping one pixel in the horizontal and vertical directions, as shown in FIG. 18A, from the leftmost pixel to the rightmost pixel in the first line from the bottom. It can be seen that the points of interest are sequentially set with the skipping width of one pixel skip. Then, when the search process for the first line from the bottom is completed, the attention points are sequentially set in such a manner that the attention points are sequentially set by skipping one pixel in the next two lines.

  Also, in the setting pattern 2, when the search process in the first line from the top is completed, that is, when the first cycle search process is completed, if there is a free time remaining, the thinning is performed as a point of interest. Pixels that have not been set are sequentially set as attention points, and search processing in the second cycle is started.

  That is, as shown in FIG. 18B, when the search process in the first cycle is completed, pixels that are not set as attention points from the lower line toward the upper line are sequentially set as attention points. The search process of the second cycle is executed. Here, in the search process in the second cycle, pixels that are not set as the attention point in the same line as the line in which the attention point was set in the first cycle may be sequentially set as the attention point by skipping one pixel. The attention point may be sequentially set by skipping one pixel in a line different from the line where the attention point is set in the first cycle. Note that in the third search process, the thinning width is not limited to skipping one pixel, and may be greater than or equal to skipping two pixels.

  Returning to FIG. 15, in step S <b> 87, the distance calculation unit 60 obtains the distance at each point of interest in the non-object area in addition to the object area. If the obtained distance is shorter than a predetermined value, the obstacle approaches. Is displayed on the display unit 70 (step S88). The processing in steps S89 to S90 is the same as that in steps S18 to S19 shown in FIG.

  As described above, according to the distance measuring device 1a, it is possible to calculate the distance of the non-object region by using the free time of the calculation time in each frame, and for example, grasp the surrounding environment such as a guardrail and a road shoulder. The above useful information can be obtained, and it is possible to notify the user of the approach of an obstacle other than the target object that is dangerous.

  In the second embodiment, the attention point is set in the non-object region. Alternatively, the attention point may be set in a non-candidate region other than the candidate region.

  In Embodiments 1 and 2 described above, the number of object regions is one, but the present invention is not limited to this and may be two or more. In this case, the processing shown in FIGS. 10 to 12 is executed for each object region in the first embodiment, and the processing of FIGS. 15 to 16 may be executed for each object region in the second embodiment. And the display part 70 should just display the distance for every object area | region.

  In the first and second embodiments, the front vehicle body is adopted as the object. However, the present invention is not limited to this, and a person may be the object, or an animal such as a dog or a cat other than a person may be the object. Good. In this case, for example, the extraction unit 20 may prepare a template for each target object that is a distance measurement target, and extract various target objects by template matching.

  In the first and second embodiments, considering that the heights of the cameras 11 and 12 are the same, the search range has the same vertical height as the point of interest and is parallel to the horizontal direction. Although it is set on a straight line, the present invention is not limited to this, and a quadrangular region having a certain width in the vertical direction around the straight line may be set as the search range.

  In the first and second embodiments, the number of cameras is two, but is not limited to this, and may be three or more. In this case, the image captured by any one of the cameras is used as a standard image, and the reference image captured by the other camera is used as the reference image to cause the extraction unit 20 to the distance calculation unit 60 to perform the above-described processing, Find the distance.

1 shows a block diagram of a distance measuring device according to an embodiment of the present invention. FIG. It is a block diagram which shows the detailed structure of the distance measuring device shown in FIG. 3 shows two images from which an object region has been extracted by the extraction unit shown in FIG. It is the figure which showed the attention point set to the object area in the case where the setting density of the attention point is constant. It is the figure which showed the attention point set to the object area | region when the thinning width | variety is changed according to the area of the object area | region. It is the flowchart which showed the flow of the process of POC. It is the graph which showed the POC value. It is a schematic diagram explaining SAD. It is a schematic diagram explaining the ranging method by a stereo method. It is a flowchart which shows operation | movement of a distance measuring device. It is a flowchart which shows a search process. It is a flowchart which shows a similarity calculation process. It is the figure which showed the reference | standard window and reference window which are set to a reference | standard image and a reference image. It is a figure explaining a search range. It is a flowchart which shows operation | movement of the distance measuring device by Embodiment 2 of this invention. 16 is a flowchart showing details of the third search process shown in FIG. 15. It is the figure which showed the attention point set to a non-object area | region using the setting pattern 1 with time. It is the figure which showed the attention point set to a non-object area | region using the setting pattern 2 with time.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 1a Distance measuring device 13 Arithmetic processing device 14 Display apparatus 20 Extraction part 30 Area calculation part 40 Density setting part 50 Corresponding point search part 60 Distance calculation part 70 Display part 110 Reference camera 120 Reference camera CP Attention point D1 Object area DR Search range O1 Center point TP Corresponding point W1 Base window W2 Reference window

Claims (13)

A distance measuring device that measures a distance to an object to be measured using images photographed using a plurality of photographing means,
An object region in which any one of the plurality of input images photographed at the same timing by the photographing unit is used as a standard image and another input image is used as a reference image, and the target object is photographed from the standard image Extracting means for extracting
Area calculating means for calculating the area of the object region extracted by the extracting means;
Search means for sequentially setting the attention points in the object region extracted by the extraction means, and searching for the corresponding points of each attention point from each reference image;
A distance calculating means for calculating a distance to the object based on a plurality of parallaxes between the attention point and the corresponding point;
The searching means changes a setting density of attention points according to the area of the object region calculated by the area calculation means , and the number of attention points set in the object region is the area of the object region. A distance measuring device that sets a thinning width of a point of interest so that the number is constant regardless of the number .   The distance measuring apparatus according to claim 1, wherein the search unit increases the setting density of the attention point as the area of the object region decreases.   The distance according to claim 2, wherein the search unit increases the setting density of the attention points by reducing a thinning width when setting the attention points as the area of the object region decreases. Measuring device.   In the search process, a reference window is set around the point of interest in the reference image, a reference window having the same size as the reference window is set in the reference image, and the reference is within a search range set in the reference image. 2. The process of searching for corresponding points by repeatedly executing a similarity calculation process for obtaining a similarity between an image in the reference window and an image in the reference window while shifting the window. The distance measuring device according to any one of?   The similarity calculation process is a process of calculating the similarity based on a phase component obtained by frequency-decomposing an image in the reference window and an image in the reference window. 4. The distance measuring device according to 4.   6. The distance according to claim 5, wherein the similarity calculation processing performs frequency decomposition using one of fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform. Measuring device.   The distance measurement apparatus according to claim 6, wherein the similarity calculation process calculates the similarity using a phase-only correlation method.   5. The distance measuring apparatus according to claim 4, wherein the similarity calculation processing calculates the similarity using SAD (Sum of Absolute Difference). The search means sequentially sets attention points in a non-object region other than the object region for a period from the end of the search for the corresponding point of each attention point set in the object region to the start of processing for the next frame. And search corresponding points of each attention point from each reference image,
The distance measurement unit according to claim 1, wherein the distance calculation unit calculates a distance of the target point from a target point set in the non-object region and a corresponding point of the target point. apparatus.   The distance measuring apparatus according to claim 9, wherein the search unit sequentially sets a point of interest from the lower side to the upper side of the non-object compensation region.   The distance measuring device according to claim 1, wherein the object includes at least one of an automobile, a motorcycle, a bicycle, and a person.   The search means hierarchically generates a plurality of the reference images and the reference images having different resolutions so that the resolution increases from the lower hierarchy toward the upper hierarchy, and searches for corresponding points in the lower hierarchy Based on the result, a search range smaller than the search range of the hierarchy is set in the reference image of the next higher hierarchy, and the search process for searching for a corresponding point within the search range is hierarchically performed toward the upper hierarchy. The distance measuring device according to claim 1, wherein the corresponding point is searched by executing the distance measuring device. A distance measurement method for measuring a distance to an object to be measured using images photographed using a plurality of photographing means,
An object region in which any one of the plurality of input images photographed at the same timing by the photographing unit is used as a standard image and another input image is used as a reference image, and the target object is photographed from the standard image An extraction step to extract
An area calculating step for calculating an area of the object region extracted by the extracting step;
A search step for sequentially setting a point of interest in the object region extracted by the extraction step, and searching for a corresponding point of each point of interest from each reference image;
A distance calculating step for obtaining a distance to the object based on a plurality of parallaxes between the attention point and the corresponding point;
The searching step changes the setting density of the attention points according to the area of the object region calculated by the area calculation step , and the number of attention points set in the object region is the area of the object region. A distance measurement method characterized by setting a thinning width of a point of interest so that the number is constant regardless of the number .