JP4831084B2 - Corresponding point search device - Google Patents

Description

  The present invention relates to a corresponding point search device that searches a reference image for corresponding points of a target point set in a standard image.

  In recent years, stereo matching is known in which an object is photographed using a stereo camera and a distance to the object is calculated using the obtained input image. In stereo matching, among a plurality of images taken by a stereo camera, one input image is used as a reference image, the other input image is used as a reference image, an attention point is set in the reference image, and the reference point is set around the set attention point. In addition to setting a window for an image, the same window is set for a reference image, and a matching process of both images is obtained to search for a corresponding point of the target point from the reference image.

  In this search process, if the window is set toward the end of the input image, the window protrudes from the input image. In this case, the window includes an outer area that protrudes from the image area of the input image, an edge appears at the boundary between the input image and the outer area, and the image changes discontinuously. There is a problem that an erroneous search in which a different point is searched as a corresponding point occurs, and the search accuracy of the corresponding point is lowered.

As a technique for solving this problem, the following four conventional techniques are known. The first conventional method is a method of reducing discontinuity by performing a blurring process in which an image in a window is blurred from a certain region toward the periphery using a predetermined filter. The second conventional method is a method of filling the entire outer region with a fixed value (for example, 0 value) (for example, Patent Document 1). A third conventional method is a method in which an interpolation pixel value is obtained in advance from information of an input image in a window set at a position not including an outer region, and the outer region is set using the interpolation pixel value. It is. The fourth method is a method of filling the outer region by folding the input image in a mirror shape within the window.
JP 2004-102323 A

  However, in the first conventional method, since the blurring process using the filter is performed, the amount of calculation increases, and the outer area is not interpolated using the information of the input image. An edge appears and an erroneous search occurs. Further, in the second conventional method, when the difference between the fixed value and the pixel value of the pixel at the end of the input image is large, an edge appears in the image in the window, and an erroneous search occurs.

  In the third conventional method, there is a case where the information of the input image in the window is not reflected at all in the interpolation pixel value. In this case, an edge appears in the image in the window, and an erroneous search occurs. . In the fourth conventional method, since an inverse pattern appears in the outer area, the point in the outer area may be searched as a corresponding point.

  An object of the present invention is to provide a corresponding point search device capable of searching for corresponding points with high accuracy without increasing the calculation cost even when the window is set so as to protrude from the input image. .

(1) A corresponding point search apparatus according to the present invention includes an acquisition unit that acquires one input image as a standard image and another input image as a reference image among a plurality of input images capturing the same scene, and the standard image Sequentially set attention points on the reference image, set a window on the base image so as to include the set attention points, set a window on the reference image, and shift the windows set on the reference image, By determining the degree of coincidence of images, search means for searching for corresponding points of the target point from the reference image, and whether or not the window includes an outer region that protrudes from the image region of the input image. A determination unit configured to determine each time the window is shifted; and when the determination unit determines that the outside area is included, at least an input image in the window Based on the pixel values of pixels positioned on the edge, e Bei an interpolation means for calculating an interpolated pixel value of the outer region, wherein the interpolation means, the average pixel value of the pixel at the end of the input image within said window It is calculated as an interpolated pixel value, and the interpolated pixel value is calculated so as to gradually change toward the edge of the window in the outer region.

According to this configuration, when searching for corresponding points, when a window is set near the ends of both the input images of the base image and the reference image, the outer region included in the window is the input image in the window. Are filled with the interpolated pixel value calculated based on the pixel value of at least the end pixel. Therefore, the occurrence of an edge at the boundary between the outer region and the input image is suppressed, and a corresponding point can be searched with high accuracy. In addition, since the outer region is filled by setting the interpolation pixel value in the outer region, it is possible to prevent an increase in calculation cost. Furthermore, since the interpolation pixel value is calculated every time the window set for both the standard image and the reference image is shifted, the interpolation pixel value more accurately reflects the information of the input image included in the window. As a result, the occurrence of an edge at the boundary between the outer region and the input image can be more reliably suppressed.
Further, according to this configuration, since the pixel used for calculating the interpolation pixel value is only the pixel at the end of the input image in the window, the interpolation pixel value can be obtained at high speed. Moreover, since the pixel average value of the pixels at the edge of the input image in the window is calculated as the interpolated pixel value, the occurrence of an edge at the boundary between the outer region and the input image can be more reliably suppressed.
Further, according to this configuration, the interpolation pixel value is set in the outer region so as to change gradually toward the edge of the window, and the search accuracy can be improved.

(2) Another corresponding point search device according to the present invention includes: an acquisition unit that acquires one input image as a standard image and another input image as a reference image among a plurality of input images that capture the same scene; Both windows are set by sequentially setting the attention points in the reference image, setting a window in the reference image so as to include the set attention points, setting a window in the reference image, and shifting the window set in the reference image. A search means for searching for a corresponding point of the target point from the reference image by obtaining a degree of coincidence of the image in the image, and whether or not the window includes an outer region that protrudes from the image region of the input image. A determination unit that determines whether the window is shifted, and the determination unit determines that the outside area is included, the amount of input images in the window is small. Interpolating means for calculating the interpolated pixel value of the outer region based on the pixel value of the pixel located at the end, and the interpolating means calculates the pixel average value of each pixel constituting the input image in the window. The interpolation pixel value is calculated as the interpolation pixel value, and the interpolation pixel value is calculated so as to gradually change toward the edge of the window in the outer region .
According to this configuration, since the pixel average value of the input image in the window is adopted as the interpolated pixel value, the degree of coincidence of the images in both windows uses a partial area instead of the entire image in both windows. Even if the calculation is performed locally, the occurrence of an edge at the boundary between the outer region and the input image can be more reliably suppressed.

(3) Out of a plurality of input images capturing the same scene, acquisition means for acquiring one input image as a standard image and another input image as a reference image, and a point of interest are sequentially set and set in the standard image By setting a window on the reference image so as to include the noted point, setting a window on the reference image, and determining the degree of coincidence of the images in both windows while shifting the window set on the reference image, Search means for searching for a corresponding point of the target point from the reference image, and determining whether or not the window includes an outer region that protrudes from the image region of the input image each time the window is shifted. When the determination unit and the determination unit determine that the outer region is included, the determination unit is based on the pixel value of the pixel located at least at the end of the input image in the window. And interpolating means for calculating interpolated pixel values of the outer region, wherein the interpolating means gives a certain margin in the vertical direction with reference to the pixel average value of each pixel located at the end of the input image in the window. The calculated value is calculated as the interpolation pixel value .
According to this configuration, in the pixel for which the interpolated pixel value is to be calculated, the ratio of the pixel whose pixel value is larger than the pixel average value is higher or lower than the ratio of the small pixel. Since the value of the interpolated pixel value is adjusted by giving a vertical margin to the pixel average value, it is possible to more reliably suppress the occurrence of an edge at the boundary between the outer region and the input image.

( 4 ) In addition, when the ratio of pixels having a pixel value higher than the pixel average value is higher than the ratio of low pixels at the end position of the input image in the window, the interpolation means sets the pixel average value to the pixel average value. It is preferable that a value obtained by adding a certain margin is the interpolation pixel value, and vice versa, a value obtained by subtracting a certain margin from the pixel average value is the interpolation pixel value. According to this configuration, it is possible to more reliably suppress the occurrence of an edge at the boundary between the outer region and the input image.

(5) Still another corresponding point search device according to the present invention includes an acquisition unit that acquires one input image as a standard image and another input image as a reference image among a plurality of input images that capture the same scene; Attention points are sequentially set in the reference image, a window is set in the reference image so as to include the set attention points, a window is set in the reference image, and both windows are set while shifting the window set in the reference image. The search means for searching for the corresponding point of the attention point from the reference image by determining the degree of coincidence of the images in the window, and whether the window includes an outer region that protrudes from the image region of the input image A determination unit that determines whether or not the window is shifted, and when the determination unit determines that the outside area is included, the input image in the window Based on the pixel value of the pixel located at the end even without, an interpolation means for calculating an interpolated pixel value of the outer region, the interpolation means, the pixels of each row located at the end of the input image within said window A pixel value is calculated as an interpolated pixel value corresponding to each row of the outer region, and the interpolated pixel value is calculated so as to gradually change toward the edge of the window in the outer region .

  According to this configuration, the interpolated pixel value of each row is calculated using the pixel value of the pixel of each row at the end of the input image in the window. Therefore, when the method of searching for the corresponding point by shifting the window set for the reference image to the same line as the line where the point of interest is positioned by one line is used, the interpolation pixel value setting process in the outer region is accelerated. The occurrence of an edge can be suppressed at the boundary between the outer region and the input image.

( 6 ) The search means hierarchically generates a plurality of the reference images and the reference images having different resolutions so that the resolution becomes higher from the lower hierarchy toward the upper hierarchy. Based on the corresponding point search result, a search range smaller than the search range of the hierarchy is set as a reference image one level higher, and search processing for searching for a corresponding point within the search range is performed in the upper hierarchy. It is preferable to search for corresponding points by executing in a hierarchical manner.

  According to this configuration, when a so-called multi-resolution strategy method is employed as the search process, it is possible to further reduce the occurrence of edges in a low-level input image in which more parts that are outer regions appear, and Search accuracy can be increased.

( 7 ) Further, it is preferable that the search means calculates the degree of coincidence by performing a correlation operation between an image in the window set as the reference image and an image in the window set as the reference image. According to this configuration, the degree of coincidence can be calculated with high accuracy.

( 8 ) Further, the search means calculates the degree of coincidence based on a phase component obtained by frequency-decomposing an image in the window set as the reference image and an image in the window set as the reference image. It is preferable to do.

  According to this configuration, since the degree of coincidence is calculated based on the phase component obtained by frequency-decomposing both images in the window set as the standard image and the reference image, it is affected by a luminance difference between images and noise. Difficult and robust search processing can be realized.

( 9 ) Moreover, it is preferable that the search means performs frequency decomposition using any of fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform.

  According to this configuration, the image in the window is one of a fast Fourier transform, a discrete Fourier transform, a discrete cosine transform, a discrete sine transform, a wavelet transform, and a Hadamard transform as a method for frequency-decomposing the image in the window. Since the method is employed, frequency resolution can be performed with high accuracy.

( 10 ) Moreover, it is preferable that the said search means calculates the said coincidence using a phase only correlation method. According to this configuration, since the corresponding points are searched using the phase only correlation method, the corresponding points can be searched with high accuracy.

  According to the present invention, even if the window is set so as to protrude from the input image, the corresponding points can be searched with high accuracy without increasing the calculation cost.

  Hereinafter, a corresponding point search device according to an embodiment of the present invention will be described. FIG. 1 shows a schematic configuration diagram of the corresponding point search apparatus. The corresponding point search apparatus includes two cameras C1 and C2 and a controller 10.

  In the following description, the case where the corresponding point search device is mounted on a vehicle body such as an automobile will be described as an example. However, this is only an example, and any device that performs a process for searching for a corresponding point will be described. You may mount in such an apparatus.

  For example, the cameras C1 and C2 are arranged at the same height so as to be symmetrical with respect to the traveling direction by providing a predetermined interval in the left-right direction that is orthogonal to the traveling direction of the vehicle body and parallel to the horizontal direction. A scene in front of the vehicle body is photographed at a predetermined frame rate. Here, the cameras C1 and C2 are synchronized in shooting timing so that frames are shot at the same time. Assume that the cameras C1 and C2 are calibrated in advance and the camera parameters are also known in advance.

  In addition, although the cameras C1 and C2 are two, it is not limited to this, You may employ | adopt three or more cameras. In addition, when the amount of movement of an object is calculated from a time series image, or when shooting data and registration data are compared, the number of cameras may be one.

  The controller 10 is configured by a hardware device including, for example, a CPU, a ROM, a RAM, and the like, and sequentially captures images taken by the cameras C1 and C2 via a communication cable. The controller 10 may capture image data from the cameras C1 and C2 wirelessly.

  FIG. 2 shows a functional block diagram of the controller 10. The controller 10 includes an image acquisition unit 11 (an example of an acquisition unit), a setting unit 12 (an example of a search unit), a window setting unit 13 (an example of a search unit), a determination unit 14 (an example of a determination unit), and an interpolation unit 15 ( An example of an interpolation unit) and a search processing unit 16 (an example of a search unit) are provided.

  The image acquisition unit 11 sequentially acquires two input images obtained by capturing a scene at a predetermined frame rate by the cameras C1 and C2, and includes a standard image acquisition unit 11a and a reference image acquisition unit 11b. The reference image acquisition unit 11a sequentially acquires an input image transmitted from the camera C1 as a reference image. The reference image acquisition unit 11b sequentially acquires an input image transmitted from the camera C2 as a reference image. As the standard image and the reference image, for example, digital image data in which a plurality of pixels having 256 gradation values from 0 (black) to 255 (white) are arranged in a matrix can be employed.

  The setting unit 12 includes an attention point setting unit 12a and a center point setting unit 12b. The attention point setting unit 12a sets the attention point in the reference image acquired by the reference image acquisition unit 11a. Note that the attention point indicates a pixel on the reference image to be searched for the corresponding point. Here, the point-of-interest setting unit 12a sequentially sets, for example, all pixels on the reference image or predetermined pixels thinned out at regular intervals as the point of interest so as to perform raster scanning from the upper left to the lower right.

  FIG. 3 is an explanatory diagram of attention point setting processing. As shown in FIG. 3, all the pixels on the reference image partitioned in a grid pattern or predetermined pixels thinned out at a constant interval are sequentially set as the attention point CP. In the example of FIG. 3, the upper left pixel is set as the attention point CP.

  The center point setting unit 12b sets the center point of the reference window as a reference image. Here, the center point setting unit 12b performs center raster scanning so that each pixel on the reference image is sequentially scanned from the upper left to the lower right while one attention point is set by the attention point setting unit 12a. Set as a point.

  The window setting unit 13 includes a standard window setting unit 13a and a reference window setting unit 13b. The reference window setting unit 13a sets a reference window having a size of a predetermined row × predetermined column around the target point set by the target point setting unit 12a as a reference image. FIG. 4 is a diagram showing a reference image in which a reference window is set. As shown in FIG. 4, it can be seen that the upper left pixel of the reference image I1 is set as the attention point CP, and a reference window W1 having a size of 14 × 14 around the attention point CP is set. Further, it can be seen that the reference window W1 protrudes from the reference image I1, and the reference window W1 includes an outer region D1 that protrudes from the reference image I1. The same thing can happen in the reference window.

  The reference window setting unit 13b sets a reference window having the same size as the standard window as a reference image with the center point set by the center point setting unit 12b as the center. Hereinafter, when the standard window and the reference window are not particularly distinguished from each other, they are collectively referred to as “window”, and when the standard image and the reference image are not particularly distinguished from each other, they are collectively referred to as “input image”. . In FIG. 4, for convenience of explanation, a 14 × 14 window is illustrated, but this is an example, and the size of the window may be appropriately changed to a preferable value depending on the number of pixels of the entire input image and the photographing target.

  The determination unit 14 includes determination units 14a and 14b. The determination unit 14a determines whether or not an outer region is included in the reference window set in the reference image every time the reference window is shifted. Here, the determination unit 14a may determine that an outer region is included in the reference window when, for example, even one pixel located at the end of the reference image is included.

  Similar to the determination unit 14a, the determination unit 14b determines whether or not an outer area is included in the reference window set in the reference image every time the reference window is shifted.

  When the determination unit 14 determines that the outer region is included, the interpolation unit 15 calculates the interpolation pixel value of the outer region based on the pixel value of the pixel located at least at the end of the input image in the window. . Here, the interpolating unit 15a includes an interpolating unit 15a that interpolates an outer region of the reference window, and an interpolating unit 15b that interpolates the outer region of the reference window. Details of the processing of the interpolation unit 15 will be described later. The interpolated pixel value indicates a pixel value when a pixel is virtually set at the same resolution as the input image in the outer region. A pixel virtually set in the outer region is referred to as an “interpolated pixel”.

  The search processing unit 16 searches for a corresponding point of the target point from the reference image by obtaining a degree of coincidence between the image in the reference window and the image in the reference window while shifting the reference window. Here, the search processing unit 16 calculates the degree of coincidence using any one of SAD (Sum of Absolute Difference), SSD (Sum of Squared intensity Difference), NCC (Normalized Cross Correlation), and phase-only correlation method. Can do.

  In the search processing by SAD, the degree of coincidence between the image in the standard window and the image in the reference window is obtained from the correlation value obtained by the equation (1) while shifting the reference window, and when the degree of coincidence is the highest. The center point of the reference window is searched as the corresponding point of the attention point. As shown in the equation (1), the SAD has a feature that the correlation value is calculated by subtracting the pixel values of two images as they are, so that the calculation amount is small and high-speed processing is possible.

However, M _L represents the image in the reference window, M _R represents an image in the reference window, Q is shows the horizontal size of the window, P is indicating the vertical size of the window.

  In the search processing by SSD, corresponding points are searched in the same manner as in SAD except that the degree of coincidence between the image in the standard window and the image in the reference window is obtained from the correlation value obtained by Expression (2).

As shown in equation (2), since the subtraction value of the pixel value of two images is squared, the SSD can clarify the error between both images even when the window size is small. .

  In the search process by NCC, corresponding points are searched in the same manner as SAD except that the degree of coincidence between the image in the reference window and the image in the reference window is obtained from the correlation value obtained by Expression (3).

However, [mu] M _L represents the local mean value of the image within the reference window, [mu] M _R represents a local mean value of the image in the reference window.

  As shown in Equation (3), the NCC obtains a correlation value from the variance value of two images obtained by subtracting the local average value, so that a linear change in brightness (linear change in pixel value and contrast) is obtained. This is a technique that is not affected by noise and noise.

  The search process using the phase only correlation method calculates the degree of coincidence based on the phase component obtained by frequency-decomposing the image in the window set as the reference image and the image in the window set as the reference image, Search for corresponding points. As a method for performing frequency decomposition, any of fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform can be employed.

  FIG. 5 is a diagram showing a flow of processing of the phase only correlation method. First, the image f in the standard window and the image g in the reference window are subjected to a discrete Fourier transform (DFT) to obtain an image F and an image G. Next, the image F and the image G are normalized, and an image F ′ and an image G ′ are obtained. Next, the image F ′ and the image G ′ are combined to obtain a correlation image R. Next, the correlation image R is subjected to inverse discrete Fourier transform (IDFT) to obtain a POC function r. FIG. 6 is a graph showing the POC function r.

  As shown in FIG. 6, the POC function r has a steep correlation peak, and is known to have high robustness and estimation accuracy in image matching. The height of the correlation peak increases as the correlation between the image f and the image g increases. Therefore, by adopting, for example, the height of the correlation peak as the degree of coincidence, it is possible to indicate how close the images f and g in both windows are. Then, when the reference window is shifted, the center point of the reference window when the matching degree becomes the highest is searched for as a corresponding point.

  Next, a description will be given of a decrease in search accuracy due to the inclusion of the outer region in the window. FIG. 7 is an explanatory diagram of the corresponding point search process. In FIG. 7, TP, which is a corresponding point on the reference image of the target point CP set in the base image, is shifted by N pixels in the horizontal direction. Further, in FIG. 7, the reference window W2 is set as the reference image so that the center point of the reference window W2 is positioned at CP ′, which is a point on the reference image having the same coordinates as the point of interest CP, and the search process is started. Then, the reference window W2 is shifted from the CP ′ by one pixel in the horizontal direction. In this case, when the reference window is shifted by N pixels, the degree of coincidence of both images in the standard window W1 and the reference window W2 is the highest, and TP should be searched as a corresponding point.

  However, as shown in FIG. 7, when the outer area D1 is included in the reference window W1 and an edge is generated at the boundary between the reference image and the outer area D1, the corresponding point is not TP but TP due to the influence of the edge. For example, a search is performed at CP ′ on the edge side, and an erroneous search occurs.

  Therefore, both the interpolation units 15a and 15b interpolate the outer region using the following first to fourth methods. First, the first method will be described. FIG. 8 is an explanatory diagram of the first and third methods. The first method is a method of calculating a pixel average value of pixels located at the end E1 of the input image I1 in the window W as an interpolation pixel value, and setting the interpolation pixel value to each interpolation pixel in the outer region D1. . Here, when the window W includes two horizontal and vertical sides of the input image I1 and the outer region D1 has an L shape, one line and one edge are formed in the horizontal direction at the end of the input image I1 in the window W. Pixels included in one line in the vertical direction are pixels located at the end E1. Further, when one side of the input image I1 in the horizontal or vertical direction is included in the window W and the outer region D1 has a strip shape, pixels included in one horizontal line or one vertical line at the edge of the window W. Is a pixel located at the end E1. Note that the number of lines of the pixel located at the end E1 may be a plurality of lines such as two or three lines instead of one line.

  Further, in the first method, each interpolation pixel may be set in the outer region D1 so that the interpolation pixel value gradually changes toward the end of the window W. In this case, the pixel value of the outer region D1 may be increased toward the end of the window W, or may be decreased toward the end of the window W.

  Specifically, when the interpolation pixel value is decreased toward the end of the window W, the interpolation pixel value may be calculated so as to decrease stepwise with reference to the pixel average value. Further, when the interpolation pixel value is increased toward the end of the window W, the interpolation pixel value may be calculated so as to increase stepwise with the pixel average value as a reference.

  When the pixel value is increased toward the edge of the window W, a pixel value and a pixel average value that are larger than the pixel average value by a predetermined value are used as the interpolation pixel values. The pixel value of the outer region D1 may be set so that the number of large pixel values increases.

  According to the first method, since the pixel used for calculation of the interpolation pixel value is only the pixel at the end E1 of the input image I1 in the window, the interpolation pixel value can be obtained at high speed. Further, since the pixel average value of the pixels at the end of the input image I1 in the window W is calculated as the interpolated pixel value, the occurrence of an edge at the boundary between the outer region D1 and the input image I1 can be more reliably suppressed. .

  Next, the second method will be described. FIG. 9 is an explanatory diagram of the second method. In the second method, an input image I1 in the window W, that is, a pixel average value of each pixel constituting a rectangular area indicated by the dark color E2 in FIG. 9 is calculated as an interpolated pixel value, and the outer area D1 is calculated. It is a method to set. Also in the second method, similarly to the first method, the interpolation pixel value may be set in the outer region D1 so as to gradually change toward the end.

  According to the second method, since the pixel average value of the input image I1 in the window W is adopted as the interpolated pixel value, the degree of coincidence between both images in the window W of the standard image and the reference image is determined in both windows. Even when the calculation is locally performed using a part of the area rather than the entire area of the image, the occurrence of an edge at the boundary between the outer area and the input image can be more reliably suppressed.

  Next, the third method will be described with reference to FIG. The third method is a method of calculating, as an interpolated pixel value, a value that gives a certain margin α in the vertical direction with reference to the pixel average value of each pixel located at the end E1 of the input image I1 in the window W. In this case, among the pixels located at the end E1 of the input image I1 in the window W, when the proportion of pixels whose pixel value is higher than the pixel average value is higher than the proportion of pixels lower than the pixel average value, A value obtained by adding a certain margin (= pixel average value + α) as an interpolation pixel value, and vice versa, a value obtained by subtracting a certain margin from the pixel average value (= pixel average value−α) as an interpolation pixel value What is necessary is just to calculate.

  According to the third method, since the value of the interpolated pixel value is adjusted by giving a vertical margin α to the pixel average value, the occurrence of an edge at the boundary between the outer region D1 and the input image I1 is more reliably suppressed. be able to. As the margin α, a value of 1 to 10% of the average pixel value may be adopted, or a value determined in advance according to the average pixel value may be adopted.

Next, the fourth method will be described. FIG. 10 is an explanatory diagram of the fourth method. The fourth approach is to calculate the pixel value of the pixel G _i in the i-th row on the edge of the input image I1 in the window W, as an interpolated pixel value H _i corresponding to the i-th row of the outer area D1, the outer This is a method of setting each interpolation pixel constituting the i-th row of the region D1 using the interpolation pixel value H _i .

  In this case, it is preferable that the search processing unit 16 sets the center point of the reference window to the same line as the line where the target point is located, and searches the corresponding point by shifting the reference window by one line in the line. Since the cameras C1 and C2 are installed in parallel with each other, the corresponding points are likely to be present in the same row as the attention point set in the standard image in the reference image. Therefore, it is possible to search for corresponding points at high speed by shifting the reference image so that the center point is located in this search range, with one row on the same reference image as the row where the attention point is set as the search range. It becomes.

Then, when a search process for searching for corresponding points by shifting the reference window by one line in the horizontal direction is adopted, pixel values are continuous in the horizontal direction even if horizontal stripes occur due to different values in each row in the outer region D1. If it changes, the search will not be affected. Thus, by uniformly filling the outer region D1 in the horizontal direction with the interpolated pixel value H _i , the search accuracy can be increased and the processing speed can be increased.

  In the fourth method, when the window W includes two sides of the input image I1 and the outer region D1 has an L shape, the input image I1 is adjacent to one line at the end parallel to the horizontal direction. The pixel value of each interpolation pixel in the outer region D1 to be processed may be filled with the pixel average value of the pixels located in the one line and within the window W. Specifically, in the case of FIG. 10, using the pixel average value of the pixels located in the first row of the input image I1 in the window W, the pixels of each interpolation pixel in the outer region D1 adjacent to the first row. A value should be set.

  Next, the operation of the corresponding point search device will be described. FIG. 11 is a flowchart showing the operation of the corresponding point search apparatus. The flowchart of FIG. 11 shows a case where two input images captured by the cameras C1 and C2 are processed at time t, and is executed each time an input image is captured from the cameras C1 and C2.

  First, in step S11, the standard image acquisition unit 11a acquires an input image captured by the camera C1 as a standard image, and the reference image acquisition unit 11b acquires an input image captured by the camera C2 as a reference image.

  In step S12, the attention point setting unit 12a sets the attention point at a predetermined position on the reference image, and the reference window setting unit 13a sets the reference window on the reference image with the attention point as the center.

  In step S13, the determination unit 14a determines whether or not an outer region is included in the reference window. If the outer region is included (YES in step S14), the interpolation unit 15a interpolates the outer region using any one of the first to fourth methods (step S15).

  On the other hand, when the outer area is not included in the standard window (NO in step S14), step S15 is passed, and the center point setting unit 12b sets the center point at a predetermined position on the reference image, and the reference window The setting unit 13b sets a reference window on the reference image with the center point as the center (step S16).

  In step S17, the determination unit 14b determines whether or not an outer area is included in the reference window. If the outer region is included (YES in step S18), the interpolation unit 15b interpolates the outer region using any one of the first to fourth methods (step S19).

  On the other hand, if the reference window does not include an outer region (NO in step S18), step S19 is passed, and the search processing unit 16 calculates the degree of coincidence between the images in the reference window and the reference window (step S19). S20).

  In step S21, when the reference window setting unit 13b has set the reference window to the final position and the reference window has been slid (YES in step S21), the search processing unit 16 determines the degree of coincidence in step S20. The center point having the highest degree of coincidence among the center points of the reference window for which is calculated is calculated as the corresponding point (step S22).

  On the other hand, when the reference window setting unit 13b has not set the reference window at the final position (NO in step S21), the process returns to step S16. Then, the center point setting unit 12b sets the next center point in the reference image, and the reference window setting unit 13b sets a reference window around the center point (step S16). The processes in steps S16 to S21 are repeated as described above, and the reference window is shifted on the reference image.

  In step S23, when the reference window setting unit 13a has set the reference window at the final position and the reference window has been slid (YES in step S23), the process is ended. On the other hand, if the reference window is not set to the final position (NO in step S23), the process returns to step S12.

  Note that the search processing unit 16 may search for corresponding points using a multi-resolution strategy method described below. FIG. 12 is an explanatory diagram of the multi-resolution strategy method. The multi-resolution strategy method hierarchically generates a plurality of reference images and reference images having different resolutions so that the resolution increases from the lower layer to the upper layer, and the corresponding point search results in the lower layer Based on the above, a search range smaller than the search range of the hierarchy is set in the reference image of the next higher hierarchy, and search processing for searching for corresponding points within the search range is executed hierarchically toward the upper hierarchy This is a method for searching for corresponding points. According to this method, it is possible to search for corresponding points at high speed and with high accuracy.

  When the multi-resolution strategy method is adopted with the window size in each layer being the same, the resolution of the input image I1 becomes lower toward the lower layer, so that the number of places where the window including the outer region is set increases, and the search accuracy is increased. Becomes lower. Since the upper layer inherits the search result in the lower layer and the search range is set, the search result in the lower layer is important.

  In the corresponding point search apparatus, since the outer region is interpolated using the first to fourth methods, it is possible to search for corresponding points in the lower layers with high accuracy, and the multi-resolution strategy method is adopted. Even in this case, it is possible to search for corresponding points at high speed and with high accuracy.

  FIG. 13 is a flowchart of search processing when the multi-resolution strategy method is adopted. First, in step S31, a standard image and a reference image are acquired as in step S11 shown in FIG.

  In step S32, the point-of-interest setting unit 12a and the center point setting unit 12b execute a process of reducing the resolution of the standard image and the reference image at a predetermined resolution for each layer, and a plurality of standard images and references having different resolutions Generate images hierarchically. Here, the number of layers and the resolution in each layer are determined in advance, and the resolution increases from the lower layer to the upper layer, and the base image and the reference image that have not undergone resolution conversion are the highest layer. Is done.

  In step S33, the attention point setting unit 12a sets the attention point in the reference image of the highest hierarchy. In step S34, the attention point setting unit 12a and the center point setting unit 12b set the attention layer to be processed. Here, the attention layer sequentially sets attention layers toward the highest layer, such as setting the lowest layer as the attention layer and then setting the next higher layer as the attention layer. .

  In step S35, the reference window setting unit 13a sets the reference window with the projected point on the reference image of the attention layer corresponding to the attention point set in step S33 as the attention point.

  In step S36, the determination unit 14a determines whether or not the outer region is included in the reference window set in step S35. If it is determined that an outer region exists (YES in step S37), the interpolation unit 15a interpolates the outer region using any one of the first to fourth methods (step S38). On the other hand, if it is determined that there is no outside area (NO in step S37), step S38 is passed through and search processing is executed by the search processing unit 16 (step S39). Details of the process in step S39 will be described later.

  In step S40, when the search process for the highest hierarchy is completed (YES in step S40), the process proceeds to step S41. On the other hand, if the search process for the highest hierarchy has not been completed (NO in step S40), the process returns to step S34, and the hierarchy one level higher is set as the target layer.

  In step S41, when the last point is set as the attention point among the points that are set in advance as the attention points on the reference image by the attention point setting unit 12a (YES in step S41). Processing is terminated. On the other hand, when the last point is not set as the attention point (NO in step S41), the process returns to step S33, and the next attention point is set.

  Next, the details of the search process in step S39 shown in FIG. 13 will be described. FIG. 14 is a flowchart showing details of the search process. First, in step S61, the center point setting unit 12b sets a search area in the reference image of the target layer. Here, the center point setting unit 12b sets, for example, the entire region as a search range in the reference image of the lowest hierarchy, and in the reference image of the attention layer higher than that, the center point setting layer 12b is one layer below the attention layer. Based on the projection point of the reference image of the target layer corresponding to the corresponding point calculated in step 1, a region that includes the projection point and is narrower than the search range set in the next lower layer is set as the search range. .

  In step S62, the center point setting unit 12b sets a center point at a predetermined position in the search range set in step S61, and the reference window setting unit 13b sets a reference window with the center point as the center.

  In step S63, the determination unit 14b determines whether or not an outer area is included in the reference window. If the outer area is included (YES in step S64), the interpolation unit 15b interpolates the outer area using any of the first to fourth methods (step S65).

  On the other hand, if the outer region is not included (NO in step S64), step S65 is passed, and the search processing unit 16 calculates the degree of coincidence of both images in the reference window and reference window of the target layer (step S66). ).

  In step S67, when the reference window is set to a predetermined final position in the search range by the reference window setting unit 13b and the reference window has finished sliding (YES in step S67), the search processing unit 16 Then, the center point having the highest degree of matching among the center points of the reference window whose degree of matching is calculated in step S66 is calculated as a corresponding point (step S68).

  On the other hand, if the sliding of the reference window has not ended (NO in step S67), the process returns to step S62, and the next center point is set on the reference image of the attention layer.

  As described above, according to the corresponding point search apparatus, even when the window is set so as to protrude from the input image, the corresponding point can be searched with high accuracy without increasing the calculation cost.

  In the above description, one reference image is used. However, the present invention is not limited to this, and a plurality of reference images may be used. In this case, a plurality of cameras corresponding to each reference image may be installed. Further, a plurality of center point setting units 12b to 15b corresponding to each reference image may be provided. And the search process part 16 should just obtain | require a corresponding point with respect to each reference image.

1 shows a schematic configuration diagram of a corresponding point search device according to an embodiment of the present invention. FIG. The functional block diagram of a controller is shown. It is explanatory drawing of the attention point setting process. It is the figure which showed the reference | standard image in which the reference | standard window was set. It is the figure which showed the flow of the process of the phase only correlation method. It is the graph which showed the POC function. It is explanatory drawing of a corresponding point search process. It is explanatory drawing of the 1st and 3rd method. It is explanatory drawing of a 2nd method. It is explanatory drawing of the 4th method. It is a flowchart which shows operation | movement of a corresponding point search apparatus. It is explanatory drawing of a multi-resolution strategy method. It is a flowchart of the search process at the time of employ | adopting a multi-resolution strategy method. It is a flowchart which shows the detail of a search process.

Explanation of symbols

10 controller 11 image acquisition unit 11a reference image acquisition unit 11b reference image acquisition unit 12 setting unit 12a attention point setting unit 12b center point setting unit 13 window setting unit 13a reference window setting unit 13b reference window setting units 14, 14a and 14b determination unit 15, 15a, 15b Interpolation unit 16 Search processing unit C1, C2 Camera

Claims (10)

An acquisition unit that acquires one input image as a standard image and another input image as a reference image among a plurality of input images that capture the same scene;
Attention points are sequentially set in the reference image, a window is set in the reference image so as to include the set attention points, a window is set in the reference image, and both windows are set while shifting the window set in the reference image. Search means for searching for a corresponding point of the attention point from the reference image by obtaining a degree of coincidence of images in the window;
Determining means for determining whether or not the window includes an outer region that protrudes from the image region of the input image each time the window is shifted;
Interpolating means for calculating an interpolated pixel value of the outer area based on a pixel value of a pixel located at least at an end of the input image in the window when the determining means determines that the outer area is included. for example Bei the door,
The interpolation means calculates a pixel average value of pixels at the edge of the input image in the window as the interpolation pixel value, and the interpolation pixel value so as to gradually change toward the edge of the window in the outer region. Corresponding point search device characterized by calculating . An acquisition unit that acquires one input image as a standard image and another input image as a reference image among a plurality of input images that capture the same scene;
Attention points are sequentially set in the reference image, a window is set in the reference image so as to include the set attention points, a window is set in the reference image, and both windows are set while shifting the window set in the reference image. Search means for searching for a corresponding point of the attention point from the reference image by obtaining a degree of coincidence of images in the window;
Determining means for determining whether or not the window includes an outer region that protrudes from the image region of the input image each time the window is shifted;
Interpolating means for calculating an interpolated pixel value of the outer area based on a pixel value of a pixel located at least at an end of the input image in the window when the determining means determines that the outer area is included. for example Bei the door,
The interpolation means calculates a pixel average value of each pixel constituting the input image in the window as the interpolation pixel value, and the interpolation pixel gradually changes toward the edge of the window in the outer region. A corresponding point search apparatus characterized by calculating a value . An acquisition unit that acquires one input image as a standard image and another input image as a reference image among a plurality of input images that capture the same scene;
Attention points are sequentially set in the reference image, a window is set in the reference image so as to include the set attention points, a window is set in the reference image, and both windows are set while shifting the window set in the reference image. Search means for searching for a corresponding point of the attention point from the reference image by obtaining a degree of coincidence of images in the window;
Determining means for determining whether or not the window includes an outer region that protrudes from the image region of the input image each time the window is shifted;
Interpolating means for calculating an interpolated pixel value of the outer area based on a pixel value of a pixel located at least at an end of the input image in the window when the determining means determines that the outer area is included. for example Bei the door,
The interpolating means calculates as the interpolated pixel value a value giving a certain margin in the vertical direction with reference to the pixel average value of each pixel located at the end of the input image in the window. apparatus. The interpolation means adds a certain margin to the average pixel value when the ratio of pixels having a pixel value higher than the average pixel value is higher than the ratio of low pixels at the edge of the input image in the window. 4. The corresponding point search device according to claim 3 , wherein a value obtained by subtracting a certain margin from the average pixel value is used as the interpolated pixel value when the interpolated pixel value is the interpolated pixel value. An acquisition unit that acquires one input image as a standard image and another input image as a reference image among a plurality of input images that capture the same scene;
Attention points are sequentially set in the reference image, a window is set in the reference image so as to include the set attention points, a window is set in the reference image, and both windows are set while shifting the window set in the reference image. Search means for searching for a corresponding point of the attention point from the reference image by obtaining a degree of coincidence of images in the window;
Determining means for determining whether or not the window includes an outer region that protrudes from the image region of the input image each time the window is shifted;
Interpolating means for calculating an interpolated pixel value of the outer area based on a pixel value of a pixel located at least at an end of the input image in the window when the determining means determines that the outer area is included. for example Bei the door,
The interpolation means calculates a pixel value of a pixel in each row located at an end of the input image in the window as an interpolated pixel value corresponding to each row in the outer region, and moves toward the edge of the window in the outer region. A corresponding point search device characterized in that the interpolated pixel value is calculated so as to change gradually with time . The search means hierarchically generates a plurality of the reference images and the reference images having different resolutions so that the resolution increases from the lower hierarchy toward the upper hierarchy, and searches for corresponding points in the lower hierarchy Based on the result, a search range smaller than the search range of the hierarchy is set in the reference image of the next higher hierarchy, and the search process for searching for a corresponding point within the search range is hierarchically performed toward the upper hierarchy. corresponding point search device according to any one of claims 1 to 5, characterized in that searching the corresponding points by running. The search means, according to claim 1-6, characterized in that calculating the degree of coincidence by correlation operation with the image in the window set in the image and the reference image in the window set in the reference image The corresponding point search apparatus according to any one of the above. The search means calculates the degree of coincidence based on a phase component obtained by frequency-decomposing an image in a window set as the reference image and an image in a window set as the reference image. The corresponding point search device according to any one of claims 1 to 7 . 9. The corresponding point search according to claim 8 , wherein the search means performs frequency decomposition using any one of fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform. apparatus. The corresponding point search apparatus according to claim 9 , wherein the search means calculates the degree of coincidence using a phase only correlation method.