JP4517449B2 - Correlation calculation method for images - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to a correlation calculation method in which an object is imaged by a pair of cameras, and a distance distribution from the camera to the object or a surface direction of a surface forming the object is detected based on a deviation amount (parallax) between the images. In particular, the present invention relates to a correlation calculation method for efficiently and rapidly calculating a distance distribution to an object by converting an image into multiple resolutions.
  The present invention is applied to, for example, a distance distribution detection device that calculates a distance distribution to an object around a vehicle mounted on a vehicle at high speed and high density.
[0002]
[Conventional skills]
  Conventionally, there are object recognition apparatuses using image processing. For example, a vehicle exterior monitoring device disclosed in Japanese Patent Laid-Open No. 5-265547 is one of them. It is a system that recognizes the position, size, and distance of an object by mounting two cameras on the vehicle, imaging the object in real time, and performing stereo matching processing on the two-screen images.
  If two cameras arranged at a predetermined separation distance image a front object, the object is imaged in the X direction in the left and right images. This is because parallax occurs in the two cameras. If the object is in the vicinity of the camera, the parallax is large, and if the object is far away, the parallax is small. The stereo matching process is a technique for detecting the distance to the object using this principle.
[0003]
  A conventional distance distribution measurement procedure will be briefly described. In the conventional distance distribution measurement, first, an image of an object is input from each of two cameras, and A / D conversion is performed to obtain digitized image data (left and right images). Next, for example, a 4 × 4 small area is cut out from the left and right images, and correlation calculation is performed to obtain the parallax of the small area. The parallax is the amount of shift between the small areas in the x direction, for example. From this deviation amount, the distance from the camera to the small area constituting the object, that is, the distance to the object is obtained. If the camera separation distance is B, the focal distance is f, and the amount of deviation is ΔX pixels, the distance z to the object is expressed by the following equation. This distance is obtained for each small area, and the distance distribution to the object, that is, the object shape and posture are calculated.
[Equation 1]
      z = f · B / αΔX
      α = size of one pixel of the image sensor (1)
[0004]
[Problems to be solved by the invention]
  However, the vehicle exterior monitoring device disclosed in Japanese Patent Application Laid-Open No. 5-265547 performs a collation operation in a predetermined area (for example, 4 × 4 pixels) over the entire area of a given image. In this case, collation is performed every 4 pixels, but when high density (every 1 pixel) collation is performed, the amount of calculation becomes enormous, and it is not suitable for detecting the distance in real time, for example.
  In general, in correlation calculation between images using the sum of absolute differences, if the image size is vertical H, horizontal V, the size of the collation area is vertical M, horizontal N, and the search range is D, the correlation S is given by Given. This means that the amount of calculation becomes enormous in proportion to the image size, the collation area size, and the search range.
[Expression 2]
  S = Σ^H _{y = 0}Σ^V _{x = 0}Σ^D _{d = 0}Σ^M _{m = 0}Σ^N _{n = 0}
      ｜ F_r(X + m, y + n) -F_l(X + m + d, y + n) | (2)
  Where F (x, y) is the pixel intensity at the coordinates (x, y), and F_rIs the right image, F_lMeans the left image.
[0005]
  The present invention has been made to solve the above-described problems, and its object is to convert an object image into a multi-resolution hierarchical image, and the correlation between the hierarchical images between the hierarchical images in the next stage. By determining the correlation calculation parameters, the distance distribution to the object is obtained at high speed and with high density, and the surface direction of the surface on which the object is formed is accurately obtained.
[0006]
[Means for Solving the Problems]
  In order to achieve this object, according to the object recognition device according to claim 1, an object is imaged by a plurality of imaging devices, and a plurality of small region images are cut out from the obtained one image as a collation region, and each collation is performed. Distance distribution from the imaging device to the object by a correlation calculation between the image of the region and the otheras well asA correlation calculation method in an image for detecting a surface direction of a surface forming the object, wherein a plurality of images having different resolutions are generated for the plurality of images,The pyramid structure has a hierarchical structure in which the resolution and area of the layer changes uniformly as the N value indicating the index of the layer changes.Multi-resolution image generating means for forming an image, and when N and M are integers, each parallax for each collation area is obtained by correlation calculation between N layers, and collation for M layer correlation calculation is performed based on each parallax The distance distribution to the object is determined by determining the area and its search area, and performing the correlation calculation on each of the determined matching areas and each search areaIn each layer forming the pyramid structure, the collation region and the search region for the correlation calculation of the lower layer are calculated from the result obtained by the correlation calculation of the upper layer of the adjacent layer, and the lower layer is sequentially calculated. From the distance distribution of the hierarchy in which three points can be selected from the distance distribution obtained by the calculation for each hierarchy, the correlation calculation is finally performed on the 0th layer that is the bottom surface of the pyramid structure by repeating toward Select three points, determine the plane equation that passes through the three points, and from the plane equation,Detecting a surface direction of a surface forming the objectThis is a correlation calculation method in an image.
[0007]
  According to the correlation calculation method for an image according to claim 2,Three points of the 0th layer image corresponding to the three selected points are obtained, a plane equation passing through the three points is determined from the three points of the zeroth image, and the plane forming the object is determined from the plane equation. The surface direction is detected.
[0008]
[0009]
  or,Claim 3According to the correlation calculation method in the image described in the above, the presence or absence of correlation by the correlation calculation has the minimum correlation value or / and the minimum correlation value based on the sum of absolute differences between the matching area image and the search area image as a vertex. It is determined by an apex angle related value.
  or,Claim 4According to the correlation calculation method for an image described in (1), the image of each layer is an edge image, and the collation area is set on the edge image.
[0010]
[Action and effect of the invention]
  According to the object recognition device of the first aspect, the plurality of imaging devices capture an object and obtain a plurality of images. Then, a plurality of images having different resolutions are generated from the images by multi-resolution image generation means, and images are formed in a hierarchical structure for each resolution.
  The plurality of images having different resolutions are performed by, for example, averaging pixel intensities within a predetermined region, Gaussian pyramid (subsampling conversion using a Gaussian filter of the original image), wavelet conversion (decomposing the image by spatial frequency), and the like.
  At this time, for example, the 0th layer is the original image with the highest resolution, and becomes a low-resolution image with lower resolution as it goes to the upper layer.
[0011]
  Then, a small area image is cut out from an arbitrary N-layer image and used as a collation area, and the collation area is subjected to correlation calculation along the X direction with respect to the other N-layer image. That is, the correlation calculation between the N layers is performed, and each parallax for each matching area set on the N layer image is obtained. Then, based on each parallax, a collation area for the correlation calculation between the M layers and its search area are determined. Next, a correlation calculation between the M layers is performed in each of the determined matching areas and each search area. That is, the distance distribution to the object by the M layer imageAnd thingsThe surface direction of the surface forming the body is detected. At this time, N ≠ M, and N and M may be continuous or may be discrete.
[0012]
  Here, the N layer is a low resolution image and the M layer is a high resolution image. Since the N-layer image has a low resolution and a small area, the correlation calculation between the images is completed at high speed. That is, the collation area and the search area for the correlation calculation between the M layers, which is the next process, are quickly calculated and determined. That is, the calculation in the non-correlation region in the correlation calculation between the M layers is omitted. That is, the correlation calculation between the M layers is efficiently performed. Further, since the M layer image has a higher resolution than the N layer image, the distance distribution to the object is calculated densely. That is, if the correlation calculation method for an image of the present invention is used, the distance distribution to the object can be calculated at high speed and with high density. Also, the surface direction of the surface on which the object is formed is calculated from the distance distribution at high speed and with high density.
[0013]
  or,Of the present inventionAccording to the object recognition apparatus, the hierarchical structure is a pyramid structure in which the resolution and area of the layer change uniformly as the N value indicating the index of the hierarchy changes.
  For example, the 0th layer with N = 0 has the highest resolution in the original image, and as the N value increases, the resolution becomes lower and the area becomes smaller. Since the low-resolution image on several layers has a pyramid structure, the processing area (correlation calculation processing) is linearly small, and the distance distribution to the object is calculated at high speed.
  For example, if the object has a complex shapeAboveCorrelation calculation is performed in the rank layer (N layer). And the collation area | region and search area | region of the lower layer (M layer) are decided from the result. Because of the pyramid structure, the collation area and the search area can be easily determined linearly. Therefore, the correlation calculation in the M layer can be processed more quickly and efficiently. Therefore, the distance distribution to the object can be obtained with high density at high speed.
[0014]
  or,Of the present inventionAccording to the object recognition device, in each layer forming the pyramid structure, a collation region and a search region are calculated for the correlation calculation of the lower layer immediately below by the correlation calculation of the upper layer of the adjacent layer. Do it sequentially. That is, the collation region and the search region are sequentially determined from the predetermined layer on the apex side of the pyramid structure downward, and the correlation calculation is performed. This is repeated until the final layer.
  That is, the correlation calculation is finally performed on the 0th layer, which is the bottom surface of the pyramid structure, and the distance distribution to the object or the surface direction of the surface forming the object is calculated from the result.
  In this way, the resolution and area of the image are formed in a pyramid structure, and the collation area and search area are sequentially updated from the low resolution layer (upper layer) to the high resolution layer (lower layer), so that the correlation calculation is performed. It is efficient and can reliably determine the distance distribution to the object. Further, the surface direction of the surface on which the object is formed can be reliably calculated from the distance distribution.
[0015]
  or,Claim 3According to the object recognizing device described in (1), the sum of absolute differences between the image in the collation area and the image in the search area is first obtained for determining the presence or absence of correlation. Then, the presence or absence of correlation is determined by the correlation minimum value or / and the vertex angle related value having the correlation minimum value as a vertex.
  Here, the apex angle-related value is the apex angle formed by the minimum point and the points on both sides thereof, the inclination of the side forming the apex angle, or the area of a polygon formed including the apex angle.
[0016]
  For example, the presence or absence of correlation is examined with respect to a large number of local minimum values of the correlation calculation result. If the minimum value is significantly below a predetermined threshold, it is determined that there is a correlation, and the separation distance between the collation area and the search area is set as parallax.
  For example, if there are a plurality of candidates for the minimum correlation value, the vertex angle related value including the minimum value is examined. If the apex angle is a sharper angle, the parallax is obtained with the minimum correlation value. On the contrary, if all the minimum values are larger than a predetermined threshold value, it is determined that there is no correlation.
  As described above, since the presence / absence of correlation is determined by the correlation minimum value and / or the vertex angle related value having the correlation minimum value as a vertex, the parallax, that is, the distance distribution to the object and the surface of the surface forming the object A direction is calculated.
[0017]
  or,Claim 4According to the correlation calculation method for an image described in (1), the image is an edge image, and the collation area is set on the edge image.
  The edge image is obtained, for example, by a Sobel operation process, and thereby the outline of the object, the edge, and the boundary between the regions constituting the object are emphasized. And the said collation area | region is set on the edge and the said correlation calculation process is performed. Since the collation area is set only on the edge image, not the entire area of the image, the correlation calculation amount is significantly reduced. Thereby, the distance distribution to the object and the surface direction of the object can be obtained at higher speed.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
(First embodiment)
  FIG. 1 shows an embodiment to which the correlation calculation method for images according to the present invention is applied. FIG. 2 is a system configuration diagram of the distance distribution calculation apparatus. This system is a multi-resolution that generates a hierarchical image by performing high-speed A / D conversion on video signals sent from the cameras 10, 11 and cameras 10, 11 arranged at a predetermined separation distance, and multiplexing the resolutions. A multi-resolution generation unit 20 that is an image generation unit, a spatial filter unit 30 that converts an image of each layer generated by the multi-resolution generation unit 20 into an edge image, and a correlation operation that stereo-matches each layer edged by the spatial filter unit 30 Unit 40, a distance distribution calculation unit 50 that calculates a distance distribution from the result of the correlation calculation unit 40, and a surface direction estimation unit 60 that calculates a surface direction from each distance data obtained from the distance distribution calculation unit 50.
  Here, it is assumed that the correlation calculation unit 40 includes a stereo matching unit 41i (i = 0... N) that performs stereo matching on each layer image, and stereo matching is performed between layers of the same index. Note that correlation between images and stereo matching have the same meaning. Hereinafter, the correlation calculation between the images of each layer is referred to as stereo matching.
[0019]
  The multi-resolution generation unit 20, the spatial filter unit 30, the correlation calculation unit 40, the distance distribution calculation unit 50, and the plane direction estimation unit 60 include a CPU (not shown) and a ROM and work area memory (not shown) in which an arithmetic processing program is written. The RAM is not shown. The above elements are connected via a system bus including various signal lines (not shown), and data is transferred and controlled by various programs written in the CPU and ROM.
[0020]
  FIG. 2 shows the basic operation of the correlation calculation method for images in this embodiment. The figure is a flowchart. Thereby, the distance distribution to the object and the surface direction of the surface forming the object are calculated at high speed and with high density.
  First, the program is turned on by a start switch (not shown) and executed from step s100. In step s100, the cameras 10 and 11 as imaging devices are turned on to capture an object image, and in step s110, the input image is A / D converted and taken into a frame memory (not shown). The frame memory is composed of a plurality of RAMs, and one unit (one sheet) is, for example, a total of 512 × 512 RAMs corresponding to each pixel of the CCD image sensor. An arbitrary pixel is represented by coordinates (x, y) and intensity F, and the intensity F is graded, for example, in 0 to 255 levels. Hereinafter, this gradation image is referred to as image data. At this time, the upper left corner of the screen is the origin of the xy coordinate system, and the X axis is set in the horizontal direction and the Y axis is set in the vertical direction.
[0021]
  Next, the process proceeds to step s120. In step s120, the multi-resolution image generator 20 multi-resolutions the image and stores it in the RAM in a hierarchical structure. Here, the original image is a high-resolution image with the highest resolution, and images are hierarchically formed at a plurality of resolutions based on this. These are formed by, for example, an averaging method, a Gaussian pyramid, a wavelet transform, or the like in an area in which surrounding pixels of a specific pixel are averaged to be one pixel.
[0022]
  FIG. 3 shows an example in which the averaging method is sequentially performed toward the upper layer and images are configured according to resolution. The lowest layer of the 0th layer is the original image. For example, if the size of the original image is 512 × 512, L = 9 in the figure, and the uppermost layer (initial layer) is represented by N = 3.
  Specifically, the original image is formed by sequentially compressing according to the following equation.
[Equation 3]
    F_N(X, y) = 1/4 [F_N-1(X, y) + F_N-1(X + 1, y) + F_N-1(X, y + 1) + F_N-1(X + 1, y + 1)] (3)
Where F_N(X, y) is the intensity of each pixel at the coordinates (x, y) of the N layer.
  The above equation is a case where the original image is compressed to ½ both vertically and horizontally as it goes to the upper layer. That is, the area is compressed by 1/4. This also means that the resolution decreases proportionally. An example of the image hierarchized into the 0th layer to the 3rd layer is shown in FIG.
[0023]
  Next, the process proceeds to step s130 (FIG. 2). In step s130, each layer is spatially filtered. This is because the density change of each image is emphasized to simplify the processing. The spatial filter is, for example, sharpening of an image by first-order differentiation and second-order differentiation. For example, if one-dimensional differentiation is used, an object is converted into an edge image. The edge image is, for example, a contour image of a front object, and is a boundary image between the surfaces constituting the object. Next, the process proceeds to step s140.
[0024]
  In step s140, stereo matching is performed between the layers on the edged image. Stereo matching is a correlation operation that searches for matching points between two images.
  FIG. 5 shows a general flow of stereo matching in each layer. First, in step s141, in order to minimize the amount of calculation, a block that is a collation area is set on one of the edge images. Then, the absolute value of the difference from the other edge image is obtained using the block. That is, block correlation based on the sum of absolute differences is performed, and as a result, a plurality of corresponding point candidates (hereinafter, parallax candidates) are obtained. The parallax candidates are a plurality of minimum points of the correlation value obtained by the above equation (2).
  In step s141, for the first stereo collation (block correlation) for the N layer, the collation area is a predetermined value given on the ROM, and the search area is all in the horizontal direction of the predetermined layer. For the second and subsequent stereo collations for the M layer, the collation area and search area updated as described in detail in step s144 are used. Next, the process proceeds to step s142.
[0025]
  In step s142, it is determined whether the parallax candidate obtained in step s141 satisfies a predetermined condition. That is, the reliability of each parallax is evaluated (reliability evaluation). In the reliability evaluation, one is a comparison between each correlation value of each parallax candidate and a reference correlation value (sCorr). The other one is a comparison between each correlation gradient value, which is the vertex angle related value of the parallax candidate, and the reference correlation gradient value (sGrad). The reliability is evaluated by either or both. Next, the process proceeds to step s143, and the parallax candidate having the highest evaluation degree is set as the parallax in the layer (determination of parallax). Next, the process proceeds to step s144.
[0026]
  In step s144, the collation area | region and search area | region of another layer (M layer) are updated using the parallax of the N layer. Since the resolution image has a pyramid structure, when searching for the M layer, the matching region is determined according to the inclination of the pyramid structure. For example, if the collation area is a block shape, each side of the block is multiplied by a predetermined linear coefficient to be updated as an M-layer collation area.
[0027]
  Further, the position of the search area of the M layer, which is the other layer, is roughly determined by the result of the stereo matching (parallax) and the linear coefficient of the pyramid structure. For example, the M layer is the next layer of the N layer. That is, M = (N−1). FIG. 6 shows the relationship between search areas in the N layer and the (N-1) layer. This is a standardized image size of the N layer to the (N-1) layer. That is, the N layer image is enlarged so as to match the (N-1) layer image. Where S_NIs the Nth layer collation region and S_N-1Is that of the (N-1) th layer. Since the Nth layer is compressed, it is expanded by the collation area of the (N-1) layer when normalized. For example, it is enlarged four times. In other words, the parallax is roughly obtained by performing stereo matching in a large matching region in the N layer.
[0028]
  S '_NIs a matching region by stereo matching in the N layer. That is, it is a matching area whose reliability is determined to be high. Thereby, the parallax d_NIs roughly determined. And this approximate parallax d_NIs used to update the search area in the (N-1) layer stereo matching. That is, the region T_N-1Becomes the search area in the (N-1) layer stereo matching. At this time, region T_N-1X-direction width T_{x (N-1)}Is given by:
[Expression 4]
  d_N-△ d_N≦ T_{x (N-1)}≦ d_N+ △ d_N    (4)
  △ d_N: Constant specific to N layer
    d_N: Parallax of the Nth layer
[0029]
  In other words, if the (N-1) layer is stereo-correlated within the range of the equation (4), it means that a matching region can be easily obtained. That is, it is not necessary to search over all areas in the horizontal direction. That is, the processing time is greatly shortened.
  Therefore, in the (N-1) layer, the same stereo matching is performed in the search area, and the parallax d is more detailed and faster._N-1Is required. Moreover, the parallax d_N-1Naturally satisfies the following equation.
[Equation 5]
  d_N-△ d_N≦ d_N-1≦ d_N+ △ d_N      ... (5)
  d_N-1: Parallax of the (N-1) th layer
If the M layer is the next layer, the collation area and the search area are set in step s144 in this way.
[0030]
  Next, the process proceeds to step s150 (FIG. 2). In step s150, the distance is calculated based on the above equation (1). For example, the collation area is divided into a large number in the M layer, and the distance is calculated for each parallax. That is, a distance distribution is obtained (distance distribution output). Next, the process proceeds to step s160.
  In step s160, the coefficient of the plane equation aX + bY + cZ = 0 is obtained using the distance distribution obtained in step s150, that is, a plurality of distance data. For example, it is obtained by the least square method. Thereby, the surface direction of the surface forming the object is calculated.
[0031]
  As described above, according to the present embodiment, the obtained image is resolution-multiplexed to obtain an approximate parallax in the upper-layer low-resolution image, and the collation area and search area of the other layer are determined based on the parallax. Stereo matching is performed efficiently, and as a result, the distance distribution to the object can be obtained at high speed and high density. Further, the surface direction of the surface on which the object is formed can be obtained using the distance distribution.
[0032]
  (Second embodiment)
  The correlation calculation method in the image of the first embodiment is a correlation calculation method related to the N layer and M layer of the multi-resolution image, particularly the M = N−1 layer. For example, in the case of N = 1, stereo matching is performed on one layer, and stereo matching is further performed on the 0th layer (original image) based on the result, thereby calculating the distance distribution to the object.
  The present embodiment is an example in which the above concept is extended to calculate a distance distribution or the like using a plurality of layers. That is, this is an example of calculating the distance distribution and the like more efficiently and at high speed by repeating the method of the first embodiment a plurality of times.
[0033]
  FIG. 7 shows a correlation calculation method in which a multi-resolution image formed in a pyramid structure is processed from the third layer downward to the 0th. The figure is a flowchart. This is also an example of steps s140 and s150 in FIG. 2 of the first embodiment. The other steps s100 to s130 are the same and will be omitted.
  First, in step s170, stereo matching between the third layers is performed. Step s170 is equivalent to steps s141 to s144 shown in FIG. 5 of the first embodiment. That is, the block correlation for the edge image is the same as in the first embodiment. Thereby, parallax is obtained for a predetermined block. And it transfers to step s171 and parallax is converted into a distance.
[0034]
  Next, in step s172, the distance calculation in the block is checked. The distance calculation check corresponds to the reliability evaluation of the first embodiment. That is, if the reliability (for example, the correlation value) is larger than the predetermined value, it is determined that the region is a coincidence area, and the process proceeds to step s173.
  If the reliability is smaller than the predetermined value, it is determined that there is no matching area, that is, the process proceeds to step s190. In step s190, it is checked whether or not it is the last block, and the process proceeds to the next block or ends.
[0035]
  In step s173, stereo matching is performed between the second layers. At this time, the collation area and the search area have already been updated by the third layer process (step s144). Therefore, there is no need to newly search (correlation calculation) over the entire surface in the horizontal direction, for example. Therefore, the stereo matching is executed in a very short time, and the second layer parallax is calculated. In step s175, the distance is similarly calculated from the parallax, and the reliability is similarly determined in step s176.
  The stereo matching result (distance image) in the third layer in step s170 is shown in FIG. 8A, and that in the second layer in step s173 is shown in FIG. 8B. In FIG. 8A, a to f are blocks that have been matched by the correlation calculation (distance is calculated). In the second layer, each block is divided into four to obtain a more detailed distance distribution.
[0036]
  The same process as the series of processes (steps s173 to s176) is repeated for the first layer. That is, steps s177 to s180 are the same. Finally, stereo collation is performed on the 0th layer in step s181 to obtain a more detailed parallax, and a more detailed distance distribution is obtained in step s183.
  In this way, stereo matching is sequentially performed from the third layer toward the zeroth layer, and the distance distribution to the partial object is calculated efficiently and reliably. Next, as in the first embodiment, the process proceeds to step s160 to estimate the surface direction of the surface on which the object is formed.
[0037]
  When the distance distribution is sequentially calculated as described above, the surface direction may be estimated by the following process without waiting for the result of the distance distribution in the 0th layer. The surface direction estimation method is shown in the flowchart of FIG.
  In step s200, a distance image is read from the stereo matching result in the second layer. The distance image of the second layer is divided into four as shown in FIG. 8B and is calculated in more detail. Next, the process proceeds to step s201. In step s201, it is checked whether or not there are three or more data in the distance image. If there are three or more data, the surface is specified, so the process proceeds to step s205, and the plane is applied to the plane equation as described above. If it is 3 data or less, the plane cannot be specified, and the process proceeds to step s202.
[0038]
  In step s202, the distance image of the first layer is read out. The distance image of the first layer is calculated in more detail by dividing into 16 though not shown. Next, the process proceeds to step s203. In step s203, it is checked whether or not there are three or more data in the distance image. If there are three or more data, the surface is specified in the same manner, so the process proceeds to step s205 to determine the plane. If it is 3 data or less, the plane cannot be specified, and the process proceeds to step s204.
[0039]
  In step s204, the distance image of the 0th layer is read. This distance image is a result of stereo matching performed on the original image. The area is divided into 64 areas. If there are three or more data in this area, the process proceeds to step s205, and if it is less, the process proceeds to step s207. Since the data here is the distance image with the highest accuracy, the surface direction with the highest accuracy is obtained in step s205. Next, the process proceeds to step s207.
[0040]
  In step s207, it is checked whether or not the estimation of the surface direction has been completed for all the blocks in the second layer. If it is not the final block, the process proceeds to step s200 and the same processing is repeated for the other blocks. If it is the last block, the process ends. The estimation of the surface direction may be performed in this way. If the plane direction is estimated from the intermediate layer of the pyramid structure, the time can be shortened.
[0041]
  As described above, according to the present embodiment, the collation area and search area of the next layer are sequentially determined from the lower resolution image on the upper layer of the multi-resolution image, and more detailed stereo collation is performed.
  Therefore, the distance distribution to the object can be obtained more reliably with higher density. Further, if the distance distribution is obtained in the intermediate layer having the pyramid structure reaching the 0th layer as described above, the surface direction of the surface on which the object is formed can be obtained more efficiently using the distance distribution. When estimating the surface direction in detail, it is only necessary to sequentially perform the collation operation up to the last 0th layer. Thereby, the surface direction can be estimated with higher accuracy.
[0042]
(Modification)
  Although one embodiment representing the present invention has been described above, various other modifications are conceivable.
  For example, in the calculation of the surface direction in the second embodiment, the distance images to the second layer, the first layer, and the zeroth layer are sequentially obtained, and thereby the surface direction of the object is obtained. As shown in FIG. The collation area (block) may be read from the third layer, the process may proceed to step s211 and the distance image corresponding to the block may be directly specified and read from the zeroth layer. Then, data may be extracted from the distance image and the process may proceed to step s206. This step s206 is equivalent to that of the second embodiment.
  That is, steps s200, s201, s202, s203, and s204 in FIG. 9 in the second embodiment may be shortened to steps s210 and s211 in FIG. The surface direction can be estimated more efficiently and with higher accuracy.
[0043]
  For example, in the reliability calculation of the first embodiment, the comparison with the correlation value gradient sGrad is used as the vertex angle related value, but other vertex angle related values may be used. For example, the apex angle of the minimum point, a predetermined reference correlation value (sCorr), and the area on the minimum value side surrounded by the correlation curve may be used. Further, in the first embodiment, stereo matching is performed between edge images that have been edged. However, gradation image data can also be performed.
[Brief description of the drawings]
FIG. 1 is a system block diagram to which a correlation calculation method for images according to a first embodiment of the present invention is applied.
FIG. 2 is a flowchart showing a processing procedure of a correlation calculation method for an image according to the first embodiment of the present invention.
FIG. 3 is an explanatory diagram of a multi-resolution image having a pyramid structure according to the first embodiment of the present invention.
FIG. 4 is a plan view showing an example of each layer of a multi-resolution image according to the first embodiment of the present invention.
FIG. 5 is a flowchart showing a procedure of stereo matching according to the first embodiment of the present invention.
FIG. 6 is a diagram showing the relationship between the N-layer and (N-1) -layer collation areas and search areas according to the first embodiment of the present invention.
FIG. 7 is a flowchart of a correlation calculation method for multilayer multi-resolution images according to the second embodiment of the present invention.
FIG. 8 is an explanatory diagram of a third layer distance image (FIG. 8A) and a second layer distance image (FIG. 8B) based on a stereo matching result according to the second embodiment of the present invention;
FIG. 9 is a flowchart showing a surface direction estimation method according to the second embodiment of the present invention.
FIG. 10 is a flowchart showing a surface direction estimation method according to a modification of the present invention.
[Explanation of symbols]
10,11 camera
20 Multi-resolution image generator
30 Spatial filter section
40 Correlation calculator
40i stereo matching unit
50 Distance distribution calculator
60 plane direction estimation unit

Claims (4)

An image of an object is captured by a plurality of imaging devices, and a plurality of small area images are cut out from one obtained image as a collation region, and the distance from the imaging device to the object is calculated by correlation between each collation region and the other image A correlation calculation method in an image for detecting a distribution and a surface direction of a surface forming the object,
A plurality of images with different resolutions are generated for each of the plurality of images, and the resolution and area of the layer changes uniformly as the N value indicating the index of the layer changes for each resolution. includes a multi-resolution image generation means for forming an image on,
When N and M are integers, each parallax for each collation area is obtained by correlation calculation between N layers, and the collation area and its search area for the M layer correlation calculation are determined based on each parallax, and the parallax is determined. The distance distribution to the object is obtained by performing the correlation calculation in each matching area and each search area ,
In each layer forming the pyramid structure, the collation area and the search area for the correlation calculation of the lower layer are calculated from the result obtained by the correlation calculation of the upper layer of the adjacent layer, and the results are sequentially directed toward the lower layer. By repeating, the correlation calculation is finally performed on the 0th layer which is the bottom surface of the pyramid structure,
Three points are selected from the distance distribution of the hierarchy in which three points can be selected from the distance distribution obtained by the calculation for each layer, a plane equation passing through the three points is determined, and the object is determined from the plane equation. A correlation calculation method for an image, characterized by detecting a surface direction of a surface to be formed. Three points of the 0th layer image corresponding to the selected three points are obtained, a plane equation passing through the three points is determined from the three points of the zeroth image, and the object is formed from the plane equation 2. The correlation calculation method for an image according to claim 1, wherein a surface direction of the surface to be detected is detected. Presence / absence of correlation by the correlation calculation is determined by a minimum correlation value based on a sum of absolute differences between images in a collation area and a search area or / and an apex angle related value having the minimum correlation value as a vertex. The correlation calculation method for an image according to claim 1 or 2 . Wherein each layer of the image is an edge image, the matching area correlation calculation method in an image according to any one of claims 1 to 3, characterized in that it is set on the edge image.

Images