JP2006047252A - Image processing unit - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enhance precision for evaluation reliability in misdetection of correspondence point with respect to matching processing. <P>SOLUTION: A plurality of images Gb, Gc are picked up by an imaging part 2, comprising one or a plurality of cameras 9, 10, the matching processing for the images Gb, Gc is carried out in a stereo-processing part 5, one or a plurality of feature amounts are extracted from the images Gb, Gc by a determination part 11 of a parallex correction part 7, reliability of matching processings for the plurality of images by the stereo-processing part 5 is evaluated based on the one or a plurality of feature amounts, and mismatchings generated in the matching processing are corrected by a subsequent correction part 12, based on the evaluation information from the determination part 11. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, and in particular, to improve the accuracy of detection of erroneous corresponding points and the accuracy of reliability evaluation for matching processing based on a plurality of feature amounts extracted from a plurality of images subjected to matching processing. The present invention relates to an image processing apparatus that can

  In recent years, an obstacle detection device and an altitude measurement device have been developed as image measurement devices for obtaining three-dimensional shape information, and are mounted on a moving body such as an automobile or a helicopter. These apparatuses use a stereo image method using two cameras, and by using a triangulation principle, a set of stereo images is processed to obtain three-dimensional distance information, and the obtained distance information is obtained. Based on this, self-position recognition is performed.

  In the stereo image method described above, the same object is imaged from two or more different gaze directions using a plurality of cameras, and the images in the obtained plurality of images are associated with each other, that is, corresponding points (hereinafter referred to as matching processing). Is an image processing method for obtaining the position of a measurement object in a three-dimensional space. Typical matching methods used in this case include a pixel-based matching method, an area-based matching method, and a feature- The based-based matching method is widely used, and the Area-based matching method is widely used as a method that is highly accurate and effective for obtaining the three-dimensional shape of the object for each pixel.

  In the area-based matching method, when a point corresponding to a point in one image is searched for in another image, an image (hereinafter referred to as a reference image Gb) captured by one camera (hereinafter referred to as a reference camera) is used. Set a certain local window and use this as a template to match within the search range set while changing the position on an image (hereinafter referred to as comparative image Gc) captured by another camera (hereinafter referred to as comparative camera) Specifically, as shown in FIG. 8, a plane composed of the focus Ob of the reference camera, the focus Oc of the comparison camera, and the observation point P, the image plane of the reference image Gb, and the comparison image, as shown in FIG. Scanning is performed while shifting one pixel at a time along a straight line (hereinafter referred to as epipolar line E) in which the image plane of Gc is orthogonal, and the corresponding pixel within the set search range ( (Corresponding points) and searching for the minimum or maximum points of the evaluation formulas such as SAD (sum of absolute differences), SSD (sum of squared differences), normalized correlation, etc. Making a decision. In addition, the determination as to whether or not these corresponding points are true correct is performed by threshold processing or filtering processing based on the analysis value of the extracted evaluation function, the feature amount such as the relevance to the surrounding area, and the like. When it is determined from the processing result that an incorrect corresponding score (hereinafter referred to as “mismatching”) has occurred, the mismatching is suppressed by performing a correction process.

  However, even when the stereo image method based on the above-described Area-based matching method is used, it is difficult to completely eliminate the mismatching, and 3 or more cameras that are developed as one of the countermeasures are used. The occurrence of mismatching can be suppressed by using a multi-baseline stereo image device that performs a matching process using two or more images. However, the configuration of the apparatus becomes complicated, and the number of images handled in the matching process increases, resulting in an increase in processing steps.

  Therefore, as an image processing device capable of suppressing mismatching in the stereo image method, in the distance image generation in which the measurement target images are captured from different viewpoints and the matching processing between the images is performed, the inter-image correspondence in the distance image generation The template size to be applied to the attaching process is made variable, and feature values such as rate of change, matrix eigenvalue, variance value, etc. are calculated and compared with a threshold value. An image processing apparatus that performs matching processing in a state where the appearance of a pattern is reduced has been developed (see, for example, Patent Document 1).

In addition, as an image processing apparatus capable of removing an erroneous correspondence point due to mismatching and correcting appropriately, a small area centered on a small area of interest from among stored three-dimensional image information by majority filtering means Extract a group, set the addition frequency according to the address and parallax of each small area of the small area group based on preset conditions, extract the maximum frequency parallax by adding the addition power for each parallax, An image processing apparatus that corrects the parallax of a small region of interest in accordance with the maximum power parallax has been developed (see, for example, Patent Document 2).
JP 2001-194126 A Japanese Patent Laid-Open No. 11-248446

  However, in the case of the above-described image processing apparatus, since the evaluation of reliability of the miscorresponding point and the matching process is performed based only on individual feature amounts such as the analysis value of the evaluation function, the variance of the luminance value, etc., When matching processing is performed on an object that has almost no features such as shape and color, or on an object that has the same or similar pattern in succession, the detection accuracy of the miscorresponding point and the reliability of the matching process There is a problem that the accuracy of evaluation is lowered.

  Further, there is a problem that the measurement accuracy of the calculated three-dimensional image information is lowered by performing the subsequent correction process based on the detection result and the reliability evaluation with low accuracy.

  The present invention has been made in view of the above-described points, and the subsequent processing performed based on the detection result and the reliability evaluation by improving the accuracy of detecting the miscorresponding point and the accuracy of the reliability evaluation for the matching process. An object of the present invention is to provide a stereo image processing apparatus capable of improving the measurement accuracy of the three-dimensional image information calculated by the correction process.

In order to solve the above problems, an image processing apparatus according to the invention described in claim 1 includes:
An imaging unit composed of one or more cameras that capture a plurality of images;
In an image processing apparatus comprising a stereo processing unit that performs matching processing on the plurality of images,
A determination element extraction unit that extracts one or a plurality of feature amounts from a plurality of images subjected to matching processing;
A parallax correction unit including a determination unit that performs reliability evaluation of the matching process based on the extracted feature amount, and a correction unit that corrects an erroneous response that occurs during the matching process based on evaluation information by the determination unit It is characterized by comprising.

  According to the first aspect of the present invention, since the reliability evaluation of the matching processing for the plurality of images by the stereo processing unit is determined based on the plurality of feature amounts, the determination by one feature amount is impossible or the determination result Even if the measurement object is ambiguous, it is possible to accurately detect the miscorresponding points and evaluate the reliability of the matching process.

  An image processing apparatus according to a second aspect of the invention is characterized in that the determination unit is configured using a neural network that performs reliability evaluation of matching processing on a plurality of images by the stereo processing unit. .

  According to the second aspect of the present invention, since the determination unit is configured using a neural network, it is possible to perform reliability evaluation in a short processing time even when a large amount of feature is input. .

  In the image processing apparatus according to the third aspect of the present invention, the feature amount is an analysis result of an evaluation function used for matching processing, an element indicating originality of a pixel area, or an element indicating relevance to a surrounding pixel area. It includes at least one of them.

  According to the third aspect of the present invention, since the reliability evaluation for the matching process is performed in the determination unit to which a plurality of types of feature amounts are input, determination using one feature amount is impossible or the determination result is ambiguous. Even a measurement object can be combined with other feature quantities to perform accurate miscorresponding point detection and reliability evaluation for matching processing.

  The image processing apparatus according to claim 4 is characterized in that an analysis result of an evaluation function used in the matching process includes at least one of an element indicating a value of an evaluation function or a shape of the evaluation function. To do.

  According to the invention described in claim 4, the value of the evaluation function that has been separately input and the element indicating the shape of the evaluation function are input together, so that the detection accuracy of the miscorresponding point and the matching process are performed. Therefore, it is possible to improve the accuracy of reliability evaluation.

  According to a fifth aspect of the present invention, in the image processing apparatus according to the fifth aspect, the element indicating the shape of the evaluation function is at least one of the number of local minimum points, the width of the valley including the minimum point, or the acute angle of the valley including the minimum point. It is characterized by including one.

  According to the invention described in claim 5, as the element indicating the shape of the evaluation function, the number of local minimum points, the width dimension of the valley including the minimum point, and the acute angle of the valley including the minimum point are newly added. Thus, it is possible to improve the accuracy of detection of erroneous corresponding points and the accuracy of reliability evaluation for matching processing.

  The image processing apparatus according to claim 6, wherein the element indicating the originality of the pixel region is a number of pixels in which an absolute value of a difference in a direction along the epipolar line in the pixel region exceeds a predetermined value. And at least one of an average of absolute values of differences in a direction along the epipolar line in the pixel region and a variance of luminance values in the pixel region.

  According to the invention described in claim 6, in addition to the dispersion of the luminance value in the pixel area that has been conventionally used, the pixel whose absolute value of the difference in the direction along the epipolar line in the pixel area exceeds a predetermined value And the average of the absolute value of the difference in the direction along the epipolar line in the pixel area are added as new elements, so that the detection accuracy of miscorresponding points and the accuracy of reliability evaluation for matching processing are added. Can be improved.

  In the image processing apparatus according to claim 7, the number of pixel regions and the number of local minimum points whose elements indicating the relationship with the surrounding pixel regions are less than a predetermined value of the evaluation function in the self-matching process. , Including at least one of the width dimension of the valley including the minimum point, the acute angle of the valley including the minimum point, or the number of pixel regions outputting the same parallax in the vicinity of one arbitrarily selected pixel region. And

  According to the seventh aspect of the present invention, in addition to the number of blocks that output the same parallax in the vicinity of one arbitrarily selected pixel area in the conventionally used self-matching process, the evaluation function in the self-matching process is used. The number of blocks whose value is less than the specified value, the number of local minimum points, the width of the valley including the minimum point, and the acute angle of the valley including the minimum point are added as new elements, and detection of miscorresponding points The accuracy and the accuracy of the reliability evaluation for the matching process can be improved.

  According to the present invention, in an image processing apparatus including an imaging unit including one or a plurality of cameras that capture a plurality of images, and a stereo processing unit that performs a matching process on the plurality of images captured by the imaging unit. A determination element extraction unit that extracts one or a plurality of feature amounts from a plurality of images subjected to matching processing, a determination unit that performs reliability evaluation of matching processing based on the extracted feature amounts, and an evaluation by the determination unit Since it includes a parallax correction unit that includes a correction unit that corrects miscorrespondence that occurs during matching processing based on information, even if determination by one feature amount is impossible or the determination result is ambiguous, By making a determination based on a plurality of feature amounts, it is possible to improve the accuracy of detection of miscorresponding points and the accuracy of reliability evaluation for matching processing. It is possible to improve the measurement accuracy of the three-dimensional image information calculated by the subsequent correction process is performed based on the results and reliability evaluation.

The best mode for carrying out the present invention will be described below with reference to the drawings. However, although various technically preferable limitations for implementing the present invention are given to the embodiments described below, the scope of the invention is not limited to the following embodiments and illustrated examples.
The image processing apparatus according to the present invention will be described with reference to FIGS.
As shown in FIG. 1, the image processing apparatus 1 according to the present embodiment includes an imaging unit 2, an image input unit 3, an original image memory 4, a stereo processing unit 5, a determination element extraction unit 6, and a parallax correction unit. 7 and a distance image memory 8.

  The imaging unit 2 includes two cameras 9 and 10, and these cameras 9 and 10 are cameras that can be synchronized with each other and have a variable shutter speed, a CCD (Charge Coupled Device) sensor, An image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor is incorporated. One of these cameras (hereinafter referred to as reference camera 9) captures a reference image Gb during stereo processing, and the other camera (hereinafter referred to as comparison camera 10) captures a comparison image Gc during stereo processing. The cameras 9 and 10 are arranged so that the vertical axes of the imaging planes are parallel to each other with a predetermined baseline length.

  The image input unit 3 corresponds to the analog imaging signal transmitted from the imaging unit 2 and an analog interface (not shown) having a gain control amplifier, and a log conversion table (logarithmic conversion is performed on the light and dark part of the image). (Not shown), an A / D converter (not shown) for converting analog image data into a digital image, and a correction circuit (not shown) for correcting brightness and geometrically converting the image. It is configured. In such an image input unit 3, the brightness balance of the analog image signal is adjusted for each imaging signal transmitted from the imaging unit 2 by the gain control amplifier in the analog interface, and then the contrast of the low-brightness portion is obtained by log conversion. Image correction such as improvement is performed, and further, it is converted into digital image data of a predetermined luminance gradation, for example, 256 gradation gray scale by the A / D converter, and finally the image is rotated or translated by the correction circuit. The deviation between the reference image Gb and the comparative image Gc due to the camera mounting position is corrected by geometrical transformation such as so-called affine transformation.

  The original image memory 4 includes a semiconductor memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory), a recording medium such as a magnetic disk and an optical disk, and stores the digital image data converted by the image input unit 3. It has become so.

The stereo processing unit 5 includes a city block distance calculation circuit, a minimum / maximum value detection circuit, and a shift amount determination circuit. The stereo processing unit 5 responds by obtaining a correlation between the reference image Gb and the comparison image Gc. A pixel is specified, and perspective information to the target is digitized as three-dimensional image information (hereinafter, distance image) based on a pixel shift amount (hereinafter, parallax X) that occurs according to the distance to the target. It is like that.
Among them, in the city block distance calculation circuit, as shown in FIG. 2, the horizontal 5 × vertical of each of the images Gb and Gc is applied to the two images of the reference image Gb and the comparison image Gc stored in the original image memory 4. A matching process is performed for each pixel area of 5 pixels (hereinafter referred to as a matching block), and a city block distance C, which is an index of the degree of coincidence between the reference image Gb and the comparison image Gc, is calculated as a value of SAD as an evaluation function. It has become so. The minimum / maximum value detection circuit detects the minimum value Cmin, maximum value Cmax, and average value Cave of the calculated city block distance C, and the subsequent shift amount determination circuit uses the obtained minimum value Cmin as the reference image. It is verified whether or not Gb and the comparison image Gc coincide with each other, and the parallax X is determined.

SAD is the sum of absolute values obtained by subtracting the luminance values Ib _{i, j} and Ic _{i, j} of corresponding pixels in the reference image Gb and the comparison image Gc, as shown in Equation 1 below. As shown, the SAD function is obtained on the two-dimensional coordinates with the city block distance C calculated by SAD as the vertical axis and the parallax X as the horizontal axis, and the degree of coincidence between the reference image Gb and the comparison image Gc, that is, matching The city block distance C is smaller as the processing reliability is higher, and the city block distance C is larger as the reliability is lower.

ここで、Ｉｂ_i,jは、基準画像Ｇｂにおけるマッチングブロックの横ｉ番目、縦ｊ番目の画素の輝度値を、Ｉｃ_i,jは、比較画像Ｇｃにおけるマッチングブロックの横ｉ番目、縦ｊ番目の画素の輝度値を示す。

Here, Ib _{i, j} is the luminance value of the i-th and j-th pixels of the matching block in the reference image Gb, and Ic _{i, j} is the i-th and j-th columns of the matching block in the comparison image Gc. The luminance value of the pixel is shown.

  Further, the verification performed by the deviation amount determination circuit is such that the maximum value Cmax or the minimum value Cmin of the city block distance C is a maximum value ≦ Ca and a maximum value−minimum with respect to a preset threshold value Ca or Cb. Whether or not each of the values ≧ Cb is satisfied is satisfied, and further, whether or not the luminance difference between adjacent pixels in the horizontal direction in the matching block of the reference image Gb is larger than a preset threshold value. Is called. When these three conditions are satisfied, the parallax X is output, and when the conditions are not satisfied, the data is not adopted and rejected.

  The determination element extraction unit 6 is configured to extract a feature amount that is a determination criterion when performing reliability evaluation for the matching process. The extracted feature values are roughly divided into three elements, the analysis result of SAD, which is an evaluation function, the element indicating the originality of the matching block, and the element indicating the relationship with the surrounding matching block. Will be described in detail.

First, the analysis result of SAD will be described.
The analysis result of SAD is divided into a value of city block distance C and an element related to the shape of the SAD function. Among these, the value of the city block distance C includes a minimum value Cmin, a maximum value Cmax, and an average value Cave of the city block distance C, and is calculated by a city block distance calculation circuit in the stereo processing unit 5.

On the other hand, elements relating to the shape of the SAD function include the number of minimum points, the width W1 of the valley including the minimum point, and the acute angle of the valley including the minimum point.
The number of local minimum points is defined as N points (n) (n) near the point i, where Ci is the city block distance of the i-th horizontal pixel in the matching block and the average value of the SAD function is a threshold value. against N> fixed value _{2), C iN> C i-} (N-1) ...> C i and, satisfying point _{C i + N> C i +} (N-1) ...> C i Is the number of valleys, that is, the number of valleys.

The width dimension W1 of the valley including the minimum point is the minimum n that satisfies k × C _in > C _{in-1 in the} left vicinity direction with respect to the minimum point, and k × C _{i + m} > C _{i + in the} right vicinity direction. _This is n + m obtained by adding the minimum m for which _{m + 1} holds, and the valley width dimension W1 is larger as the gradation change continuous to the object, the so-called gradation is gentler, and smaller as it is steeper. Here, k is a constant and is normally set to 1.

  The acute angle of the valley including the minimum point is calculated by two types of methods. One is the average value of the city block distances C at both ends of the width dimension W1 of the valley including the minimum point and the city block at the minimum point. This is a method of calculating by dividing the difference from the value of the distance C by the width dimension W1 of the valley including the minimum point, and the acute angle of the entire valley can be detected. The other is a method of calculating the average of the difference between the values of the city block distance C on both sides of the valley including the minimum point as shown in the following formula 2, and can detect the acute angle of the neighborhood. . The sharp angle of these valleys is such that the gentler the gradation, the smaller the acute angle of the valley, and the sharper the gradation, the larger the acute angle of the valley.

Next, the element which shows the originality of a matching block is demonstrated.
As elements indicating the originality of the matching block, the number of pixels in which the absolute value of the difference in the direction along the epipolar line E in the matching block exceeds a predetermined value and the direction along the epipolar line E in the matching block Average value of the absolute value of the difference between them and variance of the luminance value in the matching block.

Absolute value D _i of the difference in the direction along the epipolar line E in the matching _{block, j} is calculated by the following equation number 3, of the calculated difference absolute value D _{i, j} is the pixel exceeds a predetermined value The number is a feature value.

In addition, the absolute value D _{i, j} of the difference in the direction along the epipolar line E in the matching block obtained by the above-described calculation method is averaged by the following formula 4 to obtain the epipolar line E in the matching block. The average Dave of the absolute value of the difference in the direction along the direction is used as the feature amount.

  Further, using the average Dave of the absolute values of the differences in the direction along the epipolar line E in the above-described matching block, the luminance value variance V in the matching block expressed by Equation 5 below is used as a feature amount.

Finally, the element indicating the relevance with the peripheral matching block will be described.
As elements indicating the relationship with the peripheral matching block, the number of blocks similar to the own block in the self-matching process, the average value of the SAD function in the self-matching process, the value related to the shape of the SAD function in the self-matching process, And the number of blocks that output the same parallax around the block.

  In the self-matching process, the number of blocks similar to the self-block is the same image, and an arbitrarily selected matching block (hereinafter referred to as self-block) of 5 × 5 pixels along the epipolar line E In this case, a scan is performed while shifting the block one block at a time, and an association, a so-called self-matching process, is performed between the own block and a matching block existing along the epipolar line E, and the city block distance Cs is Calculated. The number of blocks in which the city block distance Cs calculated in this way is less than a predetermined value is the number of matching blocks similar to the own block in the self-matching process, and the greater the number, the lower the reliability for the matching process. The smaller this number is, the higher it becomes.

The city block distance Cs calculated by the self-matching process is the brightness value S _{i, j} of the own block and the brightness value of the matching block existing along the epipolar line E, as shown in the following equation (6). It is obtained from the sum of absolute values of differences from S _{i + 1, j} . Also, as shown in FIG. 4, a graph of the SAD function is plotted on the two-dimensional coordinates with the vertical axis of the city block distance Cs calculated by the self-matching process and the horizontal axis of the scanning direction Y along the epipolar line E. Desired.

ここで、Ｓ_i,jは、図５に示すように、縦３×横３ブロックのマッチングブロック領域の縦ｉ番目、横ｊ番目のマッチングブロックの輝度値を、Ｓ_i+1,jは、縦ｉ＋１番目、横ｊ番目のマッチングブロックの輝度値を示す。

Here, as shown in FIG. 5, S _{i, j} is the luminance value of the i-th and j-th matching blocks in the matching block region of 3 × 3 blocks, and S _{i + 1, j} is The luminance values of the i + 1th vertical and jth horizontal matching blocks are shown.

  The average value of the city block distance Cs of the self-matching process is a value obtained by averaging the city block distances Cs calculated by the self-matching process. The smaller the value, the lower the reliability for the matching process. The larger this value, the higher the reliability.

  Values relating to the shape of the SAD function in the self-matching process include the number of local minimum points, the width W2 of the valley including the minimum point, and the acute angle of the valley including the minimum point. Here, the minimum point of the SAD function in the self-matching process is a position where the own block exists, that is, a so-called self position, and the city block distance Cs is zero. The calculation method of these values is the same as the calculation method of the value related to the shape of the SAD function in the above-described normal matching process, and the reliability criterion for the matching process is also the same.

  The number of blocks that output the same parallax around the own block is output from among the eight matching blocks surrounding the own block when the central matching block in the matching block area shown in FIG. This is the number of matching blocks in which the parallax is the same as the parallax output by the own block. The greater the number, the higher the reliability for the matching process, and the lower the reliability.

  As shown in FIG. 6, the parallax correction unit 7 includes a determination unit 11 that performs detection of an erroneous correspondence point and reliability evaluation for matching processing, and a correction unit 12 that performs correction processing based on evaluation information from the determination unit 11. It is prepared for. Among these, the determination unit 11 is configured using a non-linear three-layer hierarchical neural network 13 as shown in FIG. 7, and is transmitted from the minimum / maximum value detection circuit and the determination element extraction unit 6 of the stereo processing unit 5. On the basis of the feature amount, a miscorresponding point generated in the matching process is detected, and the matching process by the stereo processing unit 5 is determined with three levels of reliability of “high”, “medium”, and “low”. For example, when it is determined as “low”, correction processing is performed by the subsequent correction unit 12, and an accurate distance image is transmitted to and stored in the subsequent distance image memory 8.

Next, the operation of the image processing apparatus in this embodiment will be described.
By pressing an input switch (not shown) of the image processing apparatus 1, the reference image Gb and the comparison image Gc that are analog data are respectively captured by the two cameras 9 and 10 in the imaging unit 2, and the image input unit 3. Sent to. The transmitted reference image Gb and comparison image Gc are subjected to image correction processing such as improving the contrast of the low-brightness portion using a log conversion table after the brightness balance is aligned by the analog interface in the image input unit. Further, after being converted into digital data of a predetermined luminance gradation by the A / D converter, affine transformation is performed by the correction circuit, which is transmitted and stored in the original image memory 4.

  The reference image Gb and the comparison image Gc stored in the original image memory 4 are scanned while being shifted one block at a time along the epipolar line E existing in the comparison image Gc by the city block distance calculation circuit in the stereo processing unit 5. Thus, the city block distance C between one pixel in the reference image Gb and one corresponding pixel in the comparison image Gc is calculated. Further, the minimum value Cmin, the maximum value Cmax and the average value Cave of the calculated city block distance C are detected by the minimum / maximum value detection circuit, and the minimum value Cmin is compared with the reference image Gb in the subsequent shift amount determination circuit. It is verified whether or not it is coincident with Gc, and the parallax X is determined.

  The parallax X determined in this way and the minimum value Cmin, the maximum value Cmax, and the average value Cave of the city block distance C detected by the minimum / maximum value detection circuit are transmitted to the parallax correction unit 7. At the same time, the determination element extraction unit 6 extracts a value related to the shape of the SAD function, which is a feature amount, an element indicating the originality of the matching block, and an element indicating the relevance to the peripheral blocks, and the parallax correction unit 7 Sent.

  In the parallax correction unit 7, the determination unit 11 detects a miscorresponding point based on these feature amounts, and performs reliability evaluation for the matching process. The detection result of these miscorresponding points and the evaluation degree of the matching process are transmitted to the subsequent correction unit 12, and the distance image is corrected according to the evaluation degree to correspond to one pixel of the reference image Gb. The distance image is stored in the distance image memory 8 as a distance image. Further, by repeating the same operation, a distance image over the entire image is calculated and stored in the distance image memory 8 to complete a series of image processing operations.

  At this time, the reliability evaluation for the detection of the miscorresponding point by the determination unit 11 of the parallax correction unit 7 and the matching process is performed by analyzing the SAD extracted by the minimum / maximum value detection circuit of the stereo processing unit 5 and the determination element extraction unit 6. As a result, the feature quantity such as the originality of the matching block and the relevance to the surrounding matching block is identified by a binary value of “correct” or “false” or a ternary value of “correct”, “false” or “neither”. 7 is input to a discriminator, that is, a non-linear three-layer hierarchical neural network 13 as shown in FIG. 7, an erroneous correspondence point is detected based on the input feature amount, and the reliability for the matching process is “high”. Judgment is made in three stages, “medium” and “low”. When the reliability is determined to be “high”, the correction process is not performed in the correction unit 12, and when it is determined to be “low”, the correction process is performed. On the other hand, when it is determined as “medium”, for example, a correction process is performed based on information on matching blocks having a high reliability around the own block, and an accurate distance image over the entire image is obtained. Is calculated.

  As described above, according to the image processing apparatus of the present embodiment, the imaging unit 2 including the two cameras 9 and 10 that capture the two images Gb and Gc, and the two images Gb captured by the imaging unit 2. , Gc, and a stereo processing unit 5 that performs matching processing on Gc, a determination element extraction unit 6 that extracts a plurality of feature amounts from the two images Gb and Gc that have been subjected to matching processing, A determination unit 11 that performs reliability evaluation of matching processing on a plurality of images Gb and Gc by the stereo processing unit 5 based on the extracted feature amounts, and a matching process that is performed based on evaluation information by the determination unit 11 And a parallax correction unit configured to include a correction unit 12 that corrects erroneous correspondence. Even when determination with one feature amount is impossible or the determination result is ambiguous, a plurality of feature amounts are included. Therefore, it is possible to improve the accuracy of detection of miscorresponding points and the accuracy of the reliability evaluation for the matching process, and thereby the subsequent correction performed based on the detection result and the reliability evaluation. Measurement accuracy in the processing can be improved.

  Note that the imaging unit 2 is not limited to the present embodiment, and is configured by a single camera that captures a plurality of images at regular time intervals, and what speed component each frame point in each image has. A velocity vector field representing the so-called optical flow may be detected.

  Further, although all the extracted feature values are input to the determination unit 11, not all the feature values are necessarily required, and only the feature values required according to the application example are input. Also good.

  Further, the design of the discriminator and learning acquisition by the discriminator are not limited to the present embodiment, but those commonly used in conventional pattern recognition, for example, the k-nearest neighbor discrimination method (k-NN). ), A support vector machine (SVM), a subspace method, or the like may be applied.

  Furthermore, the determination unit 11 does not necessarily have to be configured with a neural network, and only a part of the determination unit 11 may be configured with a neural network.

It is a block diagram which shows the structure of the image processing apparatus which concerns on this invention. It is a top view which shows the matching block of the image processing apparatus which concerns on this invention. It is a graph which shows the result of the matching process by the image processing apparatus which concerns on this invention. It is a graph which shows the result of the self-matching process by the image processing apparatus which concerns on this invention. It is a top view which shows the own block and periphery matching block of the image processing apparatus which concerns on this invention. It is a block diagram which shows the structure of the parallax correction part of the image processing apparatus which concerns on this invention. It is the schematic which shows the neural network applied to the image processing apparatus which concerns on this invention. It is a conceptual diagram which shows the principle of the stereo imaging method using two cameras.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Image pick-up part 5 Stereo processing part 6 Determination element extraction part 7 Parallax correction part 8 Image memory 9 Reference camera 10 Comparison camera 11 Determination part 12 Correction part 13 Neural network E Epipolar line Gb Reference image Gc Comparison image W1, W2 Width dimension of the valley including the minimum point

Claims (7)

An imaging unit composed of one or more cameras that capture a plurality of images;
In an image processing apparatus comprising a stereo processing unit that performs matching processing on the plurality of images,
A determination element extraction unit that extracts one or a plurality of feature amounts from a plurality of images subjected to matching processing;
A parallax correction unit including a determination unit that performs reliability evaluation of the matching process based on the extracted feature amount, and a correction unit that corrects an erroneous response that occurs during the matching process based on evaluation information by the determination unit An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the determination unit is configured using a neural network that performs reliability evaluation of matching processing on a plurality of images by the stereo processing unit.   The feature amount includes at least one of an analysis result of an evaluation function used for matching processing, an element indicating originality of a pixel area, and an element indicating relevance with a surrounding pixel area. An image processing apparatus according to 1.   The image processing apparatus according to claim 3, wherein an analysis result of the evaluation function used for the matching processing includes at least one of an element indicating a value of the evaluation function or a shape of the evaluation function.   5. The element according to claim 4, wherein the element indicating the shape of the evaluation function includes at least one of a number of local minimum points, a width dimension of a valley including a minimum point, or an acute angle of a valley including a minimum point. Image processing device.   The element indicating the originality of the pixel area is the number of pixels in which the absolute value of the difference in the direction along the epipolar line in the pixel area exceeds a predetermined value, the difference in the direction along the epipolar line in the pixel area The image processing apparatus according to claim 3, comprising at least one of an average of absolute values of values and a variance of luminance values in a pixel region.   The elements indicating the relationship with the surrounding pixel area include the number of pixel areas whose evaluation function value in the self-matching process is lower than a predetermined value, the number of minimum points, the width dimension of the valley including the minimum point, and the minimum point. 4. The image according to claim 3, comprising at least one of the number of pixel areas that output the same parallax at an acute angle of a valley or in the vicinity of one arbitrarily selected pixel area. 5. Processing equipment.

Images