JP2005250994A - Stereo image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform stereo matching with high accuracy to achieve enhancement of data reliability and reduction of variation. <P>SOLUTION: The resolution is expanded by performing interpolation between pixels to a pair of stereo images picked up by a stereo camera 10, to which input processing is performed by an image input part 20 and stored in an original image memory 25 by a resolution expansion part 30, and the stereo matching is performed by an SAD operation to the image whose resolution is expanded by a stereo matching part 35. Then, the stereo matching with high accuracy is performed to achieve enhancement of the data reliability and reduction of the variation by finding parallax at a subpixel level with high accuracy by performing interpolation processing by thee points as a point of agreement defined by the minimum value of an SAD operation value and adjacent points before and after the agreement point by an interpolation processing part 40. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a stereo image processing apparatus that performs stereo matching using image pairs having a correlation with each other.

  In general, as a three-dimensional measurement technique using an image, a correlation between a pair of images obtained by capturing an object from different positions with a stereo camera is obtained, and a triangle is obtained from parallax with respect to the same object using camera parameters such as the stereo camera mounting position and focal length. Image processing based on a so-called stereo method for obtaining a distance by the principle of surveying is known.

As image processing by this stereo method, the present applicant previously measured the three-dimensional position of an object outside the vehicle by processing an image picked up by a stereo camera mounted on the vehicle in Japanese Patent Laid-Open No. 5-11409. In this technology, stereo matching that searches the corresponding position by calculating the degree of correlation between multiple small areas is performed at high speed using a hardware circuit for a pair of stereo images. The pixel shift amount of the small area to be output is output as parallax.
Japanese Patent Laid-Open No. 5-114099

  By the way, the parallax obtained by the stereo matching is calculated as three-dimensional distance data in the real space by converting the image plane coordinates to the real space coordinates, and the vehicle exterior situation recognition system, intelligent road traffic system (ITS), robot It is applied to various recognition processes such as vision, intruder monitoring system, and aircraft altitude measurement system.

  In order to perform such recognition processing with high reliability, it is necessary to reduce mismatching and data variation in stereo matching for calculating parallax and to increase matching accuracy. In order to suppress this mismatching, it is effective to increase the matching area that is a unit for calculating the parallax, that is, to increase the number of matching data. Conventional stereo matching is based on the “pixel” of the image. Therefore, when the matching area is enlarged, there is a disadvantage that the absolute number of calculated parallaxes is insufficient and the spatial resolution of the distance data is lowered.

  The present invention has been made in view of the above circumstances, and provides a stereo image processing apparatus that enables high-precision stereo matching without causing a reduction in resolution, and that can achieve improved data reliability and reduced variation. The purpose is that.

  In order to achieve the above object, a stereo image processing apparatus according to the present invention is a stereo image processing apparatus that performs stereo matching using image pairs having a correlation with each other, and includes data of peripheral pixels between the pixels of the image pair. The virtual pixel generated by using and inserting the virtual pixel to expand the resolution of the image pair, and the image pair expanded by the resolution expanding unit, the corresponding position of each other at the resolution of the expanded resolution And stereo matching means for specifying the above.

  A stereo image processing apparatus according to the present invention is a stereo image processing apparatus that performs stereo matching using image pairs having correlation with each other, and replaces data of each pixel of the image pair with data of peripheral pixels. The image conversion means and stereo matching means for specifying the corresponding positions of the image pairs in which the pixel data is replaced by the image conversion means are provided.

  A stereo image processing apparatus according to the present invention is a stereo image processing apparatus that performs stereo matching using image pairs having correlation with each other, and each pixel data for at least one of the image pairs. Means for replacing the image data using peripheral pixel data, and stereo matching means for specifying the corresponding positions in the image pair using the replaced pixel data.

  The stereo image processing apparatus of the present invention enables high-precision stereo matching without causing a reduction in resolution, and can achieve improved data reliability and reduced variation.

  Embodiments of the present invention will be described below with reference to the drawings. 1 to 9 relate to a first embodiment of the present invention, FIG. 1 is a block diagram showing a basic configuration of a stereo image processing apparatus, FIG. 2 is an explanatory diagram showing matching between a main image and a sub image, and FIG. Is an explanatory diagram showing the distribution of city block distances, FIG. 4 is an explanatory diagram showing interpolation after matching in units of pixels, FIG. 5 is a flowchart of stereo matching and sub-pixel processing, and FIG. 6 is an explanatory diagram showing resolution expansion of the main image FIG. 7 is an explanatory diagram showing the resolution expansion of the sub-image, FIG. 8 is an explanatory diagram showing the interpolation after matching in units of sub-pixels, and FIG. 9 is a histogram showing the parallax variation.

  In FIG. 1, reference numeral 1 denotes a stereo image processing apparatus that obtains three-dimensional distance information by processing a pair of images obtained by stereo imaging of a subject. For example, a vehicle exterior situation recognition system, an intelligent road traffic system (ITS) It is applied to various systems such as robot vision, intruder monitoring system, and flying object ground altitude measurement system.

  The stereo image processing apparatus 1 includes a stereo camera 10 composed of a pair of cameras, an image input unit 20 that performs input processing on a pair of images captured by the stereo camera 10, and images processed by the image input unit. An original image memory 25 that stores the original image, a resolution expanding unit 30 that expands the resolution of the pair of original images, a stereo matching unit 35 that evaluates the degree of correlation between the pair of images and determines the corresponding positions, and a stereo matching unit Interpolation processing unit 40 that obtains sub-pixel level parallax having a resolution of one pixel unit or less by interpolation processing based on the corresponding position determined in 35, and distance information based on the parallax is associated with the image coordinates of the original image in two dimensions. Distance image memory 45 to store as an arranged image form (distance image), original image information in original image memory 25 and distance image memory With 5 distance information configured with a recognition processing unit 50 for performing various recognition processes.

  The two cameras 10a and 10b constituting the stereo camera 10 have a CCD, a CMOS image sensor, or the like as an image sensor, and are cameras that can be synchronized with each other and have a variable shutter speed. One camera 10a is used as a main camera that captures a reference image (main image) during stereo processing, and the other camera 10b is used as a sub camera that captures a comparative image (sub image) during stereo processing. In the present embodiment, it is assumed that the two cameras 10a and 10b are arranged with a predetermined baseline length on a horizontal plane with their optical axes parallel to each other.

  The image input unit 20 includes various circuits (for example, a gain control amplifier, a lookup table, a shading correction circuit, etc.) for processing two systems of analog image signals, an A / D converter that converts analog signals into digital signals, and the like. In addition, an image adjustment circuit that corrects slight misalignment of the mechanical optical positions of the cameras 10a and 10b by geometrical transformation such as image rotation and translation is provided. Composed.

  That is, the captured images from the cameras 10a and 10b input to the image input unit 20 are corrected for sensitivity differences between the cameras 10a and 10b and distortion of the camera optical system, and then digital image data having a predetermined luminance gradation. Furthermore, the error of the mechanical attachment position of the cameras 10a and 10b is corrected by image adjustment, and the coincidence of epipolar lines in the stereo image pair of the main image and the sub image is guaranteed. The stereo image pair that has undergone such input processing is stored in the original image memory 25.

  From the stereo image pair stored in the original image memory 25, parallax is obtained by searching for a corresponding position while evaluating the degree of correlation between the main image and the sub image (stereo matching). Distance information can be obtained. The correlation between the main image and the sub-image is obtained by using a well-known region search method, and the difference (absolute value) of pixel values between the small region of the main image and the small region of the sub-image as an evaluation function of the correlation. Sum (Absolute Difference) is calculated. In general, the luminance value of each pixel is often used as the pixel value.

The value of the evaluation function by SAD is referred to as the city block distance. The higher the correlation between small regions (the more similar), the smaller the value of the city block distance, and the minimum city block distance. The parallax is given by the amount of shift in the horizontal direction between the subregions. The city block distance C is defined as an orthogonal coordinate in which the horizontal direction is i-coordinate and the vertical direction is j-coordinate, and a small area in which a correlation degree is searched is i × j (i = 0). ˜n, j = 0 to n), the main image search block main (i, j) and the sub image search block sub (i, j) as shown in the following equation (1): Are calculated by shifting the SAD value by a predetermined shift value on the i-axis.
C = Σ | main (i, j) -sub (i, j) | ... (1)
Details of stereo matching between small regions (search blocks) are described in detail in Japanese Patent Laid-Open No. 5-1114099 previously filed by the present applicant.

  In the above SAD calculation, for example, for a search block of 4 × 4 pixels, a search block of a sub image (hereinafter referred to as “sub block”) with respect to a search block of a main image (hereinafter referred to as “main block”). When the SAD calculation is performed while shifting the position of (described) on the horizontal line by one pixel at a time, as shown in FIG. 2, the point where the city block distance becomes the minimum value C0 has the highest degree of correlation between the main block and the sub block. It is obtained as a corresponding position (matching point).

  The amount of shift in units of one pixel between the main block and the sub block at this coincidence point (difference between the position im of the main block in the horizontal direction and the position is of the sub block) is a parallax (pixel parallax) having a resolution of one pixel unit. give. The distance information based on the pixel parallax is less than one pixel unit in order to accurately determine the distance to a distant subject because the resolution decreases as the distance to the subject increases due to the principle of triangulation in the stereo method. It is necessary to obtain a sub-pixel level parallax having a resolution of.

  As a technique for obtaining the sub-pixel level parallax, for example, an interpolation processing technique disclosed in Japanese Patent Application Laid-Open No. 2000-2833753 by the present applicant can be employed. This technique uses the fact that the distribution of the city block distance is symmetrical around the minimum point, and identifies the position of the minimum point from the magnitude relationship of the difference in the city block distance before and after the coincidence point giving the pixel parallax. Sub-pixel level parallax can be obtained by linear approximation.

  That is, assuming that the size of one pixel is infinitely small, as shown in FIG. 3, the distribution of the city block distance C in the image plane (i, j) becomes continuous, and the i direction (the parallax detection direction) Considering the minimum points in the horizontal direction, it can be considered that the distribution is symmetrical and continuously distributed around the minimum points. Therefore, as shown in FIG. 4, the coincidence point of the city block distance C0 at the coordinate is (the point where the city block distance is minimum in pixel units) and the city block distance C1, It is possible to calculate the sub-pixel component by calculating two straight lines L1 and L2 that are symmetrical with respect to the vertical axis based on the three points with the adjacent point of C4, and obtaining the intersection of these straight lines. it can.

  However, the coincidence point of the city block distance C0 obtained by the pixel unit search is detected at the subpixel level, including the neighboring city block distances C2 and C3 indicated by broken lines in FIG. 4 as errors. Therefore, it cannot be said that it is necessarily located closest to the local minimum point, and the technique for obtaining the subpixel component by three-point interpolation using discrete data in units of pixels approximates the true local minimum point of the city block distance. It is inevitable that the surface becomes rough.

  Accordingly, in this embodiment, as will be described in detail below, the resolution expansion unit 30 performs inter-pixel interpolation on the main image and the sub image to expand the resolution, and the image with the expanded resolution is subjected to stereo. By performing stereo matching by the above-mentioned SAD calculation in the matching unit 35, data including city block distances C2 and C3 at sub-pixel levels indicated by broken lines in FIG. 4 is obtained. Then, the interpolation processing unit 40 performs an interpolation process with three points of the coincidence point defined by the minimum value of these data and the adjacent points before and after that, thereby obtaining the sub-pixel level parallax with high accuracy.

  In this embodiment, since the main purpose is parallax detection, the resolution expansion process is performed only in the horizontal direction, which is the parallax detection direction, and virtual pixels are inserted between the horizontal pixels to reduce the horizontal resolution. Expand to 1/2 pixel level. In this case, although the amount of calculation increases, resolution expansion may be performed in both the horizontal direction and the vertical direction, and matching accuracy can be further improved.

  In this embodiment, the stereo matching unit 35 initially performs normal pixel-by-pixel matching on the main block and sub-block of the original image, and the resolution of the block matched in units of one pixel is set as a resolution expansion unit. The stereo matching unit 35 performs the matching again on the in-block and the sub-block which are expanded at 30 and the resolution is expanded. This is to obtain the coincidence point at the pixel level in advance so as to reduce the calculation amount and calculation time of the subsequent sub-pixel level processing. The resolution may be expanded and the matching processing may be performed by the stereo matching unit 35 on the main block and sub-block of the image whose resolution has been expanded.

  On the other hand, the recognition processing unit 50 is configured mainly by a processor that performs software processing, and performs various recognition processes based on the original image data in the original image memory 25 and the distance distribution data in the distance image memory 45. The recognition result by the recognition processing unit 50 is used in various control systems such as a vehicle exterior situation recognition system, ITS, robot vision, intruder monitoring system, and flying object altitude measurement system.

  For example, when applied to a vehicle exterior situation recognition system or ITS, based on the three-dimensional distance information, recognition of the road shape in front of the host vehicle, the preceding vehicle, etc., recognition of the running condition of the vehicle on the road, etc. I do. In addition, when applied to robot vision, a process for recognizing the movement path and the presence or absence of obstacles is performed based on the three-dimensional distance information. A process for recognizing the presence and position of a person or an object that enters the user is performed. Furthermore, when applied to an altitude measurement system that is mounted on a moving body such as an unmanned helicopter and measures altitude, the ground surface is recognized as a plane from the distance distribution information, and processing for obtaining the vertical altitude is performed.

  Next, details of stereo matching and subpixel processing by image resolution expansion will be described with reference to the flowchart of FIG.

  The flowchart of FIG. 5 shows the flow of processing executed in time series (partially executed in parallel) in the resolution extension unit 30, the stereo matching unit 35, and the interpolation processing unit 40. First, in step S1, normal stereo matching in units of one pixel is performed between the main block main (i, j) and the sub-block sub (i, j) on the orthogonal coordinates with the upper left of the image plane as the origin. The size of the main and sub search blocks is, for example, 4 × 4 (i = 0 to 3, j = 0 to 3), and the position of the sub block is shifted by one pixel in the horizontal direction with respect to the main block. SAD calculation is performed to find a coincidence point where the city block distance is the minimum value C0, and the pixel parallax D is calculated (see FIG. 2).

  Next, the process proceeds to step S <b> 2, and the resolution expansion processing of the matching region by inter-pixel interpolation is performed for each of the main image and the sub image. As shown in FIG. 6, this resolution expansion process is performed on the main image with a block main (i, j) of 4 × 4 pixel units and an expanded block main_ext (i ′ of 8 × 4 pixel units). , j), and for the sub-image, as shown in FIG. 7, a block sub of 6 × 4 pixels, in which the 4 × 4 block of the main block is expanded by one pixel before and after in the horizontal direction for searching. (i, j) is expanded to a block sub_ext (i ′, j) of 12 × 4 1/2 pixel units.

The i ′ = (2 × i + 1) -th pixel of the extension block indicated by hatching in FIGS. 6 and 7 is a virtual inserted between the i-th pixel and the (i + 1) -th pixel of the search block of the original image. Pixel. This virtual pixel data (interpolation data) is generated based on the surrounding pixel information. In this embodiment, as shown in the following equations (2) and (3), the i-th pixel data and the horizontal pixel data Linear interpolation is performed by taking the average value of the (i + 1) th pixel data adjacent in the direction. For this pixel interpolation, it is not always impossible to use the nearest neighbor method. However, if there is a margin in calculation capacity and calculation time, pixel interpolation by the cubic interpolation method may be performed. good.
main_ext (i ′, j) = (main (i, j) + main (i + 1, j)) / 2 (2)
sub_ext (i ′, j) = (sub (i, j) + sub (i + 1, j)) / 2 (3)
However, i ′ = 2 × i + 1

Further, the pixel data of the original image can be used as the i ′ = (2 × i) -th pixel data of the extension block corresponding to the i-th pixel of the search block of the original image. As shown in the following equation (3), i ′ = (2 × i) -th pixel data of the extension block is converted into (i−1), i, (i + 1) -th three pixels of the original image. Data is replaced with a weighted average value of its own pixel data and pixel data adjacent in the horizontal direction.
main_ext (i ′, j) = (m1 × main (i−1, j) + m2 × main (i, j)
+ M3 × main (i + 1, j)) / (m1 + m2 + m3) (4)
sub_ext (i ′, j) = (m1 × sub (i−1, j) + m2 × sub (i, j)
+ M3 × sub (i + 1, j)) / (m1 + m2 + m3) (5)
However, i ′ = 2 × i
m1, m2, m3; weighted average weight
(For example, m1 = 1, m2 = 4, m3 = 1)

  The replacement of the pixel data of the original image is a weighted average process of the image, and reduces noise components that easily induce mismatching. By using the inter-pixel interpolation and the weighted average process in combination, it is possible to improve the reliability of matching and reduce variations.

  After performing the above resolution expansion processing, the process proceeds to step S3, and the 8 × 4 block SAD calculation is shifted five times in units of 1/2 pixel, one pixel before and after the pixel in the horizontal direction with respect to the block before expansion. Run with numbers to get 5 SAD values (city block distance). Then, these five SAD values are compared with each other to obtain a shift value that takes the minimum SAD value (the minimum value of the city block distance). The shift value that takes the minimum SAD value is a shift amount that gives a parallax in units of 1/2 pixel between the main extension block and the sub extension block, and if the sub pixel component is 0 in the matching in step S1, The shift value is 2.

  If the required parallax detection accuracy is ½ pixel unit, the ½ pixel unit parallax is obtained from the sub-pixel component obtained in step S2 and the pixel parallax D obtained in step S1. The processing may be terminated. Further, instead of the SAD calculation of the 4 × 4 block of the original image in step S1, an image obtained by extending the resolution of the entire original image (for example, when the original image is 512 × 200 pixels, the entire image is expanded to 1024 × 200 pixels). ) May be subjected to SAD calculation of 4 × 4 blocks, and the parallax in units of 1/2 pixel can be directly obtained.

  Next, the process proceeds from step S3 to step S4, and the three-point interpolation between the shift value that takes the minimum SAD value obtained by the SAD calculation of the extension block and the shift value that gives the SAD values before and after the minimum SAD value, A sub-pixel level parallax (sub-pixel parallax) is obtained.

  Specifically, considering that the distribution of the city block distance is symmetric around the minimum point with respect to the SAD value (city block distance) centered on the minimum SAD value (city block distance) obtained in step S3, It is examined on which side the minimum value of the city block distance exists with respect to the coordinates of the minimum SAD value.

  When the SAD value changes greatly after the coincidence point around the coincidence point having the minimum SAD value, the local minimum point is between the coordinate position of the minimum SAD value and the coordinate position before 1/2 pixel. It is determined that there is an intersection of a straight line having a slope m passing through the point of the minimum SAD value and the next SAD value and a straight line having a slope of −m passing through the point of the SAD value at the coordinate position 1/2 pixel before. The subpixel parallax S is calculated based on the difference between the coordinate position of the minimum point and the coordinate position of the minimum SAD value.

  Conversely, when the SAD value changes greatly before the coincidence point having the minimum SAD value, it is determined that the minimum point is between the coordinate position of the minimum SAD value and the coordinate position after 1/2 pixel, The intersection of the straight line with the slope m passing through the point with the minimum SAD value and the point with the previous SAD value and the straight line with the slope -m passing through the point with the SAD value at the coordinate position after 1/2 pixel is used as the coordinate of the minimum point. The sub-pixel parallax S is calculated based on the difference between the coordinate position of the minimum point and the coordinate position of the minimum SAD value.

  For example, as shown in FIG. 8, five SAD values at coordinates (i′s−2), (i′s−1), (i ′s), (i ′s + 1), and (i ′s + 2) in 1/2 pixel units. Among C′1, C′2, C′0, C′3, and C′4, the minimum SAD value is C′3, and (C′0−C′3) <(C′4−C In the case of '3), a straight line L'1 passing through the point of the SAD value C'3 and the point of the SAD value C'4 after 1/2 pixel is obtained, and the SAD value C'0 before 1/2 pixel is obtained. The intersection of the straight line L′ 1 and the straight line L′ 2 that is symmetric with respect to the vertical axis is obtained, and the i ′ coordinate of this intersection and the coordinate (i ′s + 1) of the minimum SAD value C′3 are obtained. A sub-pixel parallax S is calculated from the difference.

  In the example of FIG. 8, the SAD value C′0 at the coordinate position i ′s corresponding to the coordinate position is of the SAD value C0 detected as the minimum value in the previous pixel-unit matching is sub-pixel unit after resolution expansion processing. This shows an example in which the SAD value C′3 at the coordinate position (i ′s + 1) after 1/2 pixel is detected as the minimum value by the above matching. As is clear from this example, in the three-point interpolation performed after pixel-by-pixel matching, the SAD value C′1 is centered on the point corresponding to the SAD value C′0, as indicated by the dashed straight line in the figure. The minimum point is estimated by three-point interpolation of the corresponding point and the point corresponding to the SAD value C′4. Therefore, by performing three-point interpolation using data obtained by matching after resolution expansion, a position closer to the true minimum point can be obtained compared to three-point interpolation using data in pixel units. The error can be reduced and the approximation accuracy can be greatly improved.

  After calculating the sub-pixel parallax S, the process proceeds from step S4 to step 5, and a parallax (D + S) having a resolution of 1/2 pixel unit or less is obtained from the pixel parallax D and the sub-pixel parallax S. By repeating such processing, the parallax (D + S) is obtained over the entire image and stored in the distance image memory 45. From the parallax (D + S) stored in the distance image memory 45, the distance in the real space can be obtained by conversion into the real space coordinates.

  FIG. 9 is a histogram showing the parallax variation calculated by the above processing, and the variation distribution with respect to the true value of the parallax obtained by the three-point interpolation after matching in sub-pixel units by the resolution extension of this embodiment is outlined. The histogram value is shown. As shown in this histogram, the disparity obtained by the processing of this embodiment is greatly reduced in variation compared to the conventional three-point interpolation after matching in units of pixels indicated by the filled histogram values in FIG. I understand that. This is because, in the processing of this embodiment, the resolution of the original image is expanded, so that the approximation of the three-point interpolation for subpixel calculation is improved, and the stability is improved by increasing the number of matching pixels. Furthermore, it is considered that the prevention of mismatching by adding a weighted average process is effective.

  As described above, in the present embodiment, the resolution of the original image captured by the stereo camera is expanded by using inter-pixel interpolation and weighted average processing, and stereo matching is performed on the expanded image. Matching accuracy can be improved and parallax in units of 1/2 pixel can be accurately detected. Furthermore, parallax can be obtained accurately with a resolution of 1/2 pixel unit or less by performing three-point interpolation after stereo matching of an image with an extended resolution.

  This makes it possible to obtain subpixel-level parallax data with high reliability by reducing noise and variations without increasing the matching area in pixels and reducing spatial resolution. It is possible to obtain an accurate distance image and to lead to more advanced image recognition and control.

  Next, a second embodiment of the present invention will be described. 10 to 12 relate to a second embodiment of the present invention, FIG. 10 is a block diagram showing a basic configuration of a stereo image processing apparatus, FIG. 11 is a flowchart of stereo matching and subpixel processing, and FIG. It is a histogram which shows.

  In the second form, stereo matching is performed after weighted averaging of an original image captured by a stereo camera. Hereinafter, the same reference numerals as those of the first embodiment are given to the constituent elements having the same functions as those of the first embodiment, and the description of the first embodiment is incorporated as necessary.

  That is, the stereo image processing apparatus 1A according to the second embodiment includes a stereo camera 10 including a set of two cameras, an image input unit 20 that performs input processing on a pair of images captured by the stereo camera 10, and the image input. The original image memory 25 that stores the image processed by the image processing unit as an original image, the image conversion unit 30A that replaces the data of each pixel of the original image with the data of the surrounding pixels, and evaluates the degree of correlation between the pair of images. A stereo matching unit 35 that determines a corresponding position, an interpolation processing unit 40 that obtains a sub-pixel level parallax having a resolution of one pixel unit or less by an interpolation process based on the corresponding position determined by the stereo matching unit 35, and distance information based on the parallax. Distance image memory 4 for storing as an image form (distance image) arranged two-dimensionally in association with the image coordinates of the original image The recognition processing unit 50 is configured to perform various recognition processes using the original image information in the original image memory 25 and the distance information in the distance image memory 45, and includes a vehicle exterior situation recognition system, an intelligent road traffic system (ITS), It is applied to various systems such as robot vision, intruder monitoring system, and flying object ground altitude measurement system.

  The functions of the units other than the image conversion unit 30A are the same as those in the first embodiment described above. However, in the second embodiment, the data of each pixel of the pair of original images captured by the stereo camera 10 is received by the image conversion unit 30A. Stereo matching is performed by the stereo matching unit 35 on the pair of images in which the pixel data is replaced using the peripheral pixel data.

The replacement of the pixel data by the image conversion unit 30A is the weighted average processing of the image described in the first embodiment. For example, as shown in the following equation (6) according to the equations (4) and (5), the original image The data p (i, j) of the i-th pixel of the current data, the data p (i−1, j) of the adjacent (i−1) -th pixel, and the data of the (i + 1) -th pixel Replace with new data ext (i, j) weighted averaged by p (i + 1, j).
ext (i, j) = (m1 * p (i-1, j) + m2 * p (i, j)
+ M3 × p (i + 1, j)) / (m1 + m2 + m3) (6)
Where m1, m2, m3: weighted average weight
(For example, m1 = 1, m2 = 4, m3 = 1)

  The processing after performing the stereo matching on the weighted average processed image is the same as that in the first embodiment, and the sub-pixel level parallax having a resolution of one pixel unit or less through the three-point interpolation in the interpolation processing unit 40. can get.

  Hereinafter, stereo matching and sub-pixel processing by pixel data replacement will be described with reference to the flowchart of FIG.

  The flowchart of FIG. 11 shows a flow of processing executed in time series (partially executed in parallel) in the image conversion unit 30A, the stereo matching unit 35, and the interpolation processing unit 40. First, in step S11, with respect to the original images of both the main image and the sub image, the data of each pixel is replaced with its own pixel data and data of neighboring pixels adjacent thereto, and the above-described weighted average processing is performed. This weighted average processing may be performed over the entire image first, or may be performed for each matching region where SAD calculation is performed.

  Next, the process proceeds to step S12, and normal stereo matching in units of one pixel is performed between the main block main (i, j) and the sub-block sub (i, j) on the orthogonal coordinates having the origin at the upper left of the image plane. . The size of the main and sub search blocks is, for example, 4 × 4 (i = 0 to 3, j = 0 to 3), and the position of the sub block is shifted by one pixel in the horizontal direction with respect to the main block. A SAD operation is performed to obtain a coincidence point where the city block distance is the minimum value C0, and a parallax (pixel parallax) D in units of pixels is calculated (see FIG. 2).

  Next, the process proceeds from step S12 to step S13, and a subpixel level is obtained by three-point interpolation between a shift value that takes the minimum SAD value obtained by the SAD calculation in step S12 and a shift value that gives SAD values before and after the minimum SAD value. The parallax (subpixel parallax) S is obtained. The processing by this three-point interpolation is as described in the first embodiment (see FIGS. 3 and 4).

  After calculating the sub-pixel parallax S, the process proceeds from step S 13 to step 14, and a parallax (D + S) having a resolution of one pixel unit or less is obtained from the pixel parallax D and the sub-pixel parallax S. By repeating such processing, the parallax (D + S) is obtained over the entire image and stored in the distance image memory 45. From the parallax (D + S) stored in the distance image memory 45, the distance in the real space can be obtained by conversion into the real space coordinates.

  As shown in FIG. 12, the parallax by the three-point interpolation after the stereo matching through the above weighted average processing has a variation distribution indicated by a white histogram value in the figure with respect to the true value. As is apparent from this histogram, in the second form, compared to the first form, the resolution of the original image is not expanded, so that the approximation by the three-point interpolation is improved and the stabilization by the increase in the number of matching pixels is achieved. Although the degree of improvement in variation is small because it cannot be expected, it is better than the conventional parallax obtained by three-point interpolation after matching the original image without performing weighted average processing, as shown by the histogram value of the fill in FIG. It can be confirmed that the variation is reduced.

  As described above, in the second embodiment, stereo matching is performed after blurring the original image captured by the stereo camera, so that mismatching can be reduced, matching accuracy and reliability can be increased, and variation can be reduced. . Moreover, in matching in units of pixels, it is possible to obtain sub-pixel level parallax data with high reliability by reducing noise and variations without interfering with enlarging the matching area and reducing the distance resolution. It becomes possible to connect to more advanced image recognition and control.

  Note that the present invention is not limited to the above-described embodiment, and it is possible to extend the resolution of only one image and perform matching processing with, for example, 1/2 pixel.

  It is also possible to perform weighted average processing on one image, expand the resolution of the other image, and perform matching. In this case, the weighted average process may or may not be performed on the other image.

  Furthermore, when a color-compatible camera is used as the imaging means, it is also possible to extract a monochrome image and apply the present invention to this image. In this case, it is possible to improve the accuracy of matching that is a problem in a color-compatible camera.

The block diagram which shows the basic composition of a stereo image processing apparatus concerning 1st Embodiment of this invention. Same as above, explanatory diagram showing matching between main image and sub-image Same as above, explanatory diagram showing distribution of city block distance Same as above, explanatory diagram showing interpolation after matching in pixel units Same as above, stereo matching and sub-pixel processing flowchart As above, an explanatory diagram showing the resolution expansion of the main image Same as above, explanatory diagram showing resolution expansion of sub-image Same as above, explanatory diagram showing interpolation after matching in sub-pixel units Same as above, histogram showing disparity variation The block diagram which shows the basic composition of a stereo image processing apparatus concerning 2nd Embodiment of this invention. Same as above, stereo matching and sub-pixel processing flowchart Same as above, histogram showing disparity variation

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,1A Stereo image processing apparatus 10 Stereo camera 20 Image input part 25 Original image memory 30 Resolution expansion part 30A Image conversion part 35 Stereo matching part 40 Interpolation processing part
Agent Patent Attorney Susumu Ito

Claims (6)

A stereo image processing apparatus for performing stereo matching using image pairs having correlation with each other,
A resolution extending means for inserting a virtual pixel generated using data of peripheral pixels between each pixel of the image pair, and extending the resolution of the image pair;
A stereo image processing apparatus, comprising: stereo matching means for specifying a corresponding position with the resolution of the expanded resolution for the image pair whose resolution is expanded by the resolution expansion means. The resolution expansion means is
2. The stereo image processing apparatus according to claim 1, wherein data of a pixel adjacent to the virtual pixel is replaced by using data of a peripheral pixel.   Interpolation based on the correlation calculation value of the corresponding position specified by the stereo matching means and the correlation calculation value of the position adjacent to the corresponding position, and determining the corresponding position with a resolution lower than the expanded resolution 3. The stereo image processing apparatus according to claim 1, further comprising means. A stereo image processing apparatus for performing stereo matching using image pairs having correlation with each other,
Image conversion means for replacing data of each pixel of the image pair with data of peripheral pixels;
A stereo image processing apparatus, comprising: stereo matching means for specifying a corresponding position with respect to an image pair in which pixel data is replaced by the image conversion means.   Interpolation means for performing an interpolation process based on the correlation value calculation value of the corresponding position specified by the stereo matching means and the correlation value calculation value of the position adjacent to the corresponding position, and determining the corresponding position with a resolution of one pixel or less. The stereo image processing apparatus according to claim 4, further comprising: A stereo image processing apparatus for performing stereo matching using image pairs having correlation with each other,
Means for replacing the data of each pixel with the data of surrounding pixels for at least one of the image pairs;
A stereo image processing apparatus, comprising: stereo matching means for specifying mutual corresponding positions in the image pair using the replaced pixel data.

Images