JP5071866B2 - Distance measuring device, method and program - Google Patents

Description

  The present invention relates to a distance measuring device and method for generating a distance image representing a three-dimensional shape of a subject, and a program for causing a computer to execute the distance measuring method.

  Corresponding points that are pixels corresponding to a plurality of images (a reference image by a reference camera and a reference image by a reference camera) obtained by capturing an image of a subject using two or more cameras provided at different positions (Stereo matching) and applying the triangulation principle to the position difference (parallax) between the corresponding pixel on the reference image and the pixel on the reference image, the pixel from the reference camera or reference camera A method has been proposed in which a distance image representing a three-dimensional shape of a subject is generated by measuring the distance to a point on the subject corresponding to.

  Here, when performing stereo matching, as shown in FIG. 3, the points in the real space mapped to a certain pixel Qa on the reference image G1 are points Q1, Q2, Q3,. Since there are a plurality of continuous lines on the line of sight, the pixel Qa ′ on the reference image G2 corresponding to the pixel Qa is on a straight line (epipolar line) that is a map of the points Q1, Q2, Q3. Corresponding points are searched based on the existence. In FIG. 3, the point O1 is the viewpoint of the base camera, and the point O2 is the viewpoint of the reference camera. Here, the viewpoint is the focal point of the optical system of each camera. When performing stereo matching in this way, a correlation window W including a pixel Qa to be searched for corresponding points is set on the reference image, and the same correlation window W as that set for the reference image is set on the reference image. It moves along the epipolar line, calculates the correlation for each pixel in the correlation window on each image for each movement position, and determines the pixel at the center of the correlation window where the correlation on the reference image G2 is maximum, It is obtained as a corresponding point of the pixel Qa.

When searching for corresponding points in this way, the search accuracy of corresponding points is improved by using a correlation window of an appropriate size. Therefore, the reliability parameter is calculated from the expected value of the energy function representing the similarity between the standard image and the reference image, and the entropy determined from the parallax probability between the pixel on the standard image and the pixel on the reference image, A method of determining the size of the correlation window based on the reliability parameter has been proposed (see Patent Document 1). In addition, a technique has been proposed in which two luminance levels are set in the standard image and the reference image, and an image group between the two luminance levels is used as a comparison area for searching for corresponding points (see Patent Document 2). Also, a method has been proposed for increasing the size of the correlation window when the correlation deteriorates when calculating the correlation for corresponding point search for time series data (see Patent Document 3). In addition, an evaluation function indicating the variation in parallax for each pixel included in the correlation window is created, the value of the evaluation function is calculated while changing the size of the correlation window, and a correlation window having a size that minimizes the calculated value is calculated. A technique to be used has also been proposed (see Non-Patent Document 1).
JP-A-5-256613 JP 2000-28354 A JP-A-11-120351 T. Kanade and M. Okutomi, "A stereo Matching Algorithm with an Adaptive Window: Theory and Experiment", Proc. Of the 1991 IEEE Intl. Conf. On Robotics and Automation, 1991, pp.1088-1095

  However, with the method described in Patent Document 1, it is difficult to calculate the conditions for maximizing the reliability parameter, and the amount of calculation becomes enormous. In addition, in the standard image and the reference image, regions having the same luminance are not necessarily regions corresponding to parallax, and therefore the distance image cannot be calculated with the technique described in Patent Document 2. Moreover, since the method described in Patent Document 3 is intended for time-series data, it cannot be applied when handling still images. Furthermore, in the method described in Non-Patent Document 1, it is necessary to calculate the evaluation function for each size of each correlation window, so that the amount of calculation is enormous.

  The present invention has been made in view of the above circumstances, and an object thereof is to easily calculate the size of a correlation window for searching for corresponding points with high accuracy.

The distance measuring device according to the present invention provides a reference image for calculating a distance to a point of interest on the subject obtained by imaging the subject, and the subject from a position different from the position from which the reference image is obtained. Input means for receiving an input of at least one reference image for calculating the distance acquired by imaging
A target pixel point on the reference image on which the point of interest is projected, and a search pixel array centered on the target point and extending in a predetermined direction on the reference image, and the luminance of the pixels in the search pixel array A search pixel sequence in which a difference value between a maximum value and a minimum value exceeds a predetermined luminance interval is used, and the number of pixels in the search pixel sequence is used when searching for a corresponding point on the reference image with respect to the target point A window size setting means for setting a correlation window size as a reference for calculating a correlation between the reference image and the reference image;
Corresponding point search means for searching for the corresponding point by calculating a correlation between each pixel in the correlation window set on the reference image and each pixel in the correlation window set on the reference image; ,
And a distance calculating means for calculating the distance based on the searched corresponding points.

  When the luminance change on the image is large, it is often an edge component of the image, so that the correlation can be accurately calculated even if the size of the correlation window is small. In addition, when the size of the correlation window is small, the amount of calculation for calculating the correlation decreases. On the other hand, when the luminance change on the image is small, it is often a flat portion on the image, and therefore the correlation cannot be accurately calculated unless the size of the correlation window is increased.

  A distance measuring device according to the present invention is a search pixel sequence centered on a target point and extending in a predetermined direction on the reference image for a target point on the reference image. A reference image that is used when a search pixel sequence in which a difference value between a maximum value and a minimum value exceeds a predetermined luminance interval and the number of pixels in the search pixel sequence is searched for a corresponding point on the reference image with respect to the target point And the size of the correlation window serving as a reference for calculating the correlation of the reference image. For this reason, in a predetermined direction on the reference image, a correlation window having a larger size is set for a target point having a smaller luminance change, and as a result, a correlation window having an appropriate size corresponding to the amount of change in luminance is set. Can do. Further, since the luminance of each pixel in the correlation window always has a predetermined luminance interval, the correlation can be calculated with high accuracy. Accordingly, the size of the correlation window can be appropriately set by simple calculation, and the accuracy of the corresponding point search can be improved.

  In the distance measuring device according to the present invention, the window size setting means sets the search pixel column having an initial size, and calculates a difference value between a maximum value and a minimum value of luminance values in pixels in the search pixel column. Calculating and determining whether or not the difference value exceeds the predetermined luminance interval. If the determination is negative, the search pixel string is increased by a predetermined number of pixels in the predetermined direction, and the difference value is calculated. It is good also as a means to repeat the process which enlarges the said search pixel row | line until it exceeds the said predetermined brightness | luminance interval.

  In the distance measuring apparatus according to the present invention, the predetermined direction may be a direction corresponding to a direction in which an epipolar line obtained by mapping each pixel of the base image to the reference image extends.

  As a result, the size of the correlation window can be set by reflecting the change in luminance in the direction in which the corresponding points are searched, so that the accuracy of the corresponding point search can be further improved.

  In the distance measuring apparatus according to the present invention, the window size setting unit may be a unit that sets the predetermined luminance interval according to a luminance variation in the reference image.

  As a result, the influence of the luminance variation of the reference image can be removed from the process of setting the size of the correlation window, so that the corresponding point can be searched for accurately without being affected by the luminance variation of the reference image. The correlation window can be set.

  In the distance measuring device according to the present invention, the window size setting unit may be a unit that sets the predetermined luminance interval to be smaller as the luminance of the target point is lower.

  As a result, the influence of variations with different fluctuation levels at high brightness, such as light shot noise, can be removed from the processing for setting the size of the correlation window. Can be set.

  Further, in the distance measuring device according to the present invention, the corresponding point searching means determines whether or not the searched corresponding point is an incorrect corresponding point, and for a pixel determined to be an incorrect corresponding point, The correlation window may be enlarged and the corresponding points may be searched again.

  Thereby, although the calculation amount increases, the accuracy of the corresponding point search can be improved.

Further, in the distance measuring device according to the present invention, a control means for controlling the window size setting means, the corresponding point search means and the distance calculation means so as to calculate the distance for all the pixels of the reference image,
A distance image generating unit that generates a distance image having a pixel value as a distance of each pixel of the reference image may be further provided.

  Thereby, a distance image can be generated with high accuracy.

  Further, in the distance measuring device according to the present invention, the corresponding point searching means searches for the corresponding points collectively for each pixel in which the correlation window of the same size is set in ascending order of the size of the correlation window. It is good.

  Thereby, a corresponding point can be searched efficiently.

  Further, in this case, the corresponding point search unit may be configured to search for a pixel of the reference image in which a miscorresponding point that is miscorresponding is searched out of corresponding points searched using the correlation window of one size. The size of the correlation window may be increased by one step, and the corresponding points may be searched together with the pixels in which the correlation window having the size larger by one step is set.

  As a result, when searching for corresponding points using a correlation window of the next size, the corresponding points are searched again for the incorrect corresponding points, so it is necessary to search the corresponding points independently only for the incorrect corresponding points. Therefore, corresponding points can be efficiently searched for erroneous corresponding points, and the accuracy of the corresponding point search can be improved.

  Further, in the distance measuring device according to the present invention, the window size setting means classifies the size of the correlation window into a plurality of predetermined size groups, and sets the size of the correlation window of each pixel for each pixel. The correlation window size may be reset to the maximum size in the classified size group.

  Thereby, a corresponding point can be searched efficiently.

The distance measurement method according to the present invention includes a reference image for calculating a distance to a point of interest on the subject obtained by imaging the subject, and the subject from a position different from the position from which the reference image is obtained. Accepting input of at least one reference image for calculating the distance obtained by imaging
A target pixel point on the reference image on which the point of interest is projected, and a search pixel array centered on the target point and extending in a predetermined direction on the reference image, and the luminance of the pixels in the search pixel array A search pixel sequence in which a difference value between a maximum value and a minimum value exceeds a predetermined luminance interval is used, and the number of pixels in the search pixel sequence is used when searching for a corresponding point on the reference image with respect to the target point Is set to the size of a correlation window serving as a reference for calculating the correlation between the reference image and the reference image,
By searching for the corresponding points by calculating the correlation between each pixel in the correlation window set on the reference image and each pixel in the correlation window set on the reference image,
The distance is calculated based on the searched corresponding points.

  In addition, you may provide as a program for making a computer perform the distance measuring method by this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing an internal configuration of a stereoscopic imaging apparatus 1 to which a distance measuring device according to a first embodiment of the present invention is applied. As shown in FIG. 1, the stereoscopic imaging apparatus 1 according to the first embodiment includes two imaging units 21A and 21B, an imaging control unit 22, an image processing unit 23, a compression / decompression processing unit 24, a frame memory 25, and a media control unit. 26, an internal memory 27, and a display control unit 28.

  FIG. 2 is a diagram illustrating the configuration of the imaging units 21A and 21B. As shown in FIG. 2, the imaging units 21A and 21B include lenses 10A and 10B, diaphragms 11A and 11B, shutters 12A and 12B, CCDs 13A and 13B, analog front ends (AFE) 14A and 14B, and an A / D conversion unit 15A. 15B is provided.

  The lenses 10A and 10B are composed of a plurality of functional lenses such as a focus lens for focusing on a subject and a zoom lens for realizing a zoom function, and their positions are adjusted by a lens driving unit (not shown). In the present embodiment, the focal position is assumed to be fixed.

  In the diaphragms 11A and 11B, the diaphragm diameter is adjusted based on the diaphragm value data obtained by the AE process by a diaphragm driving unit (not shown). In the present embodiment, it is assumed that the aperture value data is fixed.

  The shutters 12A and 12B are mechanical shutters, and are driven by a shutter driving unit (not shown) according to the shutter speed obtained by the AE process. In this embodiment, the shutter speed is assumed to be fixed.

  The CCDs 13A and 13B have a photoelectric surface in which a large number of light receiving elements are two-dimensionally arranged, and subject light is imaged on the photoelectric surface and subjected to photoelectric conversion to obtain an analog imaging signal. In addition, color filters in which R, G, and B color filters are regularly arranged are arranged on the front surfaces of the CCDs 13A and 13B.

  The AFEs 14A and 14B perform processing for removing noise of the analog imaging signal and processing for adjusting the gain of the analog imaging signal (hereinafter referred to as analog processing) for the analog imaging signals output from the CCDs 13A and 13B.

  The A / D converters 15A and 15B convert the analog imaging signals subjected to analog processing by the AFEs 14A and 14B into digital signals. Note that image data obtained by the CCDs 13A and 13B of the imaging units 21A and 21B and converted into digital signals is RAW data having R, G, and B density values for each pixel. The image represented by the image data acquired by the imaging unit 21A is referred to as a standard image G1, and the image represented by the image data acquired by the imaging unit 21B is referred to as a reference image G2.

  The imaging control unit 22 controls imaging after pressing the release button.

  In this embodiment, the focus position, aperture value data, and shutter speed are fixed, but AF processing and AE processing are performed to set the focus position, aperture value data, and shutter speed each time shooting is performed. It may be. In this case, the internal memory 27 stores a table that records reference values for the focus position, aperture value data, and shutter speed, and the focus position, aperture value data, and shutter speed that differ depending on the distance to the subject and the brightness of the shooting environment. The focal position, aperture value data, and shutter speed may be set by referring to this table in accordance with the distance to the subject obtained by AF processing and AE processing and the brightness of the shooting environment. Good.

  The image processing unit 23 performs correction processing for correcting variations in sensitivity distribution of the image data and distortion of the optical system on the digital image data acquired by the imaging units 21A and 21B, and parallelizes the two images. For parallel processing. The reference numerals of the base image and the reference image after these processes are G1a and G2a, respectively. The image processing unit 23 further performs image processing such as white balance adjustment processing, gradation correction, sharpness correction, and color correction on the base image G1a and the reference image G2a. The reference codes of the base image and the reference image obtained by this image processing are G1b and G2b, respectively.

  The compression / decompression processing unit 24 applies to the image data representing the base images G1a and G1b and the reference images G2a and G2b processed by the image processing unit 23 and the image data of the distance image generated as described later. For example, compression processing is performed in a compression format such as JPEG to generate an image file. A tag storing incidental information such as shooting date and time is added to the image file based on the Exif format or the like.

  The frame memory 25 performs processing performed by the image processing unit 23 on the distance image data acquired by the imaging unit 2A and the image data representing the reference image G1 and the reference image G2 acquired by the imaging units 21A and 21B. This is a working memory used when performing various processes including.

  The media control unit 26 accesses the recording medium 29 and controls writing and reading of an image file such as a distance image.

  The internal memory 27 stores various constants set in the stereo camera 2, a program executed by the CPU 33, and the like.

  The display control unit 28 is for displaying image data stored in the frame memory 25 on the monitor 20 and displaying an image recorded on the recording medium 29 on the monitor 20.

  In addition, the stereoscopic imaging device 1 includes a stereo matching unit 30, a distance image generation unit 31, and a window size setting unit 32.

  As shown in FIG. 3, in the stereo matching unit 30, the points in the real space mapped to a certain pixel Qa on the reference image G1 are continuous on the line of sight from the point O1, such as points Q1, Q2, Q3. Therefore, the pixel Qa ′ on the reference image R corresponding to the pixel Qa exists on a straight line (epipolar line) that is a mapping of the points Q1, Q2, Q3. Based on the reference image G2, the corresponding point between the base image G1 and the reference image G2 is searched. In FIG. 3, a point O1 is a viewpoint of the imaging unit 21A that is a reference camera, and a point O2 is a viewpoint of the imaging unit 21B that is a reference camera. Here, the viewpoint is the focal point of the optical system of the imaging units 21A and 21B. Further, the intersections O3 and O4 between the straight line connecting the points O1 and O2 and the reference images G1 and G2 are epipoles. The search for corresponding points may use the standard image G1b and the reference image G2b that have been subjected to the image processing, but it is preferable to use the standard image G1a and the reference image G2a that have been processed up to the parallel processing. . In the following description, it is assumed that the search for corresponding points uses the base image G1a and the reference image G2a.

  Specifically, when searching for corresponding points, the stereo matching unit 30 moves the correlation window W having a size set by a window size setting unit 32 described later along the epipolar line, and at each moving position, the reference image Correlation for pixels in the correlation window W of G1a and reference image G2a is calculated, and the center pixel of the correlation window W at the position where the correlation on the reference image G2a is maximum corresponds to the pixel Qa on the reference image G1a Let it be a point. In addition, as a correlation evaluation value for evaluating a correlation, a difference absolute value sum, a difference square sum, etc. can be used. In this case, the smaller the correlation evaluation value, the larger the correlation.

  FIG. 4 is a diagram for explaining the positional relationship between the base image and the reference image after the parallel processing. As shown in FIG. 4, the image planes from which the standard images G1a and the reference images G2a are obtained in the imaging units 21A and 21B are orthogonal to the optical axes of the imaging units 21A and 21B, and the intersection point with the optical axis is the origin. Coordinate systems (u, v) and (u ′, v ′). Here, by the parallel processing, the optical axes of the imaging units 21A and 21B are parallel, the two image planes are on the same plane, and the u axis and the u ′ axis on the image plane face the same direction on the same straight line. Become. Further, since the epipolar line on the reference image G2a is parallel to the u ′ axis by the parallelization process, the u axis on the reference image G1a also coincides with the direction of the epipolar line of the reference image G2a. .

  Here, it is assumed that the focal lengths of the imaging units 21A and 21B are f and the base line length is b. The focal length f and the base line length b are calculated in advance as calibration parameters and stored in the internal memory 27. At this time, the position (X, Y, Z) in the three-dimensional space is expressed by the following equations (1) to (3) with reference to the coordinate system of the imaging unit 21A.

X = b · u / (u−u ′) (1)
Y = b · v / (u−u ′) (2)
Z = b · f / (u−u ′) (3)
Here, u−u ′ is the amount of deviation (parallax) in the horizontal direction of the projection point on the image plane of the imaging units 21A and 21B. Further, it can be seen from the expression (3) that the distance Z, which is the depth, is inversely proportional to the parallax.

  The distance image generation unit 31 uses the corresponding points obtained by the stereo matching unit 30 to calculate the distances from the imaging units 21A and 21B to the subject by the above formulas (1) to (3), and calculates the calculated distance. A distance image D1 is generated in association with each pixel of the CCDs 13B and 13C. Note that the pixel value of each pixel of the distance image D1 represents the distance from the imaging units 21A and 21B to the subject. The generated distance image D1 is recorded on the recording medium 29.

  The window size setting unit 32 sets the size of the correlation window used when searching for corresponding points. FIG. 5 is a diagram showing a luminance profile in one line in the u-axis direction of the reference image G1a. In the present embodiment, the size of the correlation window is set according to a change in luminance in the direction in which the epipolar line of the reference image G2a extends on the standard image G1a. Here, since the standard image G1a and the reference image G2a are processed in parallel, the direction corresponding to the direction in which the epipolar line of the reference image G2a in the standard image G1a extends is the u-axis direction.

  Note that before the parallel processing, the direction in which the size of the correlation window is determined does not match the direction in which the pixels in the image are arranged. In this case, the direction for determining the size of the correlation window is the direction of the straight line connecting the target pixel and the epipole on the reference image G1 (the direction in which the epipolar line extends). Note that the direction of the straight line connecting the target pixel and the epipole coincides with the direction in which the pixels in the image are arranged by the parallel processing (the u-axis direction on the reference image G1a).

  As shown in FIG. 5, the window size setting unit 32 sets the luminance of the pixels included in the u-axis direction of the correlation window set as the target pixel u0 for the target pixel u0 for setting the size of the correlation window on the reference image G1a. The size W of the correlation window is set so that the difference value between the maximum value and the minimum value exceeds the predetermined luminance interval K0. Here, the window size setting unit 32 actually sets the size of the correlation window sequentially from the leftmost pixel of the reference image. Details of the processing will be described later.

  Note that a predetermined value may be used as the predetermined luminance interval for determining the size of the correlation window, but it may be calculated according to the luminance variation of the reference image G1a. Here, when the ideal sensitivity distribution correction is performed, the luminance profile of the image without the luminance change is as shown by the broken line in FIG. 6, but in reality, due to the variation in sensitivity of the imaging units 21A and 21B, As shown by the solid line in FIG. 6, the luminance varies. The standard deviation when this luminance variation is regarded as a normal distribution is σ1.

  On the other hand, when the same subject is photographed a plurality of times, the pixel values of all the images are not the same, and pixel value variation (repetitive variation) occurs every time the image is photographed. The standard deviation when this variation is regarded as a normal distribution is σ2.

The pixel value of each pixel of the reference image G1a after correcting the sensitivity distribution has variations due to sensitivity variations and repeated variations of the imaging units 21A and 21B. The standard deviation σ0 of this variation is calculated by σ0 = √ (σ1 ₂ + σ2 ₂ ). The window size setting unit 32 uses 3σ (= 3 × σ0) of the standard deviation σ0 as a predetermined luminance interval for determining the size of the correlation window.

  The CPU 33 controls each unit of the stereoscopic imaging apparatus 1 according to a signal from the input unit 34 including a release button.

  The data bus 35 is connected to each unit constituting the stereoscopic imaging device 1 and the CPU 33, and exchanges various data and various information in the stereoscopic imaging device 1.

  Next, processing performed in the first embodiment will be described. FIG. 7 is a flowchart showing processing performed in the first embodiment. Here, processing after the release button is fully pressed and an instruction for imaging is performed will be described.

  When the release button 4 is fully pressed, the CPU 33 starts processing, the imaging units 21A and 21B capture the subject in accordance with an instruction from the CPU 33, and the image processing unit 23 performs correction processing and parallelization on the acquired image data. Processing and image processing are performed to obtain standard images G1a and G1b and reference images G2a and G2b (step ST1). Next, the window size setting unit 32 sets the size of the correlation window for each pixel of the reference image G1a after the parallelization processing (step ST2).

  FIG. 8 is a flowchart of correlation window size setting processing in the first embodiment. The window size setting unit 32 first sets the target pixel for setting the window size at the initial position (step ST11). Note that the initial position is a pixel at the upper left corner of the reference image G1a. Then, the window size setting unit 32 sets the number of pixels N of the search pixel sequence for searching for the size of the correlation window to an initial value of 3 (step ST12). In this case, the half width of the search pixel column is 1. Next, it is determined whether or not the search pixel row is in the reference image G1a (step ST13). If step ST13 is negative, the size of the correlation window cannot be set appropriately, so the size of the correlation window of the target pixel is set to none (step ST14), and the process proceeds to step ST19 described later. Here, setting the size of the correlation window to none means that the size of the correlation window is not set for the target pixel and the corresponding point is not searched.

  On the other hand, when step ST13 is affirmed, a difference value between the maximum value and the minimum value of the luminance values in the pixels in the search pixel row is calculated (step ST15), and whether or not the difference value exceeds the predetermined luminance interval K0. Determination is made (step ST16). If step ST16 is negative, 2 is added to the search pixel column pixel number N (N = N + 2; step ST17), the process returns to step ST13, and the processes in and after step ST13 are repeated. Thereby, as shown in FIG. 9, the number N of pixels in the search pixel column is increased by 2 as 3, 5, 7... Until step ST16 is affirmed. Note that the half width increases by one.

  If step ST16 is positive, the size W of the correlation window of the target pixel is set to the number N of pixels of the current search pixel column (step ST18). Here, as shown in FIG. 9, when the difference value between the maximum value and the minimum value of the luminance of the pixels in the search pixel column exceeds N5, the correlation window size W is 11 when N = 11. Set to

  Next, the window size setting unit 32 determines whether or not the target pixel is the rightmost pixel of the reference image G1a (step ST19). If step ST19 is negative, the target pixel is changed to the right pixel by one pixel. (Step ST20), the process returns to Step ST12, and the processes after Step ST12 are repeated.

  If step ST19 is positive, it is determined whether or not the target pixel is in the bottom row of the reference image G1a (step ST21). If step ST21 is negative, the target pixel is set to the leftmost pixel in the line one line below (step ST22), the process returns to step ST12, and the processes after step ST12 are repeated. If step ST21 is affirmed, the process ends.

  Returning to FIG. 7, following step ST <b> 2, the stereo matching unit 30 searches for a corresponding point based on the correlation window having the set size (step ST <b> 3), and the distance image generating unit 31 searches based on the searched corresponding point. A distance image D1 is generated (step ST4). Next, in response to an instruction from the CPU 33, the media control unit 26 records the distance image D1 on the recording medium 29 (image recording: step ST5), and ends the process.

  As described above, according to the first embodiment, in the u-axis direction corresponding to the direction in which the epipolar line on the reference image G1a extends, a correlation window having a larger size is set for a pixel having a smaller luminance change. It is possible to set a correlation window of an appropriate size according to the amount of change in luminance of the reference image G1a. In addition, the luminance of each pixel in the correlation window always has a predetermined luminance interval. Accordingly, the size of the correlation window can be appropriately set by simple calculation, and the accuracy of the corresponding point search can be improved.

  Further, since the size of the correlation window is set according to the luminance change in the direction corresponding to the direction in which the epipolar line extends, the correlation window reflects the luminance change in the direction in which the corresponding point in the reference image G2a is searched. As a result, the accuracy of the corresponding point search can be further improved.

  Next, a second embodiment of the present invention will be described. Note that the stereoscopic imaging apparatus according to the second embodiment of the present invention has the same configuration as the stereoscopic imaging apparatus 1 according to the first embodiment of the present invention, and the processing performed by the stereo matching unit 30 and the window size setting unit 32 Since only the difference is made, detailed description of the configuration is omitted here.

  FIG. 10 is a flowchart showing processing performed in the second embodiment. When the release button 4 is fully pressed, the CPU 33 starts processing, the imaging units 21A and 21B capture the subject in accordance with an instruction from the CPU 33, and the image processing unit 23 performs correction processing and parallelization on the acquired image data. Processing and image processing are performed to obtain standard images G1a and G1b and reference images G2a and G2b (step ST31). Next, the window size setting unit 32 sets the size of the correlation window for each pixel of the reference image G1a after the parallelization processing (step ST32). Note that the processing for setting the size of the correlation window is the same as that in the first embodiment, and a detailed description thereof will be omitted here.

   Next, the stereo matching unit 30 takes in the correlation window size list stored in the internal memory 27 (step ST33). FIG. 11 shows a correlation window size list. The size list T0 shown in FIG. 11 is obtained by associating the actually calculated size of the correlation window with the size of the correlation window used for the corresponding point search. The size of the calculated correlation window is 10 or 11 pixels. The unit is classified into a plurality of size groups, and the size of the correlation window to be used is created to be the maximum size in each size group. That is, the size list T0 is used when the calculated correlation window size is 1 to 11 pixels, and when the calculated correlation window size is 11 pixels and the calculated correlation window size is 12 to 21 pixels. The correlation window is created to have a size of 21 pixels.

  The stereo matching unit 30 sets the minimum correlation window size (used for the corresponding point search) in the size list T0 as the current correlation window size (current correlation window size setting; step ST34), and the reference In the image G1a, a pixel in which a correlation window in a size range corresponding to the current correlation window size is set is extracted as a corresponding point search target pixel (step ST35). Then, the stereo matching unit 30 searches for the corresponding points for the extracted pixels using the correlation window having the size set in step ST34 (step ST36).

  Here, the stereo matching unit 30 searches for the center pixel of the correlation window that maximizes the correlation in the reference image G2a as a corresponding point. If the maximum value of the correlation when the corresponding point is searched is not so large, the correlation is calculated. It is doubtful whether the corresponding point searched based on the above corresponds to the target pixel of the reference image G1a. Further, when such corresponding points are used, there is a possibility that the distance image D1 cannot be calculated with high accuracy.

  For this reason, the stereo matching unit 30 determines whether or not the searched corresponding point is an incorrect corresponding point by determining whether or not the correlation is equal to or less than a predetermined threshold value Th1 (step ST37). ). Then, a pixel on the reference image G1a determined to be miscorresponding is extracted (extracting miscorresponding pixel; step ST38), and the current correlation is added to the “size of correlation window to be used” in the correlation window size list T0. It is determined whether there is a size larger than the size of the window (step ST39).

  If step ST39 is affirmed, the size of the current correlation window is stored as the size of the previous correlation window, and the size of the current correlation window is set to the next larger size in the correlation window size list T0. (Reset correlation window size: step ST40). Then, the stereo matching unit 30 converts the pixels for which the correlation window in the size range corresponding to the current correlation window size in the reference image G1a and the pixels determined to be incompatible in the process of step ST37 to the corresponding points. Extracted as search target pixels (search target pixel re-extraction; step ST41). And it returns to step ST36 and repeats the process after step ST36.

  As a result, the pixels determined to be inaccurate are included in the corresponding point search processing for the pixels for which the next largest correlation window is set.

  On the other hand, if step ST39 is negative, the distance image generation unit 31 generates a distance image D1 based on the corresponding points searched so far (excluding incorrect corresponding points) (step ST42). Then, in response to an instruction from the CPU 33, the media control unit 26 records the standard image G1a, the reference image G2a, and the distance image D1 on the recording medium 29 (image recording: step ST43), and ends the process.

  Thus, by searching for corresponding points for each correlation window of the same size, it is possible to search for corresponding points efficiently.

  For pixels determined to be miscorresponding, the correlation window size is increased and the corresponding points are searched again, so that it is not necessary to search for corresponding points only for pixels that are miscorresponding. As a result, it is possible to efficiently search for corresponding points for pixels that are erroneously matched, and to improve the accuracy of the corresponding point search.

  Note that in the second embodiment, pixels whose correlation is equal to or less than the predetermined threshold Th1 are determined to be erroneously associated with each other, but the order in which the corresponding pixels are arranged is the reference image G1a and the reference image. A pixel that is reversed on G2a, a pixel that corresponds to a plurality of pixels of the reference image G2a on the base image G1a, and a pixel whose parallax is greatly fluctuating locally with respect to its surroundings are erroneously associated. It may be used as a pixel.

  In the first and second embodiments, the present invention is applied to the stereoscopic imaging device 1 including the imaging units 21A and 21B. However, the distance measuring device according to the present invention is provided separately from the camera. Also good.

  In the first and second embodiments, only the distance image D1 is recorded on the recording medium 29. However, the standard image G1b may be recorded together with the distance image D1, and the standard image G1b and the reference image G2b are recorded. May be recorded together with the distance image D1. Further, the standard image G1a or the standard image G1a and the reference image G2a may be recorded together with the distance image D1, and further, the standard image G1 or the standard image G1 and the reference image G2 before processing in the image processing unit 23 are recorded. May be.

  In the first and second embodiments, only one imaging unit for acquiring the reference image G2 is provided. However, a plurality of reference images are acquired and the distance image D1 is obtained by acquiring a plurality of reference images. You may make it produce | generate.

  In the first and second embodiments, the predetermined luminance interval K0 is set to a constant value regardless of the luminance. Generally, as the luminance increases, the luminance variation increases as shown in FIG. growing. Here, the relationship between the luminance and the luminance variation is proportional to the square root of the luminance as shown in FIG. Therefore, the luminance variation may be obtained for each luminance, and a different luminance interval may be set for each luminance. In this case, as the luminance increases, the predetermined luminance interval K0 increases in proportion to the square root of the luminance as shown in FIG.

  By changing the predetermined luminance interval K0 in this way, it is possible to eliminate the influence of variations having different fluctuation levels at high luminance, such as light shot noise, from the processing for setting the size of the correlation window. You can search for points.

  In the first embodiment, when searching for corresponding points, it is determined whether the corresponding points are miscorresponding as in the second embodiment, and it is determined that they are miscorresponding. Alternatively, the correlation window size may be made larger than the set size, and the corresponding points may be searched again.

  As described above, the embodiment of the present invention has been described. As shown in FIGS. 7, 8, and 10, the computer functions as means corresponding to the stereo matching unit 30, the distance image generation unit 31, and the window size setting unit 32. A program for performing a simple process is also one embodiment of the present invention. A computer-readable recording medium in which such a program is recorded is also one embodiment of the present invention.

1 is a schematic block diagram showing an internal configuration of a stereoscopic imaging device to which a distance measuring device according to a first embodiment of the present invention is applied. Diagram showing the configuration of the imaging unit Diagram for explaining stereo matching The figure for demonstrating the positional relationship of the reference | standard image after a parallelization process, and a reference image The figure which shows the profile of the brightness | luminance of the u-axis direction in a reference | standard image Diagram showing the luminance profile of an image with no luminance change The flowchart which shows the process performed in 1st Embodiment. Flowchart of correlation window size setting processing in the first embodiment The figure for demonstrating the setting process of the size of the correlation window in 1st Embodiment The flowchart which shows the process performed in 2nd Embodiment. Figure showing the correlation window size list The figure for demonstrating the change of the space | interval of a reference | standard brightness level

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo imaging device 21A, 21B Image pick-up part 30 Stereo matching part 31 Distance measurement part 32 Window size setting part

Claims (12)

A reference image for calculating the distance to the point of interest on the subject acquired by imaging the subject, and acquired by imaging the subject from a position different from the position from which the reference image was acquired. Input means for receiving input of at least one reference image for calculating the distance;
A target pixel point on the reference image on which the point of interest is projected, and a search pixel array centered on the target point and extending in a predetermined direction on the reference image, and the luminance of the pixels in the search pixel array A search pixel sequence in which a difference value between a maximum value and a minimum value exceeds a predetermined luminance interval is used, and the number of pixels in the search pixel sequence is used when searching for a corresponding point on the reference image with respect to the target point A window size setting means for setting a correlation window size as a reference for calculating a correlation between the reference image and the reference image;
Corresponding point search means for searching for the corresponding point by calculating a correlation between each pixel in the correlation window set on the reference image and each pixel in the correlation window set on the reference image; ,
A distance measuring device comprising distance calculating means for calculating the distance based on the searched corresponding point.   The window size setting means sets the search pixel sequence having an initial size, calculates a difference value between a maximum value and a minimum value of luminance values in pixels in the search pixel sequence, and the difference value is the predetermined luminance interval. If the determination is negative, the search pixel string is increased by a predetermined number of pixels in the predetermined direction, and the search is continued until the difference value exceeds the predetermined luminance interval. The distance measuring apparatus according to claim 1, wherein the distance measuring device is a unit that repeats the process of enlarging the pixel column.   The distance measuring device according to claim 1, wherein the predetermined direction is a direction corresponding to a direction in which an epipolar line obtained by mapping each pixel of the base image to the reference image extends. .   The distance measuring apparatus according to claim 1, wherein the window size setting unit is a unit that sets the predetermined luminance interval according to a variation in luminance in the reference image.   5. The distance measuring device according to claim 1, wherein the window size setting means is a means for setting the predetermined luminance interval to be smaller as the luminance of the target point is lower.   The corresponding point searching means determines whether or not the searched corresponding point is an incorrect corresponding point, and for the pixel determined to be an incorrect corresponding point, the size of the correlation window is increased again and the pixel is determined again. 6. The distance measuring device according to claim 1, wherein the distance measuring device is a means for searching for corresponding points. Control means for controlling the window size setting means, the corresponding point search means and the distance calculation means so as to calculate the distance for all pixels of the reference image;
The distance measuring device according to claim 1, further comprising a distance image generating unit configured to generate a distance image having a pixel value as a distance between pixels of the reference image.   The said corresponding point search means is a means to search for the said corresponding point collectively for every pixel to which the said correlation window of the same size was set in order with the size of this correlation window being small. Distance measuring device.   The corresponding point search means sets the size of the correlation window for pixels of the reference image for which an incorrect corresponding point that has been erroneously matched among the corresponding points searched using the correlation window of one size is searched. 9. The distance measuring apparatus according to claim 8, wherein the distance measuring device is a means for searching for the corresponding point together with a pixel in which the correlation window having a size larger by one step is set.   The window size setting means classifies the size of the correlation window into a plurality of predetermined size groups, and sets the size of the correlation window of each pixel to the size of the correlation window set for each pixel. 10. The distance measuring device according to claim 7, wherein the distance measuring device is a means for resetting the maximum size in the group. A reference image for calculating the distance to the point of interest on the subject acquired by imaging the subject, and acquired by imaging the subject from a position different from the position from which the reference image was acquired. , Accepting input of at least one reference image for calculating the distance,
A target pixel point on the reference image on which the point of interest is projected, and a search pixel array centered on the target point and extending in a predetermined direction on the reference image, and the luminance of the pixels in the search pixel array A search pixel sequence in which a difference value between a maximum value and a minimum value exceeds a predetermined luminance interval is used, and the number of pixels in the search pixel sequence is used when searching for a corresponding point on the reference image with respect to the target point Is set to the size of a correlation window serving as a reference for calculating the correlation between the reference image and the reference image,
By searching for the corresponding points by calculating the correlation between each pixel in the correlation window set on the reference image and each pixel in the correlation window set on the reference image,
A distance measuring method, wherein the distance is calculated based on the searched corresponding points. A reference image for calculating the distance to the point of interest on the subject acquired by imaging the subject, and acquired by imaging the subject from a position different from the position from which the reference image was acquired. Receiving at least one reference image input for calculating the distance;
A target pixel point on the reference image on which the point of interest is projected, and a search pixel array centered on the target point and extending in a predetermined direction on the reference image, and the luminance of the pixels in the search pixel array A search pixel sequence in which a difference value between a maximum value and a minimum value exceeds a predetermined luminance interval is used, and the number of pixels in the search pixel sequence is used when searching for a corresponding point on the reference image with respect to the target point A procedure for setting a correlation window size as a reference for calculating a correlation between the reference image and the reference image;
A procedure for searching for the corresponding point by calculating a correlation between each pixel in the correlation window set on the reference image and each pixel in the correlation window set on the reference image;
A program for causing a computer to execute a distance measuring method, comprising: calculating the distance based on the searched corresponding points.

Images