WO2012001982A1 - ステレオ画像処理装置およびステレオ画像処理方法 - Google Patents
ステレオ画像処理装置およびステレオ画像処理方法 Download PDFInfo
- Publication number
- WO2012001982A1 WO2012001982A1 PCT/JP2011/003751 JP2011003751W WO2012001982A1 WO 2012001982 A1 WO2012001982 A1 WO 2012001982A1 JP 2011003751 W JP2011003751 W JP 2011003751W WO 2012001982 A1 WO2012001982 A1 WO 2012001982A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- reference image
- unit
- pixel
- thinning
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/04—Interpretation of pictures
- G01C11/06—Interpretation of pictures by comparison of two or more pictures of the same area
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30268—Vehicle interior
Definitions
- the present invention relates to a stereo image processing apparatus and a stereo image processing method.
- a shift between images is calculated from a stereo image (that is, a standard image and a reference image) when the same object is photographed using a stereo camera, and a distance to the object is measured based on the shift.
- Stereo image processing apparatuses are known.
- the stereo image processing apparatus estimates, for example, the orientation of the driver's face by measuring the distance from the in-vehicle device that measures the distance to the vehicle ahead, or from the in-vehicle camera to parts of the driver's face (such as eyes and nose). Application to in-vehicle devices is under consideration.
- the base length of the camera (camera interval) has become shorter.
- the shift between the stereo images is also reduced, so that the stereo image processing apparatus is required to have a highly accurate parallax calculation function.
- stereo matching for example, a SAD (Sum of Absolute Differences) method or a POC (Phase Only Correlation) method is used. .
- SAD Sud of Absolute Differences
- POC Phase Only Correlation
- the SAD value is calculated by taking the sum of the values for the entire partial image.
- the SAD value indicates the degree of difference in image luminance. Then, the position of the rectangular window of the reference image is shifted by one pixel in the base line length direction of the camera, and the position of the first partial image in the standard image and the second in the reference image when the SAD value is minimized.
- the deviation from the position of the partial image is obtained as “pixel level parallax”.
- the SAD method has a high analysis resolution and a relatively small amount of calculation.
- the SAD method has low accuracy of sub-pixel level parallax calculation. Therefore, the accuracy of parallax calculation is limited to about 1/4 to 1/16 pixel.
- the POC method with high accuracy of parallax calculation has attracted attention.
- the first partial image and the second partial image cut out by a Hanning window or the like are subjected to a two-dimensional Fourier transform, and the obtained first partial image and the second partial image after the two-dimensional Fourier transform are obtained.
- a two-dimensional inverse Fourier transform is performed to obtain a phase-only correlation coefficient.
- an image shift amount is calculated based on the correlation peak of the phase-only correlation coefficient.
- Such a POC method (referred to as a two-dimensional POC method) has an advantage that the accuracy of parallax calculation is very high.
- the amount of parallax calculation is enormous, and it is extremely difficult to perform calculation processing in a short time.
- the two-dimensional POC method is inferior to the SAD method in terms of analysis resolution (size on a screen that enables distance measurement by distinguishing objects having different distances).
- a one-dimensional POC method in which the calculation amount of the two-dimensional POC method is reduced has also been proposed (for example, see Patent Document 1).
- this one-dimensional POC method one-dimensional Fourier transformation is performed on the first partial image and the second partial image cut out by a Hanning window or the like, and the first partial image and the second partial image after the one-dimensional Fourier transformation are performed. Of the partial images.
- a phase-only correlation coefficient is obtained by performing a one-dimensional inverse Fourier transform. That is, the amount of calculation is reduced by performing one-dimensional Fourier transform instead of two-dimensional Fourier transform.
- this one-dimensional POC method is inferior to the SAD method in terms of analysis resolution (size on a screen that enables distance measurement by distinguishing objects having different distances).
- An object of the present invention is to provide a stereo image processing apparatus and a stereo image processing method that improve the accuracy of parallax calculation while maintaining a processing amount equivalent to that of the SAD method.
- a stereo image processing apparatus forms acquisition reference images and thinning reference images by acquiring acquisition means for acquiring a stereo image including a reference image and a reference image, and thinning out the reference image and the reference image.
- acquisition means for acquiring a stereo image including a reference image and a reference image
- thinning out the reference image and the reference image By reversing the data sequence of the data sequence composed of the luminance values in the thinning-out reference image, the thinning-out means, the first calculating means for calculating the shift amount in pixel units between the thinning-out reference image and the thinning-out reference image
- Filtering means for calculating an antiphase filter coefficient, filtering the decimation reference image using the calculated antiphase filter coefficient, and outputting a filtering result; and a peak in the filtering result output from the filtering process means
- a second calculation means for calculating a shift amount of the sub-pixels comprises a.
- the stereo image processing method acquires a stereo image including a standard image and a reference image, thins the standard image and the reference image, thereby forming a thinned standard image and a thinned reference image, and performs the thinning.
- the amount of deviation between the reference image and the thinned reference image is calculated, and the phase sequence coefficient is calculated by inverting the data sequence of the data sequence composed of the luminance values in the thinned reference image, and the thinned reference image Is filtered using the calculated anti-phase filter coefficient, and a peak in the filtering result is detected, thereby calculating a shift amount in sub-pixel units between the decimation reference image and the decimation reference image.
- the present invention it is possible to provide a stereo image processing apparatus and a stereo image processing method that improve the calculation accuracy of parallax while maintaining a processing amount equivalent to that of the SAD method.
- FIG. 1 is a block diagram showing a configuration of a stereo image processing apparatus according to Embodiment 1 of the present invention.
- Flow diagram for explaining the operation of the stereo image processing apparatus Diagram for explaining the processing of the image matching unit Flow chart showing details of sub-unit calculation processing Diagram for explaining processing of filter section
- FIG. 3 is a block diagram showing a configuration of a stereo image processing apparatus according to Embodiment 2 of the present invention.
- FIG. 7 is a block diagram showing a configuration of a stereo image processing apparatus according to Embodiment 3 of the present invention.
- FIG. 6 is a block diagram showing a configuration of a stereo image processing apparatus according to Embodiment 4 of the present invention.
- FIG. 5 Block diagram showing a configuration of a stereo image processing apparatus according to Embodiment 6 of the present invention.
- Diagram for explaining cross-correlation processing The figure which compares the ranging accuracy at the time of using the conventional SAD system, a one-dimensional POC system, and the stereo image processing method of this application.
- FIG. 1 shows a configuration of a stereo image processing apparatus 100 according to Embodiment 1 of the present invention.
- the stereo image processing apparatus 100 includes a stereo image acquisition unit 101, an image matching unit 102, a filter unit 103, and a peak position detection unit 104.
- the stereo image acquisition unit 101 acquires a stereo image captured by two or more imaging systems (that is, cameras).
- a stereo image includes a standard image and a reference image in which the same object is captured by two different imaging systems.
- the image matching unit 102 calculates the “deviation amount n” between the reference image and the reference image by performing image matching processing based on the reference image and the reference image acquired by the stereo image acquisition unit 101.
- the unit of the shift amount n calculated by the image matching unit 102 is, for example, a pixel.
- the unit of “parallax” between the standard image and the reference image obtained by the peak position detection unit 104 described later is, for example, a subpixel.
- the image matching unit 102 roughly detects the deviation between the standard image and the reference image in a predetermined detection unit, and then the peak position detection unit 104 detects the deviation finely in the sub unit.
- the image matching unit 102 sets one arbitrary pixel included in the reference image as a “reference point”, and a peripheral partial image centered on the reference point (hereinafter referred to as “unit reference image”) as the reference image. Cut out from. Further, the image matching unit 102 cuts out a plurality of partial images having the same size as the unit standard image (hereinafter referred to as “unit reference images”) from different positions of the reference images. Then, the image matching unit 102 extracts a unit reference image having a maximum matching degree with the unit standard image from the plurality of unit reference images that have been cut out.
- one pixel corresponding to the “reference point” becomes the “corresponding point” on the reference image.
- an index indicating the matching degree for example, an SAD value indicating a luminance difference degree is used.
- the parallax between the base image and the reference image occurs only in the base line length direction of the camera. Therefore, when a plurality of unit reference images are cut out, the cut positions may be changed in the base line length direction. Then, the amount of deviation between the position of the reference point in the reference image and the position of the corresponding point in the reference image is calculated as the amount of deviation n described above.
- the filter unit 103 acquires the position of the reference point and the shift amount n from the image matching unit 102 and the stereo image from the stereo image acquisition unit 101.
- the filter unit 103 calculates a filter coefficient based on the standard image, and filters the reference image using the calculated filter coefficient. That is, the filter unit 103 first extracts a partial image from the reference image as a sub-pixel estimation unit reference image, and calculates a filter coefficient from the sub-pixel estimation unit reference image. Next, the filter unit 103 extracts a partial image from the reference image as a unit reference image for subpixel estimation, performs a filtering process on the subpixel estimation unit reference image using the calculated filter coefficient, and performs filtering. The result is output to the peak position detection unit 104.
- the peak position detection unit 104 detects the peak position in the filtering result acquired from the filter unit 103, thereby calculating the amount of deviation in sub-pixel units between the sub-pixel estimation unit reference image and the sub-pixel estimation unit reference image. To do.
- the peak position is a position where the filtering result is the maximum value.
- the sum of the deviation in sub-pixel units and the deviation amount n in pixel units is an accurate deviation amount between the base image and the reference image.
- FIG. 2 is a flowchart for explaining the operation of the stereo image processing apparatus 100.
- the process for an arbitrary reference point in the reference image will be described.
- the parallax is obtained for all pixels in the distance measurement target area in the reference image by sequentially changing the reference point. Calculated.
- step S1 the image matching unit 102 determines an analysis reference position.
- the analysis reference position is one coordinate point that is a target for calculating parallax in the reference image, and is the reference point described above.
- step S ⁇ b> 2 the image matching unit 102 cuts out a unit reference image that is a partial image based on the analysis reference position determined in step S ⁇ b> 1 from the reference image received from the stereo image acquisition unit 101.
- the unit of the size of the unit reference image is, for example, a pixel.
- step S3 the image matching unit 102 determines a search range and a search start position in the reference image based on the analysis reference position determined in step S2.
- the parallax of the stereo camera is determined by the base line length, which is the distance between the cameras, the focal length of the lens, and the distance from the stereo camera to the object. Therefore, the search range may be determined based on the distance from the stereo camera to the object to be measured. Further, since the object at infinity from the stereo camera is picked up at the same position of the standard image and the reference image, the search start position in the reference image may be set to the same coordinates as the standard point in the standard image.
- step S4 the image matching unit 102 cuts out a peripheral partial image centered on the search start position determined in step S3 as a unit reference image from the reference image received from the stereo image acquisition unit 101.
- the unit of the size of this unit reference image is, for example, a pixel.
- step S5 the image matching unit 102 calculates a matching degree between the unit standard image and the unit reference image.
- a matching degree for example, an SAD value indicating a luminance difference degree or a luminance similarity degree is used.
- step S6 the image matching unit 102 performs search range end determination processing. If it is determined here that the search range has not ended (step S6: NO), the image matching unit 102 shifts the position to be cut out within the search range determined in step S3 by one pixel, and starts a new operation in step S4. A simple unit reference image. In this way, the processing from steps S4 to S6 is repeated until the search range is completed.
- step S7 the image matching unit 102 specifies a unit reference image having the highest matching degree based on the plurality of matching degrees obtained by the processes in steps S4 to S6.
- the image matching unit 102 specifies a unit reference image that has the minimum or minimum brightness difference.
- step S2 to step S7 will be specifically described with reference to FIG.
- step S2 the image matching unit 102 cuts out a peripheral partial image around the analysis reference position as a unit reference image.
- a rectangular window having a predetermined size (vertical size: wv pixel, horizontal size: wh pixel) is used for cutting out the unit reference image.
- the analysis reference position is a reference point (xa, ya).
- the center of the rectangular window defined by the window function is described as being coincident with the analysis reference position, but the analysis reference position exists near the center of the rectangular window even if it is not strictly the center. It only has to be.
- step S3 the image matching unit 102 determines a search range and a search start position in the reference image based on the analysis reference position determined in step S1. For example, the same coordinates (xa, ya) as the analysis reference position in the reference image are used as the search start position (initial coordinates for cutting out the unit reference image in the reference image).
- step S4 the image matching unit 102 cuts out a peripheral partial image around the search start position as a unit reference image from the reference image.
- the same rectangular window as the rectangular window used for extracting the unit standard image is used.
- step S5 the image matching unit 102 calculates the degree of matching between the unit standard image and the unit reference image.
- this matching degree for example, an SAD value indicating the luminance difference degree is used. This SAD value is calculated by the following equation (1).
- the image matching unit 102 shifts the cutout position and cuts out a new unit reference image from the reference image.
- the position to be cut out is shifted by one pixel.
- the direction of shifting is the direction of the right arrow extending from the coordinates (xa, ya) of the reference image in FIG.
- the image matching unit 102 specifies a unit reference image having the maximum matching degree based on the plurality of matching degrees obtained by the processes in steps S4 to S6. Specifically, the image matching unit 102 specifies a unit reference image corresponding to, for example, the smallest SAD value among a plurality of SAD values. One pixel corresponding to the “reference point” in the specified unit reference image becomes a “corresponding point” on the reference image. If the coordinates of the corresponding points are (xa + n, ya), n is the amount of deviation in pixel units.
- the SAD value is used as an index of matching degree, but the present invention is not limited to this, and any one that can be used as an index of matching degree can be used.
- SSD Squared Differences
- step S8 the filter unit 103 and the peak position detection unit 104 perform sub-unit calculation processing based on the corresponding points obtained in step S7 and the reference image and reference image received from the stereo image acquisition unit 101.
- FIG. 4 is a flowchart showing details of the sub-unit calculation process.
- FIG. 5 is a diagram for explaining the concept of the sub-unit arithmetic processing.
- step S11 the filter unit 103 cuts out a sub-pixel estimation unit reference image from the reference image.
- a window function is used to cut out the unit reference image for subpixel estimation.
- the window function for example, the window function w (m) of the Hanning window represented by the equation (2) can be used.
- window function of a Hanning window is used as the window function.
- a Hamming window, a Blackman window, a Kaiser window, or the like may be used as the window function.
- These window functions are selected depending on which of the characteristics (for example, frequency power characteristics, phase characteristics, cut-out edge continuity) of the unit reference image for subpixel estimation is important. For example, the Kaiser window is selected when importance is attached to the phase characteristics, and the Hanning window is selected when importance is attached to reduction of the calculation amount.
- the image cutout processing in the image matching unit 102 is performed in units of pixels, for example, and therefore it is important to reduce the number of calculations rather than accuracy. Therefore, the first window function used in the image matching unit 102 is a window function that simply cuts out image data.
- the function be a target (ie, a function in which the first and last values of one period are zero).
- the first window function When comparing the first window function and the second window function with respect to frequency characteristics, the first window function has a narrower main lobe than the second window function, The amplitude of the side lobe is large.
- the second window function w (m) uses a Hanning window whose vertical axis size is 1 pixel (pixel) and whose horizontal axis size is “KJ” pixels. M is an integer from J to K.
- the second window function w (m) is set around the reference point (xa, ya). As a result, an image having a vertical axis size of 1 pixel and a horizontal axis size of “KJ” pixels centered on the reference point (xa, ya) is cut out as a unit reference image for subpixel estimation.
- f ′ (m) represents a luminance signal string of the sub-pixel estimation unit reference image.
- step S12 the filter unit 103 cuts out a sub-pixel estimation unit reference image centered on the corresponding point detected in step S7 from the reference image.
- the same second window function as that of the sub-pixel estimation unit reference image is used.
- the second window function is set around the corresponding point (xa + n, ya).
- g ′ (m) represents a luminance signal string of the sub-pixel estimation unit reference image.
- the second window function w (m) whose vertical axis size is 1 pixel (pixel) and whose horizontal axis size is “KJ” pixels is used, but this size is an example. Yes, it is not limited to this.
- a second window function having a vertical axis size of 3 pixels may be used. In that case, first, a partial image having a vertical axis size of 3 pixels is cut out from each of the base image and the reference image using the second window function. Then, in the cut out partial image, the average (average luminance) of the luminances of three pixels having the same value on the vertical axis of the coordinate is calculated, and the subpixel estimation with the vertical axis size constituted by the average luminance is 1 pixel. It may be a unit reference image for use or a unit reference image for subpixel estimation.
- the second window function having a vertical axis size of 1 pixel for example, in the partial images cut out in three rows including the upper row and the lower row, the values of the vertical axes of the coordinates are the same.
- An average of the luminance values of the pixels may be calculated, and a sub-pixel estimation unit reference image or a sub-pixel estimation unit reference image configured by the average luminance may be used.
- the average luminance may be calculated after weighting.
- the weighting coefficient used in this case may be determined by a window function like a two-dimensional POC.
- the filter unit 103 calculates anti-phase filter coefficients based on the sub-pixel estimation unit reference image. Specifically, the filter unit 103 rearranges (that is, inverts) the signal sequence (that is, the luminance value signal sequence) in which the luminance values of the respective coordinates in the sub-pixel estimation unit reference image are arranged in order in the reverse order. ) To calculate antiphase filter coefficients. That is, the tap length of the antiphase filter is equal to the horizontal axis size of the unit reference image for subpixel estimation (that is, the window length of the window function).
- step S14 the filter unit 103 filters the unit reference image for subpixel estimation using the antiphase filter coefficient calculated in step S13, and outputs the filtering result to the peak position detection unit 104.
- the window length (KJ) of the window function w (m) is 5 pixels
- the luminance value signal sequence of the unit reference image for subpixel estimation is “1, 2, 3, 4, 5”. It will be explained as being.
- the luminance value signal sequence x (m) of the sub-pixel estimation unit reference image is “1, 2, 3, 4, 5”.
- the filter unit 103 performs a filtering process on the luminance value signal sequence of the sub-pixel estimation unit reference image using the antiphase filter coefficient h (k).
- the signal sequence is such that the coordinates of the constituent signal are “k ⁇ 2, k ⁇ 1, k, k + 1, k + 2”.
- a filtering process is performed by multiplying the antiphase filter coefficient h (k), and a sum z (m) of each multiplication result is calculated.
- m takes an integer value.
- the sub-pixel estimation unit reference image is considered in consideration of the luminance at the coordinate points around the sub-pixel estimation unit reference image.
- the luminance value signal sequence x (m) of an image including two pixels adjacent to each other is “0, 0, 1, 2, 3, 4, 5, 0, 0”.
- the sum z (2) is 55
- the sum z (3) is 40
- the sum z (4) is 26.
- f ′ ( ⁇ k) obtained by inverting the luminance value signal sequence of the unit reference image for subpixel estimation is used as the filter coefficient h (k) of the antiphase filter.
- G ′ (m) is a luminance value of the sub-pixel estimation unit reference image.
- the anti-phase filter is a kind of so-called FIR filter and has a feature that it is a linear transition invariant system.
- the linear transition invariant system is a system in which when the input signal has a shift, the output signal is shifted by the same amount as the input signal. That is, in the specific example described above, the case where there is no deviation between the sub-pixel estimation unit reference image and the sub-pixel estimation unit reference image is described as an example. However, the sub-pixel estimation unit reference image is used for sub-pixel estimation. When there is a deviation smaller than the sampling interval with respect to the unit reference image, the same deviation occurs in the signal sequence z (m) as a filtering result.
- the tap length of the antiphase filter is set according to the size of the shift amount n in pixel units detected by the pixel unit matching. For example, when the shift amount n per pixel is small, the tap length of the antiphase filter is set to be short accordingly. That is, when obtaining the parallax of an object of the same size in real space, the parallax is smaller when the object is far away than when it is near, and the shift amount n in units of pixels is also smaller. At the same time, since the size captured in the image is also reduced, the size of the sub-pixel estimation unit reference image and the sub-pixel estimation unit reference image is changed in accordance with the magnitude of the shift amount n. The tap length of can also be changed adaptively. As a result, the parallax that matches the size of the object to be measured can be calculated.
- the filtering result is the output of the linear transition invariant system, it excludes errors in lens distortion correction, errors such as gain noise caused by CCD and other image sensors, and errors in the calculation accuracy of image extraction by windowing. In theory, it represents the true amount of deviation. Therefore, the true peak position in the sub-pixel unit can be obtained by interpolating the numerical value between the pixels according to the sampling theorem for the output of the antiphase filter discretized in the pixel unit.
- step S15 the peak position detection unit 104 detects the peak position in the filtering result. Based on this peak position, it is possible to detect a shift amount in subpixel units between the base image and the reference image.
- Sinc function is used to detect this peak position.
- the sinc function is a function defined by sin ( ⁇ x) / ⁇ x, and is a function used when returning the discretized sampling data to the original continuous data. It is proved by the sampling theorem that the original continuous data can be completely restored by performing a convolution operation between the sampled discrete data and the sinc function.
- the signal data at the pixel unit interval can be interpolated, and the filtering result z (m) in which the signal data is theoretically interpolated even in the sub-pixel unit. ) True peak position can be derived.
- FIG. 7 is a diagram for explaining the peak position detection using the sinc function
- a curve 701 is a result of the convolution operation of the signal sequence z (m) that is the filtering result calculated in S14 and the sinc function. It is an example.
- the numerical value between the filtering results z (1) and z (2) and the signal data between z (2) and z (3) are interpolated.
- the peak position detection unit 104 detects the peak position of the filtering result z (m) obtained by interpolating the signal data by a binary search.
- the peak position detection unit 104 repeats the same process as described above, with the position C as a new binary search reference point.
- the number of repetitions of this process can be set according to the required subpixel accuracy. That is, if the required sub-pixel accuracy is 1/2 pixel, the above-described processing may be performed once. If the accuracy is 1/4 pixel, it is required twice. If it is 1/8 pixel, it is required 3 times. The number of repetitions is determined according to the subpixel accuracy. Then, the peak position detection unit 104 treats the last obtained midpoint as the detected peak position ⁇ .
- the method for detecting the peak position using the sinc function and the binary search method has been described.
- the present invention is not limited to this, and the peak position may be searched for by a combination of a sinc function and a gradient method.
- any maximum value detection method can be used after interpolating the signal data of the pixel unit interval by convolving the discretized filtering result with a sinc function.
- the peak position detection unit 104 may detect the peak position using quadratic curve approximation. Thereby, the amount of calculation can be reduced.
- the discretized filtering result is fitted with a quadratic curve, and the position of the maximum value of the quadratic curve is detected as the peak position. Thereby, the peak position can be obtained with an accuracy equal to or less than the discretization interval.
- FIG. 8 is a diagram for explaining peak position detection using quadratic curve approximation.
- the value z (0) at the position m 0 where the filtering result z (m) at the pixel unit interval is maximum, and the filtering result at a position shifted by one pixel from the maximum position to the left and right.
- a quadratic curve is obtained through three points z (+1) and z ( ⁇ 1), and a position where the quadratic curve has a maximum value is detected as a peak position ⁇ .
- This peak position ⁇ is calculated by the following equation (5).
- the parallax of the analysis reference position in the reference image is obtained by adding the deviation n in pixel units and the shift ⁇ in subpixel units.
- step S9 the distance measurement target region end determination process is performed, and if there is an unprocessed region that has not yet been processed in steps S1 to S8, the analysis reference position is shifted to display the unprocessed region. The process from step S1 to step S8 is performed.
- image matching unit 102 calculates the amount of deviation in units of pixels between the standard image and the reference image.
- the filter unit 103 calculates an anti-phase filter coefficient by inverting the data sequence of the data sequence composed of the luminance values in the standard image, and filters the reference image using the calculated anti-phase filter coefficient.
- the peak position detection unit 104 detects a peak in the filtering result, thereby calculating a shift amount in sub-pixel units between the standard image and the reference image.
- the parallax is obtained by adding the shift amount in pixel units and the shift amount in sub-pixel units.
- FIG. 9 shows a configuration of stereo image processing apparatus 200 according to Embodiment 2 of the present invention.
- the stereo image processing apparatus 200 includes a data deletion unit 201.
- the data deletion unit 201 forms the thinned standard image and the thinned reference image by thinning the image data of the standard image and the reference image. Specifically, the data deletion unit 201 thins out the image data of the standard image and the reference image by sampling at a predetermined sampling period.
- the band-limited data signal has a larger amount of data but an equal amount of information than the data signal obtained by thinning out the band-limited data.
- the focal length of the lens When using a camera, it is possible to remove high-band components of image data (data signal) by adjusting the focal length of the lens. That is, by making a blurred image (an image with reduced resolution) before thinning out the data signal, the data signal before thinning and the data signal after thinning can be regarded as equivalent. For example, when a VGA image having a horizontal resolution of 640 ⁇ vertical 480 is reduced to a QVGA image having a horizontal resolution of 320 ⁇ vertical 240, the image is obtained by adjusting the focal length of the lens so that the resolution is halved. The obtained image data may be sampled every other pixel.
- the thinning standard image and the thinning reference image are output to the image matching unit 102 and the filter unit 103. Therefore, in the stereo image processing apparatus 200, the image matching unit 102 and the filter unit 103 target the thinning reference image and the thinning reference image.
- Stereo image processing apparatus 200 The operation of the stereo image processing apparatus 200 having the above configuration will be described. Note that the stereo image acquisition unit 101 according to the present embodiment acquires a blurred image (an image with reduced resolution) from which high-frequency components have been removed by adjusting the focal length of the camera lens, and the data deletion unit 201 It shall be handed over.
- a blurred image an image with reduced resolution
- FIG. 10 is a flowchart for explaining the operation of the stereo image processing apparatus 200.
- step S21 the data deleting unit 201 forms the thinned reference image and the thinned reference image by thinning out the image data of the standard image and the reference image.
- FIG. 11 is a diagram comparing an optical signal (image signal) (FIG. 11A) obtained when the thinning process is not executed and an optical signal (FIG. 11B) obtained when the thinning process is executed.
- Each of the upper XY planes in FIGS. 11A and 11B is an image (a base image or a reference image) taken with the focal length of the lens adjusted so that the resolution is halved. That is, the image shown on the XY plane is a state in which high-frequency components are removed from the original image.
- a plurality of rectangles included in the XY plane indicate pixels.
- the optical signal in FIG. 11A is obtained from pixels (that is, all pixels) indicated by white rectangles included in the XY plane (that is, thinning processing is not performed).
- the optical signal in FIG. 11B is obtained from pixels (in other words, every other pixel) indicated by white rectangles included in the XY plane (that is, thinning processing is performed).
- the high frequency components of the original image are removed by halving the resolution.
- the optical signals reproduced in each case can be regarded as equivalent by the sampling theorem.
- the amount of image data can be reduced even when averaging is performed with a plurality of pixels (for example, 4 pixels) instead of the thinning process.
- this process is not a linear process, signal processing is performed in subsequent processes. Processing based on the theory becomes meaningless.
- the resolution may affect the pixel pitch, which may degrade the accuracy.
- the resolution is halved, the pixel pitch is doubled, so the distance error is about twice regardless of the parallax calculation accuracy of the matching method.
- phase correlation is accurately determined only from the image data in other bands. It is possible to obtain the phase characteristics necessary for the calculation. That is, by performing the thinning process, the matching accuracy can be theoretically maintained even if the amount of image data is reduced and the amount of calculation is reduced.
- the data deletion unit 201 is provided before the image matching unit 102 and the filter unit 103, and thins out the image data of the reference image and the reference image.
- a thinning standard image and a thinning reference image are formed.
- the filter unit 103 performs a filtering process using an antiphase filter, which is a matching process based on the phase correlation.
- the filter unit 103 performs a filtering process using an anti-phase filter, which is a matching process based on the phase correlation, the parallax calculation accuracy can be maintained even if the thinning process is performed.
- FIG. 12 shows a configuration of stereo image processing apparatus 300 according to Embodiment 3 of the present invention.
- the stereo image processing apparatus 300 includes a high frequency component deletion unit 301.
- the high frequency component deletion unit 301 performs a process of extracting only the low frequency component by suppressing the high frequency component among the low frequency component and the high frequency component constituting the image data (data signal) of the standard image and the reference image.
- the high frequency component means a component having a frequency equal to or higher than 1 ⁇ 2 of the sampling frequency in the data deletion unit 201.
- the high frequency component deletion unit 301 is configured by, for example, a low-pass filter (low-pass filter). This filter may be a linear filter such as an FIR filter.
- FIG. 13 is a flowchart for explaining the operation of the stereo image processing apparatus 300.
- step S31 the high frequency component deletion unit 301 suppresses only the low frequency component by suppressing the high frequency component among the low frequency component and the high frequency component constituting the image data (data signal) of the standard image and the reference image. Perform the extraction process.
- FIG. 14 shows an amplitude characteristic with respect to the spatial frequency (FIG. 14A) obtained when the high-frequency component suppression process is not executed, and an amplitude characteristic with respect to the spatial frequency (when the high-frequency component suppression process is executed). It is a figure which compares FIG. 14B).
- FIG. 14B shows the amplitude characteristics with respect to the spatial frequency when the frequency component of 1 ⁇ 2 or more of the sampling frequency in the subsequent data deletion unit 201 is suppressed.
- the XY plane shown in the upper part of FIG. 14A is image data (data signal) in which high-frequency components are not suppressed
- the XY plane shown in the upper part of FIG. 14B is image data (data signals) in which high-frequency components are suppressed.
- the subsequent processing of the data deleting unit 201 is the same as that in the second embodiment.
- phase characteristics necessary for matching processing based on phase correlation can be obtained from only low frequency components. Therefore, the parallax can be calculated with high accuracy.
- the high frequency component is not removed by the linear filter, the high frequency component is superimposed on the low frequency component as aliasing noise when performing the thinning process, and therefore the phase characteristics in the low frequency component change, As a result, the parallax accuracy is degraded.
- the low frequency component is suppressed by suppressing the high frequency component among the low frequency component and the high frequency component constituting the signal of the standard image and the reference image. Process to extract only the components.
- each of the base image and the reference image is composed of a plurality of channel images (for example, an R (red) channel image, a G (green) channel image, and a B (blue) channel image).
- a plurality of channel images for example, an R (red) channel image, a G (green) channel image, and a B (blue) channel image.
- FIG. 15 shows a configuration of stereo image processing apparatus 400 according to Embodiment 4 of the present invention.
- the stereo image processing apparatus 400 includes a color image data deletion unit 401.
- the color image data deletion unit 401 extracts only one channel image (for example, an R channel image) from the standard image and the reference image. As a result, other channel images can be deleted, so that the processing amount in the processing of the image matching unit 102 and the processing of the filter unit 103 in the subsequent stage can be reduced.
- one channel image for example, an R channel image
- FIG. 16 is a flowchart for explaining the operation of the stereo image processing apparatus 400.
- step S41 the color image data deletion unit 401 extracts only one channel image from the standard image and the reference image.
- FIG. 17 is a diagram for explaining a Bayer array image obtained by a color filter.
- the Bayer array image is configured with R pixels, G pixels, G pixels, and B pixels as one constituent unit (that is, a color pixel unit).
- the color image data deletion unit 401 extracts only the R channel image from the image data of the standard image and the reference image.
- R pixels are arranged every other pixel. Therefore, extracting an optical signal (image signal) of only the R channel image is equivalent to a process of thinning out the image data of the standard image and the reference image.
- FIG. 18 is an image diagram when extracting only the R channel image, and the same effects as the contents described in FIG. 14 can be obtained.
- each of the standard image and the reference image is composed of a plurality of channel images
- the color image data deletion unit 401 is By extracting only one channel image from the plurality of channel images, a thinning reference image and a thinning reference image are formed. Accordingly, the amount of image data can be reduced, and the processing amount of the image matching unit 102 and the filter unit 103 can be reduced.
- the filter unit 103 performs a filtering process using an anti-phase filter, which is a matching process based on the phase correlation.
- Phase characteristics sufficient to maintain matching accuracy based on phase correlation can be obtained from only one channel image. Therefore, the parallax calculation accuracy can be maintained even when the above-described thinning process is performed.
- the S / N ratio can be improved by taking the sum of the luminance signals between the pixels arranged in the direction perpendicular to the baseline length direction.
- the S / N ratio can be improved by taking the sum of the luminance signals between the pixels arranged in the vertical direction. That is, taking the above-described color pixel unit as an example, the parallax calculation accuracy can be improved by taking the sum of the pixel value of the R pixel and the pixel value of the G pixel existing under the R pixel. Note that in the case where the sum of luminance signals is obtained between pixels arranged in a direction perpendicular to the baseline length direction, the sum of pixel values may be obtained between pixels having the same wavelength.
- the object whose parallax is calculated has a surface that exists at substantially the same distance from the camera.
- the S / N ratio can be improved by taking the sum of the pixels arranged in the baseline length direction.
- the left-right direction is the baseline length direction.
- the S / N ratio can be improved by taking the sum of the luminance signals between the pixels arranged in the left-right direction. That is, taking the above-described color pixel unit as an example, the parallax calculation accuracy can be improved by taking the sum of the pixel value of the R pixel and the pixel value of the G pixel existing under the R pixel.
- the horizontal coordinate (X coordinate) is different, and pixels having the same wavelength do not take the sum. This is because the sampling interval of the optical signal is not every other pixel, but is equivalent to sampling while averaging two consecutive pixels, and the frequency characteristics of the optical signal cannot be maintained. For example, R (red), G (green), and G (green) in the Bayer array are not summed.
- the Bayer array shown here is an example, and the present invention is not limited to this.
- each of the base image and the reference image includes a plurality of channel images (for example, an R (red) channel image, a G (green) channel image, and a B (blue) channel image. ).
- the color pixel unit is composed of four pixels, that is, an R pixel, a first G pixel, a second G pixel, and a B pixel, the R pixel, the first G pixel, and the B pixel.
- the luminance values are linearly combined in the first pixel group.
- FIG. 19 shows the configuration of a stereo image processing apparatus 500 according to Embodiment 5 of the present invention.
- the stereo image processing apparatus 500 includes a data deletion unit 501, an image matching unit 502, and a filter unit 503.
- the data deleting unit 501 forms one channel composite image by linearly combining the luminance values in the first pixel group in two pixel groups into which a plurality of pixels constituting the color pixel unit are divided.
- This channel composite image is formed for each of the standard image and the reference image. That is, a channel composition reference image and a channel composition reference image are formed.
- the first pixel group includes R pixel, first G pixel, and B pixel. Consists of pixels.
- the second pixel group includes second G pixels.
- the image matching unit 502 basically has the same function as the image matching unit 102. However, the image matching unit 502 sets the channel synthesis standard image and the channel synthesis reference image as processing targets, and performs matching processing based on the channel synthesis standard image and the channel synthesis reference image. The image matching unit 502 outputs the reference point and the “deviation amount n” to the filter unit 503 as a result of the matching process. Note that the image matching unit 502 does not process the reference image of the second pixel group and the reference image of the second pixel group.
- the filter unit 503 calculates the first filter coefficient and the second filter coefficient based on the channel synthesis reference image for the first pixel group of the reference image and the image of the second pixel group, respectively.
- the filter unit 503 obtains a first filtering result by filtering the channel synthesis reference image using the first filter coefficient.
- the filter unit 503 obtains a second filtering result by filtering the images of the second image group of the reference image using the second filter coefficient, using the second filter coefficient.
- the filter unit 503 obtains a final filtering result by adding the first filtering result and the second filtering result.
- the final filtering result is output to the peak position detection unit 104.
- the peak detection accuracy can be improved by detecting the peak position using the final filtering result obtained by adding the first filtering result and the second filtering result.
- FIG. 20 shows a configuration of stereo image processing apparatus 600 according to Embodiment 6 of the present invention.
- the stereo image processing apparatus 600 includes a cross-correlation unit 601.
- the cross-correlation unit 601 cuts out a unit reference image for subpixel estimation from a unit reference image based on corresponding points. Then, the cross-correlation unit 601 calculates the cross-correlation between the sub-pixel estimation unit reference image and the sub-pixel estimation unit reference image.
- FIG. 21 is a diagram for explaining the cross-correlation processing.
- the cross-correlation unit 601 cuts out a unit reference image for subpixel estimation from the unit reference image based on the corresponding points.
- the same second window function as that in the case of the sub-pixel estimation unit reference image is also used for extracting the sub-pixel estimation unit reference image.
- the second window function is set to the corresponding point (xa + n, ya).
- an image having a vertical axis size of 1 pixel and a horizontal axis size of “KJ” pixels centered on the corresponding point (xa + n, ya) is extracted as a sub-pixel estimation unit reference image.
- the cross-correlation unit 601 calculates the cross-correlation between the sub-pixel estimation unit reference image and the sub-pixel estimation unit reference image. This cross-correlation is calculated by equation (7).
- Equation (3) the inside of ⁇ is equivalent to Equation (3), and the range of addition of ⁇ is from -J to -K.
- J and K indicate the window function range centered on zero. J and K have opposite signs.
- the order of addition is different only in the mathematical expression, and a calculation result equivalent to the expression (3) can be obtained. That is, the calculation by the antiphase filter can be replaced with the calculation of the cross correlation. Therefore, even when cross-correlation is used, high-precision sub-pixel level matching can be performed as in the method using the anti-phase filter.
- FIG. 22 shows a result of comparison of ranging accuracy when the conventional SAD method, the one-dimensional POC method, and the stereo image processing method of the present application (hereinafter, this method) are used.
- the distance measurement accuracy is indicated by the characteristics of the distance to the distance measurement object and the standard deviation of the distance measurement result.
- the results shown in FIG. 22 are calculated on the basis of a stereo image that is taken while the distance measurement target is a vehicle and the distance from the stereo camera is changed at 10 m intervals.
- the standard deviation of the distance measurement result is used in order to eliminate error factors of correction of lens distortion and parallelization correction of the stereo camera.
- a method with a small standard deviation of the distance measurement result is a highly accurate method.
- the standard deviation is a variation in distance measurement results.
- the standard deviation of the distance measurement result is calculated by using, as sample points, pixels in the vehicle area that are visually extracted from among the pixels included in the captured stereo image. For the subpixel estimation of this method, quadratic curve approximation with the least amount of calculation was used. As shown in FIG. 22, the POC method ( ⁇ mark) and the present method ( ⁇ mark) show equivalent characteristics, and the standard deviation is smaller than that of the SAD method ( ⁇ mark).
- FIG. 23 shows the result of comparing the calculation time of the parallax calculation when the SAD method, the one-dimensional POC method, and the present method are used.
- the stereo image processing method according to the present invention has a calculation time equivalent to that of the SAD method and can achieve distance measurement accuracy equivalent to that of the one-dimensional POC method. There is an effect.
- the stereo image acquired by the stereo image acquisition unit is directly input to the image matching unit and the filter unit.
- the following processing may be performed as preprocessing before input.
- a parallelization correction unit may be provided in front of the image matching unit and the filter unit, and the parallelization correction unit may perform a distortion correction process for the lens of the stereo image and a parallelization correction process for making the optical axis parallel.
- the distortion correction process is a process for correcting an object that is a straight line in real space so that it appears in a straight line in a camera image using calibration data prepared in advance.
- the parallel correction process a coordinate conversion process for converting coordinates so that an object having a constant distance in the optical axis direction of the camera is an object image of the same size no matter where the image is captured in the camera image; And an image shift process in which an object at an infinite point is placed at the same position in two camera images with the optical axes of the two being parallel.
- the parallelization correction is performed by the image shift process after the coordinate conversion process, but the present invention is not limited to this, and the parallelization correction may be performed by the coordinate conversion simultaneously with the lens distortion correction.
- the method is not particularly limited as long as both the correction of the distortion of the lens and the correction of the positional relationship between the two cameras can be performed.
- contrast correction performed by normal image processing or edge enhancement using a Laplacian filter may be performed.
- contrast correction the dynamic range of the luminance change between the standard image and the reference image can be matched, so that more accurate image matching can be performed.
- edge enhancement using a Laplacian filter a direct current component (that is, a difference in brightness between the reference image and the reference image) resulting from individual differences between cameras can be excluded, and more accurate image matching is performed. be able to.
- luminance information sampled in units of pixels (at integer value positions) is converted into luminance information at real value positions.
- an interlinear method using linear interpolation or a bicubic method using luminance information around the conversion target position may be used.
- the stereo image processing device described in each of the above embodiments may be provided with a matching error detection unit that detects a pixel level matching error based on the filtering result.
- the matching error detection unit determines that the pixel level matching is a false matching when the output from the filter unit is not symmetric (that is, bilaterally symmetric).
- the matching error detection unit detects a matching error when the minimum position of the SAD value and the peak position of the output from the filter unit are different at the pixel level (that is, when no peak appears in the output from the filter unit). Judge that there is. Thereby, it is not necessary to perform back matching processing, and the amount of calculation corresponding to that is reduced.
- each functional block used in the description of each of the above embodiments is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them. Although referred to as LSI here, it may be referred to as IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.
- the method of circuit integration is not limited to LSI, and implementation with a dedicated circuit or a general-purpose processor is also possible.
- An FPGA Field Programmable Gate Array
- a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.
- the stereo image processing apparatus and stereo image processing method of the present invention are useful for improving the parallax calculation accuracy while maintaining a processing amount equivalent to that of the SAD method.
- Stereo image processing apparatus 101 Stereo image acquisition unit 102, 502 Image matching unit 103, 503 Filter unit 104 Peak position detection unit 201, 501 Data deletion unit 301 High frequency component deletion unit 401 Color image Data deletion unit 601 cross-correlation unit
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Measurement Of Optical Distance (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
Description
[ステレオ画像処理装置100の構成]
図1は、本発明の実施の形態1に係るステレオ画像処理装置100の構成を示す。図1において、ステレオ画像処理装置100は、ステレオ画像取得部101と、画像マッチング部102と、フィルタ部103と、ピーク位置検出部104とを有する。
ステレオ画像取得部101は、2つ以上の撮像系(つまり、カメラ)で撮影されたステレオ画像を取得する。ステレオ画像には、2つの異なる撮像系によって同一対象物が撮影された基準画像及び参照画像が含まれる。
画像マッチング部102は、ステレオ画像取得部101で取得された基準画像及び参照画像に基づいて画像マッチング処理を行うことにより、基準画像と参照画像との「ズレ量n」を算出する。画像マッチング部102で算出されるズレ量nの単位は、例えば、ピクセルである。一方、後述するピーク位置検出部104によって得られる、基準画像と参照画像との「視差」の単位は、例えば、サブピクセルである。すなわち、画像マッチング部102では、基準画像と参照画像とのズレが所定の検出単位で粗く検出され、その後に、ピーク位置検出部104によってズレがサブ単位で細かく検出される。
フィルタ部103は、画像マッチング部102から基準点の位置、及びズレ量n、並びに、ステレオ画像取得部101からステレオ画像を取得する。
ピーク位置検出部104は、フィルタ部103から取得されたフィルタリング結果におけるピーク位置を検出することにより、サブピクセル推定用単位基準画像とサブピクセル推定用単位参照画像とのサブピクセル単位のズレ量を算出する。ここで、ピーク位置は、フィルタリング結果が最大値となる位置である。このサブピクセル単位のズレと、ピクセル単位のズレ量nとの和が、基準画像と参照画像の正確なズレ量である。
以上の構成を有するステレオ画像処理装置100の動作について説明する。なお、以下では、画像横方向をX軸、画像縦方向をY軸として、1画素が1座標点であるものとして説明する。
ステップS1で、画像マッチング部102は、分析基準位置を決定する。分析基準位置とは、基準画像において視差を算出する対象となる1座標点であり、上述した基準点である。
ステップS2で、画像マッチング部102は、ステレオ画像取得部101から受け取る基準画像から、ステップS1で決定された分析基準位置を基準とした部分画像である、単位基準画像を切り出す。この単位基準画像の大きさの単位は、例えば、ピクセルである。
ステップS3で、画像マッチング部102は、ステップS2で決定された分析基準位置に基づいて、参照画像におけるサーチ範囲及びサーチ開始位置を決定する。ステレオカメラの視差は、カメラ間の距離である基線長およびレンズの焦点距離、ならびにステレオカメラから対象物までの距離により決定される。よって、サーチ範囲は、ステレオカメラから測距の対象物までの距離に基づいて決定すればよい。また、ステレオカメラから無限遠にある対象物は基準画像と参照画像の同じ位置に撮像されるため、参照画像におけるサーチ開始位置は基準画像における基準点と同じ座標を設定すればよい。
ステップS4で、画像マッチング部102は、ステレオ画像取得部101から受け取る参照画像から、ステップS3で決定されたサーチ開始位置を中心とした周辺の部分画像を単位参照画像として切り出す。この単位参照画像の大きさの単位は、例えば、ピクセルである。
ステップS5で、画像マッチング部102は、単位基準画像と単位参照画像とのマッチング度を算出する。このマッチング度には、例えば、輝度相違度を示すSAD値や輝度類似度が用いられる。
ステップS6で、画像マッチング部102は、サーチ範囲の終了判定処理を行う。ここでサーチ範囲が終了していないと判定される場合(ステップS6:NO)には、画像マッチング部102は、ステップS3で決定したサーチ範囲内において切り出す位置を1ピクセルずらして、ステップS4で新たな単位参照画像を切り出す。このようにして、ステップS4~S6までの処理は、サーチ範囲が終了するまで繰り返される。
ステップS7で、画像マッチング部102は、ステップS4~S6の処理によって得られた複数のマッチング度に基づいて、マッチング度が最大となる単位参照画像を特定する。マッチング度として輝度相違度を用いられている場合には、画像マッチング部102は、輝度相違度が極小ないし最小となる単位参照画像を特定する。
ステップS8で、フィルタ部103及びピーク位置検出部104は、ステップS7で得られた対応点、並びに、ステレオ画像取得部101から受け取る基準画像及び参照画像に基づいて、サブ単位演算処理を行う。
ステップS11で、フィルタ部103は、基準画像からサブピクセル推定用単位基準画像を切り出す。
ステップS12で、フィルタ部103は、参照画像から、ステップS7で検出された対応点を中心としたサブピクセル推定用単位参照画像を切り出す。サブピクセル推定用単位参照画像の切出し処理において、サブピクセル推定用単位基準画像の場合と同じ第2の窓関数が用いられる。ただし、第2の窓関数は、対応点(xa+n、ya)を中心に設定される。これにより、サブピクセル推定用単位参照画像として、対応点(xa+n,ya)を中心に、縦軸サイズが1ピクセルであり横軸サイズが“K-J”ピクセルである画像が切り出される。図5において、g’(m)は、サブピクセル推定用単位参照画像の輝度信号列を表している。
ステップS13で、フィルタ部103は、サブピクセル推定用単位基準画像に基づいて逆位相フィルタ係数を算出する。具体的には、フィルタ部103は、サブピクセル推定用単位基準画像における各座標の輝度値を順番に並べた信号列(つまり、輝度値信号列)を逆の順番に並べ換える(つまり、反転させる)ことにより、逆位相フィルタ係数を算出する。すなわち、逆位相フィルタのタップ長は、サブピクセル推定用単位基準画像の横軸サイズ(つまり、窓関数の窓長)と等しい。
ステップS14で、フィルタ部103は、ステップS13で算出された逆位相フィルタ係数を用いてサブピクセル推定用単位参照画像をフィルタリングし、フィルタリング結果をピーク位置検出部104へ出力する。
ステップS15で、ピーク位置検出部104は、フィルタリング結果におけるピーク位置を検出する。このピーク位置に基づいて、基準画像と参照画像とのサブピクセル単位のズレ量を検出することができる。
ステップS9では、測距対象領域の終了判定処理が行われ、未だステップS1~ステップS8までの処理が行われていない未処理領域が存在する場合には、分析基準位置をずらしてその未処理領域についてステップS1~ステップS8までの処理を行う。
実施の形態2では、基準画像及び参照画像を間引いた後に、画像マッチング部102の処理及びフィルタ部103の処理を行う。これにより、処理量が削減される。
図9は、本発明の実施の形態2に係るステレオ画像処理装置200の構成を示す。図9において、ステレオ画像処理装置200は、データ削除部201を有する。
以上の構成を有するステレオ画像処理装置200の動作について説明する。なお、本実施の形態のステレオ画像取得部101は、カメラのレンズの焦点距離を調整することにより、高周波成分を取り除いたぼけた画像(解像度を落とした画像)を取得し、データ削除部201に受け渡すものとする。
実施の形態3では、実施の形態2における間引き処理の前段に、基準画像及び参照画像の画像データ(データ信号)を構成する低周波数成分及び高周波数成分の内、高周波数成分を抑制することにより、低周波数成分のみを抽出する処理を行う。これにより、ステレオ画像を他の用途にも応用する場合などで、レンズの焦点距離を調整することにより、ぼけた画像にすることが望ましくない場合にも精度劣化を防止して演算量を削減することができる。
図12は、本発明の実施の形態3に係るステレオ画像処理装置300の構成を示す。図12において、ステレオ画像処理装置300は、高周波成分削除部301を有する。
以上の構成を有するステレオ画像処理装置300の動作について説明する。図13は、ステレオ画像処理装置300の動作説明に供するフロー図である。
実施の形態4では、特に、基準画像及び参照画像のそれぞれが、複数のチャネル画像(例えば、R(赤)チャネル画像、G(緑)チャネル画像、B(青)チャネル画像)から構成される。そして、実施の形態4では、複数のチャネル画像の内の1つのみが抽出されるか、又は、種類が異なる複数のチャネル画像が平均化される。これにより、実施の形態2における間引きと同等の効果が得られる。
図15は、本発明の実施の形態4に係るステレオ画像処理装置400の構成を示す。図15において、ステレオ画像処理装置400はカラー画像データ削除部401を有する。
以上の構成を有するステレオ画像処理装置400の動作について説明する。図16は、ステレオ画像処理装置400の動作説明に供するフロー図である。
実施の形態5では、実施の形態4と同様に、基準画像及び参照画像のそれぞれが、複数のチャネル画像(例えば、R(赤)チャネル画像、G(緑)チャネル画像、B(青)チャネル画像)から構成される。実施の形態5では、カラー画素ユニットがR画素,第1のG画素,第2のG画素,B画素の4画素から構成されている場合に、R画素,第1のG画素,及びB画素から成る第1の画素グループと、第2のG画素から成る第2の画素グループとにグルーピングし、第1の画素グループ内で輝度値の線形合成を行う。
実施の形態6では、フィルタ算出処理及びフィルタリング処理の代わりに、相互相関処理を行う。
図20は、本発明の実施の形態6に係るステレオ画像処理装置600の構成を示す。図20において、ステレオ画像処理装置600は、相互相関部601を有する。
以上の構成を有するステレオ画像処理装置600の動作について説明する。図21は、相互相関処理の説明に供する図である。
(1)上記各実施の形態では、ステレオ画像取得部で取得されたステレオ画像を画像マッチング部及びフィルタ部へ直接入力したが、入力前の前処理として次の処理が行われてもよい。
101 ステレオ画像取得部
102,502 画像マッチング部
103,503 フィルタ部
104 ピーク位置検出部
201,501 データ削除部
301 高周波成分削除部
401 カラー画像データ削除部
601 相互相関部
Claims (6)
- 基準画像及び参照画像を含むステレオ画像を取得する取得手段と、
前記基準画像及び前記参照画像を間引くことにより、間引き基準画像及び間引き参照画像を形成する間引き手段と、
前記間引き基準画像と前記間引き参照画像とのピクセル単位のズレ量を算出する第1の算出手段と、
前記間引き基準画像内の輝度値から成るデータ列のデータ順序を反転させることにより逆位相フィルタ係数を算出し、前記間引き参照画像を前記算出された逆位相フィルタ係数を用いてフィルタリングし、フィルタリング結果を出力するフィルタリング処理手段と、
前記フィルタリング処理手段から出力されたフィルタリング結果におけるピークを検出することにより、前記間引き基準画像と前記間引き参照画像とのサブピクセル単位のズレ量を算出する第2の算出手段と、
を具備するステレオ画像処理装置。 - 前記間引き手段の前段に設けられ、前記間引き手段に入力される前記基準画像及び前記参照画像の信号を構成する低周波数側成分及び高周波数側成分の内、前記高周波数側成分を抑制することにより、前記低周波数側成分のみを抽出し、前記高周波数側成分が除かれた、前記基準画像及び前記参照画像を前記間引き手段へ出力する抽出手段、
をさらに具備する請求項1に記載のステレオ画像処理装置。 - 前記基準画像及び前記参照画像のそれぞれは、複数のチャネル画像から構成され、
前記間引き手段は、前記基準画像及び前記参照画像から、前記複数のチャネル画像の内で1つのチャネル画像のみを抽出することにより、前記間引き基準画像及び前記間引き参照画像を形成する、
請求項1に記載のステレオ画像処理装置。 - 前記基準画像及び前記参照画像のそれぞれは、複数のチャネル画像から構成され、
前記間引き手段は、前記複数のチャネル画像を合成して1つの合成チャネル画像を形成することにより、前記間引き基準画像及び前記間引き参照画像を形成する、
請求項1に記載のステレオ画像処理装置。 - 前記基準画像及び前記参照画像のそれぞれは、複数のカラー画素ユニットから構成され、
各カラー画素ユニットは、赤画素、第1の緑画素、第2の緑画素、青画素の4画素から構成され、
前記赤画素、前記第1の緑画素、及び前記青画素は第1の画素グループを構成し、前記第2の緑画素は第2の画素グループを構成し、
前記間引き手段は、前記基準画像の前記第1の画素グループ内でチャネル合成基準画像を形成することにより、前記チャネル合成基準画像と前記第2の画素グループの前記基準画像とから成る前記間引き基準画像を形成し、前記参照画像の前記第1の画素グループ内でチャネル合成参照画像を形成することにより、前記チャネル合成参照画像と前記第2のグループの前記参照画像とから成る前記間引き参照画像を形成し、
前記第1の算出手段は、前記チャネル合成基準画像及び前記チャネル合成基準画像に基づいて、前記ピクセル単位のズレ量を算出し、
前記フィルタリング処理手段は、前記チャネル合成基準画像及び前記第2の画素グループの前記基準画像から、第1の逆位相フィルタ係数及び第2の逆位相フィルタ係数を算出し、
前記チャネル合成参照画像を前記第1の逆位相フィルタ係数を用いてフィルタリングし、前記第2のグループの前記参照画像を前記第2の逆位相フィルタ係数を用いてフィルタリングすることにより得られた第1のフィルタリング結果及び第2のフィルタリング結果を足し合わせることによって得られた最終的なフィルタリング結果を前記第2の算出手段へ出力する、
請求項1に記載のステレオ画像処理装置。 - 基準画像及び参照画像を含むステレオ画像を取得し、
前記基準画像及び前記参照画像を間引くことにより、間引き基準画像及び間引き参照画像を形成し、
前記間引き基準画像と前記間引き参照画像とのピクセル単位のズレ量を算出し、
前記間引き基準画像内の輝度値から成るデータ列のデータ順序を反転させることにより逆位相フィルタ係数を算出し、前記間引き参照画像を前記算出された逆位相フィルタ係数を用いてフィルタリングし、
フィルタリング結果におけるピークを検出することにより、前記間引き基準画像と前記間引き参照画像とのサブピクセル単位のズレ量を算出する、
ステレオ画像処理方法。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP11800452.2A EP2592595A4 (en) | 2010-06-30 | 2011-06-30 | Stereo image processing apparatus and method of processing stereo image |
CN201180025461.9A CN102906782B (zh) | 2010-06-30 | 2011-06-30 | 立体图像处理装置及立体图像处理方法 |
US13/702,168 US8903135B2 (en) | 2010-06-30 | 2011-06-30 | Stereo image processing apparatus and method of processing stereo image |
JP2011548467A JP5810314B2 (ja) | 2010-06-30 | 2011-06-30 | ステレオ画像処理装置およびステレオ画像処理方法 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010149425 | 2010-06-30 | ||
JP2010-149425 | 2010-06-30 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012001982A1 true WO2012001982A1 (ja) | 2012-01-05 |
Family
ID=45401720
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2011/003751 WO2012001982A1 (ja) | 2010-06-30 | 2011-06-30 | ステレオ画像処理装置およびステレオ画像処理方法 |
Country Status (5)
Country | Link |
---|---|
US (1) | US8903135B2 (ja) |
EP (1) | EP2592595A4 (ja) |
JP (1) | JP5810314B2 (ja) |
CN (1) | CN102906782B (ja) |
WO (1) | WO2012001982A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2012060093A1 (ja) * | 2010-11-05 | 2012-05-10 | パナソニック株式会社 | ステレオ画像処理装置及びステレオ画像処理方法 |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2013214234A (ja) * | 2012-04-03 | 2013-10-17 | Sony Corp | 移動制御装置、移動制御方法、およびプログラム |
JP6017399B2 (ja) * | 2013-10-23 | 2016-11-02 | オリンパス株式会社 | 撮像装置及び位相差検出方法 |
CN104977795A (zh) * | 2014-04-08 | 2015-10-14 | 立普思股份有限公司 | 立体摄影系统及其方法 |
JP2017099616A (ja) * | 2015-12-01 | 2017-06-08 | ソニー株式会社 | 手術用制御装置、手術用制御方法、およびプログラム、並びに手術システム |
CN105844633B (zh) * | 2016-03-21 | 2019-03-26 | 西安电子科技大学 | 基于De序列和相位编码的单帧结构光深度获取方法 |
US10957068B2 (en) | 2017-01-06 | 2021-03-23 | Canon Kabushiki Kaisha | Information processing apparatus and method of controlling the same |
JP6894707B2 (ja) * | 2017-01-06 | 2021-06-30 | キヤノン株式会社 | 情報処理装置およびその制御方法、プログラム |
CN110211053B (zh) * | 2019-04-28 | 2023-09-15 | 航天智造(上海)科技有限责任公司 | 用于三维测量的快速精确相位匹配方法 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000171214A (ja) * | 1998-12-08 | 2000-06-23 | Meidensha Corp | 対応点検索方法及びこれを利用した三次元位置計測方法 |
JP2008123141A (ja) | 2006-11-09 | 2008-05-29 | Yamatake Corp | 対応点探索方法および3次元位置計測方法 |
WO2010113389A1 (ja) * | 2009-03-31 | 2010-10-07 | パナソニック株式会社 | ステレオ画像処理装置およびステレオ画像処理方法 |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050088515A1 (en) * | 2003-10-23 | 2005-04-28 | Geng Z. J. | Camera ring for three-dimensional (3D) surface imaging |
JP4328692B2 (ja) * | 2004-08-11 | 2009-09-09 | 国立大学法人東京工業大学 | 物体検出装置 |
JP3937414B2 (ja) * | 2004-08-11 | 2007-06-27 | 本田技研工業株式会社 | 平面検出装置及び検出方法 |
CN101404777B (zh) * | 2008-11-06 | 2010-12-01 | 四川虹微技术有限公司 | 一种基于深度图像的绘制的视图合成方法 |
CN101729918A (zh) * | 2009-10-30 | 2010-06-09 | 无锡景象数字技术有限公司 | 一种实现双目立体图像校正和显示优化的方法 |
-
2011
- 2011-06-30 EP EP11800452.2A patent/EP2592595A4/en not_active Withdrawn
- 2011-06-30 US US13/702,168 patent/US8903135B2/en not_active Expired - Fee Related
- 2011-06-30 JP JP2011548467A patent/JP5810314B2/ja not_active Expired - Fee Related
- 2011-06-30 WO PCT/JP2011/003751 patent/WO2012001982A1/ja active Application Filing
- 2011-06-30 CN CN201180025461.9A patent/CN102906782B/zh not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000171214A (ja) * | 1998-12-08 | 2000-06-23 | Meidensha Corp | 対応点検索方法及びこれを利用した三次元位置計測方法 |
JP2008123141A (ja) | 2006-11-09 | 2008-05-29 | Yamatake Corp | 対応点探索方法および3次元位置計測方法 |
WO2010113389A1 (ja) * | 2009-03-31 | 2010-10-07 | パナソニック株式会社 | ステレオ画像処理装置およびステレオ画像処理方法 |
Non-Patent Citations (1)
Title |
---|
See also references of EP2592595A4 |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2012060093A1 (ja) * | 2010-11-05 | 2012-05-10 | パナソニック株式会社 | ステレオ画像処理装置及びステレオ画像処理方法 |
US9148653B2 (en) | 2010-11-05 | 2015-09-29 | Panasonic Intellectual Property Management Co., Ltd. | Stereo image processing device and stereo image processing method |
JP5874039B2 (ja) * | 2010-11-05 | 2016-03-01 | パナソニックIpマネジメント株式会社 | ステレオ画像処理装置及びステレオ画像処理方法 |
Also Published As
Publication number | Publication date |
---|---|
EP2592595A4 (en) | 2017-11-15 |
EP2592595A1 (en) | 2013-05-15 |
CN102906782A (zh) | 2013-01-30 |
US8903135B2 (en) | 2014-12-02 |
JPWO2012001982A1 (ja) | 2013-08-22 |
CN102906782B (zh) | 2016-03-02 |
US20130094713A1 (en) | 2013-04-18 |
JP5810314B2 (ja) | 2015-11-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5810314B2 (ja) | ステレオ画像処理装置およびステレオ画像処理方法 | |
JP5567179B2 (ja) | ステレオ画像処理装置およびステレオ画像処理方法 | |
US9275463B2 (en) | Stereo image processing device and stereo image processing method | |
US8355564B2 (en) | Corresponding point searching method and three-dimensional position measuring method | |
JP6258209B2 (ja) | ステレオ画像処理装置およびステレオ画像処理方法 | |
JP5874039B2 (ja) | ステレオ画像処理装置及びステレオ画像処理方法 | |
EP2657901B1 (en) | Stereo image processing apparatus and stereo image processing method | |
US20090226097A1 (en) | Image processing apparatus | |
JP2001208524A (ja) | 画像処理装置及び画像処理方法 | |
KR101532686B1 (ko) | 경계 적응적 필터를 이용한 영상 처리 장치 및 그 방법 | |
de Lima et al. | An edge directed super resolution technique for multimedia applications | |
CN114841894A (zh) | 一种光谱畸变的校正方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201180025461.9 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011548467 Country of ref document: JP |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11800452 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13702168 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011800452 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |