JP4749986B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that performs preprocessing on each captured image in order to obtain parallax of the same object existing in both of the two captured images respectively captured by the two image capturing apparatuses.

  2. Description of the Related Art Conventionally, for example, a system for capturing an image of the front side of a vehicle with two imaging devices arranged in parallel at the front of the vehicle and monitoring an object such as a person existing on the front side of the vehicle based on the captured images Is known (see, for example, Patent Documents 1 and 2). In such a system, based on two captured images, the parallax (parallax in the parallel direction of two imaging devices) of the same object existing in the captured images is detected. Then, the relative distance of the object with respect to the vehicle is calculated from the detected parallax.

  In this case, in order to detect parallax, it is necessary to search for a point in the other captured image corresponding to a point on an object in one of the captured images (hereinafter referred to as a point of interest). As this search method, as seen in Patent Document 1, a difference method or a normalized correlation method is generally used. In these methods, the degree of coincidence of distribution of pixel values (such as luminance values) in a local region including a target point in the one captured image and a local region in the search region of the other captured image is determined by so-called SAD, Judgment is made based on the value of a correlation function such as NCC, and a point in the local region of the other captured image having the highest degree of matching is determined in the one captured image with respect to the local region including the target point in one captured image. This is a method of searching as a point in the other captured image corresponding to the point of interest. In addition, in the thing of patent document 2, SAD is used.

Further, in Patent Document 2, in order to compensate for variations in pixel values between both captured images due to differences in characteristics between two imaging devices, the A / D conversion values of the outputs of each imaging device are assigned to each imaging device. A technique for correcting with a corresponding lookup table is known.
JP 2000-329852 A Japanese Patent Laid-Open No. 5-114099

  Incidentally, a variety of noises are included in the captured image of each imaging apparatus. In addition, even if the same object is imaged by two imaging devices due to variations in characteristics of individual imaging devices, the pixel values such as the luminance values of the pixels of the object image in each captured image are both Variation occurs between captured images. Such noise included in each captured image and variations in pixel values between the both captured images become an error factor in parallax detection. Therefore, in order to increase the detection accuracy of parallax, it is desirable to appropriately compensate for the influence of error factors such as noise included in each captured image and variations in pixel values between both captured images.

  However, in the technique shown in Patent Document 1, since the parallax is detected using the captured image of each imaging device as it is, the influence of the error factor is not compensated. For this reason, it is often difficult to accurately detect the parallax.

  Further, in the technique disclosed in Patent Document 2, a lookup table for each imaging device is used to compensate for variations in pixel values between the imaging devices, but noise included in each captured image. The effects of are not compensated. For this reason, it is difficult to further increase the detection accuracy of the parallax with the technique found in Patent Document 2. Further, the look-up table as found in Patent Document 2 is necessary for each image pickup device, so that it lacks versatility and is disadvantageous in terms of cost.

  The present invention has been made in view of such a background, and by performing appropriate preprocessing on the captured images obtained from the two imaging devices, the influence of noise included in each captured image and the relationship between the two captured images are obtained. It is an object to provide a versatile image processing apparatus that can appropriately compensate for the influence of variations in pixel values and detect parallax with high accuracy based on each pre-processed image data. To do.

  First, the basic concept of the present invention will be described. Although detailed description will be described later, according to the knowledge of the present inventor, the same object is imaged by the first imaging device and the second imaging device arranged in parallel in a predetermined direction (for example, the left-right direction and the up-down direction). In this case, if the pixel value of each pixel of the first captured image obtained from the first imaging device and the same object as the pixel are captured, the second captured image (captured image obtained from the second imaging device) ) Is generally approximated by a normal distribution having a central value (average value) of a certain value. In each captured image, the stochastic distribution of the difference between the pixel value of each pixel and the pixel value of a pixel having a relatively small number of pixels d from the pixel (hereinafter, the second stochastic distribution). May be approximated by a normal distribution having a center value (average value) of 0 and a standard deviation or variance according to the number of pixels d (interval).

  As described above, when the first stochastic distribution and the second stochastic distribution are approximated by a normal distribution, for each pixel in a predetermined region (at least a region where parallax is to be detected) of each captured image, By performing the following pre-processing, the influence of noise included in each captured image and the influence of variations in pixel values between both captured images are appropriately compensated, and the influence is small (the S / N ratio is high). ) Image data after preprocessing can be obtained.

  In the preprocessing, for each pixel in a predetermined area of each captured image, a pixel value I (p + d) of a pixel having an interval of a certain number of pixels d in the positive direction in the predetermined direction. And a pixel value I (pd) of a pixel having an interval of the number of pixels d in the negative direction in the predetermined direction, or a value obtained by scaling the difference value is a pixel value of the pixel. A pre-processing for setting values is preferable. Further, the pre-processing is performed in each captured image by calculating a stochastic distribution of a difference between a pixel value of each pixel and a pixel value of a pixel having an interval of the number of pixels d in the predetermined direction with respect to the pixel. Among the values of d that can be approximated by the above normal distribution (a normal distribution whose center value (average value) is 0 and has a standard deviation or variance according to the number of pixels d (interval)) It is preferable to use the largest value of d.

Therefore, in order to achieve the above object, the image processing apparatus according to the first invention of the present application is first imaged by each of the first imaging device and the second imaging device arranged in parallel in a predetermined direction. In order to obtain the parallax of the same object existing in both the captured image and the second captured image, an image processing device that performs preprocessing on each captured image,
For any one of the first captured image and the second captured image, each pixel value I (p) of the plurality of pixels in the one captured image and each pixel On the other hand, the stochastic distribution of the difference value from the pixel value I (p + d) of the pixel having an interval of the pixel number d in the predetermined direction has an average value of 0 and corresponds to the value of the pixel number d. The maximum value dmax of the number of pixels d that can be approximated by a normal distribution having a standard deviation is used as sampling data of pixel values of a plurality of sets of pixels spaced in the predetermined direction in the one captured image. Preliminary processing means to be determined based on;
For each pixel in a predetermined area of each captured image, a pixel value I (p + dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in a positive direction in the predetermined direction, and the predetermined value A difference from the pixel value I (p-dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in the negative direction is obtained, and the obtained difference value or the difference value is scaled. Preprocessing execution means for obtaining, as preprocessed image data used for obtaining the parallax of the same object, an image having a value obtained as a pixel value of each pixel in a predetermined area of each captured image. (First invention).

  According to the first aspect of the invention, the maximum value dmax of the pixel number d obtained by the preliminary processing unit is used for the preprocessing by the preprocessing execution unit. That is, for each pixel in a predetermined region of each captured image, the pixel value I of a pixel having an interval of the number of pixels of the maximum value dmax in the positive direction in the predetermined direction (the parallel direction of both imaging devices). The difference value between (p + dmax) and the pixel value I (p-dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in the negative direction in the predetermined direction, or the difference value is scaled Pre-processed image data having the adjusted value as the pixel value of the pixel is created from each imaging device. By such pre-processing, the influence of noise included in each captured image and the influence of variations in pixel values between both captured images are appropriately compensated, and after such pre-processing that the influence is small (high S / N ratio) Image data can be obtained. As a result, the detection accuracy can be improved by detecting the parallax using the preprocessed image data. In this case, since it is not necessary to prepare the characteristics of the individual imaging devices in advance, the versatility of the image processing device of the first invention can be improved.

  Note that the “pixel value” in the first invention may generally be a luminance value, but when the captured image is a color image, hue, saturation, or the like may be used as the pixel value. The “scale adjustment” means a process of multiplying the difference value by a predetermined gain, a process of adding a predetermined offset value to the difference value, or a synthesis process of these processes. The same applies to the second to sixth inventions described later.

Here, as described above, when the second stochastic distribution is approximated by a normal distribution, in each captured image, the pixel value I (p) of each of a plurality of pixels and the predetermined value for each pixel The variance of the difference from the pixel value I (p + d) of the pixels having an interval of the number of pixels d in the direction is ω ² (d ² ), and the pixel values I (p (p)) of the plurality of pixels of the one captured image ) and, when the pixel value I of a pixel that exists the interval of one pixel in the predetermined direction for each pixel (p + 1) dispersion of the difference between the omega ² (1 ^2), distributed omega ² (d ² ) is approximately equal to a value obtained by multiplying the variance ω ² (1 ² ) by the value of the number of pixels d. Accordingly, the maximum value dmax of the number of pixels d such that the second stochastic distribution can be approximated by a normal distribution is, in other words, the pixels in the variance ω ² (d ² ) and the variance ω ² (1 ² ). It can be considered that the maximum value dmax of d is such that the absolute value of the difference from the value obtained by multiplying the value of the number d is 0 or a value close thereto.

Therefore, the image processing apparatus according to the second invention of the present application includes a first captured image and a second captured image respectively captured by the first imaging device and the second imaging device arranged in parallel in a predetermined direction. In order to obtain the parallax of the same object existing in both, the image processing device that pre-processes each captured image,
For any one of the first captured image and the second captured image, each pixel value I (p) of the plurality of pixels in the one captured image and each pixel On the other hand, the variance of the difference from the pixel value I (p + d) of the pixels having an interval of the number of pixels d in the predetermined direction is ω ² (d ² ), and each of the plurality of pixels of the one captured image is The variance of the difference between the pixel value I (p) and the pixel value I (p + 1) of a pixel having an interval of one pixel in the predetermined direction with respect to each pixel is defined as ω ² (1 ² ). The number of pixels d such that the absolute value of the difference between the variance ω ² (d ² ) and the value obtained by multiplying the variance ω ² (1 ² ) by the number of pixels d is smaller than a predetermined value. Preliminary processing means for obtaining a maximum value dmax based on sampling data of pixel values of a plurality of sets of pixels spaced in the predetermined direction in the one captured image;
For each pixel in a predetermined area of each captured image, a pixel value I (p + dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in a positive direction in the predetermined direction, and the predetermined value A difference from the pixel value I (p-dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in the negative direction is obtained, and the obtained difference value or the difference value is scaled. Preprocessing execution means for obtaining, as preprocessed image data used for obtaining the parallax of the same object, an image having a value obtained as a pixel value of each pixel in a predetermined area of each captured image. (2nd invention).

  According to the second aspect of the invention, the maximum value dmax obtained by the preliminary processing means corresponds to the maximum value of the number of pixels d such that the second stochastic distribution can be approximated by a normal distribution. Therefore, by using this maximum value dmax and pre-processing each captured image in the same manner as in the first invention, the same effect as in the first invention can be obtained.

In the second aspect of the invention, the preliminary processing means may obtain the maximum value dmax as follows, for example. That is, the preliminary processing means obtains the variance ω ² (1 ² ) based on sampling data of pixel values of a plurality of sets of pixels having an interval of one pixel in the predetermined direction in the one captured image. And the variance ω ² (d ² ) corresponding to each set value of the d value is set so that the value of the pixel number d is sequentially increased by an integer value of 2 or more. And a means for obtaining on the basis of sampling data of pixel values of a plurality of sets of pixels having an interval of the number of pixels of the set value in the predetermined direction in the captured image, and obtained in accordance with each set value of the value of d Means for determining whether an absolute value of a difference between the variance ω ² (d ² ) and a value obtained by multiplying the calculated variance ω ² (1 ² ) by the set value is smaller than the predetermined value; , The setting value immediately before the setting value of d when the judgment result is negative It may be composed of a means for determining a serial maximum value dmax (third invention).

According to the third aspect of the invention, first, the variance ω ² (1 ² ) is calculated based on sampling data of pixel values of a plurality of sets of pixels having an interval of one pixel in the predetermined direction in the one captured image. Desired. Then, while the value of d is sequentially set as 2, 3, 4,..., The variance ω ² (d ² ) corresponding to each set value is set in the predetermined direction in the one captured image. It is obtained based on sampling data of pixel values of a plurality of sets of pixels having an interval of the number of pixels of the set value. Further, the absolute value of the difference between the variance ω ² (d ² ) obtained corresponding to each set value of d and the value obtained by multiplying the variance ω ² (1 ² ) by the set value is the predetermined value. It is determined whether or not the value is smaller than the value (whether or not the difference is substantially zero). In this case, when the set value of d exceeds a certain value, the degree of approximation between the second probabilistic distribution and the normal distribution becomes low, so the result of the above determination becomes negative. Therefore, in the third aspect of the invention, the set value immediately before the set value of d when the above determination is negative is determined as the maximum value dmax. Thereby, the maximum value dmax for preprocessing can be appropriately determined by a simple exploratory method.

  In the first to third aspects of the invention, only one of the first captured image and the second captured image is used to obtain the maximum value dmax of the d value for preprocessing. Can also be used to determine the value of dmax.

Accordingly, the image processing apparatus according to the fourth aspect of the present invention provides a first captured image and a second captured image respectively captured by the first imaging device and the second imaging device arranged in parallel in a predetermined direction. In order to obtain the parallax of the same object existing in both, the image processing device that pre-processes each captured image,
For each of the first captured image and the second captured image, a pixel value I (p) of each of the plurality of pixels in each captured image and a pixel in the predetermined direction with respect to each pixel The stochastic distribution of the difference value from the pixel value I (p + d) of pixels having an interval of several d is a normal distribution having an average value of 0 and a standard deviation corresponding to the value of the number of pixels d. A candidate value for the maximum value dmax of the number d of pixels to be approximated is obtained for each captured image based on sampling data of pixel values of a plurality of sets of pixels that are spaced in the predetermined direction in each captured image. And a preliminary processing means for determining a smaller candidate value among the candidate values of the maximum value dmax corresponding to each obtained captured image as the maximum value dmax for preprocessing,
A pixel value I (p + dmax) of a pixel having an interval of the number of pixels of the maximum value dmax for preprocessing in a positive direction in the predetermined direction with respect to each pixel in the predetermined region of each captured image. And the pixel value I (p-dmax) of the pixel having the interval of the maximum value dmax for preprocessing in the negative direction in the predetermined direction, and the difference value obtained or the difference Preprocessing execution means for obtaining an image having a value obtained by adjusting the scale as a pixel value of each pixel in a predetermined area of each captured image as preprocessed image data used for obtaining the parallax of the same object (4th invention).

  According to the fourth aspect of the invention, the preliminary processing means obtains a candidate value for the maximum value dmax for each captured image, and uses the smaller one of the candidate values as the maximum value dmax for preprocessing. Determine as. Therefore, a suitable preprocessing maximum value dmax is determined for both the first captured image and the second captured image. Then, using the maximum value dmax for preprocessing, the same effect as in the first invention can be obtained by performing preprocessing on each captured image as in the first invention.

In addition, as described with reference to the second invention, in each captured image, the maximum value dmax of the number of pixels d such that the second stochastic distribution can be approximated by a normal distribution is, in other words, the variance ω ² ( The maximum value dmax of d is such that the absolute value of the difference between d ² ) and the value obtained by multiplying the variance ω ² (1 ² ) by the value of the number of pixels d is 0 or close to this value. You can also.

Therefore, an image processing apparatus according to a fifth aspect of the present invention provides a first captured image and a second captured image respectively captured by a first imaging device and a second imaging device arranged in parallel in a predetermined direction. In order to obtain the parallax of the same object existing in both, the image processing device that pre-processes each captured image,
For each of the first captured image and the second captured image, a pixel value I (p) of each of the plurality of pixels in each captured image and a pixel in the predetermined direction with respect to each pixel A variance of a difference from a pixel value I (p + d) of pixels having an interval of several d is ω ² (d ² ), and each pixel value I (p) of the plurality of pixels of the one captured image is When the variance of the difference between each pixel and the pixel value I (p + 1) of the pixel having an interval of one pixel in the predetermined direction is ω ² (1 ² ), the variance ω ² (d ² ) and a candidate value for the maximum value dmax of the number of pixels d such that the absolute value of the difference between the variance ω ² (1 ² ) multiplied by the number of pixels d is smaller than a predetermined value, For each captured image, each captured image is obtained based on sampling data of pixel values of a plurality of sets of pixels that are spaced apart in the predetermined direction. A pretreatment means for determining the smaller candidate values of ones of the candidate values of the maximum value dmax to respond as the maximum value dmax of pretreatment,
A pixel value I (p + dmax) of a pixel having an interval of the number of pixels of the maximum value dmax for preprocessing in a positive direction in the predetermined direction with respect to each pixel in the predetermined region of each captured image. And a pixel value I (p-dmax) of a pixel having an interval of the maximum number of dmax pixels for preprocessing in the negative direction in the predetermined direction, and the difference value obtained or Pre-processing for obtaining, as pre-processed image data, an image having a value obtained by adjusting a difference value as a pixel value of each pixel in a predetermined region of each captured image as a pixel value of the same object And an execution means (fifth invention).

  According to the fifth aspect of the present invention, the candidate value of the maximum value dmax obtained for each captured image by the preliminary processing means is such that the second stochastic distribution can be approximated by a normal distribution in each captured image. This corresponds to the maximum value of the number d. Therefore, by determining the smaller candidate value of these maximum value dmax candidate values as the preprocessing maximum value dmax, the first captured image and the second captured image are the same as in the fourth aspect of the invention. A pre-processing maximum value dmax suitable for any of the images is determined. Then, using the maximum value dmax for preprocessing, the same effect as in the first invention can be obtained by performing preprocessing on each captured image as in the first invention.

In the fifth invention, the preliminary processing means may determine the dmax candidate value for each captured image by the same method as in the third invention. That is, in the fifth aspect, the preliminary processing means includes a plurality of sets of the variance ω ² (1 ² ) for each captured image and an interval of one pixel in the predetermined direction in each captured image. Means for obtaining the sampling value of the pixel value of each pixel, and for each of the captured images, the value of d is set while sequentially increasing the value of the number of pixels d by an integer value of 2 or more. The variance ω ² (d ² ) corresponding to each set value is obtained based on sampling data of pixel values of a plurality of sets of pixels having an interval of the number of pixels of the set value in the predetermined direction in each captured image Means, and for each captured image, the variance ω ² (d ² ) determined corresponding to each set value of the value of d and the calculated variance ω ² (1 ² ) are multiplied by the set value. Means for determining whether or not the absolute value of the difference from the value is smaller than the predetermined value; Means for determining, for each captured image, a setting value immediately before the set value of d when the determination result is negative as a candidate value of the maximum value dmax corresponding to each captured image; What is necessary is just to comprise from the means which determines the smaller candidate value of the candidate values of the said maximum value dmax determined corresponding to the captured image as the said maximum value dmax for pre-processing (6th invention).

  According to the sixth aspect of the invention, for each captured image, a dmax candidate value is determined by the same processing as in the third aspect of the invention. The smaller one of these candidate values is determined as the maximum value dmax for preprocessing. Therefore, the maximum value dmax for preprocessing can be appropriately determined using a simple exploratory method.

  In each of the inventions described above, the detection of parallax using the preprocessed image data of each captured image may be performed using a method similar to the conventional method. That is, with respect to an arbitrary pixel (a pixel capturing an image of an object) in any one of the first captured image and the second captured image, the distribution of pixel values in the local region including the pixel By searching for a local region of the other captured image that maximizes the degree of coincidence using a correlation function such as SAD, the pixel of the other captured image corresponding to the pixel of the one captured image is determined. The difference between the positions of the pixels in the predetermined direction may be detected as parallax. In this case, it is particularly preferable to use SAD with a small calculation processing load.

  A first embodiment of the image processing apparatus of the present invention will be described with reference to FIGS.

  FIG. 1 is a block diagram showing the system configuration of the apparatus of this embodiment. With reference to the figure, in the present embodiment, in order to monitor an object such as a person existing in front of a vehicle 1 such as an automobile, two imaging devices 2R and 2L are mounted on the front portion of the vehicle 1. ing. The imaging devices 2R and 2L are configured by a CCD camera or the like. In the present embodiment, these imaging devices 2R and 2L are arranged side by side in the left-right direction of the vehicle 1 with their optical axes directed in the front-rear direction of the vehicle 1. The imaging devices 2R and 2L are located on the right side and the left side of the vehicle 1, respectively. The vehicle 1 is equipped with an image processing unit 3 to which video signals of images taken by these imaging devices 2R and 2L are input. The image processing unit 3 is an electronic circuit unit including a microcomputer and the like, recognizes an object such as a person existing on the front side of the vehicle 1 based on the input video signal, and the vehicle 1 of the recognized object. Processing such as detecting a spatial relative position with respect to is executed.

  The image processing unit 3 includes an image acquisition unit 4, an image preprocessing unit 5, a parallax calculation unit 6, an object extraction unit 7, and a tracking unit 8 as functions realized by the hardware and software.

  Here, a schematic process of each part of the image processing unit 3 will be described below.

  Outputs (video signals) of the imaging devices 2R and 2L are taken into the image acquisition unit 4 at a predetermined processing cycle. The image acquisition unit 4 performs A / D conversion on the video signal (analog signal) of each pixel of the input captured images of the imaging devices 2R and 2L, and the A / D converted digital data is stored in an image memory (not shown). Keep in memory. The digital data is data indicating the pixel value of each pixel of each captured image, and the pixel value is a luminance value in the present embodiment. Hereinafter, the captured image obtained from the imaging device 2R is referred to as a right image, and the captured image obtained from the imaging device 2L is referred to as a left image. In the present embodiment, these right image and left image are grayscale images.

  Thus, the right image and the left image stored and held in the image memory by the image acquisition unit 4 are input to the image preprocessing unit 5. The image preprocessing unit 5 performs preprocessing on the input right image and left image as will be described later, and creates preprocessed image data. The image processing unit 3 realizes the function as the image processing apparatus of the present invention by the function of the image preprocessing unit 5.

  The preprocessed image data is input to the parallax calculation unit 6. The parallax calculation unit 6 is captured by one of the right image and the left image, for example, each pixel of the right image, based on the preprocessed image data of the right image and the preprocessed image data of the left image. The parallax of the object is detected. Then, based on the detected parallax, the distance of the object captured by each pixel of the right image is calculated, and a so-called distance map (a map in which the calculated distance is associated with each pixel of the right image) is created. . A specific example of the parallax detection process will be described later.

  The distance map calculated by the parallax calculation unit 6 is input to the object extraction unit 7. The object extraction unit 7 extracts an object present in the right image based on the input distance map. For example, in the right image, a continuous pixel group having the same distance from the vehicle 1 (distance difference within a predetermined range) and continuous is extracted as an individual object.

  The output of the object extraction unit 7 (data indicating the existence area in the right image of each object) is input to the tracking unit 8. The tracking unit 8 determines the identity of the objects extracted by the object extracting unit 7 from each of the right images obtained at different times, and performs tracking for the same object for the time. Specifically, the identity of the object A extracted from the right image obtained at time k1 and the object B extracted from the right image obtained at time k2 is determined, and when it is determined that they are the same The identification labels of the objects A and B are the same. In this case, the identity of the objects A and B is determined based on, for example, feature quantities such as the shapes of the objects A and B and the luminance distribution (luminance distribution on the right image).

  Note that the distance information of the object extracted based on the right image and the left image as described above is used, for example, to determine the possibility of contact between the object and the vehicle 1, and the possibility of contact is high. The determined object is visually or audibly notified to the driver of the vehicle 1. Alternatively, an auxiliary driving operation (handle operation, brake operation, etc.) of the vehicle 1 is performed via an actuator (not shown).

  Next, the processing of the image preprocessing unit 5 will be described in detail below. First, the principle of the processing of the image preprocessing unit 5 will be described.

  In the following description, it is assumed that an xy coordinate system in which the horizontal direction (left-right direction) is the x direction and the vertical direction (up-down direction) is the y direction is set for each of the right image and the left image. In this case, the origin of the xy coordinate system for the right image and the origin of the xy coordinate system for the left image have the same positional relationship with respect to the right image and the left image (for example, the upper left corner of each of the left and right images). Point, the center point of each image, etc.). The pixel at the position (x, y) of the right image is expressed as GR (x, y), and the luminance value is expressed as IR (x, y). Similarly, the pixel at the position (x, y) of the left image is expressed as GL (x, y), and the luminance value is expressed as IL (x, y). Furthermore, the brightness value of each pixel GR (x, y) after preprocessing of the right image is IR ′ (x, y), and the brightness value of each pixel GL (x, y) after preprocessing of the left image is IL ′. Expressed as (x, y). However, when there is no need to distinguish between the right image and the left image, GR (x, y) and GL (x, y) are typically represented by G (x, y), IR (x, y), IL (x, y) is typically represented by I (x, y), and IR ′ (x, y) and IL ′ (x, y) are typically represented by I ′ (x, y).

  In this embodiment, since the parallel direction of the imaging devices 2R and 2L is the x direction, the parallax to be calculated by the parallax calculation unit 6 is a parallax in the x direction (left and right direction) between the left and right images. In this case, it is assumed that the preprocessing applied to the left and right images in order to increase the parallax detection accuracy is generally expressed by the following equation (1).

In the above formula (1), α is a certain integer value of “1” or more, and a _d (d = −α,..., 0,..., Α) is a weighting coefficient. This expression (1) is obtained for the pixel G (x, y) with 2α + 1 pixels within a range having an interval of the number of pixels α from the pixel G (x, y) in the positive direction and the negative direction of the x axis. Weighted average values (linear combination values) of the respective luminance values of the pixels (pixels continuously arranged in the x direction from the pixel G (x−α, y) to the pixel G (x + α, y)), This represents preprocessing for setting a new luminance value I ′ (x, y) of the pixel G (x, y). Note that this preprocessing does not depend on the value of the y coordinate, and therefore the y coordinate value is often omitted in the following description. For example, I (x, y) is abbreviated as I (x).

The preprocessing represented by equation (1) is defined by a set of weighting factors a _d (this is commonly referred to as an operator). In the present embodiment, among the preprocessing expressed by the equation (1), image data obtained by the preprocessing (more specifically, from the luminance value IR ′ (x, y) after the preprocessing of the right image). A set of weighting factors a _d that are considered to be able to detect parallax with as high accuracy as possible from a set of data and data composed of luminance values IL ′ (x, y) after preprocessing of the left image) And pre-process the left and right images. Such pre-processing is determined based on a probability model expressed by the following equations (2) and (3).

IL (x) = IR (x) + ε (x), ε (x) ≒ N (μ, σ) (2)
I (x + d) = I (x) + τ (d), τ (d) ≈N (0, ω (d ² )) (3)

In these formulas (2) and (3), N (a, b) (a and b are general variables) means a normal distribution function whose median (average value) and standard deviation are a and b, respectively. . Equation (2) is obtained when the same object image is captured with parallax 0 between the pixel GR (x, y) of the right image and the pixel GL (x, y) of the left image. The probabilistic distribution of the difference ε (x) from the respective luminance values IR (x, y) and IL (x, y) of GR (x, y) and GL (x, y) is the center value (average value) ) And standard deviation are respectively approximated by a normal distribution function N (μ, σ) with μ and σ. In this case, ε (x) corresponds to noise caused by a difference in characteristics between the imaging devices 2R and 2L. In Equation (2), the noise causes an average offset of μ between the luminance value IL (x, y) and the luminance value IR (x, y), and the variance (variation) of the offset is It is expressed as noise with a normal distribution of σ ² .

Further, ω (d ² ) in the equation (3) means a function whose value increases monotonously as the absolute value of d increases. Note that ω (d ² ) = 0 when d = 0. Therefore, the expression (3) is obtained when the two pixels G (x, y) and G (x + d, y) separated by d pixels in the x direction are relatively close in the left and right images. The probabilistic distribution of the difference τ (d) between the luminance values I (x, y) and I (x + d, y) of G (x, y) and G (x + d, y) Value) and standard deviation are approximated by a normal distribution function with 0 and ω (d ² ), respectively, and the variance (variation) ω ² (d ² ) of the difference τ (d) is the absolute value of d Is expressed as the value of G increases (as pixels G (x, y) and G (x + d, y) increase). In other words, this expression (3) is obtained when the pixels G (x, y) and G (x + d, y) are close to each other (when the absolute value of d is close to “1”). There is a high probability that an image of the same object is captured at the pixels G (x, y) and G (x + d, y), and as a result, their luminance values I (x, y) and I (x + It expresses that the difference τ (d) between d, y) has a high probability of becoming a value close to zero. Furthermore, the expression (3) is captured by the pixels G (x, y) and G (x + d, y) as the pixels G (x, y) and G (x + d, y) are further away from each other. The probability that different objects are different from each other increases, and the probability that the absolute value of the difference τ (d) between the luminance values I (x, y) and I (x + d, y) increases will increase. expressing.

  When the above equations (1), (2), and (3) are assumed, the S / N of the image data (preprocessed image data) after the left and right images are preprocessed by the equation (1). The ratio has a correlation with the amount represented by the expression on the right side of the expression (4), as shown by the following expression (4).

  That is, the square value of the S / N ratio is proportional to the amount represented by the expression on the right side of Expression (4). The denominator of the right side of this equation (4) is the luminance value IR ′ (x, y) of the pixel GR (x, y) in the preprocessed image data of the right image and the preprocessed image data of the left image. An amount corresponding to the variance of the difference between the luminance value IL ′ (x, y) of the pixel GL (x, y) is shown. This amount depends on the dispersion (variation) of ε in the equation (2). In addition, the numerator on the right side of Expression (4) is the pixel G (x, y), G (x + d, y) that is spaced in the x direction in each image data after preprocessing of the left and right images. An amount corresponding to the variance of the difference (change amount) in luminance value is shown. This amount depends on the dispersion (variation) of τ in the equation (3).

Here, in order to accurately detect the parallax (parallax in the x direction in the present embodiment) from the image data after the preprocessing of the left and right images,
(A) The variation in luminance value difference between the pre-processed image data of the right image and the pre-processed image data of the left image is as small as possible. (B) The pre-processed image data of the right image The average value of the difference in luminance value with the pre-processed image data of the left image is as close to 0 as possible (C) In the left and right image data after the pre-processing, the change in the luminance value in the x direction is remarkable. It is preferable that it appears easily. In this case, the value of the denominator on the right side of Equation (4) is required to be as small as possible from the above condition (A). Further, from the condition (C), it is required that the value of the numerator on the right side of the formula (4) is as large as possible. Further, from the condition (B), at least the sum Σa _d = a _−α +... + A ₀ +... + A of the weight coefficients a _d (d = −α,..., 0,. It is required that _α is “0”.

In order to satisfy these requirements as much as possible, a set of weight coefficients a _d (pre-processing operator) in the above equation (1) is set, and pre-processing using the operator is performed on the left and right images. Thus, the S / N ratio of the pre-processed image data can be increased, and as a result, the detection accuracy of parallax can be increased.

A set of weighting factors a _d satisfying the above requirements is given by the following equation (5). At this time, the expression (1) becomes the following expression (6).

with respect to a _{d ≠ ± α a d, a} d = 0
against d = a _{± α a d, a d =} ± 1 ( Composite same order)
...... (5)

I ′ (x, y) = I (x + α, y) −I (x−α, y) (6)

The pre-processing equation (6) thus obtained is separated as much as possible in the x-direction in order to improve the detection accuracy of the parallax in the x-direction (increase the S / N ratio of the image data after the pre-processing). This shows that it is preferable to perform preprocessing such as taking a difference in luminance values of pixels on the left and right images.

  However, Expression (6) is obtained on the assumption of the probability models of Expressions (2) and (3). In particular, in the probability model of the formula (3), generally, when the absolute value of d is too large, a deviation from an actual event occurs. Therefore, in order to suitably obtain image data in which the S / N ratio is increased by the preprocessing of Expression (6), the maximum value (limit value) of d at which the probability model of Expression (3) can be approximately established is set. It is preferable to use as the value of α in formula (6).

  Therefore, in the present embodiment, the left and right sides of the left and right are expressed by the following equation (7) in which the maximum value dmax (> 0) of d that the probability model of equation (3) can approximately hold is used as the value of α in equation (6). Each image is preprocessed.

I '(x, y) = I (x + dmax, y) -I (x-dmax, y) (7)

The maximum value dmax of d in the preprocessing of the equation (7) can be determined as follows. First, according to the probability model of the equation (3), the following equation (8) is established for the variance ω ² (d ² ) of the difference τ between the luminance values I (x + d) and I (x).

ω ² (d ² ) = d · ω ² (1 ² ) (8)

That is, when the probability model of Expression (3) is established, the luminance values of two pixels G (x + d, y) and G (x, y) separated by d pixels in the x direction in the left and right images. Difference variance ω ² (d ² ) is a luminance value of two pixels G (x + 1, y) and G (x, y) (two pixels adjacent in the x direction) separated by one pixel in the x direction. Of the difference ω ² (1 ² ). Therefore, whether or not the probability model of Equation (3) is approximately established for a certain value d is whether ω ² (d ² ) and d · ω ² (1 ² ) are substantially equal. Judgment can be made by (whether or not the difference between them is substantially zero). Specifically, whether or not the probability model of the equation (3) is approximately established can be determined by whether or not the inequality of the following equation (9) is established for a certain value d. . Note that “c” in the equation (9) is a predetermined value in the vicinity of 0 (> 0).

| Ω ² (d ² ) −d · ω ² (1 ² ) | <c (9)

Also, for any value of d, the variance ω ² (d ² ) is the variance of τ in the above equation (3), so the luminance values of the right image and the left image acquired from the imaging devices 2R and 2L, respectively. Using the sampling data of I (x, y), it can be obtained by the following equation (10).

N is the number of sampling data of a set of luminance values I (x + d) and I (x). The denominator on the right side of Equation (10) means the sum of the squares of the differences between the luminance values I (x + d) and I (x) in each sampling data. Supplementally, the variance ω ² (d ² ) obtained by this equation (10) is the sample variance.

Accordingly, the value of d is provisionally set in order from “1”, ω ² (d ² ) is obtained by the equation (10), and whether or not the inequality of the equation (9) is satisfied is determined. The maximum value dmax of d for which the probability model of 3) can be approximately established can be determined in an exploratory manner. Note that when the value of d is “1”, it is obvious that the inequality of equation (9) holds, so in practice, only when the value of d is set to an integer value of 2 or more, What is necessary is just to judge whether the inequality of Formula (9) is satisfy | filled.

  Based on what has been described above, the processing of the image preprocessing unit 5 will be specifically described.

  As shown in the flowchart of FIG. 2, the image preprocessing unit 5 performs preprocessing for the left and right images.

First, the image preprocessing unit 5 uses the above formula based on the data of the luminance value I (x, y) of one of the right image and the left image acquired by the image acquisition unit 4, for example, the right image. By calculating the right side of (10), the variance ω ² (1 ² ) when the value of d in the probability model of Equation (3) is “1” is obtained (STEP 1). That is, in the right image, the difference in luminance values (I (x + 1, y) −I (x, y)) of pixels GR (x + 1, y) and GR (x, y) adjacent to each other in the x direction. Find the variance of. In this case, in this embodiment, for example, all sets of pixels GR (x + 1, y) and GR (x, y) in the entire region of the right image are set as sample targets for the calculation of Expression (10), and the sample targets The luminance values (I (x + 1, y), I (x, y)) of each set of pixels are used as sampling data for calculating the variance ω ² (1 ² ).

Next, the image preprocessing unit 5 sets “1” as the provisional value of the maximum value dmax (STEP 2), and further sets the maximum value dmax as the value of d for which the variance ω ² (d ² ) is to be calculated. Is set to a value increased by “1” from the current value (temporary value set in the immediately preceding STEP 2) (STEP 3). Then, the image preprocessing unit 5 calculates the variance ω ² (d ² ) with respect to the value of d set in STEP 3 (= dmax + 1, which is an integer value of 2 or more) (STEP 4). In this case, as in the case of STEP 1, all the sets of pixels GR (x + d, y) and GR (x, y) in the entire region of the right image are sampled for the calculation of Expression (10), The luminance values (I (x + d, y), I (x, y)) of each set of pixels to be sampled are used as sampling data for calculating the variance ω ² (d ² ).

Next, the image preprocessing unit 5 determines whether or not the inequality of the equation (9) is established for the variance ω ² (1 ² ) obtained in STEP 1 and the variance ω ² (d ² ) obtained in STEP 4. Judgment is made (STEP 5). If the determination result is affirmative, the image preprocessing unit 5 sets the current value of d (value determined by the latest processing of STEP 3) as the provisional value of the maximum value dmax (STEP 6). . In other words, the provisional value of the maximum value dmax is increased by “1” from the current value. Then, the image preprocessing unit 5 executes the processing from STEP3 again.

In this way, by repeating the processing of STEP 3 to STEP 6, the value of d when obtaining the variance ω ² (d ² ) in STEP 4 is incremented by “1” as an integer value of 2 or more. When the value of d exceeds a certain value, the determination result in STEP5 becomes negative. Thus, when the determination result in STEP 5 becomes negative, the current value of dmax (this is the value immediately before the current value of d) is finally used for the preprocessing of the equation (7). The value of dmax to be determined is determined (STEP 7). As a result, the maximum value dmax of d that approximately establishes the probability model of Equation (3) is determined exploratively based on the luminance value data of the right image.

  After determining the maximum value dmax in this way, the image preprocessing unit 5 performs the preprocessing of the equation (7) on the left and right images (STEP 8).

  The above is the details of the processing of the image preprocessing unit 5 in the present embodiment. By such pre-processing, post-processed image data having a high S / N ratio and suitable for parallax calculation is obtained. Note that the processing of STEP 1 to STEP 7 is processing constituting the preliminary processing means in the first to third inventions. STEP 8 is a process constituting the pre-processing execution means in the first to third inventions.

  FIG. 3 is a diagram schematically illustrating an image of preprocessed image data obtained by such preprocessing. Specifically, FIG. 3A shows a right image captured by the imaging device 2R. In this example, the right image includes a two-lane road 20 on which a vehicle (own vehicle) carrying the imaging devices 2R and 2L is traveling, and another vehicle 21 traveling in front of the vehicle, The forests 22 existing on both sides of the road 20 are imaged. In addition, although illustration of a left image is abbreviate | omitted, the road 20, the other vehicle 21, and the forest 22 are imaged also in the left image.

  FIG. 3B shows an image of image data obtained by performing preprocessing on the right image of FIG. As shown in the figure, the image data is an image in which the contour of the object shown in the right image appears relatively remarkably.

Supplementally, in the present embodiment, the right image is used when the variance ω ² (d ² ) (including the case where d = 1) is obtained by the equation (10), but the left image may be used. . In addition, as a sample object when obtaining the variance ω ² (d ² ), all the sets of the pixels GR (x + d, y) and GR (x, y) in the entire region of the right image are used. A plurality of sets of pixels in a partial region of the image (for example, a region excluding the peripheral region of the image) may be sampled. In this case, when the location of the object in each image is roughly recognized by binarization processing of the left and right images, a plurality of sets of pixels in the region including the recognized location May be obtained as a sample object, and the variance ω ² (d ² ) may be obtained.

  In the present embodiment, the preprocessing is performed using the single dmax value for the pixels in the entire region of each of the left and right images, but for each column of pixels arranged in the x direction of each image. Alternatively, pre-processing may be performed using the value of dmax determined separately. In this case, the dmax value for each column in the x direction is determined by executing the processing in STEP 1 to STEP 7 in FIG. 2 based on the image data in each column in the x direction of the right image or the left image. You just have to do it. Then, the preprocessing of Expression (7) may be executed for the left and right images using the determined dmax value for each column in the x direction.

  Using the preprocessed image data obtained by the image preprocessing unit 5 as described above, the processing of the parallax calculation unit 6 is performed as follows, for example.

  Either one of the pre-processed image data of each of the left and right images, for example, each pixel GR (x, y) in the pre-processed image data of the right image is centered on that pixel GR (x, y) Is set in the image of the preprocessed image data of the right image. Then, a search area of a predetermined size is set for the image of the preprocessed image data of the left image. This search area is set to an area that is wider in the x direction than the correlation determination area and includes image data equivalent to the image data in the correlation determination area.

  And, in the search area of the image of the pre-processed image data of the left image, the brightness value distribution that best matches the distribution of the brightness value in the correlation determination area of the image of the pre-processed image data of the right image A region (a local region having the same size as the correlation determination region) is searched by a known correlation calculation method such as SAD, and the pixel GL (x ′, y) of the left image corresponding to the center point of the searched region Is determined as a pixel capturing the same object as the pixel of the right image corresponding to the center point of the correlation determination area.

  Further, in this way, the x coordinate value x ′ of the pixel GL (x ′, y) of the left image determined corresponding to each pixel GR (x, y) of the right image and the pixel GR ( The difference (x′−x) [pixel] from the x coordinate value x of x, y) is calculated as the parallax Δ between the left and right images related to the object captured by the pixel GR (x, y) of the right image. The

  Furthermore, the distance z from the vehicle 1 of the object captured by the pixel GR (x, y) of the right image is calculated from the parallax Δ by the following equation (11).

z = (f × D) / (Δ × p) (11)

Note that f is the focal length of the imaging devices 2R and 2L, D is the base line length (interval of the optical axis) of the imaging devices 2R and 2L, and p is the pixel pitch (length of one pixel).

  The distance map is created by the processing of the parallax calculation unit 6 described above.

  In the process of the parallax calculation unit 6, the preprocessed image data obtained by the preprocess of the equation (7) is used, so that the parallax can be detected with high accuracy.

  In order to verify this, the inventor of the present application places another vehicle in a position 6 m away from the vehicle 1 in front of the vehicle 1 while the vehicle 1 is stationary, and obtains it by the imaging devices 2R and 2L. The parallax relating to the other vehicle was calculated from the preprocessed image data of the right image and the left image obtained by the above-described method. Furthermore, in order to compare with this, the parallax regarding the other vehicle was calculated using the right image and the left image obtained by the imaging devices 6R and 6L as they are. Then, the sum of squares of errors with respect to the parallax converted from the actual distance (the true value of the parallax) was calculated.

  As a result, the sum of squares of the parallax error when the preprocessed image data is used is 1919, and the square sum of the parallax errors when the right image and the left image are used as they are is 5688. From this, it can be seen that the preprocessing in the present embodiment is effective for increasing the accuracy of detection of parallax.

  Next, a second embodiment of the present invention will be described with reference to the flowchart of FIG. Note that the present embodiment is different from the first embodiment only in the processing of the first embodiment and the image preprocessing unit 5. Therefore, the description of the same parts as those in the first embodiment is omitted.

  In the first embodiment, an example in which the value of dmax in the preprocessing equation (7) is determined using only one of the right image and the left image is shown. The image is used to determine the value of dmax.

  FIG. 4 is a flowchart showing the processing of the image preprocessing unit 5 in the present embodiment. Referring to the figure, the same processing as STEP 1 to STEP 7 in FIG. 2 in the first embodiment is executed from STEP 11 to STEP 17. Thereby, the candidate value dmaxR of the maximum value dmax of d based on the right image is determined.

Next, from STEP18 to STEP24, the same processing as STEP11 to STEP17 is executed using the left image. The processing of STEPs 18 to 24 is different from the processing of STEP 11 to STEP 17 only in the image used for obtaining the variance ω ² (d ² ). Then, the candidate values dmaxL of the maximum value dmax of d based on the left image are determined by the processing of STEPs 18 to 24.

  Note that the processing of STEP 11 to STEP 17 and the processing of STEP 18 to 24 may be executed in reverse order, or may be executed in parallel by time-sharing processing or the like.

  Next, the image processing unit 5 finally determines the smaller of the candidate value dmaxR based on the right image and the candidate value dmaxL based on the left image as the value of dmax for preprocessing in Expression (7). (STEP 25). After that, the image pre-processing unit 5 uses the dmax value determined in this way to perform the pre-processing of Equation (7) on the left and right images (STEP 26).

  The configuration and processing other than those described above are the same as those in the first embodiment. Note that the processing in STEPs 11 to 25 is processing that constitutes preliminary processing means in the fourth to sixth inventions. STEP 26 is processing that constitutes pre-processing execution means in the fourth to sixth inventions.

  In this embodiment, since dmax is determined using both the right image and the left image, suitable preprocessing can be performed on both images. Also in this embodiment, the parallax detection accuracy can be increased as in the first embodiment.

  In each of the embodiments described above, the preprocessing is performed according to the equation (7). However, a process of multiplying the calculation result on the right side of the equation (7) by a predetermined gain or a predetermined offset value is used. The scale adjustment of the pixel value I ′ (x, y) after the pre-processing may be performed by performing the adding process or the combined process.

  In each of the above embodiments, the pre-processing is performed on the pixels G (x, y) in the entire area of the right image and the left image. Of these images, an object for which parallax is to be detected. The pre-processing may be executed only for the pixel G (x, y) in a predetermined area (partial image area) in which the image is included.

  In each of the above embodiments, the case where the imaging devices 2R and 2L are arranged in parallel in the left-right direction has been described as an example. However, these imaging devices are arranged in parallel in the up-down direction so as to detect parallax in the up-down direction. May be. In that case, the maximum d value dmax that satisfies the inequality of the above equation (9) in the y direction (vertical direction) is obtained, and the preprocessing by the following equation (12) is performed for each captured image using the dmax. Should be applied.

I '(x, y) = I (x, y + dmax) -I (x, y-dmax) (12)

In each of the above embodiments, the luminance value is used as the pixel value. However, when each captured image is a color image, the hue or saturation of each pixel may be used as the pixel value.

The block diagram which shows the system configuration | structure of the apparatus in embodiment of this invention. 3 is a flowchart showing processing of an image preprocessing unit according to the first embodiment of the present invention. FIG. 3A is a diagram illustrating an example of a captured image, and FIG. 3B is a diagram illustrating an example of an image after pre-processing of the captured image. The flowchart which shows the process of the image pre-processing part in 2nd Embodiment of this invention.

Explanation of symbols

  2R, 2L ... Imaging device, 5 ... Image preprocessing unit, STEP1 to STEP7 ... Preliminary processing means, STEP8 ... Preprocessing execution means, STEP11 to STEP25 ... Preliminary processing means, STEP26 ... Preprocessing execution means.

Claims (6)

In order to obtain the parallax of the same object existing in both the first captured image and the second captured image respectively captured by the first imaging device and the second imaging device arranged in parallel in a predetermined direction An image processing apparatus that performs preprocessing on each captured image,
For any one of the first captured image and the second captured image, each pixel value I (p) of the plurality of pixels in the one captured image and each pixel On the other hand, the stochastic distribution of the difference value from the pixel value I (p + d) of the pixel having an interval of the pixel number d in the predetermined direction has an average value of 0 and corresponds to the value of the pixel number d. The maximum value dmax of the number of pixels d that can be approximated by a normal distribution having a standard deviation is used as sampling data of pixel values of a plurality of sets of pixels spaced in the predetermined direction in the one captured image. Preliminary processing means to be determined based on;
For each pixel in a predetermined area of each captured image, a pixel value I (p + dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in a positive direction in the predetermined direction, and the predetermined value A difference from the pixel value I (p-dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in the negative direction is obtained, and the obtained difference value or the difference value is scaled. Preprocessing execution means for obtaining, as preprocessed image data used for obtaining the parallax of the same object, an image having a value obtained as a pixel value of each pixel in a predetermined area of each captured image. An image processing apparatus. In order to obtain the parallax of the same object existing in both the first captured image and the second captured image respectively captured by the first imaging device and the second imaging device arranged in parallel in a predetermined direction An image processing apparatus that performs preprocessing on each captured image,
For any one of the first captured image and the second captured image, each pixel value I (p) of the plurality of pixels in the one captured image and each pixel On the other hand, the variance of the difference from the pixel value I (p + d) of the pixels having an interval of the number of pixels d in the predetermined direction is ω ² (d ² ), and each of the plurality of pixels of the one captured image is The variance of the difference between the pixel value I (p) and the pixel value I (p + 1) of a pixel having an interval of one pixel in the predetermined direction with respect to each pixel is defined as ω ² (1 ² ). The number of pixels d such that the absolute value of the difference between the variance ω ² (d ² ) and the value obtained by multiplying the variance ω ² (1 ² ) by the number of pixels d is smaller than a predetermined value. Preliminary processing means for obtaining a maximum value dmax based on sampling data of pixel values of a plurality of sets of pixels spaced in the predetermined direction in the one captured image;
For each pixel in a predetermined area of each captured image, a pixel value I (p + dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in a positive direction in the predetermined direction, and the predetermined value A difference from the pixel value I (p-dmax) of a pixel having an interval of the number of pixels of the maximum value dmax in the negative direction is obtained, and the obtained difference value or the difference value is scaled. Preprocessing execution means for obtaining, as preprocessed image data used for obtaining the parallax of the same object, an image having a value obtained as a pixel value of each pixel in a predetermined area of each captured image. An image processing apparatus. The preliminary processing means obtains the variance ω ² (1 ² ) based on sampling data of pixel values of a plurality of sets of pixels having an interval of one pixel in the predetermined direction in the one captured image; ,
While setting the value of the number of pixels d to be sequentially increased by an integer value of 2 or more, the variance ω ² (d ² ) corresponding to each set value of the value of d is set to the one captured image. Means for obtaining based on sampling data of pixel values of a plurality of sets of pixels having an interval of the number of pixels of the set value in the predetermined direction;
The absolute value of the difference between the variance ω ² (d ² ) determined corresponding to each set value of the value d and the value obtained by multiplying the calculated variance ω ² (1 ² ) by the set value is Means for determining whether the value is smaller than the predetermined value;
3. The image processing according to claim 2, further comprising means for determining, as the maximum value dmax, a setting value immediately before a setting value of d when the determination result is negative. apparatus. In order to obtain the parallax of the same object existing in both the first captured image and the second captured image respectively captured by the first imaging device and the second imaging device arranged in parallel in a predetermined direction An image processing apparatus that performs preprocessing on each captured image,
For each of the first captured image and the second captured image, a pixel value I (p) of each of the plurality of pixels in each captured image and a pixel in the predetermined direction with respect to each pixel The stochastic distribution of the difference value from the pixel value I (p + d) of pixels having an interval of several d is a normal distribution having an average value of 0 and a standard deviation corresponding to the value of the number of pixels d. A candidate value for the maximum value dmax of the number d of pixels to be approximated is obtained for each captured image based on sampling data of pixel values of a plurality of sets of pixels that are spaced in the predetermined direction in each captured image. And a preliminary processing means for determining a smaller candidate value among the candidate values of the maximum value dmax corresponding to each obtained captured image as the maximum value dmax for preprocessing,
A pixel value I (p + dmax) of a pixel having an interval of the number of pixels of the maximum value dmax for preprocessing in a positive direction in the predetermined direction with respect to each pixel in the predetermined region of each captured image. And the pixel value I (p-dmax) of the pixel having the interval of the maximum value dmax for preprocessing in the negative direction in the predetermined direction, and the difference value obtained or the difference Preprocessing execution means for obtaining an image having a value obtained by adjusting the scale as a pixel value of each pixel in a predetermined area of each captured image as preprocessed image data used for obtaining the parallax of the same object An image processing apparatus comprising: In order to obtain the parallax of the same object existing in both the first captured image and the second captured image respectively captured by the first imaging device and the second imaging device arranged in parallel in a predetermined direction An image processing apparatus that performs preprocessing on each captured image,
For each of the first captured image and the second captured image, a pixel value I (p) of each of the plurality of pixels in each captured image and a pixel in the predetermined direction with respect to each pixel A variance of a difference from a pixel value I (p + d) of pixels having an interval of several d is ω ² (d ² ), and each pixel value I (p) of the plurality of pixels of the one captured image is When the variance of the difference between each pixel and the pixel value I (p + 1) of the pixel having an interval of one pixel in the predetermined direction is ω ² (1 ² ), the variance ω ² (d ² ) and a candidate value for the maximum value dmax of the number of pixels d such that the absolute value of the difference between the variance ω ² (1 ² ) multiplied by the number of pixels d is smaller than a predetermined value, For each captured image, each captured image is obtained based on sampling data of pixel values of a plurality of sets of pixels that are spaced apart in the predetermined direction. A pretreatment means for determining the smaller candidate values of ones of the candidate values of the maximum value dmax to respond as the maximum value dmax of pretreatment,
A pixel value I (p + dmax) of a pixel having an interval of the number of pixels of the maximum value dmax for preprocessing in a positive direction in the predetermined direction with respect to each pixel in the predetermined region of each captured image. And a pixel value I (p-dmax) of a pixel having an interval of the maximum number of dmax pixels for preprocessing in the negative direction in the predetermined direction, and the difference value obtained or Pre-processing for obtaining, as pre-processed image data, an image having a value obtained by adjusting a difference value as a pixel value of each pixel in a predetermined region of each captured image as a pixel value of the same object An image processing apparatus comprising: an execution unit. The preliminary processing unit is configured to sample the variance ω ² (1 ² ) for each captured image, and sampling data of pixel values of a plurality of sets of pixels having an interval of one pixel in the predetermined direction in each captured image. A means of obtaining based on
Wherein each captured image, the while setting the value of the number of pixels d as will by sequentially increasing by 2 or more integer values, the dispersion omega ² corresponding to each set of the values of the d (d ²⁾ A means for obtaining, based on sampling data of pixel values of a plurality of sets of pixels having an interval of the number of pixels of the set value in the predetermined direction in each captured image;
A value obtained by multiplying the variance ω ² (d ² ) obtained corresponding to each set value of the value d and the obtained variance ω ² (1 ² ) for each captured image by the set value. Means for determining whether the absolute value of the difference between and is smaller than the predetermined value;
Means for determining, for each captured image, a setting value immediately before a setting value of d when the determination result is negative as a candidate value of the maximum value dmax corresponding to each captured image; 6. The image according to claim 5, further comprising means for determining a smaller candidate value among the candidate values of the maximum value dmax determined corresponding to the captured image as the maximum value dmax for preprocessing. Processing equipment.