WO2015182771A1

WO2015182771A1 - Image capturing device, image processing device, image processing method, and computer program

Info

Publication number: WO2015182771A1
Application number: PCT/JP2015/065660
Authority: WO
Inventors: 和由大塚
Original assignee: 日本電産エレシス株式会社
Priority date: 2014-05-30
Filing date: 2015-05-29
Publication date: 2015-12-03

Abstract

[Problem] To acquire parallax information straightforwardly by stereo matching. [Solution] The present invention is provided with: a first image-capturing unit (11) for receiving light of a first wavelength band and capturing an image; a second image pickup unit (12) for receiving light of a second wavelength band different from the first wavelength band and capturing an image; an image correction unit (21) for correcting differences between a first captured image captured by the first image-capturing unit (11) and a second captured image captured by the second image-capturing unit (12); a common feature quantity acquisition unit (23) for acquiring a common feature quantity for each of a first corrected image obtained from the first captured image and a second corrected image obtained from the second captured image, the corrected images having been obtained by the image correction unit (21); and a parallax information acquisition unit (24) for obtaining parallax information by stereo matching using the common feature quantity that was acquired for the first corrected image by the common feature quantity acquisition unit (23) and the common feature quantity that was acquired for the second corrected image by the common feature quantity acquisition unit (23).

Description

Imaging apparatus, image processing apparatus, image processing method, and computer program

The present invention relates to an imaging device, an image processing device, an image processing method, and a computer program.

For example, in the conventional technique described in Patent Document 1, a first imaging unit that captures an object including a visible light component and a near-infrared component and an object including a visible light component and not including a near-infrared component are captured. A second imaging unit, and measures the distance to the subject by a stereo matching method based on visible light components in the daytime, while projecting an infrared pattern on the subject with a near-infrared auxiliary light source at night, The distance to the subject is measured by a pattern light projection method based on the components.

JP 2013-156109 A

However, since the above-described conventional technique uses two types of methods (stereo matching method and pattern light projection method), the apparatus configuration becomes complicated. In addition, a near-infrared auxiliary light source for projecting an infrared pattern is required for the pattern light projection method.

The present invention has been made in view of such circumstances, and it is an object of the present invention to provide an imaging device, an image processing device, an image processing method, and a computer program capable of simply obtaining parallax information by a stereo matching method. .

According to one embodiment of the present invention, in an imaging device having a first imaging unit, a second imaging unit, and an image processing device, the first imaging unit receives light in a first wavelength range. The second captured image is received by the step of acquiring the first captured image and the second imaging unit receives light in a second wavelength range different from the first wavelength range. An image correction step for acquiring a first corrected image and a second corrected image for each by correcting a difference between the first captured image and the second captured image, and the first A common feature amount acquisition step for acquiring a first common feature amount and a second common feature amount for each based on a common feature amount of the corrected image and the second corrected image; and the first common feature Stereo matching using the second common feature quantity and the second common feature quantity. A parallax information acquiring step of acquiring the parallax information is an imaging method with.

According to an embodiment of the present invention, in the imaging method, the first common feature amount and the second common feature amount are absolute values of differential values of luminance values.

According to one embodiment of the present invention, the first common feature amount or the second common feature amount is based on an absolute value of a differential value of all of R, G, and / or at least one of the luminance values. It is the acquired imaging method.

According to an embodiment of the present invention, the first wavelength region and the second wavelength region partially overlap, and information indicating the amount of received light having the wavelength of the overlapping portion is the first common feature. And an imaging method used as the second common feature amount.

According to an embodiment of the present invention, the imaging surface of the first imaging unit and the imaging surface of the second imaging unit are on different planes, and the parallax information acquisition step includes the first common feature. A sub-step for determining a search pixel p1 of the first common feature quantity image made up of a quantity, and a second common feature quantity image made up of the second common feature quantity corresponding to the search pixel p1 according to epipolar constraints A sub-step for obtaining the search pixel p2 at, and a degree of coincidence between the first common feature amount image and the second common feature amount image with the search pixels p1 and p2 as the center, and a second step based on the degree of coincidence. A sub-step of obtaining the position of the search pixel p2 of the common feature amount image, and acquiring parallax information by the stereo matching.

According to an embodiment of the present invention, the parallax information acquisition step includes the pixel value of the first corrected image or the pixel value of the second corrected image and the parallax value of the parallax image acquired by the stereo matching. And clustering the pixel values of the first corrected image or the pixel values of the second corrected image surrounded by a set of pixel points as a result of the clustering as a region representing one object It is.

According to one embodiment of the present invention, a first imaging unit that receives light in a first wavelength range and acquires a first captured image, and a second wavelength that is different from the first wavelength range. By correcting the difference between the second imaging unit that receives the light of the region and acquires the second captured image, the first captured image, and the second captured image, the first captured image An image correction unit for obtaining a first correction image obtained from the captured image and a second correction image obtained from the second captured image; and the first correction image and the second correction image. Stereo matching is performed by using a common feature amount acquisition unit that acquires a first common feature amount and a second common feature amount for each, and the first common feature amount and the second common feature amount. It is an imaging device provided with the parallax information acquisition part which acquires parallax information.

According to an embodiment of the present invention, a first captured image captured by receiving light in the first wavelength range and light in a second wavelength range different from the first wavelength range are received. The second captured image captured in this manner is retained, and the difference between the captured images is corrected to obtain from the first corrected image obtained from the first captured image and the second captured image. A first common feature amount and a second common feature amount based on a common feature amount of the image correction unit that acquires the second corrected image and the first corrected image and the second corrected image, respectively. A common feature amount acquisition unit for acquiring feature amounts;

A parallax information acquisition unit that acquires parallax information by stereo matching using the first common feature quantity and the second common feature quantity.

According to one embodiment of the present invention, a program recorded in a non-volatile storage medium and executed by a computer, receiving light in a first wavelength range and acquiring a first captured image; Receiving light in a second wavelength range different from the first wavelength range to obtain a second captured image, and correcting a difference between the first captured image and the second captured image Thus, based on the image correction step for obtaining the first correction image and the second correction image for each, and the common feature amount of the first correction image and the second correction image, the first correction image and the second correction image are obtained. Using the common feature amount acquisition step of acquiring one common feature amount and the second common feature amount, and using the first common feature amount and the second common feature amount, the parallax information is obtained by stereo matching. A parallax information acquisition step to be acquired; A program recorded in a computer readable storage medium having.

According to an embodiment of the present invention, there is provided a program recorded in a non-volatile storage medium and executed by a computer, wherein the first common feature amount and the second common feature amount are differential values of luminance values. It is a program recorded on a computer-readable storage medium that is an absolute value.

According to an embodiment of the present invention, there is provided a program recorded in a non-volatile storage medium and executed by a computer, wherein the first common feature amount or the second common feature amount is R, G, or B. It is a program recorded on a computer-readable storage medium that is acquired based on the absolute value of the derivative value of all or at least one of the luminance values.

According to the present invention, an effect that the parallax information can be simply obtained by the stereo matching method is obtained.

1 is a block diagram illustrating a configuration of an imaging apparatus 1 according to an embodiment of the present invention. 1 is a configuration diagram illustrating a configuration of an imaging apparatus 1 according to a first embodiment of the present invention. It is a figure which shows the visual field of the imaging part which concerns on 1st Embodiment of this invention. It is a figure which shows the vehicle coordinate system which is the world coordinate system which concerns on 1st Embodiment of this invention. It is a flowchart which shows the procedure of the parameter acquisition process which concerns on 1st Embodiment of this invention. It is a figure which shows each coordinate system which concerns on 1st Embodiment of this invention. It is a chart which shows an example of a Laplacian filter coefficient concerning a 1st embodiment of the present invention. It is a flowchart of the parallax information acquisition method which concerns on 1st Embodiment of this invention. It is a flowchart of the parallax information acquisition method which concerns on 1st Embodiment of this invention. It is a conceptual diagram of the parallax search method which concerns on 1st Embodiment of this invention. It is a figure which shows the visual field FOV110 of the

visible light cameras

110 and 120, FOV120.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of an imaging apparatus 1 according to an embodiment of the present invention. In FIG. 1, the imaging device 1 includes a first imaging unit 11, a second imaging unit 12, and an image processing device 20. The image processing apparatus 20 includes an image correction unit 21, a parameter acquisition unit 22, a common feature amount acquisition unit 23, and a parallax information acquisition unit 24.

The first imaging unit 11 receives and captures light in the first wavelength range. The second imaging unit 12 receives and captures light in a second wavelength range different from the first wavelength range of the first imaging unit 11. Examples of combinations of light in the first wavelength range and light in the second wavelength range include the following light combination examples 1 and 2.
Light combination example 1: “visible light” as light in the first wavelength region and “far infrared light” as light in the second wavelength region.
Light combination example 2: “visible light” as light in the first wavelength range and “near infrared” as light in the second wavelength range.

It should be noted that light combinations other than the light combination examples 1 and 2 described above may be used. For example, you may combine arbitrary two light from visible light, near infrared rays, short wavelength infrared rays, medium wavelength infrared rays, long wavelength infrared rays, far infrared rays, and ultraviolet rays.

The first captured image captured by the first imaging unit 11 and the second captured image captured by the second imaging unit 12 are input to the image processing device 20. The image correction unit 21 corrects the difference between the first captured image and the second captured image. The parameter acquisition unit 22 acquires parameters for stereo matching. The common feature amount acquisition unit 23 acquires a common feature amount for each of the first correction image obtained from the first captured image and the second correction image obtained from the second captured image by the image correction unit 21. To do. The disparity information acquisition unit 24 acquires disparity information by stereo matching using the common feature amount acquired for the first correction image by the common feature amount acquisition unit 23 and the common feature amount acquired for the second correction image. To do.

The image processing apparatus 20 according to the present embodiment may be realized by dedicated hardware, or configured by a memory and a CPU (central processing unit) to realize the functions of the image processing apparatus 20. The function may be realized by the CPU executing the computer program.

Hereinafter, each embodiment according to the present invention will be described.

[First Embodiment]
1st Embodiment is an Example which applied the imaging device 1 shown by FIG. 1 to the vehicle. FIG. 2 is a configuration diagram showing the configuration of the imaging apparatus 1 according to the first embodiment of the present invention. In FIG. 2, the vehicle 30 is provided with a first imaging unit 11 and a second imaging unit 12. In the first embodiment, the first imaging unit 11 receives visible light and images, and the second imaging unit 12 receives far infrared light and images. The first imaging unit 11 is installed at the center upper end of a windshield (Front shield) 31 of the vehicle 30. The second imaging unit 12 is installed at a position offset to the left from the center of the front bumper 32 of the vehicle 30. In the first embodiment, the first imaging unit 11 is a reference camera, and the second imaging unit 12 is a comparison camera.

The image processing apparatus 20 is provided in a driving support unit 41 provided in the vehicle 30. The first captured image captured by the first imaging unit 11 and the second captured image captured by the second imaging unit 12 are input to the driving support unit 41. The first captured image and the second captured image input to the driving support unit 41 are input to the image processing device 20 provided in the driving support unit 41.

The vehicle 30 is provided with a CAN (Controller Area Network) 42. The driving support unit 41 and the other control units 43 a to 43 f of the vehicle 30 are connected to the CAN 42. The driving support unit 41 transmits a control signal to the brake control unit 43a and the electric power steering control unit 42b through the CAN 42, for example. Other examples of the control unit include an engine control unit and an inter-vehicle distance control unit.

The first imaging unit 11 and the second imaging unit 12 are connected to the CAN 42, and the first imaging unit 11 transmits the first captured image to the driving support unit 41 via the CAN 42, so that the second imaging is performed. The unit 12 may transmit the second captured image to the driving support unit 41 via the CAN 42. Moreover, you may comprise either the 1st imaging part 11 or the 2nd imaging part 12, and the driving assistance unit 41 as the same apparatus. For example, the first imaging unit 11 and the driving support unit 41 may be configured as an integrated device.

FIG. 3 is a diagram illustrating a field of view (FOV) of the imaging unit according to the first embodiment. In FIG. 3, the field of view FOV11 of the first imaging unit 11 and the field of view FOV12 of the second imaging unit 12 are different. Therefore, the first captured image captured by the first image capturing unit 11 and the second captured image captured by the second image capturing unit 12 are different in view angle and number of pixels. Therefore, in the first embodiment, in addition to calibration for correcting distortion caused by the optical system in each of the

imaging units

11 and 12, the difference in the angle of view and the number of pixels between the first captured image and the second captured image are determined. Perform correction processing.

Next, the operation of the image correction unit 21 will be described. The image correction unit 21 performs distortion correction. This distortion correction will be described. First, a distortion correction method in a general stereo camera will be described. In general, a stereo camera shows a target such as a chessboard pattern with respect to the left and right cameras, and the optical system distortion rate “κ ₁ , κ ₂ , κ ₃ ,..., S, f by a technique such as“ Tsai, Zhang ”. , K _x , k _y , c _x , c _y ”. However, “κ ₁ , κ ₂ , κ ₃ ,...” Is a distortion rate parameter proportional to “r ² , r ⁴ , r ⁶ ,. r is a distance from the optical axis coordinates (c _x , c _y ) to the image coordinates (u, v), and is calculated by [Equation 1]. The optical axis coordinates (c _x , c _y ) are expressed in an image coordinate system with the upper left corner of the image as the origin.

s is a skew distortion and is represented by a distortion rate “s = tan θ” due to the rotation angle θ of the optical axis. k _x and k _y are pixel sizes, and usually “k _x = k _y ” in many cases. f is a focal length. The focal length f may be expressed as a ratio with respect to the pixel size k _x .

Further, for each camera coordinate system, a pan angle, a pitch angle, and a roll angle with respect to the world coordinate system are obtained. Then, the focal length f and the pixel size k _x of the captured image of the left and right _cameras, k _y becomes common, and the affine transformation as an imaging surface and the epipolar line (epipolar line) is parallel (affine transformation) is for A parameter or a dedicated LUT (look up table) to be set in the dedicated logic (operation circuit or operation program) is obtained. Then, the parameter or the dedicated LUT is set in the dedicated logic. This is a preparation for obtaining an image suitable for stereo matching. The affine transformation is to perform linear mapping transformation such as rotation, enlargement or reduction, and shear accompanied by parallel movement.
The above is the distortion correction method in a general stereo camera.

On the other hand, in the first embodiment, the world coordinate system shown in FIG. 4 is used. FIG. 4 is a diagram illustrating a vehicle coordinate system that is a world coordinate system according to the first embodiment. In FIG. 4, a world coordinate system {[X, Y, Z], origin O ₀ } is defined for the vehicle 30. In the first embodiment, the first imaging unit 11 (reference camera) is the right camera, and the second imaging unit 12 (comparison camera) is the left camera. The first imaging is the left and right cameras. Usually, the focal length f and the pixel sizes k _x and k _y are different between the unit 11 and the second imaging unit 12. If the captured images respectively captured by the first imaging unit 11 and the second imaging unit 12 are left as they are, it is inconvenient in matching calculation and parallax calculation in stereo matching. Therefore, the focal length of the comparison image focal length f and the pixel size k _x, k _y is (the first image captured by the first imaging unit 11) reference image (utilizing the second imaging unit 12 and the second captured image) f and pixel size k _x, to be the same as k _y, set parameters or dedicated LUT set dedicated logic for affine transformation. As a result, the matching operation in stereo matching can use a template having the same number of pixels for the reference image and the comparison image, and the parallax calculation can be performed simply by counting the number of pixels.

Generally, when the focal length f is the same and the imaging surface and the epipolar line are parallel, the imaging surfaces of the left and right cameras are on the same plane. However, in the first embodiment, the epipolar lines can be made parallel by the first imaging unit 11 (reference camera) and the second imaging unit 12 (comparison camera). The surfaces cannot be coplanar. Specifically, each captured image exists in a stepped state before and after. Therefore, in the first embodiment, a method described later based on epipolar geometry is used as a stereo matching method in a state where the imaging surfaces are not the same plane.

The image correction unit 21 performs distortion correction (WARP) by performing affine transformation on the comparison image (second captured image by the second imaging unit 12) using the parameters obtained above or the dedicated LUT. . The focal length f and the pixel sizes k _x and k _y of the comparison image (second corrected image) after the distortion correction are the reference image (first captured image (first corrected image by the first imaging unit 11)). The same as the focal length f and the pixel sizes k _x , k _y . Thereby, the difference in the angle of view and the number of pixels between the first captured image captured by the first imaging unit 11 and the second captured image captured by the second imaging unit 12 is corrected. Become.

The above is the description of the operation of the image correction unit 21.

Next, the operation of the parameter acquisition unit 22 according to the first embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing a procedure of parameter acquisition processing according to the first embodiment.

(Step S11) The parameter acquisition unit 22 calculates the parameters (E ₁ , K ₁ , E ₂ , K ₂ ). Calculation of these parameters (E ₁ , K ₁ , E ₂ , K ₂ ) will be described. FIG. 6 is a diagram illustrating each coordinate system according to the first embodiment. In FIG. 6, the world coordinate system (vehicle coordinate system) {[X, Y, Z], origin O ₀ } and the reference camera (first imaging unit 11) coordinate system {[U ₁ , V ₁ , W ₁ ], Origin O ₁ (focus)}, comparative camera (second imaging unit 12) coordinate system {[U ₂ , V ₂ , W ₂ ], origin O ₂ (focus)}, and reference image (first Captured image of imaging unit 11) coordinate system {[u ₁ , v ₁ , 1], origin o ₁ (upper left of captured image)} and comparative image (captured image of second imaging unit 12) coordinate system {[u ₂ , v ₂ , 1], the origin o ₂ (upper left of the captured image)}. In FIG. 6, S ₁ is a screen (imaging surface) of the reference camera (first imaging unit 11), [c _x1 , c _y1 , 1] is an image center of the screen S ₁ , and 51 is a reference camera optical axis. S ₂ is a screen (imaging _surface) of the comparison camera (second imaging unit _{12), [c x2, c} y2, 1] is the image center of the screen _{S 2,} 52 is a comparative optical axis of the camera. 53 is an epipolar plane. e ₁ and e ₂ are epipolar points (sometimes called epipoles). (E ₁ -e ₂ ) is an epipolar line. Incidentally, in FIG. 6, the focal length f and the pixel size _k x of the screen _{S 2} in comparison cameras, _{k y} is the focal length f and the pixel size of the screen _{S 1} of the base camera _k x, equal to the adjusted and _{k y} As shown.

In FIG. 6, the position of the object 50 in the world coordinate system is P _0, and the position of the object 50 in each camera coordinate system is P _c (c is 1 (reference camera coordinate system) or 2 (comparison camera coordinate system). )). Then, the relationship between the positions P ₀ and P _c of the object 50 is expressed by [Equation 2].

In [Equation 2], R _c is a rotation matrix composed of a pan angle, a pitch angle, and a roll angle generated by attaching each camera (the first imaging unit 11 and the second imaging unit 12) to the vehicle 30. is there. T _c represents the attachment position of the pre-camera principal point with respect to the origin O ₀ of the world coordinate system.

In addition, E _c in the case of transformation as in [Equation 3] is referred to as an external camera parameter.

The relationship between the position P _c of the object 50 in the camera coordinate system, a position p _c of the object 50 in the image coordinate system on the screen S _c is represented by [Expression 4].

Here, h is the third element w _c of P _c . f is the same focal length after adjustment on the screens S ₁ and S ₂ . k _x and k _y are the same pixel sizes after adjustment on the screens S ₁ and S ₂ . c _x, _{c y} is the image center on the screen _{S c} (optical axis position).
K _c in [Expression 4] is referred to as an internal camera parameter.

As a result, the parameter matrices E ₁ and K ₁ shown in [Equation 5] are obtained for the reference camera (first imaging unit 11), and [Equation 5] is obtained for the comparative camera (second imaging unit 12). ] parameter matrix _E 2, _{K 2} shown in obtained.

(Step S12) The parameter acquisition unit 22 calculates a basic matrix (E ₀ ). The calculation of this basic matrix (E ₀ ) will be described. First, [Expression 7] is obtained by modifying [Expression 6] in which [c = 1] in [Expression 3].

Next, when [Equation 7] is substituted into the equation [c = 2] in [Equation 3], [Equation 8] is obtained.

[Equation 8] In [Equation 8], [Equation 9] is obtained when [Equation 9].

The [number 10] is an equation for coordinate transformation to the position P ₂ of the comparison camera coordinate system from a position P ₁ of the base camera coordinate system. R ₀ and T ₀ are coordinate transformation matrices, R ₀ has 3 rows and 3 columns, and T ₀ has 3 rows and 1 column.

In FIG. 6, a vector from the origin O ₁ to the position P ₁ of the reference camera coordinate system, a vector from the origin O ₂ to the position P ₂ of the comparison camera coordinate system, and a reference from the origin O ₂ of the comparison camera coordinate system. Since the vector reaching the origin O ₁ of the camera coordinate system is on the same epipolar plane 53, the inner product of the outer product becomes zero. From this, the basic matrix E ₀ of the epipolar equation is obtained as shown below.

First, among the three vectors shown in [Equation 11], the outer product of any two vectors becomes [Equation 12].

However, “T ₀ × T ₀ = 0”.

Next, taking the inner product for [Equation 12] yields [Equation 13].

Next, taking the inner product for [Equation 13] yields [Equation 14].

By this [Equation 14], a basic matrix E ₀ of the epipolar equation is obtained. E ₀ has 3 rows and 3 columns.

(Step S13) The parameter acquisition unit 22 calculates a basic matrix (F ₀ ). The calculation of the basic matrix (F ₀ ) will be described. 6, the image coordinates of the object 50 on the screen S ₁ of the base camera coordinate system and p _1, when the image coordinates of the object 50 on the screen S ₂ in comparison camera coordinate system and p _2, [Expression 15 ].

When this [Equation 15] is transformed, it becomes [Equation 16].

[Substitution 16] Substituting [Equation 16] into [Equation 13] yields [Equation 17].

From this [Equation 17], the basic matrix F ₀ is obtained as [Equation 18]. F ₀ has 3 rows and 3 columns.

With this basic matrix F ₀ , an epipolar constraint I ₂ (see FIG. 6) when searching for the position p ₂ on the comparison image with respect to the position p ₁ on the reference image can be obtained. The epipolar constraint I ₂ is obtained as a 3 × 1 matrix by [Equation 19].

The above is the description of the operation of the parameter acquisition unit 22.

Next, the operation of the common feature quantity acquisition unit 23 will be described. The common feature amount acquisition unit 23 acquires a common feature amount from each of the first correction image and the second correction image in which the difference in the angle of view and the number of pixels is corrected by the image correction unit 21. Focal length f and the pixel size k _x of the second corrected image (comparison _image), k _y is the focal length f and the pixel size k _x of the first corrected image (reference _image), to be the same as correction and k _y Yes.

In the first embodiment, the first image pickup unit 11 receives visible light to pick up an image (visible light camera), and the second image pickup unit 12 receives far infrared light to pick up an image (far infrared camera). ). In this way, the physical quantities that are meant by the luminance values of the captured images captured by receiving light in different wavelength bands are different. The luminance value of the captured image of the visible light camera is a value corresponding to the amount of visible light, and means the color and brightness of the object surface. On the other hand, the luminance value of the far-infrared camera is a value corresponding to the amount of far-infrared emitted from the object surface by black body radiation, and means the temperature of the object surface. Therefore, it is meaningless to perform stereo matching using the luminance value of each captured image as it is. Therefore, in the first embodiment, it has been conceived that information on the contour of an object is used as a common feature amount that exists in common in each captured image. Specifically, since the brightness value of each captured image varies greatly with the contour of the object, the absolute value of the differential value of the brightness value is used as the common feature amount.

Hereinafter, the common feature amount acquisition method according to the first embodiment will be described. The common feature amount acquisition unit 23 performs a Laplacian filter process that is a second order differential in the u and v directions on each of the first correction image and the second correction image. In this Laplacian filter processing, the Laplacian filter coefficient is multiplied by one-to-one with the pixel of the corrected image, and the sum of the products is calculated (convolution: convolution integration). FIG. 7 is a chart showing an example of Laplacian filter coefficients according to the first embodiment. The example of FIG. 7 is a 5 × 5 Laplacian filter coefficient.

Next, the common feature amount acquisition unit 23 calculates the absolute value of the differential value of the first corrected image after the Laplacian filter processing. This absolute value is the common feature amount of the first corrected image. The image composed of the common feature amount of the first corrected image (first common feature amount image) is used for stereo matching in the parallax information acquisition unit 24. Further, the common feature amount acquisition unit 23 calculates the absolute value of the differential value of the second corrected image after the Laplacian filter processing. This absolute value is the common feature amount of the second corrected image. The image composed of the common feature amount of the second corrected image (second common feature amount image) is used for stereo matching in the parallax information acquisition unit 24.

Note that the first captured image received by visible light and captured can be acquired as an R, G, B color image. In this case, the first corrected image corrected by the image correction unit 21 can also be acquired as an R, G, B color image. Therefore, as another embodiment, the common feature amount acquisition unit 23 first performs Laplacian filter processing on each of the first correction images acquired as the R, G, and B color images. As a result, data composed of absolute values of differential values corresponding to R, G, and B is calculated as an image composed of the first common feature amount. Based on the absolute value data of the differential values of R, G, and B, the most effective first common feature amount can be obtained when performing stereo matching. At this time, all of the absolute value data of the differential values of R, G, and B may be used, or any one or two of R, G, and B may be selected and used. As a result, a more accurate stereo matching process can be performed.

The above is the description of the operation of the common feature amount acquisition unit 23.

Next, the operation of the parallax information acquisition unit 24 will be described. The disparity information acquisition unit 24 performs stereo matching using the first common feature amount image and the second common feature amount image obtained by the common feature amount acquisition unit 23, and acquires disparity information. 8 and 9 are flowcharts of the disparity information acquisition method according to the first embodiment. Hereinafter, the parallax information acquisition method according to the first embodiment will be described with reference to FIGS. 8 and 9.

(Step S <b> 21) The image processing apparatus 20 inputs the first captured image captured by the first imaging unit 11 and the second captured image captured by the second imaging unit 12.

(Step S22) The image correction unit 21 performs distortion correction (WARP) on the first captured image and the second captured image input in Step S21. For example, the image correction unit 21 performs affine transformation using a dedicated LUT on the second captured image. LUT the dedicated, the focal length f and the pixel size k _x of the second captured _image, k _y are the affine transformation, the focal length f and the pixel size k _x of the first captured _image, the same as so as the k _y In addition, it has been prepared in advance.

(Step S23) The common feature amount acquisition unit 23 performs Laplacian filter processing on each of the first correction image and the second correction image obtained by the distortion correction in step S22.

(Step S24) The common feature amount acquisition unit 23 calculates the absolute value of the differential value of the first corrected image after the Laplacian filter processing in Step S23, and from this absolute value (common feature amount of the first corrected image). A first common feature amount image is obtained. Further, the common feature amount acquisition unit 23 calculates the absolute value of the differential value of the second corrected image after the Laplacian filter processing in step S23, and the first value composed of this absolute value (the common feature amount of the second corrected image). Two common feature amount images are obtained.

(Step S25) The parallax information acquisition unit 24 determines a search start reference pixel “p ₁ = [u ₁ v ₁ 1] ^T ” in the first common feature amount image (reference image) obtained in step S24. For example, the upper left pixel of the first common feature quantity image is set as a search start reference pixel. The search start reference pixel is the first search pixel of the first common feature amount image.

(Step S26) Epipolar constraint when the parallax information acquisition unit 24 searches for the position p ₂ on the second common feature amount image (comparison image) with respect to the search pixel p ₁ of the first common feature amount image. determine the I _2. The epipolar constraint I ₂ is obtained by the above [Equation 19]. Epipolar constraint I ₂ is the coefficient of the comparison image search line. Note that the parallax information acquisition unit 24 holds a basic matrix F ₀ calculated in advance.

(Step S27) The parallax information acquisition unit 24 sets the search start lateral position u ₂ to “u ₂ = u ₁ ” in the second common feature amount image. This search start lateral position is the first search lateral position of the second common feature amount image.

(Step S28) the disparity information acquisition unit 24, a search vertical position v ₂ corresponding to the search lateral position u ₂ at the second common feature amount image, determined according to epipolar constraints I ₂ as shown in FIG. 10. FIG. 10 is a conceptual diagram of the parallax search method according to the first embodiment.
The search vertical position v ₂ is calculated by “(c− (a × u ₁ )) / b” using elements (epipolar constraint line coefficients) a, b, and c of the epipolar constraint I ₂ .

Here, when “p ₂ = [u ₂ v ₂ 1] ^T ” calculated in step S28 is outside the range of the second common feature amount image, the next step S29 is skipped and the process proceeds to step S30. (Not shown in FIG. 8). In the second common feature image, a search position “p ₂ = [u” having an initial search horizontal position “u ₂ = u ₁ ” and a search vertical position v ₂ corresponding to the search horizontal position u _2. ₂ v ₂ 1] ^T ”is a search start position (initial pixel position).

(Step S29) The parallax information acquisition unit 24 calculates the degree of coincidence between the first common feature amount image and the second common feature amount image with the pixels of interest p ₁ and p ₂ as the center. As a method of calculating the degree of matching, for example, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross-Correlation: normal) Examples thereof include cross-correlation), ZNCC (Zero-mean Normalized Cross-Correlation), and SGM (Semi-Global Matching).

(Step S30) The parallax information acquisition unit 24 increments the search lateral position u2 of the _second common feature amount image.

(Step S31) parallax information acquiring unit 24, the search lateral position u ₂ at the second common feature value image to determine whether the reached to the end of the second common feature amount image. As a result of this determination, if the end of the second common feature amount image has been reached, the process proceeds to step S32 in FIG. 9, and if not, the process returns to step S28.

(Step S32) The parallax information acquisition unit 24 selects the pixel of interest of the second common feature quantity image that has the highest degree of coincidence among the degrees of coincidence calculated for the search pixel p1 of the _first common feature quantity image. p ₂ (maximum coincidence comparison coordinate position) is obtained.

(Step S33) The parallax information acquisition unit 24 obtains the distance from the search start position (initial pixel position) to the maximum coincidence comparison coordinate position in the second common feature amount image. Then, the parallax information acquisition unit 24 divides the obtained distance by the pixel size k _x . The disparity information acquisition unit 24 sets the quotient that is the result of the division as a disparity value for the search pixel p1 of the _first common feature amount image.

(Step S34) The parallax information acquisition unit 24 stores the parallax value obtained in step S33 in a parallax image. The storage position of the parallax value is the same position as the search pixel p1 of the _first common feature amount image.

(Step S35) The parallax information acquisition unit 24 increments the search pixel “p ₁ = [u ₁ v ₁ 1] of the first common feature amount image.

(Step S36) The parallax information acquisition unit 24 determines whether the search pixel “p ₁ = [u ₁ v ₁ 1] is within the range of the first common feature amount image. If it is within the range of the common feature quantity image, the process returns to step S26 in Fig. 8 to continue searching for the first common feature quantity image, whereas it is outside the range of the first common feature quantity image. In this case, the search for the first common feature amount image is terminated (the processes in FIGS. 8 and 9 are terminated).

A parallax image corresponding to the first common feature amount image is obtained by the processing of FIGS. From this parallax image, three-dimensional information of the subject (for example, three-dimensional distance information to the subject) can be acquired.

As described above, in the first embodiment, stereo matching is performed using the absolute value of the differential value of the luminance value as a common feature amount. For this reason, there are few parallax values contained in a parallax image compared with the case where stereo matching is performed with a luminance value. Therefore, the parallax information acquisition unit 24 may perform a process of supplementing the information of the parallax image. Specifically, the parallax information acquisition unit 24 uses the pixel value (the luminance value or hue of the image captured by the visible light camera) of the first corrected image (reference image) from which the first common feature amount image is obtained. Or the pixel value of the second corrected image (comparison image) from which the second common feature amount image is obtained (the luminance value of the image captured by the far-infrared camera and representing the surface temperature of the subject). Based on the parallax value of the parallax image obtained in the first embodiment, that is, distance information, pixels that are relatively equidistant are obtained by clustering as a pixel point set, that is, a point sequence equivalent to a contour. Next, the parallax information acquisition unit 24 uses the pixel value of the first corrected image or the luminance value of the second corrected image surrounded by the pixel point set as a result of the clustering as a region representing one object. Clustering.

The parallax image after clustering has a parallax value for pixels corresponding to the contour of the subject, and further has identification information indicating that the same object is present in the pixel areas clustered as the same cluster. Thereby, the three-dimensional distance information to the subject is obtained from the parallax value of the pixel corresponding to the contour. Furthermore, it can be determined that pixel areas having the same identification information are the same object.

[Example of acquisition method of three-dimensional distance information]
Here, an example of a method for acquiring three-dimensional distance information using the parallax image according to the first embodiment will be described. In general, when parallax values are obtained in a planar shape in a parallax image, it is known that a road surface and an object can be identified using methods such as “v-disparity” and “virtual disparity”. . However, the parallax image according to the first embodiment has a parallax value only in the contour of the subject, and the parallax value is not obtained in a planar shape. For this reason, in the parallax image according to the first embodiment, for a pixel having a parallax value, the parallax value is compared with the parallax value of a pixel connected to the pixel, and based on the comparison result, (For example, identification of a road surface and an object). For example, three-dimensional distance information is obtained from the pixel corresponding to the contour in which the parallax value is stored. From the distribution of the three-dimensional distance information, the subject is identified by determining whether a certain distribution is planar such as a road surface or is standing vertically. Then, the three-dimensional distance information of the contour of the identified subject is set as the three-dimensional distance information to the subject.

According to the first embodiment described above, by providing the visible light camera and the far-infrared camera, the parallax that can be used for calculating the three-dimensional information such as the distance measurement to the subject by the stereo matching method from the captured image of each camera day and night. Information can be obtained simply.

In the first embodiment described above, a visible light camera is used as a reference camera and a far-infrared camera is used as a comparison camera. Conversely, a far-infrared camera is used as a reference camera and a comparison camera is visible. You may comprise so that an optical camera may be used.

[Second Embodiment]
In the second embodiment, another example of the common feature amount will be described. In the second embodiment, the first wavelength range of light received by the first imaging unit 11 and the second wavelength range of light received by the second imaging unit 12 are partially overlapped. In the second embodiment, information indicating the received light amount of light having the wavelength of the overlapping portion is used as a common feature amount.

For example, the light in the first wavelength range related to the first imaging unit 11 is visible light, and the light in the second wavelength range related to the second imaging unit 12 is near-infrared light and visible light adjacent to the near-infrared light. This is the red end (end on the long wavelength side). In this case, the first wavelength region and the second wavelength region overlap the red end portion of the visible light included in the second wavelength region. From this, the red pixel in the captured image of the first imaging unit 11 and the pixel of the captured image of the second imaging unit 12 are correlated. This highly correlated pixel is used for stereo matching as a common feature.

In addition, since the 2nd imaging part 12 of this example receives a part of near infrared rays and visible light, it can be arrange | positioned next to a visible light camera (1st imaging part 11). Thereby, since the image plane of both cameras can be made the same, it is possible to obtain a parallax image by the conventional stereo matching method.

According to the second embodiment described above, an effect that the parallax information can be simply obtained by the stereo matching method is obtained.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design changes and the like within a scope not departing from the gist of the present invention.

For example, the driving support unit 41 shown in FIG. 2 performs a stereo camera recognition process using the parallax information acquired by the image processing device 20, and based on the result of the stereo camera recognition process, via the CAN 42, A control signal may be transmitted to each of the control units 43a to 43f.

For example, the driving support unit 41 acquires the three-dimensional information of the subject from the parallax information acquired by the image processing device 20. Next, the driving support unit 41 recognizes a travelable range such as a road surface, a road obstacle such as a guard rail, and an obstacle different from the road surface such as a preceding vehicle or an oncoming vehicle from the acquired three-dimensional information. . Next, the driving support unit 41 obtains a relative distance and a relative speed with each of the recognized objects. Next, the driving support unit 41 supports traveling set as safe traveling such as acceleration, following, deceleration, stop, or avoidance based on the obtained relative distance and relative speed.

Further, the driving support unit 41 accelerates the vehicle 30 in accordance with the function of running along the lane on the road (Lane Keep) and the road conditions (congestion status, position of the preceding vehicle, presence of an interrupted vehicle, etc.) You may make it have the function (Auto | Cruise | Control) which controls deceleration, and the function which supports the driving | running | working set as comfortable driving | running | working.

Further, the driving support unit 41 has a function of data linking with the navigation device, a function of calculating a traveling course on the road surface from the course information set in the navigation device and the lane information recognized by the driving support unit 41, and the calculated You may make it have a function which supports automatic driving | running | working so that it may drive | work a driving course.

In addition, as shown in FIG. 11, the case where the

visible light cameras

110 and 120 having different visual fields FOV 110 and FOV 120 are arranged on the front windshield of the vehicle 30 will be described. The visual field FOV 110 of the visible light camera 110 is wider than the visual field FOV 120 of the visible light camera 120. In this case, in the image correction by the image correction unit 21, distortion is performed so that the angle of view of the image captured by the visible light camera 110 with the wide angle becomes the same as the angle of view of the image captured with the visible light camera 120 with the narrow angle. Create an LUT to be used for correction. Since the

visible light cameras

110 and 120 are arranged side by side on the same windshield in the vehicle interior, the image planes of both cameras can be the same. Accordingly, a parallax image can be obtained by a conventional stereo matching method, and it is possible to acquire three-dimensional information of a subject from the parallax image and identify a road surface and an object.

Further, a computer program for realizing the functions of the above-described image processing apparatus 20 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed. Good. Here, the “computer system” may include an OS and hardware such as peripheral devices.

“Computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a DVD (Digital Versatile Disk), and a built-in computer system. A storage device such as a hard disk.

Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time.

The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

Further, the program may be for realizing a part of the above-described functions. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... 1st imaging part, 12 ... 2nd imaging part, 20 ... Image processing apparatus, 21 ... Image correction part, 22 ... Parameter acquisition part, 23 ... Common feature-value acquisition part, 24 ... Parallax Information acquisition unit, 30 ... vehicle, 31 ... windshield, 32 ... front bumper, 41 ... driving support unit, 42 ... CAN, 43a to 43f ... control unit

Claims

In an imaging device having a first imaging unit, a second imaging unit, and an image processing device,
Receiving light in a first wavelength range by the first imaging unit to obtain a first captured image;
Receiving light of a second wavelength range different from the first wavelength range by the second imaging unit, and acquiring a second captured image;
An image correction step of acquiring a first corrected image and a second corrected image for each by correcting a difference between the first captured image and the second captured image;
A common feature amount acquiring step for acquiring a first common feature amount and a second common feature amount for each based on a common feature amount of the first corrected image and the second corrected image;
A parallax information acquisition step of acquiring parallax information by stereo matching using the first common feature quantity and the second common feature quantity;
An imaging method comprising:
The imaging method according to claim 1, wherein the first common feature amount and the second common feature amount are absolute values of a differential value of a luminance value.
2. The imaging according to claim 1, wherein the first common feature amount or the second common feature amount is acquired based on an absolute value of a differential value of luminance values of all or at least one of R, G, and B. Method.
The first wavelength region and the second wavelength region partially overlap each other, and information indicating the amount of received light having the wavelength of the overlapping portion is used as the first common feature amount and the second common feature amount. The imaging method according to claim 1 to be used.
The imaging surface of the first imaging unit and the imaging surface of the second imaging unit are on different planes,
The parallax information acquisition step includes
A sub-step of determining a search pixel p1 of a first common feature quantity image composed of the first common feature quantity;
Sub-step for obtaining a search pixel p2 on the second common feature quantity image composed of the second common feature quantity corresponding to the search pixel p1 according to the epipolar constraint;
The degree of coincidence between the first common feature amount image and the second common feature amount image is calculated around the search pixels p1 and p2, and the position of the search pixel p2 in the second common feature amount image is calculated based on the degree of coincidence. Sub-step for
5. The imaging method according to claim 1, further comprising: acquiring parallax information by the stereo matching.
The parallax information acquisition step clusters the pixel value of the first corrected image or the pixel value of the second corrected image and the parallax value of the parallax image acquired by the stereo matching, and the clustering result Clustering pixel values of the first corrected image or pixel values of the second corrected image surrounded by a set of pixel points as a region representing one object,
The imaging method according to any one of claims 1 to 5.
A first imaging unit that receives light in the first wavelength region and acquires a first captured image;
A second imaging unit that receives light in a second wavelength range different from the first wavelength range and obtains a second captured image;
By correcting the difference between the first captured image and the second captured image, the first corrected image obtained from the first captured image and the second captured image obtained from the second captured image are displayed. An image correction unit that acquires the two corrected images;
A common feature amount acquisition unit that acquires a first common feature amount and a second common feature amount for each of the first correction image and the second correction image;
A parallax information acquisition unit that acquires parallax information by stereo matching using the first common feature quantity and the second common feature quantity;
An imaging apparatus comprising:
A first picked-up image picked up by receiving light in the first wavelength region, and a second picked-up image picked up by receiving light in a second wavelength region different from the first wavelength region; Is acquired, and the first corrected image obtained from the first captured image and the second corrected image obtained from the second captured image are acquired by correcting the difference between the captured images. An image correction unit to perform,
A common feature amount acquisition unit that acquires a first common feature amount and a second common feature amount for each based on a common feature amount of the first correction image and the second correction image;
A parallax information acquisition unit that acquires parallax information by stereo matching using the first common feature quantity and the second common feature quantity;
An image processing apparatus.
A program recorded in a non-volatile storage medium and executed by a computer,
Receiving light in a first wavelength range to obtain a first captured image;
Receiving light in a second wavelength range different from the first wavelength range to obtain a second captured image;
An image correction step of acquiring a first corrected image and a second corrected image for each by correcting a difference between the first captured image and the second captured image;
A common feature amount acquiring step for acquiring a first common feature amount and a second common feature amount for each based on a common feature amount of the first corrected image and the second corrected image;
A parallax information acquisition step of acquiring parallax information by stereo matching using the first common feature quantity and the second common feature quantity;
A program recorded on a computer-readable storage medium.
A program recorded in a non-volatile storage medium and executed by a computer,
The program recorded in the computer-readable storage medium according to claim 9, wherein the first common feature amount and the second common feature amount are absolute values of differential values of luminance values.
A program recorded in a non-volatile storage medium and executed by a computer,
The first common feature amount or the second common feature amount is acquired based on an absolute value of a differential value of all or at least one of R, G, and B,
The program recorded on the computer-readable storage medium of Claim 9.