JP6791341B2 - Calibration method, calibration equipment, and program - Google Patents

Description

The present invention relates to calibration methods, calibration devices, measuring tools and programs.

A stereo camera that can measure the distance to the subject is used. For example, a technology for controlling an automobile by measuring the distance to a subject in front of the vehicle with a stereo camera mounted on the automobile (hereinafter referred to as "in-vehicle stereo camera") has been put into practical use. For example, the distance measured by the in-vehicle stereo camera is used for warning the driver, controlling the brake, steering, etc. for the purpose of preventing a vehicle collision and controlling the inter-vehicle distance.

In general, an in-vehicle stereo camera is often installed inside the windshield of an automobile. This is because when the in-vehicle stereo camera is installed outside the vehicle, higher durability is required in terms of waterproofing and dustproofing. The stereo camera installed inside the car captures the scenery outside the car through the windshield. In general, windshields have a complex curved shape and are distorted in shape as compared to optical components such as lenses in cameras. Therefore, the windshield causes distortion in the captured image taken through the windshield. In addition, the distortion characteristics of the captured image change depending on the installation position and orientation when the stereo camera is installed in the vehicle. In order to calibrate these distortions contained in the captured image, after installing the stereo camera in a predetermined position of the vehicle, the distortion of the captured image is calibrated by using the captured image taken through the windshield in that state. It is necessary to calculate the calibration parameters for (correction).

Generally, as a method of calculating calibration parameters for calibrating the distortion of a captured image, a method of using a chart on which a specific mark or the like for measuring a distance is written is known. In this method, the position of the mark (subject) on the captured image theoretically calculated from the relative position of the mark and the stereo camera and the position of the mark on the captured image actually captured by the stereo camera. Calculate the calibration parameters for calibrating the distortion of the captured image based on the difference. That is, the calibration parameters that determine the conversion to eliminate the difference are calculated.

In Patent Document 1, each of a pair of image data output from a pair of cameras constituting a stereo camera is converted using calibration parameters based on the deviation of the coordinates between one image data and the other image data. Discloses a device that adjusts the optical distortion and positional deviation of a stereo camera by image processing.

However, if there is an error in the relative position between the stereo camera and the chart, an error will occur in the coordinates of the subject on the captured image, which is theoretically calculated, and therefore an error will also occur in the calibration parameters for calibrating the distortion of the captured image. It ends up. In particular, there is a problem that an error is likely to occur in the relative position between the stereo camera mounted on an object such as a vehicle and the chart.

The present invention has been made in view of the above, and an object of the present invention is to provide a calibration method, a calibration device, a measuring tool, and a program capable of calculating highly accurate calibration parameters for calibrating a stereo camera.

In order to solve the above-mentioned problems and achieve the object, the present invention is a calibration method for calibrating a stereo camera, the step of capturing an image including a subject located in the shooting range of the stereo camera, and the above. A step of acquiring the distance from the subject to the stereo camera by using the distance from the subject to the intermediate measurement point between the subject and the stereo camera and the distance from the intermediate measurement point to the stereo camera. , A step of determining calibration parameters for calibrating the stereo camera based on the distance and the captured image .

According to the present invention, there is an effect that highly accurate calibration parameters for calibrating a stereo camera can be calculated.

FIG. 1 is a diagram showing an example of the relationship between the measuring tool of the first embodiment, the stereo camera, and the calibration device. FIG. 2 is a diagram showing an example of the configuration of the stereo camera of the first embodiment. FIG. 3 is a diagram for explaining the principle of distance measurement using a stereo camera. FIG. 4 is a diagram showing an example of the measuring tool of the first embodiment. FIG. 5 is a diagram showing an example of the distance measuring device of the first embodiment. FIG. 6 is a diagram showing an example of the configuration of the calibration device of the first embodiment. FIG. 7 is a diagram showing an example of the subject coordinate system of the first embodiment. FIG. 8 is a diagram showing an example of a method of determining three-dimensional coordinates indicating the position of the first camera of the first embodiment. FIG. 9 is a diagram showing an example of the camera coordinate system of the first camera of the first embodiment. FIG. 10 is an overall schematic flowchart of the calibration method according to the first embodiment. FIG. 11 is a flowchart showing an example of a calibration method in the calibration device of the first embodiment. FIG. 12 is a flowchart showing an example of the overall flow of the calibration method of the first embodiment. FIG. 13 is a diagram for explaining an example in which the distance measuring device of the modified example of the first embodiment measures the distance from the first camera (second camera) using the intermediate measurement point. FIG. 14 is a diagram showing an example of the relationship between the measuring tool of the second embodiment, the stereo camera, and the calibration device. FIG. 15A is a cross-sectional view showing a cross section of the angle measuring plate of the second embodiment. FIG. 15B is a front view showing the front surface of the angle measuring plate of the second embodiment. FIG. 16 is a front view of the first member of the second embodiment. FIG. 17 is a diagram for explaining a case where the holes 106, 107, and 108 formed in the second member are viewed from the position of the optical center of the first camera. FIG. 18A is a diagram showing the shape of the hole 106 when the hole 106 is viewed from the position of the optical center of the first camera. FIG. 18B is a diagram showing the shape of the hole 107 when the hole 107 is viewed from the position of the optical center of the first camera. FIG. 18C is a diagram showing the shape of the hole 108 when the hole 108 is viewed from the position of the optical center of the first camera. FIG. 19A is a diagram showing an image of an angle measuring plate when optical blur is not included. FIG. 19B is a diagram showing an image of an angle measuring plate when optical blur is included. FIG. 20 is a diagram for explaining the magnitude of the radius of the foot of the mountain having the brightness of FIG. 19B. FIG. 21 is a diagram for explaining the relationship between the position of the luminance peak on the imaging surface and the inclination of the angle measuring plate. FIG. 22 is a diagram for explaining a method of determining an equation of a plane indicating the position of the measuring tool. FIG. 23 is a diagram showing an example of the configuration of the calibration device of the second embodiment. FIG. 24 is an overall schematic flowchart of the calibration method according to the second embodiment. FIG. 25 is a flowchart showing an example of a calibration method in the calibration device of the second embodiment. FIG. 26 is a cross-sectional view showing a cross section of the angle measuring plate of the third embodiment. FIG. 27 is a diagram for explaining the angle of refraction of light. FIG. 28 is a diagram for explaining the light emitted from the angle measuring plate of the third embodiment. FIG. 29 is a diagram for explaining the relationship between the inclination of the measuring tool and the deviation of the position of the luminance peak. FIG. 30 is a diagram for explaining the relationship between the moire period and the moving range of the luminance peak. FIG. 31 is a diagram for explaining the positions of the luminance peaks of the adjacent holes. FIG. 32 is a diagram showing an example of the hardware configuration of the stereo camera and the calibration device of the first to third embodiments.

The calibration method, the calibration device, the measuring tool, and the embodiment of the program will be described in detail with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a diagram showing an example of the relationship between the measuring tool 20 of the first embodiment, the stereo camera 10, and the calibration device 30. FIG. 1 is an example of calibrating a photographed image taken by a stereo camera 10 (vehicle-mounted stereo camera) mounted inside the windshield of an automobile. The measuring tool 20 is installed so as to be within the shooting range of the stereo camera 10. For example, the measuring tool 20 is installed at a distance of about 2 m from the stereo camera 10 so as to be substantially facing each other. The measuring tool 20 is used to acquire measurement data for determining calibration parameters for calibrating the stereo camera 10. The measurement data is input to a computer as the calibration device 30, and the calibration parameters are determined by the computer. First, the stereo camera 10 to be calibrated will be described.

FIG. 2 is a diagram showing an example of the configuration of the stereo camera 10 of the first embodiment. The stereo camera 10 of the present embodiment includes a first camera 1, a second camera 2, a storage unit 3, an external I / F 4, a correction unit 5, and a calculation unit 6. The first camera 1 captures a subject and acquires a first captured image. The second camera 2 captures a subject and acquires a second captured image. The first camera 1 and the second camera 2 are arranged in parallel so that the optical axes are parallel. The shooting timings of the first camera 1 and the second camera 2 are synchronized, and the same subject is shot at the same time.

The storage unit 3 stores the first captured image, the second captured image, and the calibration parameters. The calibration parameter is a parameter used when correcting the distortion of the first captured image and the second captured image. The calibration parameters are determined by the calibration method of this embodiment. The external I / F4 is an interface for inputting / outputting data of the storage unit 3. The calibration parameters used in the stereo camera 10 are determined by the calibration method of the present embodiment and are stored in the storage unit 3 using the external I / F4.

The correction unit 5 reads out the first captured image, the second captured image, and the calibration parameters from the storage unit 3. The correction unit 5 corrects the first captured image and the second captured image by an image correction formula according to the calibration parameter. The image correction formula is a formula for correcting the first captured image (second captured image) by converting the coordinates of the first captured image (second captured image). For example, when the coordinates of the first captured image (second captured image) are corrected by the affine transformation, the image correction formula can be expressed by a matrix, so that the calibration parameter is a component of the matrix. When the coordinates of the first captured image (second captured image) are corrected by a non-linear transformation, the calibration parameter is a coefficient such as a polynomial representing the transformation. The correction unit 5 may correct either the first captured image or the second captured image. That is, the image correction formula may be an image correction formula for correcting the other shot image based on one of the shot images. The correction unit 5 inputs the corrected first captured image and the corrected second captured image to the calculation unit 6.

The calculation unit 6 calculates the parallax for each subject from the corrected first captured image and the corrected second captured image. Here, parallax and the principle of distance measurement using parallax will be described.

FIG. 3 is a diagram for explaining the principle of measuring the distance using the stereo camera 10. In the example of FIG. 3, the first camera 1 (focal length f, optical center O ₀ , imaging surface S ₀ ) is arranged with the Z axis as the optical axis direction. Further, the second camera 2 (focal length f, optical center O ₁ , imaging surface S ₁ ) is arranged with the Z axis as the optical axis direction. The first camera 1 and the second camera 2 are arranged at positions parallel to the X-axis and separated by a distance B (baseline length). Hereinafter, the coordinate system of FIG. 3 is referred to as a “camera coordinate system”. Further, the coordinate system based on the optical center of the first camera 1 is referred to as the "first camera coordinate system". The coordinate system based on the optical center of the second camera 2 is called the "second camera coordinate system".

The subject A located at a position separated from the optical center O ₀ of the first camera 1 by a distance d in the optical axis direction forms an image at P ₀ , which is the intersection of the straight line A O ₀ and the imaging surface S ₀ . On the other hand, the second camera 2, the same subject A is forms an image at a position _{P 1} on the imaging surface _{S 1.}

Here, the intersection of a straight line passing through the optical center O ₁ of the second camera 2 and parallel to the straight line AO ₀ and the imaging surface S ₁ is defined as P ₀ '. Also the distance _{P 1} and D and _{P 0} '. The distance D represents the amount of positional deviation (parallax) on the images of the same subject taken by two cameras. The triangle _A-O 0 -O ₁ and a triangle _{O 1} -P ₀ '-P 1 are similar. Therefore, the following equation (1) holds.

That is, the distance d to the subject A can be obtained from the baseline length B, the focal length f, and the parallax D. When the first camera 1 and the second camera 2 are accurately arranged, the distance d calculated in the first camera coordinate system (the distance between the optical center O ₀ of the first camera 1 and the subject A in the optical axis direction). ) And the distance d (distance in the optical axis direction between the optical center O ₁ of the second camera 2 and the subject A) calculated in the second camera coordinate system.

The above is the principle of distance measurement by the stereo camera 10. In order to accurately obtain the distance d to the subject A, the first camera 1 and the second camera 2 must be accurately arranged. However, the first camera 1 (second camera 2) may be displaced in the direction of rotation about the X-axis, Y-axis, or Z-axis. As a result, the coordinates of the first captured image (second captured image) are shifted substantially vertically and horizontally. Further, in the case of an in-vehicle stereo camera that shoots a subject through the windshield, distortion of the first shot image (second shot image) also occurs due to the influence of the windshield. The stereo camera 10 corrects the misalignment error caused by the deviation of the first captured image (second captured image) due to the assembly tolerance of the two cameras and the distortion of the first captured image (second captured image) due to the windshield. The first captured image (second captured image) is corrected by signal processing using the calibration parameters.

Returning to FIG. 2, the calculation unit 6 generates a parallax image representing the parallax for each pixel based on the density value of the pixels of the captured image (first captured image or second captured image) used as a reference when calculating the parallax. .. Further, the calculation unit 6 calculates the distance to the subject by using the parallax image and the equation (1).

Next, the measuring tool 20 will be described. FIG. 4 is a diagram showing an example of the measuring tool 20 of the first embodiment. The measuring tool 20 of the present embodiment has a quadrangular plate-like structure. The shape and material of the measuring tool 20 may be arbitrary. That is, the measuring tool 20 may be any member having a surface for acquiring data used for calibration. The surface of the measuring tool 20 has five marks 21. The mark 21 is used as a measurement chart for calculating parallax. The shape, number, and position of the marks 21 are not limited to the aspects of the present embodiment and may be arbitrary. Further, the surface of the measuring tool 20 has a shading pattern for facilitating detection of the corresponding points on the second captured image corresponding to the points on the first captured image. Further, a distance measuring device 22a, a distance measuring device 22b, a distance measuring device 22c, and a distance measuring device 22d are provided at four corners of the measuring tool 20 on the quadrangular plate, respectively. Hereinafter, when the distance measuring device 22a, the distance measuring device 22b, the distance measuring device 22c, and the distance measuring device 22d are not distinguished, they are simply referred to as the distance measuring device 22.

FIG. 5 is a diagram showing an example of the distance measuring device 22 of the first embodiment. The distance measuring device 22 has a biaxial rotation holding mechanism that can rotate up, down, left, and right around a preset measurement point 23. The distance measuring device 22 of the present embodiment measures the distance by using the TOF (Time Of Flight) of the laser beam 24. The distance measuring method of the distance measuring device 22 may be arbitrary. For example, the distance measuring device 22 may measure the distance using ultrasonic waves.

Distance measurement device 22 optical center O ₀ of the first camera 1 _(see FIG. 3) the distance indicating the distance to the information (hereinafter referred to as "first distance information.") And, the second camera 2 the optical center O _{1 (FIG.} The distance information (hereinafter referred to as "second distance information") indicating the distance to (see 3) is acquired. The reason why the distance measuring devices 22 are provided at the four corners of the measuring tool 20 in FIG. 4 is that the measuring points 23 in FIG. 5 are separated from each other as much as possible. As a result, a plurality of first distance information (second distance information) having different values as much as possible can be acquired, and the calibration device 30 described later is placed between the first camera 1 (second camera 2) and the measuring tool 20. It is possible to improve the accuracy when calculating the distance (distance in the optical axis direction of the first camera 1 or distance in the optical axis direction of the second camera 2). The number and position of the distance measuring devices 22 are not limited to the embodiment of the present embodiment and may be arbitrary.

FIG. 6 is a diagram showing an example of the configuration of the calibration device 30 of the first embodiment. The calibration device 30 of the present embodiment includes a reception unit 31, a first camera position calculation unit 32, a first camera orientation calculation unit 33, a second camera position calculation unit 34, a second camera orientation calculation unit 35, and a distance calculation unit 36. It includes an ideal parallax calculation unit 37, a parallax calculation unit 38, and a determination unit 39. The calibration device 30 is an information processing device (computer).

The reception unit 31 measures a plurality of (four in the present embodiment) first distance information, a plurality of (four in the present embodiment) second distance information, a first captured image including the measuring tool 20 as a subject, and measurement. The second captured image including the tool 20 as a subject, the three-dimensional coordinate information of a plurality of (five in this embodiment) marks 21 in the subject coordinate system, and the distance measuring device 22 (measurement point 23) 3 in the subject coordinate system. Accepts dimensional coordinate information. The reception unit 31 measures, for example, a plurality of first distance information, a first photographed image, a second photographed image, three-dimensional coordinate information of a plurality of marks 21, and a distance measurement according to the operation of the user of the calibration device 30. It accepts the input indicating the three-dimensional coordinate information of the device 22. Here, the subject coordinate system will be described.

FIG. 7 is a diagram showing an example of the subject coordinate system of the first embodiment. The example of FIG. 7 is an example in which the origin of the three-dimensional coordinates is taken at the lower left end of the measuring tool 20. Four distance measuring devices 22a (measurement point 23a), distance measuring device 22b (measurement point 23b), distance measuring device 22c (measurement point 23c), and distance measuring device 22d (measurement point 23d) indicating the positions of the subject coordinate system. The three-dimensional coordinates can be obtained accurately. That is, the four three-dimensional coordinates in the subject coordinate system are known. Similarly, the three-dimensional coordinate information of a plurality of (five in this embodiment) marks 21 in the subject coordinate system is also known.

Returning to FIG. 6, the reception unit 31 inputs the three-dimensional coordinate information of a plurality of (five in this embodiment) marks 21 in the subject coordinate system to the distance calculation unit 36. The reception unit 31 inputs the first captured image and the second captured image to the parallax calculation unit 38.

Further, the reception unit 31 inputs the first distance information and the three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system to the first camera position calculation unit 32. Further, the reception unit 31 inputs the second distance information and the three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system to the second camera position calculation unit 34.

Further, the reception unit 31 inputs the first captured image and the three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system to the first camera orientation calculation unit 33. Further, the reception unit 31 inputs the second captured image and the three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system to the second camera orientation calculation unit 35.

The first camera position calculation unit 32 uses a plurality of (four in this embodiment) first distance information and three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system to obtain the first camera 1. the three-dimensional coordinates indicating the position of the optical center O ₀ (hereinafter referred to as "first camera coordinate".), it is calculated by using the object coordinate system.

FIG. 8 is a diagram showing an example of a method of determining three-dimensional coordinates indicating the position of the first camera 1 of the first embodiment. d0 indicates the first distance information acquired by the distance measuring device 22a. That is, the distance from the measurement point 23a to the first camera 1 is shown. d1 indicates the first distance information acquired by the distance measuring device 22b. That is, the distance from the measurement point 23b to the first camera 1 is shown. d2 indicates the first distance information acquired by the distance measuring device 22c. That is, the distance from the measurement point 23c to the first camera 1 is shown.

The three-dimensional coordinates indicating the position of the first camera 1 can theoretically be calculated as follows using the subject coordinate system. First, a point set 25a representing a spherical surface having a radius d0 is obtained with the measurement point 23a as the center. Next, a point set 25b representing a spherical surface having a radius d1 centered on the measurement point 23b is obtained. Next, a point set 25c representing a spherical surface having a radius d2 is obtained with the measurement point 23c as the center. Next, the point set included in both the point set 25a and the point set 25b is obtained. This point set is d0 & d1 in FIG. d0 & d1 is a point set (arc) represented by the intersection of a point set representing a spherical surface having a radius d0 and a point set representing a spherical surface having a radius d1. Next, the point set included in both the point set 25b and the point set 25c is obtained. This set of points is d1 & d2 in FIG. d1 & d2 is a point set (arc) represented by the intersection of a point set representing a sphere having a radius d1 and a point set representing a sphere having a radius d2. Finally, the first camera coordinates can be calculated by finding the intersection C of the arc represented by d0 & d1 in FIG. 8 and the arc represented by d1 & d2. That is, theoretically, the first camera coordinates can be calculated if there are three first distance information.

However, considering the measurement error of the distance measuring device 22, it is desirable to calculate the coordinates of the first camera using more measurement points 23 (four in the present embodiment). Therefore, the first camera position calculation unit 32 calculates the first camera coordinates by, for example, performing the least squares approximation according to the following equation (2) to calculate the intersection C.

Here, n is the number of measurement points 23. pi is the three-dimensional coordinate of the i-th measurement point 23. di is the distance from the i-th measurement point 23 measured by the distance measuring device 22 to the first camera 1.

Returning to FIG. 6, the first camera position calculation unit 32 inputs the first camera coordinates calculated using the subject coordinate system to the first camera orientation calculation unit 33. Further, the first camera position calculation unit 32 inputs the first camera coordinates calculated using the subject coordinate system to the distance calculation unit 36.

The first camera orientation calculation unit 33 receives the first captured image and the three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system from the reception unit 31. Further, the first camera orientation calculation unit 33 receives the first camera coordinates calculated using the subject coordinate system from the first camera position calculation unit 32.

The first camera orientation calculation unit 33 uses the camera coordinate system to obtain the three-dimensional coordinates of the measurement points 23 (23a, 23b, 23c, 23d) and the measurement points 23 (23a, 23b, 23c, 23d) on the first captured image. The orientation (optical axis direction) of the first camera 1 is calculated from the two-dimensional coordinates of the image of 23d) and the focal length of the first camera 1. Specifically, first the first camera orientation calculation unit 33, the camera coordinate system three-dimensional coordinates of the measuring point 23, which is calculated using the subject coordinate system, in which a first optical center O ₀ of the camera 1 as the origin Convert to. That is, this camera coordinate system is the coordinates with the position of the pinhole in the pinhole camera model as the origin. The camera coordinate system at this point is not yet in the orientation of the camera coordinate system due to the deviation of the orientation of the first camera. Next, the first camera orientation calculation unit 33 calculates the three-axis rotation angle r = (α1, β1, γ1) for adjusting the camera coordinate system to the orientation of the camera coordinate system due to the deviation of the orientation of the first camera. To do.

FIG. 9 is a diagram showing an example of the camera coordinate system of the first camera 1 of the first embodiment. The three-dimensional coordinates of the measurement point 23 (subject) are (x, y, z), the two-dimensional coordinates on the imaging surface 40 are (u, v), and the focal length of the first camera 1 is f. At this time, the position of the image 41 on the imaging surface 40 of the measurement point 23 can be expressed by the following equation (3).

Generally, the position of the optical center of the camera, the focal length f of the camera, the three-dimensional coordinates p = (x, y, z) of the measurement point 23, and the camera direction (three-axis rotation angle r = (α1, β1, γ1)). ), The two-dimensional coordinates (u, v) on the imaging surface 40 can be calculated using the equation (3). Α1 indicates the rotation angle with respect to the X axis, β1 indicates the rotation angle with respect to the Y axis, and γ1 indicates the rotation angle with respect to the Z axis.

On the contrary, from the position of the optical center of the camera, the focal length f of the camera, the three-dimensional coordinates p = (x, y, z) of the measurement point 23, and the two-dimensional coordinates (u, v) on the imaging surface 40, The camera direction (three-axis rotation angle r = (α1, β1, γ1)) can be specified by using the equation (3).

Using the relationship of equation (3), from the three-axis rotation angle r = (α1, β1, γ1) and the three-dimensional coordinates p = (x, y, z) of the measurement point 23, 2 on the imaging surface 40 Let F be the function that calculates the dimensional coordinates (u, v) ((u, v) = F (r, p)).

The first camera orientation calculation unit 33 calculates the three-axis rotation angle r = (α1, β1, γ1) by performing the least squares approximation according to the following equation (4).

Here, n is the number of measurement points 23. pi is the three-dimensional coordinate of the i-th measurement point 23. (Ui, vi) are two-dimensional coordinates on the imaging surface 40 corresponding to the i-th measurement point 23.

Since the camera direction (three-axis rotation angle r) is three variables, the camera direction can be determined from the equation (3) if there are two-dimensional coordinates of two points on the imaging surface 40 as a binding condition. The reason why the first camera orientation calculation unit 33 calculates the three-axis rotation angle r = (α1, β1, γ1) by the equation (4) is that the first captured image is captured through the front glass. That is, since the first captured image contains distortion due to the windshield, the triaxial rotation angle r = (α1, β1, γ1) is approximated to the least squares by the equation (4) using a large number of measurement points 23. It is desirable to calculate.

If the distortion differs depending on (u, v), for example, if it is assumed in advance that the distortion in the peripheral portion of the image is larger than that in the center of the screen, the measurement point 23 is reflected in the central portion of the first captured image accordingly. The position of the measurement point may be arranged as follows. Further, in the equation (4), a weight may be applied according to the measurement point 23.

Further, although the measurement point 23 of the distance measuring device 22 is used as the measurement point 23, any measurement point 23 may be used as long as the coordinates are known in the subject coordinate system. For example, an arbitrary measurement point 23 suitable for measurement on the measuring tool 20 or an arbitrary measurement point 23 suitable for measurement not on the measuring tool 20 may be used.

Returning to FIG. 6, the first camera orientation calculation unit 33 transfers the direction (3-axis rotation angle r = (α1, β1, γ1)) of the first camera 1 calculated by the above equation (4) to the distance calculation unit 36. input.

The second camera position calculating unit 34 from the second distance information of a plurality (four in this embodiment), that three-dimensional coordinates indicating the position of the optical center O ₁ of the second camera 2 (hereinafter "second camera coordinates". ) Is calculated using the subject coordinate system. Since the method of calculating the second camera coordinates is the same as the method of calculating the first camera coordinates, detailed description thereof will be omitted. The second camera position calculation unit 34 inputs the second camera coordinates calculated using the subject coordinate system to the second camera orientation calculation unit 35. Further, the second camera position calculation unit 34 inputs the second camera coordinates calculated using the subject coordinate system to the parallax calculation unit 38.

The second camera orientation calculation unit 35 receives the second captured image and the three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system from the reception unit 31. Further, the second camera orientation calculation unit 35 receives the second camera coordinates calculated using the subject coordinate system from the second camera position calculation unit 34.

The second camera orientation calculation unit 35 uses the camera coordinate system to obtain the three-dimensional coordinates of the measurement points 23 (23a, 23b, 23c, 23d) and the measurement points 23 (23a, 23b, 23c, 23d) on the second captured image. The orientation (optical axis direction) of the second camera 2 is calculated from the two-dimensional coordinates of the image of 23d) and the focal length of the second camera 2. Since the method of calculating the orientation of the second camera 2 is the same as the method of calculating the orientation of the first camera 1, detailed description thereof will be omitted. The second camera orientation calculation unit 35 inputs the direction of the second camera 2 (three-axis rotation angle r = (α2, β2, γ2)) to the distance calculation unit 36.

The distance calculation unit 36 receives the three-dimensional coordinate information of the plurality of marks 21 in the subject coordinate system from the reception unit 31. Further, the distance calculation unit 36 receives the first camera coordinates calculated using the subject coordinate system from the first camera position calculation unit 32. Further, the distance calculation unit 36 receives the direction of the first camera 1 (three-axis rotation angle r = (α1, β1, γ1)) from the first camera orientation calculation unit 33. Further, the distance calculation unit 36 receives the second camera coordinates calculated using the subject coordinate system from the second camera position calculation unit 34. Further, the distance calculation unit 36 receives the direction of the second camera 2 (three-axis rotation angle r = (α2, β2, γ2)) from the second camera orientation calculation unit 35.

Distance calculating unit 36 for each of a plurality of marks 21, using the first camera coordinate system based on the first camera coordinates and the first camera 1 orientation, the optical center O ₀ of the first camera 1 and the plurality of marks The distance d in the optical axis direction of the first camera 1 from 21 is calculated.

Specifically, first, the distance calculation unit 36 rotates the camera coordinate system with the first camera coordinates as the origin by the direction of the first camera 1 (three-axis rotation angle r = (α1, β1, γ1)). Convert to the first camera coordinate system. That is, the first camera coordinate system has the first camera coordinates as the origin, the optical axis direction of the first camera 1 as the Z axis, and the straight line passing through the horizontal origin of the plane perpendicular to the Z axis including the origin as the X axis. It is a coordinate system with a straight line passing through the origin in the vertical direction of a plane perpendicular to the Z axis including the origin as the Y axis. Next, the distance calculation unit 36 uses the first camera coordinate system and the equation (1) to determine the distance d in the optical axis direction of the first camera 1 between the optical center O ₀ of the first camera 1 and the plurality of marks 21. Calculate (see FIG. 3). As described above, the relative position between the subject (plurality of marks 21) and the stereo camera 10 (optical center O ₀ of the first camera 1) becomes clear. The distance calculation unit 36 inputs distance information indicating the distance d between each of the plurality of marks 21 and the stereo camera 10 into the ideal parallax calculation unit 37.

Note the distance calculation unit 36 by using the second camera coordinate system based on the second camera coordinate and the second camera 2 orientation, the second optical center O ₁ and a plurality of marks 21 of the camera 2 of the second camera 2 The distance d in the optical axis direction may be calculated.

The ideal parallax calculation unit 37 receives the above-mentioned distance information from the distance calculation unit 36. The ideal parallax calculation unit 37 obtains a plurality of ideal parallax showing the ideal parallax between the plurality of marks 21 included in the first captured image and the plurality of marks 21 included in the second captured image based on the above-mentioned distance information. Calculate using the formula (1) for each mark 21. The ideal parallax calculation unit 37 inputs the ideal parallax to the determination unit 39.

The parallax calculation unit 38 receives the first captured image and the second captured image from the reception unit 31. The parallax calculation unit 38 calculates the parallax between the plurality of marks 21 included in the first captured image and the plurality of marks 21 included in the second captured image for each of the plurality of marks 21. The parallax calculation unit 38 inputs the parallax to the determination unit 39.

The determination unit 39 receives the ideal parallax from the ideal parallax calculation unit 37, and receives the parallax from the parallax calculation unit 38. Further, the determination unit 39 receives the first photographed image and the second photographed image from the reception unit 31. The determination unit 39 determines the calibration parameters for correcting the first captured image and the second captured image based on the parallax and the ideal parallax. For example, the determination unit 39 determines a calibration parameter for correction so that the difference between the parallax and the ideal parallax becomes zero.

When the parallax is generated in the vertical direction (Y-axis direction), the determination unit 39 is a calibration parameter for correcting the parallax in the vertical direction to 0 regardless of the distance to the subject (plurality of marks). To determine. The reason for this is that it is assumed that parallax occurs only in the horizontal direction (X-axis direction).

FIG. 10 is an overall schematic flowchart of the calibration method according to the first embodiment. The stereo camera 10 acquires a captured image (step S101). Specifically, the first camera 1 acquires the first captured image, and the second camera 2 acquires the second captured image.

Next, the calibration device 30 measures the relative position between the stereo camera 10 and the subject (step S102). Specifically, the calibration device 30, a first optical center O ₀ of the camera 1 of the stereo camera 10, the relative positions of the plurality of marks 21 on the sizing device 20 measures a step S6~ step S10 described later ..

Next, the calibration device 30 determines the calibration parameters based on the relative position (step S103). Specifically, the calibration device 30 ideally bases the parallax between the plurality of marks 21 included in the first captured image and the plurality of marks 21 included in the second captured image based on the relative positions measured in step S102. The calibration parameter for correcting at least one of the first captured image and the second captured image is determined so as to match the ideal parallax indicating the parallax of.

Next, the details of the calibration method in the calibration device 30 of the present embodiment will be described with reference to the flowchart. FIG. 11 is a flowchart showing an example of a calibration method in the calibration device 30 of the first embodiment.

The reception unit 31 receives the coordinate information on the measuring tool 20 (step S1). The coordinate information includes three-dimensional coordinate information of a plurality of marks 21 (five in the present embodiment) in the subject coordinate system and a plurality of distance measuring devices 22 (measurement points 23 in the present embodiment) in the subject coordinate system. It is the three-dimensional coordinate information of. Further, the reception unit 31 receives a plurality of (four in this embodiment) first distance information (step S2). Further, the reception unit 31 receives a plurality of (four in this embodiment) second distance information (step S3). Further, the reception unit 31 receives the first captured image including the measuring tool 20 as a subject (step S4). Further, the reception unit 31 receives a second captured image including the measuring tool 20 as a subject (step S5).

Next, the first camera position calculation unit 32 uses the plurality of first distance information and the three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system to obtain the optical center O ₀ of the first camera 1. The first camera coordinates indicating the position of are calculated using the subject coordinate system (step S6).

Next, the second camera position calculation unit 34 uses the plurality of second distance information and the three-dimensional coordinate information of the distance measuring device 22 (measurement point 23) in the subject coordinate system to obtain the optical center O ₁ of the second camera 2. The second camera coordinates indicating the position of are calculated using the subject coordinate system (step S7).

Next, the first camera orientation calculation unit 33 uses the camera coordinate system to obtain two of the three-dimensional coordinates of the measurement points 23 (23a, 23b, 23c, 23d) and the image of the measurement points 23 on the first captured image. The orientation (optical axis direction) of the first camera 1 is calculated from the dimensional coordinates and the focal distance of the first camera 1 (step S8).

Next, the second camera orientation calculation unit 35 uses the camera coordinate system to obtain two of the three-dimensional coordinates of the measurement points 23 (23a, 23b, 23c, 23d) and the image of the measurement points 23 on the second captured image. The orientation (optical axis direction) of the second camera 2 is calculated from the dimensional coordinates and the focal distance of the second camera 2 (step S9).

Then, the distance calculating unit 36 for each of a plurality of marks 21, using the first camera coordinate system based on the first camera coordinates and the first camera 1 orientation, the first optical center O ₀ of the camera 1 The distance d in the optical axis direction of the first camera 1 with the plurality of marks 21 is calculated (step S10). From step S6 to step S10, the relative position between the subject (plurality of marks 21) and the stereo camera 10 (optical center O ₀ of the first camera 1) becomes clear.

Next, the ideal parallax calculation unit 37 includes a plurality of marks 21 included in the first captured image and a plurality of marks 21 included in the second captured image based on the distance d in the optical axis direction of the first camera 1 calculated in step S10. The ideal parallax indicating the ideal parallax with the mark 21 of is calculated for each of the plurality of marks 21 using the equation (1) (step S11).

Next, the parallax calculation unit 38 calculates the parallax between the plurality of marks 21 included in the first captured image and the plurality of marks 21 included in the second captured image for each of the plurality of marks 21 (step S12).

Next, the determination unit 39 determines the calibration parameters for correcting the first captured image and the second captured image so that the difference between the parallax and the ideal parallax becomes 0 (step S13).

Note in step S10 described above, the distance calculation unit 36 uses the second camera coordinate system based on the second camera coordinate and the second camera 2 orientation, the second camera 2 the optical center O ₁ and a plurality of marks 21 The distance d in the optical axis direction of the second camera 2 may be calculated.

Next, the overall flow of the calibration method of the present embodiment using the above-mentioned measuring tool 20 and the above-mentioned calibration device 30 will be described. FIG. 12 is a flowchart showing an example of the overall flow of the calibration method of the first embodiment.

First, the measuring tool 20 is installed in front of the vehicle equipped with the stereo camera 10 so as to be substantially facing each other (step S21). Next, the measuring tool 20 is, the optical center _{O 0} of the stereo camera 10, measuring the distance between the plurality of measurement points 23 on the sizing device 20 (step S22). Specifically, by rotating the sizing device 20 the four corners of the distance measuring device 22 as appropriate, measures the distance to the optical center O ₀ of the stereo camera 10 to windshield of the vehicle. Next, the stereo camera 10 takes a picture of the measuring tool 20 through the windshield while keeping the position of the measuring tool 20 as it is (step S23).

Next, the measurement data measured in step S22 is input to the calibration device 30 (step S24). At this time, the coordinates of the subject coordinate system indicating the plurality of marks 21 on the measuring tool 20 and the coordinates of the subject coordinate system indicating the plurality of measuring points 23 on the measuring tool 20 are also input at the same time. Next, the captured image captured in step S23 is input to the calibration device 30 (step S25).

Next, the calibration device 30 calculates a sizing device 20, the optical center _{O 0} of the stereo camera 10, the relative position (step S26). Specifically, the calibration device 30 calculates the position and direction of the stereo camera 10 (steps S6 to S9 in FIG. 11). Then, the calibration device 30 calculates the distance d in the optical axis direction of the first camera 1 between the measuring tool 20 (plural marks 21) and the optical center O ₀ of the stereo camera 10 (step S10 in FIG. 11). ..

Next, the calibration device 30 calculates the ideal parallax based on the relative position calculated in step S26 (step S27). Specifically, the calibration device 30 calculates the ideal parallax by the method of step S11 in FIG. Next, the calibration device 30 calculates the parallax based on the captured image input in step S25 (step S28). Specifically, the calibration device 30 calculates the parallax by the method of step S12 in FIG. Next, the calibration device 30 determines the calibration parameters for calibrating the stereo camera 10 based on the relative position and the captured image (step S29). Specifically, the calibration device 30 determines a calibration parameter for calibrating the captured image captured by the stereo camera 10 based on the ideal parallax calculated from the relative position and the parallax calculated from the captured image (). Step S13 in FIG. 11).

As described above, in the calibration method of the first embodiment, the relative position between the measuring tool 20 installed so as to be within the shooting range of the stereo camera 10 and the stereo camera 10 is measured, and the relative position and the stereo A calibration parameter for calibrating the stereo camera 10 is determined based on a captured image including the measuring tool 20 as a subject taken by the camera 10. As a result, it is possible to calculate highly accurate calibration parameters even for the stereo camera 10 mounted on a vehicle in which it is difficult to secure the installation position accuracy.

In the first embodiment, the case of calibrating the stereo camera 10 mounted on the automobile has been described. However, the calibration method of the first embodiment may be applied to the stereo camera 10 mounted on an arbitrary object not limited to a vehicle (moving body) such as an automobile. Further, the method of the present embodiment can be applied to the stereo camera 10 which is not mounted on the object if it is desired to perform the calibration with higher accuracy.

(Modified example of the first embodiment)
Next, a modified example of the calibration method of the first embodiment will be described. In the calibration method of the modified example of the first embodiment, the distance measuring device 22 does not measure the distance to the optical center of the first camera 1 (second camera 2), but instead of the first camera 1 (second camera 2). Measure the distance to the intermediate measurement point between the optical center. This is because the optical center of the camera is generally located inside the lens, and it is difficult to measure directly.

FIG. 13 is a diagram for explaining an example in which the distance measuring device 22 of the modified example of the first embodiment measures the distance from the first camera 1 (second camera 2) by using the intermediate measurement point. is there. The example of FIG. 13 is an example in which the intermediate measurement point 61 is taken in the direction perpendicular to the measuring device 20 and is taken in the vicinity of the position 62 of the first camera 1 (second camera 2). For example, the intermediate measurement point 61 is located on the windshield.

The distance measuring device 22 is distance information indicating the distance between the intermediate measuring point provided near the optical center of the first camera 1 and the measuring point 23 on the measuring tool 20 (hereinafter referred to as “first intermediate distance information”). To measure. Further, the distance measuring device 22 is referred to as distance information (hereinafter referred to as "second intermediate distance information") indicating the distance between the intermediate measuring point provided near the optical center of the second camera 2 and the measuring point 23 on the measuring tool 20. ) Is measured.

The reception unit 31 receives the first intermediate distance information and the second intermediate distance information. Further, the reception unit 31 receives the distance from the intermediate measurement point 61 to the optical center of the first camera 1 (second camera 2) as distance information separately acquired from the measured value, the design value, and the like.

The first camera position calculation unit 32 (second camera position calculation unit 34) first determines the coordinates indicating the position of the intermediate measurement point 61 by the equation (2). Next, the first camera position calculation unit 32 (second camera position calculation unit 34) has coordinates indicating the position of the intermediate measurement point 61 and from the intermediate measurement point 61 to the optical center of the first camera 1 (second camera 2). The coordinates of the optical center of the first camera 1 (second camera 2) are calculated by the distance information indicating the distance of.

The distance from the intermediate measurement point 61 to the optical center of the first camera 1 (second camera 2) may be measured independently by the camera coordinate system without using the subject coordinate system of the measuring tool 20. When the difference between the direction of the camera optical axis 63 and the direction of the measuring tool 20 and the straight line perpendicular to the measuring tool 20 is small, the position 65 on the straight line perpendicular to the measuring tool 20 is set to the first camera 1 (second camera 2). It may be regarded as the position of the optical center of. This is because the error 64 from the actual position of the first camera 1 (second camera 2) becomes negligibly small.

Further, the distance between the intermediate measuring point 61 and the measuring point 23 of the measuring tool 20 may be measured by a tape measure or the like without using the distance measuring device 22.

(Second Embodiment)
Next, the second embodiment will be described. In the second embodiment, the measuring tool used for calibrating the stereo camera 10 is different from the first embodiment. In the calibration of the stereo camera 10 of the second embodiment, either the first camera 1 or the second camera 2 is used. In the description of the second embodiment, the first camera 1 will be used, but the second camera 2 may be used.

FIG. 14 is a diagram showing an example of the relationship between the measuring tool 120 of the second embodiment, the stereo camera 10, and the calibration device 30. In the second embodiment, the measuring tool 120 is used instead of the measuring tool 20. The description of the stereo camera 10 and the calibration device 30 is the same as the description of FIG. 1, and will be omitted. The measuring tool 120 is used for measuring the relative position with respect to the stereo camera 10 like the measuring tool 20 of the first embodiment, but the form is different from the measuring tool 20 of the first embodiment. The measuring tool 120 of the second embodiment includes an angle measuring plate 101 and a first member 102. The angle measuring plate 101 is used for measuring an angle indicating a deviation of the measuring tool 120 tilted in the horizontal direction and an angle indicating a deviation of the measuring tool 120 tilted in the vertical direction. The first member 102 is used as a chart for calibrating the stereo camera 10.

The configuration of the measuring tool 120 will be described in detail with reference to FIGS. 15A to 16. FIG. 15A is a cross-sectional view showing a cross section of the angle measuring plate 101 of the second embodiment. FIG. 15B is a front view showing the front surface of the angle measuring plate 101 of the second embodiment. FIG. 16 is a front view of the first member 102 of the second embodiment.

The angle measuring plate 101 includes a light source 103 and a second member 104. The light source 103 is a planar diffusion light source having a uniform luminance distribution. That is, the light source 103 emits light of uniform intensity regardless of the position on the surface of the first member 102 (light in which the difference in light intensity depending on the position on the surface is within a predetermined range).

The second member 104 is installed so as to cover the light source 103, and emits the light of the light source 103 from the plurality of holes 105. Each hole 105 is formed at a predetermined interval in a direction perpendicular to the surface of the first member 102. In the example of FIG. 15B, circular holes 105 having a diameter b are formed so as to be aligned vertically and horizontally at a pitch a. The number, shape, and arrangement of the holes 105 are not limited to the form shown in FIG. 15B and may be arbitrary. The material of the second member 104 may be arbitrary. The material of the second member 104 is, for example, metal.

The angle measuring plate 101 (second member 104) is arranged at the center of the surface of the first member 102. Further, the first member 102 has a mark 111 on the upper portion of the angle measuring plate 101 (second member 104). The mark 111 is used as a reference point for calculating the distance between the first member 102 (measuring tool 120) and the stereo camera 10 (first camera 1) to be calibrated. Further, the surface of the first member 102 has a shading pattern for facilitating detection of the corresponding points on the second captured image corresponding to the points on the first captured image, similarly to the surface of the measuring tool 20 of the first embodiment. Has.

Figure 17 is a diagram for explaining a case viewed first camera 1 of the optical center _O hole 106 bored in the second member 104 from the position of _0, the hole 107 and the hole 108. The position of the hole 106, the line of sight from the position of the optical center O _0, since the angle between the surface of the second member 104 is perpendicular, the light source 103 behind the second member 104 is visible in the form of FIG. 18A .. The positions of the holes 107, since the line of sight from the position of the optical center _{O 0} enters obliquely into the hole 107 of the second member 104, the light source 103 behind the second member 104 is visible in the form of Figure 18B. The positions of the holes 108, since the line of sight from the position of the optical center _{O 0} does not enter into the hole 108 of the second member 104, the light source 103 behind the second member 104 can not be seen (FIG. 18C).

That when photographing the angle measurement plate 101 by the first camera 1 of sufficiently high resolution (second member 104) than the pitch a of the hole, a sight line and a surface of the second member 104 from the optical center O ₀ of the first camera 1 The area of the image of the hole near the point where is orthogonal to is large. Then Yuki missing area of the image of the hole as the distance from the place where the line of sight and the surface of the second member 104 from the first optical center O ₀ of the camera 1 are orthogonal, the image of the hole in sufficiently large distant position It disappears.

Here, the pitch a of the holes 105 of the second member 104 will be described. The pitch a of the hole 105 in FIG. 15B is set narrower than the resolution limit of the stereo camera 10 (first camera 1). For example, in the case of a (half) viewing angle of 20 degrees, a shooting distance (calibration distance) of 2 m, and a 640x480 pixel sensor, the pitch a is smaller than the pixel pitch if the pitch a is about 2 mm or less according to the following equation (5).

When the pitch a is smaller than the pixel pitch, when the angle measuring plate 101 (second member 104) is photographed by the stereo camera 10 (first camera 1), the photographed image becomes as shown in FIG. 19A. However, in reality, the resolution limit is exceeded even if the pitch a is about 2 mm or less due to the influence of pixel aperture characteristics, blurring of the imaging optical system, optical blurring such as optical LPF (Low Pass Filter) in the case of a color camera. It will be. Then, the individual holes cannot be discriminated on the captured image, and a large peak (luminance distribution) of one large brightness as shown in FIG. 19B in which the image of FIG. 19A is blurred is formed. The apex of the luminance peak in FIG. 19B corresponds to, for example, the vicinity of the hole 106 in FIG. Further, the foot of the mountain of brightness in FIG. 19B corresponds to, for example, the vicinity of the hole 108 in FIG.

FIG. 20 is a diagram for explaining the size of the radius c of the foot of the mountain having the brightness of FIG. 19B. The thickness of the second member 104 is 40 mm, and the diameter of the hole 108 is 1 mm. Further, the (half) viewing angle of the stereo camera 10 (first camera 1) is 20 degrees, and the pixel sensor is 640 x 480 pixels. At this time, the radius c of the base of the mountain of brightness is about 22 pixels from the following equation (6).

Thus, rather than the image of the individual holes 105 in FIG. 15B, by considering a pile of continuous large luminance normalizing it, the optical center O ₀ of the stereo camera 10 (first camera 1) of the holes 105 Even when the position not just above corresponds to the direction perpendicular to the surface of the angle measuring plate 101, it can be considered that the position of the luminance peak represents the direction perpendicular to the surface of the angle measuring plate 101. Thus irrespective of the position of the optical center O ₀ of the stereo camera 10 (first camera 1), the position of the luminance peak of the luminance peaks of the photographed image obtained by photographing the angle measuring plate 101 (luminance distribution), the angle measurement plate It can be regarded as representing the direction perpendicular to the plane of 101.

Random noise included in each pixel value by approximating the entire luminance peak with a function such as the Gaussian function (exp (-r ² )) and estimating the average distribution of a large number of pixel values. It is possible to reduce the influence of the above and accurately estimate the position of the brightness peak.

FIG. 21 is a diagram for explaining the relationship between the position of the luminance peak of the imaging surface 40 and the inclination of the angle measuring plate 101. The coordinates of the imaging surface 40 are expressed in pixel units with the center of the imaging surface 40 as the origin. In this case the coordinates _(i p, _{j p)} positions of the luminance peaks on the imaging plane 40 (captured image), a vertical line drawn from the optical center _{O 0} to the angle measurement plate 101 of the stereo camera 10 (first camera 1) Indicates the position of the foot. Therefore, if the angle measuring plate 101 and the stereo camera 10 (first camera 1) face each other, that is, the normal to the surface of the measuring tool 120 and the optical axis of the stereo camera 10 (first camera 1) are parallel. If so, the position of the brightness peak should be at the center (origin) of the captured image. That position of the luminance peak _(i p, _{j p)} if the long offset from the center (origin) of the photographed image, the coordinates _(i p, _{j p)} of the captured image (imaging plane 40) indicating the position of the luminance peak from , The angle indicating the horizontal deviation of the measuring tool 120 tilted from the facing direction and the angle indicating the vertical deviation can be found. That is, when the focal length of the stereo camera 10 (first camera 1) (pixel units) is is _f, it is possible to determine the direction of the normal of the angle measuring plate 101 by _{(i p, j p, f} ). In other words, the orientation of the stereo camera 10 (first camera 1) facing the angle measuring plate 101 installed at an angle from the facing direction can be known.

Next, a method of determining the equation of a plane indicating the position of the angle measuring plate 101 (measuring tool 120) will be specifically described. FIG. 22 is a diagram for explaining a method of determining an equation of a plane indicating the position of the measuring tool 120. The plane equation representing the sizing device 120 in the coordinate system with the origin of the optical center O ₀ of the stereo camera 10 (first camera 1) represented by the following formula (7).

Normal direction of angle measurement plate 101 can be expressed as as described _{(i p, j p, f} ) in FIG. 21. Accordingly, since the normal vector of this plane can be determined by _{(i p, j p, f} ), which is _{(a, b, c) =} (i p, j p, f). Next, in order to determine the variable d of the equation of a plane, the mark 111 of the measuring tool 120 (first member 102) is measured with a laser range finder or the like, and this distance is defined as d _c . Further, the coordinates indicating the position of the mark 111 on the captured image are defined as ( _ic , j _c ). If the focal length of the stereo camera 10 (first camera 1) (pixel units) is is f, the vector _{(i c, j c, f} ) the direction of the distance _d point _{c (x c, y c,} z c) mark It is a coordinate indicating the position of 111. That is, the coordinates (x _c , y _c , z _c ) on the plane indicating the position of the mark 111 can be calculated by the following equation (8).

Therefore, the variable d of the equation of a plane can be determined from the following equation (9) to the following equation (10). From the above, the equation of a plane (a, b, c, d) representing the measuring tool 120 can be determined.

From the captured image of the angle measuring plate 101, it is possible to calculate the angle indicating the deviation of the angle measuring plate 101 in the horizontal direction and the angle indicating the deviation of the angle measuring plate 101 in the vertical direction. I don't know the distance. Therefore, in the above description, the position of the mark 111 on the captured image and the distance information d _c to the mark 111 are used. As another method, a method of actually measuring the d ₁ or the like in FIG. 22 are also possible. Further, if the position accuracy of the measuring tool 120 is high (and the angle accuracy is poor) with respect to the allowable accuracy of calibration of the stereo camera 10 (first camera 1), the distance between the mark 111 and the stereo camera 10 is actually set. A fixed value may be used as the distance information without measurement.

Next, the configuration of the calibration device 30 of the second embodiment for calibrating the stereo camera 10 (first camera 1) by using the above method will be described. FIG. 23 is a diagram showing an example of the configuration of the calibration device of the second embodiment. The calibration device 30 of the second embodiment includes a reception unit 131, a measurement unit 136, an ideal parallax calculation unit 137, a parallax calculation unit 138, and a determination unit 139.

The reception unit 131 receives captured images (first captured image captured by the first camera 1 and second captured image captured by the second camera 2) including the measuring tool 120 captured by the stereo camera 10 as a subject. The reception unit 131 receives the above-mentioned distance information _{d c} (see FIG. 22). Reception unit 131 inputs the first captured image and distance information _{d c} in the measurement unit 136. Further, the reception unit 131 inputs the captured images (first captured image and second captured image) to the parallax calculation unit 138.

Measurement unit 136 receives the first captured image and distance information _{d c} from the reception unit 131. The measuring unit 136 determines the direction (normal vector) perpendicular to the surface of the measuring tool 120 (angle measuring plate 101) by the method described with reference to FIG. 21 based on the position of the maximum brightness of the first captured image. As a result, the measuring unit 136 determines the deviation of the direction of the measuring tool 120 from the facing position of the stereo camera 10 (first camera 1) (an angle indicating a deviation inclined in the horizontal direction and an angle indicating a deviation inclined in the vertical direction). measure. The equations of the plane representing the position of the sizing device 120 in the first camera coordinate system (the coordinate system with the origin of the first optical center O ₀ of the camera 1), in Figure 22 based on the normal vector and the distance information d _c Determine by the method described. The measurement unit 136 inputs information indicating the equation of the plane to the ideal parallax calculation unit 137.

The ideal parallax calculation unit 137 receives from the measurement unit 136 a plane equation representing the position of the measurement tool 120. The ideal parallax calculation unit 137 calculates the ideal parallax, which indicates the parallax when the plane represented by the equation is photographed, by the method described with reference to FIG. The ideal parallax calculation unit 137 inputs the ideal parallax to the determination unit 139.

The parallax calculation unit 138 receives the captured images (first captured image and second captured image) from the reception unit 131. The parallax calculation unit 138 calculates the parallax by detecting the corresponding points of the second captured image corresponding to the points of the first captured image by using the shading pattern of the measuring tool 120 (first member 102). The parallax calculation unit 138 inputs the parallax to the determination unit 139.

The determination unit 139 receives the ideal parallax from the ideal parallax calculation unit 137 and the parallax from the parallax calculation unit 138. Further, the determination unit 139 receives the first photographed image and the second photographed image from the reception unit 131. The determination unit 139 determines the calibration parameters for correcting the first captured image and the second captured image based on the parallax and the ideal parallax. The determination unit 139 determines the calibration parameters for correcting at least one of the first captured image and the second captured image so that the difference between the parallax and the ideal parallax becomes 0, for example.

FIG. 24 is an overall schematic flowchart of the calibration method according to the second embodiment. The stereo camera 10 acquires a captured image (step S201). Specifically, the first camera 1 acquires the first captured image, and the second camera 2 acquires the second captured image.

Next, the calibration device 30 measures the relative position between the stereo camera 10 and the subject (step S202). Specifically, the calibration device 30, a first optical center _{O 0} of the camera 1 of the stereo camera 10, the relative positions of the sizing device 120 is measured in step S32~ step S34 described later.

Next, the calibration device 30 determines the calibration parameters based on the relative position (step S203). Specifically, the calibration device 30 calculates by detecting the corresponding points of the second captured image corresponding to the points of the first captured image by using the shading pattern of the measuring tool 120 (first member 102). The calibration parameter for correcting at least one of the first captured image and the second captured image is determined so that the parallax to be measured matches the ideal parallax indicating the ideal parallax based on the relative position measured in step S202.

Next, the details of the calibration method in the calibration device 30 of the second embodiment will be described with reference to the flowchart. FIG. 25 is a flowchart showing an example of a calibration method in the calibration device 30 of the second embodiment.

The reception unit 131 receives the captured images (first captured image and second captured image) including the measuring tool 120 as a subject (step S31). The reception unit 131 receives the distance information _{d c} (step S32).

Next, the measuring unit 136 measures the deviation of the orientation of the measuring tool 120 from the facing position of the stereo camera 10 (step S33). Specifically, the measuring unit 136 sets the direction (normal vector) perpendicular to the surface of the measuring tool 120 (angle measuring plate 101) by the method described with reference to FIG. 21 based on the position of the maximum brightness of the first captured image. By deciding, the deviation of the orientation of the measuring tool 120 is measured.

Next, the measuring unit 136 sets the equation of a plane representing the position of the measuring tool 120 in the first camera coordinate system (coordinate system with the optical center O ₀ of the first camera 1 as the origin), the normal vector and the distance information d _c. Is determined by the method described with reference to FIG. 22 (step S34).

Next, the ideal parallax calculation unit 137 calculates the ideal parallax showing the parallax when the plane represented by the equation determined in step S34 is photographed by the method described in FIG. 3 (step S35). Next, the parallax calculation unit 138 calculates the parallax by detecting the corresponding points of the second captured image corresponding to the points of the first captured image by using the shading pattern of the measuring tool 120 (first member 102). (Step S36).

Next, the determination unit 139 determines a calibration parameter for correcting at least one of the first captured image and the second captured image so that the difference between the parallax and the ideal parallax becomes 0 (step S37).

Since the overall flow of the calibration method of the second embodiment using the above-mentioned measuring tool 120 and the above-mentioned calibration device 30 is the same as the description of FIG. 12 of the first embodiment, the description thereof will be omitted.

As described above, in the calibration method of the second embodiment, the deviation of the orientation of the measuring tool 120 from the facing position of the stereo camera 10 is measured based on the position of the maximum brightness of the first captured image. Further, the calibration parameters for calibrating the stereo camera 10 are determined based on the parallax calculated from the first captured image and the second captured image and the ideal parallax in consideration of the deviation of the orientation of the measuring tool 120. As a result, highly accurate calibration parameters can be easily calculated even for the stereo camera 10 mounted on the vehicle, for which it is difficult to secure the installation position accuracy. Further, according to the present embodiment, even for a stereo camera 10 that is not mounted on a vehicle, more accurate calibration can be realized in a simple manner. In the present embodiment, the hole of the angle measuring plate is a round hole, but this is not the case. For example, a square hole or the like can also be used.

(Third Embodiment)
Next, the third embodiment will be described. In the third embodiment, a case where the measuring tool 220 (angle measuring plate 101) having a different configuration of the measuring tool 120 (angle measuring plate 101) of the second embodiment is used will be described. In the description of the third embodiment, the parts different from the second embodiment will be described.

FIG. 26 is a cross-sectional view showing a cross section of the angle measuring plate 201 of the third embodiment. Since the front view of the angle measuring plate 201 is the same as that of FIG. 15B, it is omitted. The angle measuring plate 201 includes a light source 103, a light-shielding plate 202, transparent glass 203, and a light-shielding plate 204. Since the light source 103 is the same as that of the second embodiment, the description thereof will be omitted. The light-shielding plate 202, the transparent glass 203, and the light-shielding plate 204 correspond to the second member 104 of the second embodiment.

The angle measuring plate 201 of the third embodiment uses a transparent glass plate in which opaque light-shielding regions (light-shielding plate 202 and light-shielding plate 204) are arranged on both sides. The transparent glass 203 is arranged to fix the positions of the light-shielding plate 202 and the light-shielding plate 204. By filling the space between the light-shielding plate 202 and the light-shielding plate 204 with glass, it is resistant to mechanical deviation, and measurement errors due to temperature, deformation over time, etc. can be reduced. The transparent glass 203 may be any transparent object.

The angle measuring plate 201 of the third embodiment has holes on the light-shielding surfaces on both the front and back surfaces so that light incident on the angle measuring plate 201 in a direction substantially orthogonal to the angle measuring plate 201 passes through the facing holes (for example, holes 208 and 209). Then, it is the same as the angle measuring plate 101 of the first embodiment. However, since the light is refracted at the interface between the transparent glass 203 and the air, the brightness distribution of the captured image is different from that of the second embodiment.

FIG. 27 is a diagram for explaining the angle of refraction of light. As is well known (Snell's law), the direction in which transmitted light is emitted changes due to the refraction of the glass-air interface. Assuming the specific refractive index R, the relationship between the angles θ ₁ and θ ₂ in FIG. 27 is given by the following equation (11).

Therefore, if the total thickness of the light-shielding plate 202, the transparent glass 203, and the light-shielding plate 204 is the same as the second member 104 of the second embodiment and the hole diameter, the holes farther away can be seen by the refraction of the glass. (See FIG. 28). Therefore, the size of the mountain base of the brightness distribution of the captured image is larger than that of the second embodiment. However luminance peak position, nature corresponds to the normal direction of the light shielding plate 204 regardless of the position of the optical center O ₀ of the stereo camera 10 (first camera 1) is the same.

Further, unlike the second member 104 of the first embodiment, the angle measuring plate 201 of the third embodiment transmits light passing through a hole not facing the hole in the direction shown by the dotted line in FIG. 28 in addition to the front direction. For example, in the hole 207, the light of the hole 209 facing the hole 208 also passes through the hole 207 due to the influence of refraction.

Therefore, unlike the case of the second embodiment, a moire pattern in which light and darkness repeats periodically is observed in the image taken by the angle measuring plate 201. Therefore, there is a possibility that a plurality of luminance peaks exist in the shooting range. However, if the accuracy of the installation angle between the stereo camera 10 (first camera 1) and the measuring tool 220 is known in advance, the moire cycle corresponding to the movement range of the position of the luminance peak according to the range of the installation deviation is considered. Therefore, it is possible to avoid confusing the position of the brightness peak of the facing hole with the brightness peak corresponding to the adjacent hole.

FIG. 29 is a diagram for explaining the relationship between the inclination of the measuring tool 220 and the deviation of the position of the luminance peak. As shown in FIG. 29, when the installation angle of the measuring tool 220 has an angle deviation of ± X degrees or less from the facing direction, the position of the luminance peak also exists in the range of X degrees or less from the facing position (center of the screen). That is, as shown in FIG. 30, the hole spacing may be adjusted so that the luminance peaks of adjacent holes come to positions that are at least twice the predicted luminance peak misalignment. The position of the brightness peak of the adjacent hole is determined by the angle φ in FIG. 31 (to be exact, ftan (φ)), and the angle φ is the hole pitch (p) of the light-shielding surface, the glass plate thickness (d), and the glass refractive index. It has the relationship between (n) and the following equation (12).

From the formula (12), it can be seen that the glass plate thickness (d) and the hole pitch (p) may be determined so as to satisfy the following formula (13).

In the above description, the position of the adjacent brightness peak is predicted only considering the accuracy of the installation angle. However, in reality, in addition to the installation angle of the measuring tool 220, all possible installation deviations such as the installation angle of the stereo camera 10, the translational deviation of the measuring tool 220, and the translational deviation of the stereo camera 10 are taken into consideration. It is necessary to predict the existence range of the luminance peak indicating the opposite position. Then, if the glass plate thickness (d) and the hole pitch (p) are determined so that the brightness peak of the adjacent hole does not fall within the range of the predicted brightness peak position, the hole facing the stereo camera 10 can be obtained. The position of the luminance peak within the corresponding prediction range can be uniquely determined.

When producing the light-shielding plate 202 (204), techniques such as printing and photo-etching can be used as a method of forming a light-shielding region on the flat plate. In these techniques, it is generally easier to realize a hole with a small diameter and a narrow pitch as compared with the second member 104 of the second embodiment created by a method such as drilling a hole in a thick plate. The size of the peak of brightness is determined by the ratio (and refractive index) of the hole diameter to the plate thickness. For example, even if the plate thickness (total thickness of the light-shielding plate 202, the transparent glass 203, and the light-shielding plate 204) is 6 mm and the hole radius is 0.05 mm, a brightness distribution substantially similar to that in the second embodiment can be obtained.

As described above, in the measuring tool 220 of the third embodiment, even if the angle measuring plate 201 is a thinner plate that is lighter and smaller than the angle measuring plate 101 of the second embodiment, the angle of the second embodiment is used. Calibration accuracy equivalent to that of the measuring plate 101 can be realized.

If the light is simply blocked, the light-shielding plate 202 and the light-shielding plate 204 may be arranged at the same positions as the light-shielding regions on both sides of the glass without arranging the transparent glass 203.

Finally, an example of the hardware configuration of the calibration device 30 of the first to third embodiments will be described. FIG. 32 is a diagram showing an example of the hardware configuration of the stereo camera 10 and the calibration device 30 of the first to third embodiments.

The calibration device 30 of the first to third embodiments includes a control device 51, a main storage device 52, an auxiliary storage device 53, a display device 54, an input device 55, and a communication device 56. The control device 51, the main storage device 52, the auxiliary storage device 53, the display device 54, the input device 55, and the communication device 56 are connected to each other via the bus 57.

The stereo camera 10 of the first to third embodiments includes a photographing device 151, a photographing device 152, a communication device 153, a control device 154, a main storage device 155, an auxiliary storage device 156, a display device 157, and an input device 158. The photographing device 151, the photographing device 152, the communication device 153, the control device 154, the main storage device 155, the auxiliary storage device 156, the display device 157, and the input device 158 are connected to each other via the bus 159.

The control device 51 (control device 154) is a CPU. The control device 51 (control device 154) executes a program read from the auxiliary storage device 53 (auxiliary storage device 156) to the main storage device 52 (main storage device 155). The main storage device 52 (main storage device 155) is a memory such as a ROM or a RAM. The auxiliary storage device 53 (auxiliary storage device 156) is an HDD (Hard Disk Drive), a memory card, or the like. The display device 54 (display device 157) displays the state of the calibration device 30 (stereo camera 10) and the like. The input device 55 (input device 158) accepts input from the user. The communication device 56 of the calibration device 30 and the communication device 153 of the stereo camera 10 communicate with each other via a wired or wireless network.

The photographing device 151 corresponds to the first camera 1 (see FIG. 2). The photographing device 152 corresponds to the second camera 2 (see FIG. 2).

The program executed by the stereo camera 10 and the calibration device 30 of the first to third embodiments is a file in an installable format or an executable format, and is a CD-ROM, a memory card, a CD-R, or a DVD (Digital Versaille Disk). ) Etc. are stored in a computer-readable storage medium and provided as a computer program product.

Further, the program executed by the stereo camera 10 and the calibration device 30 of the first to third embodiments is stored on a computer connected to a network such as the Internet, and is provided by being downloaded via the network. You may. Further, the program executed by the stereo camera 10 and the calibration device 30 of the first to third embodiments may be provided via a network such as the Internet without being downloaded.

Further, the programs of the stereo camera 10 and the calibration device 30 of the first to third embodiments may be configured to be provided by incorporating them into a ROM or the like in advance.

The program executed by the calibration device 30 of the first embodiment includes the above-mentioned functional blocks (reception unit 31, first camera position calculation unit 32, first camera orientation calculation unit 33, second camera position calculation unit 34, first. The module configuration includes two camera orientation calculation units 35, an ideal parallax calculation unit 37, a parallax calculation unit 38, and a determination unit 39). The program executed by the calibration device 30 of the second to third embodiments includes the above-mentioned functional blocks (reception unit 131, measurement unit 136, ideal parallax calculation unit 137, parallax calculation unit 138, and determination unit 139). It has a modular structure.

Further, the program executed by the stereo camera 10 of the first to third embodiments has a modular configuration including each functional block (correction unit 5 and calculation unit 6) described above.

In the actual hardware, each of the above-mentioned functional blocks is executed by the control device 51 (control device 154) reading a program from the above-mentioned storage medium, so that each of the above-mentioned functional blocks becomes the main storage device 52 (main storage device 155). ) Loaded on. That is, each of the above functional blocks is generated on the main storage device 52 (main storage device 155).

It should be noted that a part or all of the above-mentioned functional blocks may be realized by hardware such as an IC (Integrated Circuit) without being realized by software.

1 1st camera 2 2nd camera 3 Storage unit 4 External I / F
5 Correction unit 6 Calculation unit 10 Stereo camera 20 Measuring tool 21 Mark 22 Distance measuring device 23 Measuring point 24 Laser light 30 Calibration device 31 Reception unit 32 1st camera position calculation unit 33 1st camera orientation calculation unit 34 2nd camera position calculation Unit 35 Second camera orientation calculation unit 36 Distance calculation unit 37 Ideal parallax calculation unit 38 Parallax calculation unit 39 Decision unit 51 Control device 52 Main storage device 53 Auxiliary storage device 54 Display device 55 Input device 56 Communication device 57 Bus 101 Angle measurement board 102 First member (chart)
103 Light source 104 Second member (shading plate)
120 Measuring tool 131 Reception unit 136 Measuring unit 137 Ideal parallax calculation unit 138 Parallax calculation unit 139 Determining unit 201 Angle measuring plate 202 Shading plate 203 Transparent glass 204 Shading plate 220 Measuring tool

Japanese Patent No. 41009077

Claims (9)

A calibration method for calibrating a stereo camera
A step of capturing an captured image including a subject located in the shooting range of the stereo camera, and
A step of acquiring the distance from the subject to the stereo camera by using the distance from the subject to the intermediate measurement point between the subject and the stereo camera and the distance from the intermediate measurement point to the stereo camera. When,
A step of determining calibration parameters for calibrating the stereo camera based on the distance and the captured image .
Calibration method including. The calibration method according to claim 1, wherein the measurement of the distance from the subject to the intermediate measurement point is performed using a laser beam. The calibration method according to claim 1, wherein the measurement of the distance from the subject to the intermediate measurement point is performed using a tape measure. A step of measuring the distance from the intermediate measurement point to the stereo camera is further provided.
The calibration method according to any one of claims 1 to 3. The calibration method according to any one of claims 1 to 3, wherein the distance from the intermediate measurement point to the stereo camera is acquired in advance. The stereo camera is mounted inside the moving body and
The subject is located outside the moving body,
The calibration method according to any one of claims 1 to 5. The stereo camera is mounted inside the windshield of the vehicle.
The intermediate measurement point is on the windshield
6. The calibration method according to claim 6. A calibration device that calibrates a stereo camera
An imaging unit that captures an captured image including a subject located in the imaging range of the stereo camera, and an imaging unit.
Acquisition to acquire the distance from the subject to the stereo camera by using the distance from the subject to the intermediate measurement point between the subject and the stereo camera and the distance from the intermediate measurement point to the stereo camera. Department and
A determination unit that determines calibration parameters for calibrating the stereo camera based on the distance and the captured image.
Calibration equipment including. For processors that calibrate stereo cameras
A step of capturing an captured image including a subject located in the imaging range of the stereo camera, and
A step of acquiring the distance from the subject to the stereo camera by using the distance from the subject to the intermediate measurement point between the subject and the stereo camera and the distance from the intermediate measurement point to the stereo camera. When,
A step of determining calibration parameters for calibrating the stereo camera based on the distance and the captured image.
Program to carry out.