JP6515650B2 - Calibration apparatus, distance measuring apparatus and calibration method - Google Patents

Description

  The present invention relates to a calibration device for calibrating in real time a distance measurement device using a stereo camera mounted on a mobile body such as a vehicle, a distance measurement device equipped with the calibration device, and a calibration method thereof.

  In recent years, automobiles equipped with a collision prevention device, an inter-vehicle distance control device, and the like have been marketed as devices for assisting in the improvement of automobile safety, and research and development aimed at achieving automatic driving are being carried out actively. . These devices automatically measure the distance to the car ahead etc., alert the driver when the distance between vehicles becomes short, etc., and control the brakes and steering etc. for collision avoidance when the distance between them becomes even shorter. It is a thing. That is, in these devices, automatic measurement of the inter-vehicle distance is essential, and is also an essential processing in automatic driving.

  Radars, stereo cameras, etc. are used in automatic measurement of inter-vehicle distance, but stereo cameras detect the size, position, speed, etc. of a plurality of three-dimensional objects in an instant, and further, side wall which becomes the boundary of traveling area, It has an advantage that it is possible to accurately detect road surface marks such as road shoulders and white lines.

  A stereo camera is a combination of a plurality of (usually two) cameras into one, which captures an object with a plurality of lenses, and measures the distance to the object from a plurality of captured images.

Here, measurement of distance by a stereo camera will be described. FIG. 1 is a view for explaining the principle of distance measurement by a stereo camera composed of two cameras installed in parallel and equal positions. The cameras 11 and 12 are arranged at equal distances parallel to each other by a distance B (i.e. the base length is B), and the focal lengths are the same f. When the subject A is placed at a position away from the installation surface of the cameras 11 and 12 by the distance Z in the optical axis direction, the image of the subject A is the optical center O _{1 of the} camera 11. and the P ₁ is an intersection of the straight line and the imaging surface 13 connecting the object a, respectively imaged on P ₂ is an intersection of the straight line and the imaging surface 14 connecting the camera 12 in the optical center O ₂ and the object a. Then, if a straight line parallel to the straight line AP ₂ and passing through the optical center O ₁ is drawn and the point of intersection of the straight line and the imaging surface 13 is P ₂ ′, P ₂ ′ is at the same position as P ₂ at the camera 12 The disparity between P ₁ and P ₂ ′ is parallax. Usually, parallax means a difference in direction and is often expressed by an angle, but here, the distance D between P ₁ and P ₂ ′ is called parallax. By measuring the distance D (parallax D), the distance Z to the subject A can be determined by the following equation ₁ because the triangle AO ₂ O ₁ and the triangle O ₁ P ₂ ′ P ₁ are similar.

In order for Equation 1 to hold, the two cameras must be placed exactly parallel as shown in FIG. 1 to ensure that the images do not shift. However, if it is attempted to prevent the occurrence of a gap by firmly fixing the two cameras, it costs money, so it is a realistic solution to correct and adjust the image after imaging. Adjustment by image correction is carried out before shipment of the stereo camera, but as the stereo camera is mounted on a car and used, displacement may occur due to the influence of vibration, temperature change, etc. As such, it is desirable that the stereo camera be adjusted automatically while in use.

  The deviation includes rotation deviation due to rotation (roll) around the optical axis, vertical deviation due to vertical movement (tilt or pitch), and lateral deviation due to horizontal movement (pan or yaw). Among them, when the camera is installed in parallel parallel position, rotational shift and vertical shift are easy to detect because stereo matching does not work well, but lateral shift is not detected by stereo matching because stereo matching can be performed normally. Since the deviation directly adds to the parallax, the influence on the calculated distance is large. In particular, the effect is large when measuring the distance of an object at a long distance. Therefore, it is necessary to detect the parallax error (parallax offset) due to the lateral shift and correct the parallax.

  As for the detection of the parallax offset, the parallax for an object at a position that can be regarded as infinity should be 0, but if it is not 0, the value of the parallax becomes the parallax offset. The parallax offset can be calculated by measuring the parallax of an object at a position where it can be regarded as. However, such objects can not always be photographed. Therefore, a method has been proposed in which a parallax offset is calculated using an object in the vicinity while traveling to correct the parallax.

  In Japanese Patent Laid-Open No. 9-126759 (Patent Document 1), the left and right white lines in the vicinity of the host vehicle are recognized, the left and right white lines are linearly approximated by the least squares method, and the intersection point of the approximation straight lines is taken as an infinite point. The relative shift angle between the cameras is calculated and corrected. In Japanese Patent Application Laid-Open No. 2001-169310 (Patent Document 2), the parallax of a stationary object whose size is known, such as a traffic light and a utility pole, is calculated at two times, and the amount of movement of the vehicle between the two times is calculated from the vehicle speed. The parallax offset is calculated based on the calculated parallax and the amount of movement of the vehicle to correct the parallax.

  In patent document 1 and patent document 2, since correction | amendment is performed using specific objects, such as a white line and a signal apparatus, correction | amendment can not be performed in the environment without a specific object. That is, the method of Patent Document 1 can not be used in an environment in which there is no white line or there is only a white line on one side, and the method of Patent Document 2 can not be used on a highway or the like without traffic lights or telephone poles.

  Japanese Patent No. 4894771 (Patent Document 3) provides a method of calculating a parallax offset without performing recognition processing of a specific object. In Patent Document 3, the parallax of a plurality of feature points in an input image and a two-dimensional optical flow (the motion of an object represented by a vector) are calculated, and the feature points on a flat road surface are calculated. The disparity offset is calculated using that the vertical component and the disparity and the square of the disparity calibrated with the disparity offset and the vertical component of the optical flow respectively have a specific relationship. Thereby, the parallax can be corrected without recognizing a specific object, but the input image must include a flat road surface, and a three-dimensional object such as a building or a telephone pole can not be used effectively. The accuracy of the correction degrades in traffic jams where most of the road surface is hidden.

  In patent 5440461 (patent document 4) and Unexamined-Japanese-Patent No. 2013-224919 (patent document 5), in order to solve the above-mentioned problem, it does not depend on a specific shape thing, but variously photographed during traveling A method of detecting a parallax offset using an image has been proposed. In Patent Document 4 and Patent Document 5, parallax offsets are calculated using a plurality of objects stationary with respect to the ground. That is, in order to cancel the influence of the parallax offset with respect to images taken by the stereo camera from two points while traveling the same subject, in Patent Document 4, the measurement data is in a space in which the difference between the parallax and the parallax is two axes. In Patent Document 5, measurement data is modeled using a parallax ratio or a difference of inverse parallax instead of a parallax difference. Then, the disparity offset is calculated from the parameters of the function representing the model.

JP-A-9-126759 JP 2001-169310 A Patent No. 4894771 Patent No. 5440461 JP, 2013-224919, A

Naoki Nishikubo, Kotaro Oshida, Yuhei Oka, Keiji Jikichi, "Auto-calibration of stereo camera with FPGA", Proceedings of the 29th Annual Conference of the Robotics Society of Japan, Tokyo, September 2011

  However, the methods of Patent Document 4 and Patent Document 5 can be applied only to stationary objects, and further, since a modeling method is used, a large amount of measurement data is required to calculate the parallax offset with high accuracy. Calculation amount is increased.

  The present invention has been made under the circumstances as described above, and it is an object of the present invention to specify a specific object such as a white line or a traffic light, a flat road, etc. in calculating the parallax offset while the moving object is moving. It is not necessary to set the restriction condition that the existence of the shape object is required or the application range is limited to the stationary object, etc., and various images captured by the stereo camera (stereo image pickup device) can be used. A calibration apparatus and calibration method capable of accurately and stably obtaining a parallax offset even if there are few data, and a distance measurement apparatus that carries the calibration apparatus and corrects the parallax with high accuracy in real time and performs accurate distance measurement It is to provide.

  The present invention relates to a calibration apparatus for calibrating parameters of a stereo imaging apparatus used in a distance measurement apparatus mounted on a mobile body and performing distance measurement, and the above object of the present invention is that it is imaged by the stereo imaging apparatus A three-dimensional object feature point extraction unit for extracting at least two feature points on a three-dimensional object in a reference image 1 and obtaining screen coordinates 1 of the respective feature points, and a comparison image 1 captured by the stereo image capturing device And a reference image 2 captured by the stereo image capturing device after a predetermined time has elapsed from the time of capturing the reference image 1 and the comparative image 1 using the disparity calculating unit that calculates the disparity 1 of the feature points. The comparison image 2 is input, the optical flow of each feature point is detected using the reference image 2 or the comparison image 2, and the reference image is detected using the reference image 2 and the comparison image 2 Using the movement detection unit for calculating the screen coordinates 2 and the parallax 2 of each feature point in 2; and using the screen coordinates 1, the parallax 1, the screen coordinates 2 and the parallax 2 A parallax offset calculation unit that calculates a parallax offset used to calibrate a parameter, the parallax offset calculation unit using a conversion equation that converts screen coordinates of an image captured by the stereo image capturing device into three-dimensional coordinates. This is achieved by calculating the disparity offset using an evaluation function with the disparity offset as an unknown number defined based on a difference formula representing the difference between the amounts of movement of two points in the three-dimensional coordinate.

  Further, in the above object of the present invention, the evaluation function is defined as a sum of squares of differences of the movement amounts, and the evaluation is made by numerical analysis using the screen coordinates 1, the parallax 1, the screen coordinates 2 and the parallax 2 By obtaining a parallax offset that minimizes the function, or the evaluation function is configured by the difference equation, each difference equation is calculated using the screen coordinates 1, the parallax 1, the screen coordinates 2, and the parallax 2 The individual parallax offset to be 0 is calculated, and the representative value of the individual parallax offset is used as the parallax offset, or the parallax offset calculation unit calculates the cubic of each of the feature points from the screen coordinates 1 and the parallax 1 The original coordinates 1 are calculated, the three-dimensional coordinates 2 of the feature points are calculated from the screen coordinates 2 and the parallax 2, and the movement calculated from the three-dimensional coordinates 1 and the three-dimensional coordinates 2 The combination of feature points whose difference is larger than a predetermined threshold value is not used for calculation of the parallax offset, or the movement detection unit determines the predetermined time after capturing the reference image 1 and the comparison image 1 A predetermined number of reference images and comparison images are input at time intervals, and only optical flow detection is performed on reference images and comparison images other than the reference image N and comparison image N input last. This is achieved more effectively by calculating the parallax 2 of each feature point only for the comparative image N.

  In addition, by mounting the above-described calibration device, it is possible to achieve a distance measurement device that corrects the parallax with high accuracy in real time and performs accurate distance measurement.

  According to the calibration device of the present invention, the parallax offset is calculated using feature points on a solid object, and even a moving solid object can be used, so almost all the images captured by the stereo image pickup device are It can be used. In addition, since the parallax offset can be calculated if there are at least two feature points, the parallax offset can be accurately calculated even if there are few measurement data.

  Further, according to the distance measuring device equipped with the calibration device of the present invention, since the parallax can be corrected with high accuracy in real time, accurate distance measurement can be performed.

It is the schematic which shows the principle of measurement of the distance by a stereo camera. It is a figure which shows the setting of the coordinate axis of a screen coordinate and a three-dimensional coordinate. It is a block diagram which shows the structural example (1st Embodiment) of this invention. It is a flowchart which shows the operation example (1st Embodiment) of this invention. It is an image figure showing an example of a matching field in stereo matching. It is an image figure which shows the calculation method in sub-pixel precision. It is an image figure which shows the example of the matching area | region in optical flow detection. It is a figure which shows the simulation result of this invention. It is a block diagram which shows the structural example (2nd Embodiment) of this invention. It is a flowchart which shows a part (operation | movement in a parallax offset calculation part) of the operation example (2nd Embodiment) of this invention. It is a block diagram which shows the structural example (3rd Embodiment) of this invention. It is a flowchart which shows the operation example (3rd Embodiment) of this invention.

  In the present invention, even if the stereo imaging device mounted on the moving object moves, the shape of the solid object to be captured does not change, so the movement distance and movement direction of the points (feature points) on the solid object are the same. Parallax offset is calculated using the fact that there is. That is, since the position of the feature point on the captured image (captured image) is defined by two-dimensional coordinates (screen coordinates), the feature point on the captured image moves when the stereo imaging device moves. The distance varies depending on where the feature point is initially located, but the distance the actual feature point moved whose position is defined by three-dimensional coordinates (three-dimensional coordinates) does not change the shape of the solid object , Is the same regardless of the initial position of the feature point. Furthermore, the moving direction is also the same. And since parallax is used in calculation of the position of the feature point in three-dimensional coordinates from the position of the feature point in screen coordinates, the parallax offset is added to the disparity, and the conversion formula from screen coordinates to three-dimensional coordinates is used. If the movement distance in three-dimensional coordinates is formulated, the optimum solution of the parallax offset can be obtained from the values of the screen coordinates of a plurality of feature points under the condition that the movement distance is the same.

  In the present invention, it is assumed that shading correction, inter-camera sensitivity difference correction, distortion correction, and the like are performed on a captured image in a stereo image capturing apparatus. Shading is a phenomenon in which the brightness decreases as it gets closer to the edge of the screen of the captured image. The inter-camera sensitivity difference means that differences occur in how luminance is detected because there are individual differences among cameras of the same product. Since the shading and the inter-camera sensitivity difference cause erroneous detection in stereo matching described later, they are corrected in the stereo image pickup device. Further, the image obtained from the camera is affected by the lens of the camera, and the amount of distortion increases as the distance from the center of the image increases, so this distortion is also corrected. Furthermore, rotational deviation and vertical deviation are also detected by the stereo image pickup apparatus and calibrated.

  Extraction of feature points to be measured is performed by the method of “feature point detection” described in Non-Patent Document 1, for example. That is, regions having strong edges in two directions perpendicular to each other are extracted as feature points from the captured image.

  Next, conversion from screen coordinates to three-dimensional coordinates will be described.

FIG. 2 is a view showing setting of coordinate axes of screen coordinates and three-dimensional coordinates, which corresponds to the camera 11 in FIG. As shown in FIG. 2, through the optical center O ₁ of the camera 11, in a plane on intersecting perpendicular to the optical axis of the camera 11, the optical center O ₁ as the origin, x-axis of the three-dimensional coordinates in the horizontal direction, vertical The direction is y-axis of three-dimensional coordinates, and the optical axis direction is z-axis of three-dimensional coordinates. In addition, assuming that the optical axis of the camera 11 passes through the center of the imaging surface 13 and the imaging surface 13 is set to intersect the optical axis perpendicularly, x on the imaging surface 13 with the center as the origin, x of three-dimensional coordinates. The opposite direction parallel to the axis is i axis of screen coordinates, and the opposite direction parallel to the y axis of three dimensional coordinates is j axis of screen coordinates. Since the subject appears upside down on the imaging plane, in the captured image, the i-axis and the j-axis are in the same direction as the x-axis and the y-axis respectively when viewed from the subject. In such a coordinate axis setting, when the position (coordinates) of the subject A in three-dimensional coordinates is (X, Y, Z) and the position (coordinates) in the screen coordinates is (I, J), X and Y Is given by the following equation 2 and equation 3, and Z can be obtained by the above equation 1.

Here, as in the case of FIG. 1, B is a base length and D is a parallax.

If there is no error (parallax offset) in the measured parallax D, the coordinates of the subject A in three-dimensional coordinates can be calculated from the coordinates of the screen coordinates using Equations 1 to 3, but the parallax offset D ₀ is added In the case, coordinates (X, Y, Z) in three-dimensional coordinates are as shown in the following expressions 4 to 6.

The values of the screen coordinates at time t of the feature points α _m and α _n extracted from the captured image are (I _m (t), J _m (t)) and (I _n (t), J _n (t), respectively. Assuming that the measured parallaxes are D _m (t) and D _n (t), respectively, change amounts of coordinates of feature points α _m and α _n in three-dimensional coordinates from time t ₁ to t ₂ (movement The difference in amount) is as shown in the following numbers 7-9.

Here, Ex _mn , Ey _mn and Ez _mn are respectively the difference in change amount in the x coordinate, the difference in change amount in the y coordinate, and the difference in change amount in the z coordinate.

If the feature point alpha _m and alpha _n are on the same three-dimensional object, the moving distance of the feature points alpha _m and alpha _n each of the three-dimensional coordinates are the same, yet because the moving direction is the same, Ex _mn , Ey _mn and Ez _mn are all zero. Therefore, the parallax offset can be determined by calculating D ₀ satisfying Ex _mn = Ey _mn = Ez _mn = 0, but is calculated from Ex _mn = 0, Ey _mn = 0, and Ez _mn = 0, respectively. D ₀ does not necessarily have the same value. This is because errors occur in the measured data due to various factors such as measurement accuracy and calculation accuracy. Similarly, when the number of feature points is three or more, a mismatch also occurs at D ₀ calculated from a plurality of combinations of feature points. Therefore, the optimal D ₀ is determined from the actual measurement data by the method of determining the optimal solution, and the optimal D ₀ is defined as the disparity offset.

First, a method of obtaining an optimal solution of D _{0 by} a numerical analysis method will be described.

L number of feature points α _1, α _{2, ...,} to define the objective function (cost function) represented by the following Expression 10 with respect to alpha _L.

Here, m and n are integers of 1 or more and L or less,
Means the sum of the sum of squares at each combination of m and n satisfying m <n. Then, to calculate the D ₀ of the evaluation function and minimize, to the D ₀ and parallax offset. As a method of numerical analysis, the steepest descent method, the Newton method, etc. are used.

Next, a method of calculating the disparity offset by calculating the representative value will be described. That is, a plurality of disparity offsets (individual disparity offsets) D ₀ are calculated, and a representative value (average value, mode value, etc.) of the individual disparity offsets is taken as the disparity offset.

Calculate D ₀ satisfying Ex _mn = 0, Ey _mn = 0, and Ez _mn = 0 with the combination of m and n satisfying m <n, and let the calculated representative value of all D _{0 be} the parallax offset . In the calculation of the representative value, the calculation may be performed excluding the individual parallax offset assumed to be unsuitable. For example, the average value of the individual disparity offsets is calculated, and the average value calculated again except for the individual disparity offsets whose absolute value of deviation from the average value is equal to or more than a predetermined value is taken as the disparity offset. Alternatively, the representative value may be calculated simply by excluding the individual parallax offset whose absolute value is equal to or more than a predetermined value. Such an individual parallax offset is because there is a possibility that the condition is not satisfied, for example, the target feature points are present on different moving objects.

The parallax offset may be calculated by a method combining the method by numerical analysis and the method by individual parallax offset calculation. For example, instead of Equation 10, the parallax offset (individual disparity offset) D ₀ which minimizes each evaluation function is calculated using the following three equations 11 as an evaluation function, and the representative value of the individual disparity offset is used as the parallax offset .

The combination of m and n may be divided into a plurality of groups, individual parallax offsets may be calculated using Equation 10 as an evaluation function in each group, and the representative value of the calculated individual parallax offsets may be used as the parallax offset. For example, if the feature points are present in a plurality of places on the captured image, each group is set as one group. Also in this combined method, individual disparity offsets which are estimated to be unsuitable may be excluded.

  In the calculation of the optimal parallax offset, calculation may be performed using data of at least one coordinate without using data of all the x coordinate, y coordinate, and z coordinate.

  The stereo image pickup device picks up an image at predetermined time intervals and outputs the image. Although the parallax offset is also calculated (updated) at a predetermined time interval, the time interval of the parallax offset calculation may be the same as the time interval of image capturing, but may be longer than the time interval of image capturing. For example, when the processing time required to calculate the parallax offset is longer than the time interval for capturing an image, the time interval for calculating the parallax offset is long.

  As described above, in the calibration device of the present invention, the feature points to be subjected to the parallax offset calculation may be present on a three-dimensional object, and the feature points are automatically extracted. Therefore, almost all images captured by the stereo image capturing device You can use the image of In addition, as long as at least two feature points need to be present and the parallax offset can be calculated using the three coordinates (x coordinate, y coordinate, z coordinate) of the feature point, the parallax offset can be accurately performed even if the measurement data is small. Can be calculated.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 3 is a block diagram showing a configuration example (first embodiment) of the present invention. In this configuration example, the calibration device 20 includes the memory 210, the solid object feature point extraction unit 220, the disparity calculation unit 230, the movement detection unit 240, and the disparity offset calculation unit 250, and the reference output from the stereo image capturing device 10. An image and a comparison image are input, and a parallax offset is calculated.

  The stereo image capturing apparatus 10 includes two cameras, the focal length of the two cameras is the same f, the distance between the cameras (base length) is B, and the optical axes are adjusted in advance so as to be parallel. There is. The stereo image pickup device 10 is installed on a movable body such as a vehicle so that the two cameras are in parallel equidistant position. The camera on the right (corresponding to the position of the camera 11 in FIG. 1) captures the reference image in the traveling direction of the moving body, and the camera on the left (corresponding to the position of the camera 12 in FIG. 1) captures the comparison image. The reference image and the comparison image are captured at predetermined time intervals at the same timing and subjected to correction (shading correction, inter-camera sensitivity difference correction, distortion correction, etc.) and rotational shift and vertical shift calibration as described above, It is output.

  The reference image and the comparison image output from the stereo image capturing device 10 are stored in the memory 210. The three-dimensional object feature point extraction unit 220 extracts a plurality of feature points (feature points 1) from the reference image (reference image 1) stored in the memory 210. The disparity calculating unit 230 calculates the disparity (disparity 1) of the feature point 1 by stereo matching using the reference image 1 stored in the memory 210 and the comparison image (comparison image 1) corresponding thereto. The movement detection unit 240 includes an optical flow detection unit 241 and a parallax detection unit 242. The optical flow detection unit 241 reads from the memory 210 a reference image (reference image 2) captured after the reference image 1, The feature point (feature point 2) corresponding to the feature point 1 is detected from the reference image 2 by the detection method, and the parallax calculating unit 242 uses the reference image 2 and the comparison image (comparison image 2) corresponding thereto. The two parallaxes (parallax 2) are calculated by stereo matching. The parallax offset calculation unit 250 calculates the parallax offset from the values of the screen coordinates of the feature point 1 and the feature point 2 and the parallax 1 and the parallax 2 by numerical analysis.

  An operation example of the calibration apparatus 20 having such a configuration will be described with reference to the flowchart of FIG.

  The reference image and the comparison image that are picked up and output at predetermined time intervals by the stereo image pickup device 10 are sequentially stored in the memory 210. The memory 210 stores the reference image and the comparison image, which are captured at the same timing, in correspondence with each other.

The three-dimensional object feature point extraction unit 220 reads the reference image SP1 captured at time t1 from the memory 210, and a plurality of feature points (feature points 1) according to the method of “feature point detection” described in Non-Patent Document 1. Are extracted (step S10). In the reference image, two-dimensional coordinates (screen coordinates) having coordinate axes (i-axis, j-axis) similar to those of the imaging plane shown in FIG. 2 are set, and the screen of each extracted feature point 1 Screen coordinate data (screen coordinates) TC1 having the coordinate values (I1 _k , J1 _k ) (k = 1, 2,..., K, and K are the number of feature points) at the coordinates are output.

  The screen coordinate data TC1 is input to the disparity calculating unit 230, the optical flow detecting unit 241, and the disparity offset calculating unit 250.

The disparity calculating unit 230 reads the comparison image CP1 captured at time t1 associated with the reference image SP1 and the reference image SP1 from the memory 210, and uses the screen coordinate data TC1 to perform feature matching by stereo matching. Calculate the disparity of 1. The stereo matching method uses a region-based method in which an image is divided into small regions (windows) for matching, and a SAD (Sum of Absolute Difference) function is used as a matching evaluation function. First, the position (reference position 1) of the feature point 1 in the reference image SP1 is specified by the screen coordinate data TC1. After specifying the reference position 1, as shown in FIG. 5, in the comparison image CP1, the image data (brightness) of the region on the right side in the horizontal direction from the reference position 1 and the image data of the reference position 1 in the reference image SP1 ( Matching with the luminance) (step S20). That is, SAD values of both image data are calculated by the SAD function, and a position where the SAD value is minimum in the matching area is set as the position (comparison position 1) of the feature point 1 in the comparison image CP1. From the positions of the two cameras in the stereo image pickup device 10, the matching area is limited to the area shown in FIG. Then, the parallax is calculated from the deviation between the reference position 1 and the comparison position 1, but in order to increase the deviation accuracy, the parallax with sub-pixel accuracy is obtained (step S30). FIG. 6 is a diagram simulating a part of SAD values in the matching area shown in FIG. 5, and it is assumed that the minimum SAD value is obtained at s2. In FIG. 6, it is assumed that the SAD function can be linearly approximated near the minimum value, and the position of the one with the largest SAD value (s3 in FIG. 6) among s2 of the position where the SAD value is minimum and the position before and after it. Are drawn by a straight line (line s2s3 in FIG. 6), and the slope is the reverse sign of the slope of the straight line and the SAD value is smaller (in FIG. 6) a straight line is drawn. The comparison position is 1. The deviation between the true comparison position 1 and the reference position 1 is taken as parallax. The SAD function near the minimum value may be approximated not by linear approximation but by a quadratic curve or the like. The calculated parallax D1 _k (k = 1, 2,..., K) of each feature point 1 is output to the parallax offset calculation unit 250 as parallax data PA1.

The optical flow detection unit 241 which has input the screen coordinate data TC1 reads the reference image SP1 and the reference image SP2 captured at time t2 (t2> t1) from the memory 210, and detects the feature points from the reference image SP2 by the optical flow detection method. A feature point (feature point 2) corresponding to 1 is detected. First, the position (reference position 1) of the feature point 1 in the reference image SP1 is specified by the screen coordinate data TC1. After specifying the reference position 1, similarly to the stereo matching method, with the SAD function as an evaluation function, in the reference image SP2, image data of an area other than the reference position 1 and image data of the reference position 1 in the reference image SP1. Matching is performed (step S40). That is, the SAD values of both image data are calculated by the SAD function, and the place where the SAD value is minimum is taken as the temporary position of the feature point 2. At this time, as shown in FIG. 7, the matching area may be limited to the area predicted as the movement destination of the feature point 1. When the temporary position of the feature point 2 is detected, the position (reference position 2) of the feature point 2 is obtained with sub-pixel accuracy based on the temporary position, as in the parallax detection unit 230 (step S50). That is, based on the SAD values at the upper, lower, left, and right positions of the temporary position, the reference position 2 is determined by linear approximation or the like in both the i-axis direction and the j-axis direction. The calculated coordinate values (I2 _k , J2 _k ) (k = 1, 2,..., K) of the reference position 2 are output to the parallax calculating unit 242 and the parallax offset calculating unit 250 as screen coordinate data TC2.

The parallax calculating unit 242 reads the comparison image CP2 captured at time t2 associated with the reference image SP2 and the reference image SP2 from the memory 210, and uses the screen coordinate data TC2 to perform stereo matching similarly to the parallax calculating unit 230. The parallax of each feature point 2 is calculated by the following method (steps S60 and S70). The operation content is the same as that of the disparity calculating unit 230. The calculated parallax D2 _k (k = 1, 2,..., K) of each feature point 2 is output to the parallax offset calculation unit 250 as parallax data PA2.

  The parallax offset calculation unit 250 uses the input screen coordinate data TC1, parallax data PA1, screen coordinate data TC2 and parallax data PA2 to minimize the parallax offset that minimizes the evaluation function F (D0) defined by the following equation 12. D0 is determined by a numerical analysis method (the steepest descent method, Newton method, etc.) (step S80).

Here, a and b are integers of 1 or more and K or less,
Denotes the sum of the sum of squares at each combination of a and b satisfying a <b.

  The calculated parallax offset D0 is used to correct the parallax used for distance measurement in the distance measuring device on which the calibration device 20 is mounted.

  When the parallax offset is calculated at the same time interval as the time interval at which the stereo imaging device 10 captures an image, the reference image 2 is taken as the reference image 1 and the comparison image 2 is taken as the comparison image 1 at the next time Steps S10 to S80 are repeated using the obtained images as the reference image 2 and the comparison image 2. In this case, the feature point extraction (step S10) of the solid object feature point extraction unit 220 and the parallax calculation (steps S20 and S30) of the parallax calculation unit 230 are omitted, and the reference position 2 detected by the optical flow detection unit 241 and the parallax calculation The parallax calculated by the unit 242 may be diverted.

As described above, the parallax offset D0 is calculated not by using a numerical analysis method, but by calculating individual parallax offsets that satisfy EX _ab = 0, EY _ab = 0, and EZ _ab = 0 (a <b) The representative value (average value, mode value, etc.) of all the individual disparity offsets may be set as the disparity offset D0. In this method, as described above, individual disparity offsets that are presumed to be unsuitable may be excluded. The parallax offset may be calculated by a method combining the method by numerical analysis and the method by individual parallax offset calculation.

  In the first embodiment, the SAD function is used as an evaluation function in stereo matching in the parallax calculation units 230 and 242 and optical flow detection in the optical flow detection unit 241. However, an SSD (Sum of Squared Difference) function is used. You may use Alternatively, instead of using the SAD function or the SSD function, the deviation may be calculated by the phase only correlation method. In addition, although the position is determined with sub-pixel accuracy, it is not necessary to determine sub-pixel accuracy when processing time is prioritized over accuracy.

Further, the optical flow detection unit 241 detects the feature point 2 from the reference image SP2, but may detect it from the comparison image CP2. In this case, when limiting the matching area, the parallax data PA1 output from the parallax calculating unit 230 is used, and the parallax calculating unit 242 detects the feature point 2 in the reference image SP2, and the parallax D2 _k k = 1, 2,..., K) is to be calculated.

  In the operation example in the first embodiment, the three-dimensional object feature point extraction unit 220, the parallax calculation unit 230, the optical flow detection unit 241, and the parallax calculation unit 242 each output data after processing for all feature points is completed. However, the coordinate value is output each time the solid object feature point extraction unit 220 extracts the feature point 1, and the processes of the parallax calculation unit 230, the optical flow detection unit 241, and the parallax calculation unit 242 may be executed. good. That is, steps S10 to S70 are executed in units of feature points, and after processing for all feature points is completed, step S80 is executed.

  FIG. 8 is a diagram showing simulation results of the present invention.

  A solid object is placed 10 m and 20 m in front of the moving object, the moving object is advanced 5 m, and a reference image and a comparison image are respectively taken at two points before and after advancing. When B × f = 100 and disparity offset D0 = 5 are set, and one feature point is set to each solid object (2 points), two points are set (4 points) and three points are set (3 points) The simulation was performed at 6 points). The parallax measurement accuracy is 0.2 px (pixel). In FIG. 8, the horizontal axis is the parallax offset D0, and the vertical axis is the density (depth) of the evaluation function. As shown in FIG. 8, it can be seen that there is a peak near D0 = 5 in all three cases, and the parallax offset is detected with high accuracy.

  In the first embodiment, before the disparity offset calculation unit calculates the disparity offset, the combination of inappropriate feature points is detected, and the combination is not used in the calculation of the disparity offset, so that the accuracy of calculation of the disparity offset can be calculated. You can raise it. Specifically, the difference between the amount of change (amount of movement) from time t1 to t2 in the three-dimensional coordinates of the two feature points in the three-dimensional coordinates and y coordinates is calculated, and the difference is equal to or greater than a predetermined threshold The combination of feature points is not used in the calculation of disparity offset. Such feature points do not satisfy the condition because they may be present on different moving objects.

  A block diagram of a configuration example (second embodiment) to which a function of excluding the combination of the above-mentioned inappropriate feature points is added is shown in FIG. The configuration of the second embodiment is the same as that of the first embodiment shown in FIG. 3 except for the parallax offset calculation unit 350, and thus the description of the same configuration is omitted.

  The disparity offset calculating unit 350 includes a movement amount checking unit 351, a disparity offset estimating unit 352, and a memory 353. The movement amount inspection unit 351 uses the values of screen coordinates of the feature point 1 and the feature point 2, the parallax 1 and the parallax 2, and the parallax offset stored in the memory 353, and the x coordinate in three dimensions of each feature point The amount of change from time t1 to t2 in the coordinate and z coordinate is calculated, and a combination of feature points whose difference (absolute value) in the amount of change is equal to or greater than a predetermined threshold value is extracted. The parallax offset estimation unit 352 is a combination excluding the combination of the feature points extracted by the movement amount inspection unit 351 and is a parallax offset by numerical analysis or the like from the values of the screen coordinates of the feature point 1 and the feature point 2 and the parallax 1 and parallax 2 Calculate The calculated parallax offset is output, and is stored in the memory 353, and is used to calculate the amount of change of the movement amount inspection unit 351 in the next parallax offset calculation. In addition, 0 is used as an initial value of the parallax offset used by change amount calculation.

  An operation example of the disparity offset calculation unit 350 will be described with reference to the flowchart of FIG.

The movement amount inspection unit 351 receives the screen coordinate data TC1, the parallax data PA1, the screen coordinate data TC2 and the parallax data PA2, reads the parallax offset D0 stored in the memory 353, and x coordinate in three dimensional coordinates of each feature point The change amounts VX _k , VY _k and VZ _k (k = 1, 2,..., K) from time t1 to t2 in the y and z coordinates are calculated from the following equation 13 (step S810).

Then, at least one of differences (absolute values) ΔVX _ab , ΔVY _ab and ΔVZ _ab (where a and b are integers of 1 or more and K or less, a <b) between the amounts of change of two feature points calculated from A combination of feature points which are equal to or greater than a predetermined threshold θ is extracted (step S820), and is output as excluded combination data EP.

The predetermined threshold θ may not be common to ΔVX _ab , ΔVY _ab, and ΔVZ _ab , and different threshold may be used.

  The parallax offset estimation unit 352 receives the screen coordinate data TC1, the parallax data PA1, the screen coordinate data TC2, the parallax data PA2, and the excluded combination data EP, and combines the feature points other than the combination of the feature points included in the excluded combination data EP. Then, using the screen coordinate data TC1, the parallax data PA1, the screen coordinate data TC2, and the parallax data PA2, the parallax offset D0 is determined by the numerical analysis method or the like as in the parallax offset calculation unit 250 in the first embodiment Step S830).

  In the second embodiment, extraction of an inappropriate combination of feature points and calculation of a parallax offset are separately performed, but extraction of an inappropriate combination of feature points in calculation of a parallax offset is performed and extraction is performed. You may make it not use the calculated combination for calculation of parallax offset. That is, steps S820 and S830 may be performed in combination of feature points.

  In the calculation of the parallax offset, if the movement distance of the feature point is too short, the accuracy of the calculated parallax offset may fall, and the longer the movement distance of the feature point, the higher the accuracy. However, when the movement distance is long, it takes time to detect the feature point after movement from the image, and there is also a possibility that the feature point can not be detected accurately. Therefore, after extracting feature points, only detection of feature points (detection of optical flow) is performed on a predetermined number of images output from the stereo image pickup apparatus, and parallax offset is calculated on the subsequent images. By performing the above, it is possible to increase the accuracy of the calculated parallax offset without degrading the detection accuracy of the feature point.

  A block diagram of a configuration example (third embodiment) of the present invention that performs such processing is shown in FIG. With respect to the first embodiment shown in FIG. 3, switches 410 to 415 are added to the third embodiment, and a memory 443 is further added to the movement detection unit. The same components as those of the first embodiment are denoted by the same reference numerals and the description thereof will be omitted.

An operation example of the third embodiment will be described with reference to the flowchart of FIG. In the third embodiment, it was extracted from the captured image of the feature point in time t _1, the time t _2, t 3, _..., images captured at t _N only performs detection of the feature point, time It is assumed that detection of parallax offset is performed on an image captured at t _{N + 1} .

When three-dimensional object feature point extraction unit 220 reads from the memory 210 the image captured at time _{t 1} as a reference image SP1 and comparison image CP1, switches 410 and 411 in ON, the switch 412,413,414,415 is OFF and (Step S100). The three-dimensional object feature point extraction unit 220 extracts feature points, the parallax calculation unit 230 calculates parallax, and the screen coordinate data TC1 and the parallax data PA1 are output to the parallax offset calculation unit 250 (step S110). Screen coordinate data TC1 is output to the optical flow detection section 441, from the optical flow detection unit 441, like the optical flow detector 241 in the first embodiment, the memory 210 the reference image SP2 captured at time t ₂ The feature point is detected using the reference image SP1 as well, and the screen coordinate data TC2 is output (step S120). The output screen coordinate data TC2 is stored in the memory 443.

Next, the switches 410, 411, 412, 413 and 414 are turned off, and only the switch 415 is turned on (step S130). Then, only feature point detection of the optical flow detection unit 441 is performed. That is, the optical flow detection unit 441 reads from the memory 210 the image captured at time _{t 2} as a reference image SP1, reads out the image captured at time _{t 3} as a reference image SP2, further screen coordinate data TC2 from the memory 443 Are read out, and feature points are detected from the reference image SP2 (step S140). At this time, the position of the feature point in the reference image SP1 is specified using the screen coordinate data TC2. The coordinate values of the detected feature point are output as screen coordinate data TC 2 and stored in the memory 443. The stored screen coordinate data TC2 is used to specify the position of the feature point at the next time.

After that, step S140 is repeated for the image captured at each time until the image captured at time tN _-1 is read out from the memory 210 as the reference image SP1.

When reading from the memory 210 the image captured at time _{t N} as a reference image SP1, switches 410 and 411 in the OFF, the switch 412,413,414,415 becomes ON (step S150). Then, the optical flow detection unit 441 detects a feature point by the same processing as step S140, and outputs the screen coordinate data TC2 (step S160). The screen coordinate data TC2 is input to the disparity calculating unit 242 and the disparity offset calculating unit 250. The parallax calculation unit 242 reads the image captured at time t _{N + 1} from the memory 210 as the reference image SP2 and the comparison image CP2, calculates the parallax using the input screen coordinate data TC2, and calculates the parallax offset calculation unit as the parallax data PA2. It outputs to 250 (step S170). The parallax offset calculation unit 250 obtains the parallax offset D0 using the input screen coordinate data TC1, the parallax data PA1, the screen coordinate data TC2, and the parallax data PA2 (step S180).

In the third embodiment, the switch of the processing for the captured image is performed by a switch, but information on the time is added to the image, and the processing to be performed in each part is switched based on the information. Also good.

The calibration apparatus according to the present invention has been described above. However, the present invention is not limited to the above-described embodiment, and it goes without saying that various other configurations and methods can be adopted without departing from the scope of the present invention. Further, the calibration device according to the present invention can be implemented by software (computer program) using a computer system, and ASIC (Application Specific Integrated Circuit), GPU (Graphics Processing Unit), FPGA (Field Programmable) It can of course be implemented by hardware such as Gate Array).

Reference Signs List 11 and 12 camera 10 stereo image capturing device 20, 30, 40 calibration device 210, 353, 443 memory 220 three-dimensional object feature point extraction unit 230, 242, parallax detection unit 240, 440 movement detection unit 241, 441 optical flow detection unit 250 , 350 Parallax offset calculation unit 351 Movement amount inspection unit 352 Parallax offset estimation unit 410, 411, 412, 413, 414, 415 switch

Claims (11)

In a calibration device for calibrating parameters of a stereo imaging device used in a distance measurement device mounted on a mobile body and performing distance measurement,
A three-dimensional object feature point extraction unit that extracts at least two feature points on a three-dimensional object in the reference image 1 captured by the stereo image capturing device, and obtains the screen coordinates 1 of the respective feature points;
A disparity calculating unit that calculates the disparity 1 of the feature points using the comparison image 1 captured by the stereo image capturing device;
The reference image 2 and the comparison image 2 captured by the stereo image capturing device are input after a predetermined time has elapsed from the time when the reference image 1 and the comparison image 1 were captured, and the reference image 2 or the comparison image 2 is input. A movement detection unit that detects an optical flow of each of the feature points using the reference image 2 and the comparison image 2 and calculates the screen coordinates 2 and the parallax 2 of the feature points in the reference image 2 When,
A parallax offset calculation unit configured to calculate a parallax offset used for calibration of a parameter of the stereo imaging device using the screen coordinates 1, the parallax 1, the screen coordinates 2 and the parallax 2;
The disparity offset calculation unit is a difference equation representing a difference between movement amounts of two points in the three-dimensional coordinates using a conversion equation that converts screen coordinates of an image captured by the stereo image capturing device into three-dimensional coordinates. A calibration apparatus characterized in that the disparity offset is calculated by an evaluation function with the disparity offset as an unknown number defined based on.   The evaluation function is defined as a sum of squares of differences of the movement amounts, and a parallax offset which makes the evaluation function minimum is obtained by numerical analysis using the screen coordinates 1, the parallax 1, the screen coordinates 2 and the parallax 2. The calibration device according to claim 1.   The evaluation function is composed of the differential equation, and using the screen coordinates 1, the parallax 1, the screen coordinates 2 and the parallax 2 to calculate an individual parallax offset where the respective difference formulas are 0, the individual parallax offset The calibration apparatus according to claim 1, wherein the disparity offset is a representative value of   The disparity offset calculation unit calculates three-dimensional coordinates 1 of the feature points from the screen coordinates 1 and the disparity 1 and calculates three-dimensional coordinates 2 of the feature points from the screen coordinates 2 and the disparity 2 4. The combination according to any one of claims 1 to 3, wherein a combination of feature points whose difference in movement amount calculated from the three-dimensional coordinate 1 and the three-dimensional coordinate 2 is larger than a predetermined threshold is not used for calculating the parallax offset. Calibration device.   The movement detection unit inputs a predetermined number of reference images and comparison images at predetermined time intervals after the time of capturing the reference image 1 and the comparison image 1, and finally inputs the reference image N and the comparison image N. The optical flow is detected only for reference images and comparison images other than the above, and calculation of parallax 2 of each feature point is performed only for the reference image N and the comparison image N. Calibration device as described in.   A distance measuring device comprising the calibration device according to any one of claims 1 to 5. In a calibration method for calibrating parameters of a stereo image pickup device mounted on a moving body and used in a distance measurement device that performs distance measurement,
A solid object feature point extraction step of extracting at least two feature points on a solid object in the reference image 1 captured by the stereo image capturing device, and determining screen coordinates 1 of the feature points;
A parallax calculation step of calculating the parallax 1 of the feature points using the comparative image 1 captured by the stereo image capturing device;
The reference image 2 and the comparison image 2 captured by the stereo image capturing device are input after a predetermined time has elapsed from the time when the reference image 1 and the comparison image 1 were captured, and the reference image 2 or the comparison image 2 is input. A movement detection step of detecting an optical flow of each of the feature points using the reference image 2 and the comparison image 2 to calculate screen coordinates 2 and a parallax 2 of the feature points in the reference image 2 When,
A parallax offset calculating step of calculating a parallax offset to be used for calibration of a parameter of the stereo imaging device using the screen coordinates 1, the parallax 1, the screen coordinates 2 and the parallax 2;
The disparity offset calculating step is a difference formula representing a difference between movement amounts of two points in the three-dimensional coordinates using a conversion formula for converting screen coordinates of an image captured by the stereo image capturing apparatus into three-dimensional coordinates. And calculating the disparity offset using an evaluation function with the disparity offset as an unknown number defined on the basis of.   The evaluation function is defined as a sum of squares of the time variation, and a parallax offset which makes the evaluation function minimum is obtained by numerical analysis using the screen coordinates 1, the parallax 1, the screen coordinates 2 and the parallax 2. The calibration method according to Item 7.   The evaluation function is composed of the differential equation, and using the screen coordinates 1, the parallax 1, the screen coordinates 2 and the parallax 2 to calculate an individual parallax offset where the respective difference formulas are 0, the individual parallax offset The calibration method according to claim 7, wherein the disparity offset is a representative value of   The disparity offset calculating step calculates three-dimensional coordinates 1 of the feature points from the screen coordinates 1 and the disparity 1 and calculates three-dimensional coordinates 2 of the feature points from the screen coordinates 2 and the disparity 2 10. A combination of feature points whose difference in movement amount calculated from the three-dimensional coordinate 1 and the three-dimensional coordinate 2 is larger than a predetermined threshold is not used for calculation of the parallax offset. Calibration method. In the movement detection step, a predetermined number of reference images and comparison images are input at predetermined time intervals after the time of capturing the reference image 1 and the comparison image 1, and the reference image N and the comparison image N input last The optical flow detection is performed only on the reference image and the comparison image other than the reference image, and the calculation of the parallax 2 of each feature point is performed only on the reference image N and the comparison image N. The calibration method described in.