JP2018032253A - Calibration device, calibration method, and calibration program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To carry out calibration using an appropriate feature point.SOLUTION: A calibration device includes: an image acquisition part for acquiring a photographic image acquired by photographing with a camera fixed to a vehicle; a feature point detection part for detecting a feature point from the photographic image; a feature point series generation tracking part for generating a feature point series by tacking the feature point by a time series; a height information calculation part for calculating height information of the generated feature point series; an advisability determination part for determining the advisability of the feature point series based on the height information; and a calibration part for calibrating an external parameter of the camera using the feature point determined to be advisable by the advisability determination part.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a calibration device, a calibration method, and a calibration program.

  In recent years, vehicle control such as automatic parking and automatic driving using an in-vehicle camera has been studied. In order to control the vehicle using the captured image of the camera, it is necessary to accurately determine external parameters such as the camera mounting position and mounting angle, but the physical mounting accuracy is insufficient for vehicle control. There can be. Patent Document 1 discloses a method of tracking a plurality of detected feature points in an image input from a camera and estimating external parameters of the camera from the time-series coordinate positions of the feature points.

International Publication No. 2012/139636

  Patent Document 1 does not describe how to select feature points suitable for external parameter calibration.

According to the first aspect of the present invention, the calibration device includes an image acquisition unit that acquires a captured image obtained by capturing with a camera fixed to a vehicle, and a feature point detection that detects a feature point from the captured image. A feature point sequence generation tracking unit that tracks the feature points in time series to generate a feature point sequence, a feature point sequence height calculation unit that calculates height information of the generated feature point sequence, A quality determination unit that determines the quality of the feature point series based on height information, and a calibration unit that calibrates the external parameters of the camera using the feature point series that has been affirmed by the quality determination unit.
According to the second aspect of the present invention, the calibration method is a calibration method for a camera fixed to a vehicle, wherein a captured image obtained by capturing with the camera is acquired, and a feature point is obtained from the captured image. Detecting the feature points in time series to generate a feature point series, calculating height information of the generated feature point series, and determining whether the feature point series is good or bad based on the height information. The external parameters of the camera are calibrated using the feature point series that has been affirmed in the determination of pass / fail.
According to the third aspect of the present invention, the calibration program is a calibration program for calibrating external parameters of a camera fixed to a vehicle, and a computer captures a captured image obtained by the camera. Obtaining a feature point from the captured image, tracking the feature point in time series, generating a feature point series, calculating height information of the generated feature point series, and based on the height information The quality of the feature point series is judged, and the external parameters of the camera are calibrated using the feature point series that has been affirmed by the judgment of quality.

  According to the present invention, external parameters can be calibrated using appropriate feature points.

Hardware configuration diagram of the calibration device according to the first embodiment Functional block diagram of the calibration device according to the first embodiment 3A is a diagram showing the positional relationship between the vehicle and the feature points, and FIG. 3B is a plan view of FIG. Flowchart showing calibration processing operation The figure which shows the hardware constitutions of the calibration apparatus in 2nd Embodiment. Functional block diagram of the calibration device according to the second embodiment Diagram showing how to provide the program

(First embodiment)
Hereinafter, a first embodiment of the calibration apparatus will be described with reference to FIGS.

(Hardware configuration)
FIG. 1 is a diagram illustrating a hardware configuration of the calibration apparatus 1. The calibration device 1 is mounted on the vehicle 2. The vehicle 2 includes a calibration device 1, a camera 3, a speed sensor 5, and a gyro sensor 6. The calibration device 1, the camera 3, the speed sensor 5, and the gyro sensor 6 are connected by a signal line.

  The calibration device 1 includes a CPU 11, a ROM 12, a RAM 13, and a flash memory 14. The ROM 12 stores a program 12P and internal parameters of the camera 3. The CPU 11 realizes functions to be described later by expanding and executing the program 12P stored in the ROM 12 in the RAM 13. The flash memory 14 is a nonvolatile storage unit, for example, and stores a parameter 14P. The parameter 14 </ b> P is an external parameter of the camera 3 calibrated by the calibration device 1. However, in the initial state, the design value of the external parameter or the calibration result at the time of shipment is stored as the parameter 14P. The external parameters include information on the posture angle of the camera 3 with respect to the vehicle 2 and the mounting height of the camera 3 with respect to the vehicle 2. The posture angle is a yaw angle, a roll angle, and a pitch angle.

  The camera 3 captures the periphery of the vehicle 2 at a predetermined capturing period, for example, a 33.3 ms period, and outputs an image (hereinafter referred to as a captured image) obtained by capturing to the calibration device 1. The internal parameters of the camera 3, that is, the image sensor size and the focal length are known and stored in the ROM 12 of the calibration apparatus 1. The camera 3 is attached to a predetermined position of the vehicle 2 in a predetermined posture according to a prior design, but it is inevitable that an error occurs in the attachment. For this reason, the external parameters of the camera 3 based on the design values are known, but the external parameters taking into account the error of attachment to the vehicle 2 are not clear. Therefore, as will be described later, the calibration device 1 calculates an external parameter in consideration of an attachment error to the vehicle 2, that is, calibrates the external parameter.

  The speed sensor 5 is a wheel speed sensor using, for example, a rotary encoder, and measures the speed of the vehicle 2 and outputs it to the calibration device 1. The gyro sensor 6 is, for example, a vibration type gyroscope that can detect at least the angular velocity of the yaw angle, and outputs the detected angular velocity to the calibration device 1.

(Function block)
FIG. 2 is a diagram illustrating functions realized by the CPU 11 of the calibration apparatus 1 executing a program stored in the ROM 12 as function blocks.
The calibration apparatus 1 functions as an image acquisition unit 102, a feature point detection unit 103, a feature point position storage control unit 104, a feature point tracking unit 105, a tracking completion determination unit 106, and a feature point series creation. Unit 107, coordinate conversion unit 108, external parameter storage control unit 109, feature point series height calculation unit 110, vehicle speed information holding unit 112, travel state holding unit 113, and feature point series registration determination unit 114 , A feature point series storage unit 115, a calibration timing creation unit 116, a calibration execution unit 117, a calibration result holding unit 118, and an output unit 119. Hereinafter, the operation of each functional block will be described. However, the operations of the feature point series height calculation unit 110 and the calibration execution unit 117 will be described in detail later.

The image acquisition unit 102 reads a captured image output from the camera 3 and outputs it to the feature point detection unit 103. Since the camera 3 continuously captures images at a predetermined time interval, the image acquisition unit 102 receives captured images that are continuous in time series.
The feature point detection unit 103 performs image processing on the captured image read by the image acquisition unit 102 and extracts feature points. The feature points here are, for example, intersections of edges such as broken-line corners and pillar corners drawn on the road surface, that is, corner feature points. The corner feature point can be extracted by applying a Harris operator which is a known technique, for example. The feature point information extracted by the feature point detection unit 103 is sent to the feature point position storage control unit 104 and the feature point tracking unit 105.

  The feature point position storage control unit 104 stores the feature point information extracted by the feature point detection unit 103, for example, the coordinates in the captured image, and the feature point tracking information tracked by the feature point tracking unit 105 in the RAM 13. The feature point tracking information is information indicating the same feature point in different captured images, for example, information indicating the coordinates of the feature point for each captured image. The feature point position storage control unit 104 outputs the feature point information stored in the RAM 13 to the feature point tracking unit 105 and the tracking completion determination unit 106.

  The feature point tracking unit 105 tracks the same feature point from a plurality of captured images that are continuous in time series, and outputs feature point tracking information to the feature point position storage control unit 104. For tracking feature points, information on feature points in the latest photographed image input from the feature point detection unit 103, information on feature points in past photographed images stored in the RAM 13 by the feature point position storage control unit 104, and The motion information of the vehicle 2 input from the traveling state holding unit 113 is used. For tracking feature points, known tracking methods such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), and LK (Lucas-Kanade) can be used. The feature point tracking unit 105 performs tracking processing for all the feature points in the latest captured image input from the feature point detection unit 103, and then the tracking completion determination unit 106 starts operation.

The tracking completion determination unit 106 reads the feature point information stored in the RAM 13 using the feature point position storage control unit 104, determines the feature point that has been tracked, that is, the feature point that cannot be tracked with the latest photographed image. To extract. The situation in which the feature point assumed here cannot be tracked is, for example, that a certain feature point detected from an object stationary on the road deviates from the shooting range of the camera 3 because the vehicle 2 continued to travel. This is a situation where the captured image is no longer captured. Information of a series of feature points determined to have been tracked by the tracking completion determination unit 106, that is, the coordinates of the same feature point in different captured images is sent to the feature point sequence creation unit 107.
The feature point series creation unit 107 collects a series of tracked feature points as one feature point series for the feature points determined to be tracked by the tracking completion determination unit 106 and outputs the feature points to the coordinate conversion unit 108.

  The coordinate conversion unit 108 converts the coordinates of each feature point constituting one feature point series output from the feature point series creation unit 107 from the two-dimensional coordinates in the captured image to the three-dimensional coordinates based on the camera 3, that is, Convert to real space coordinates. However, it is assumed that all feature points are present on the road surface, and coordinate conversion is performed with a constraint that the height of the feature points is zero. This coordinate transformation is realized by calculation using a rotation matrix. The elements of the rotation matrix are determined using internal parameters and external parameters of the camera 3. The value stored in the ROM 12 is used as the internal parameter, and the value stored as the parameter 14P in the flash memory 14 is used as the external parameter. The coordinates of each feature point constituting one feature point series, which has been coordinate transformed by the coordinate transformation unit 108, are output to the feature point series height calculation unit 110.

  The external parameter storage control unit 109 writes the parameter 14P to the flash memory 14 and reads the parameter 14P from the flash memory 14, that is, reads / writes the external parameter. The external parameter storage control unit 109 provides the parameter 14P to the coordinate conversion unit 108 for coordinate axis conversion. The external parameter storage control unit 109 provides the calibration execution unit 117 with the parameter 14P as an initial value for calibration. The external parameter storage control unit 109 stores the external parameter calibrated by the calibration execution unit 117 in the flash memory 14 as a new parameter 14P, that is, overwrites it.

The feature point series height calculation unit 110 calculates the height of the feature point series coordinate-converted by the coordinate conversion unit 108 using the speed of the vehicle 2 obtained from the vehicle speed information holding unit 112. This calculation will be described using Equation 2 described later. The coordinates of each feature point of the feature point series transformed by the coordinate transformation unit 108 and the height of the feature point series calculated by the feature point series height calculation unit 110 are output to the feature point series registration determination unit 114. The The processing of the feature point series height calculation unit 110 will be described in detail later.
The vehicle speed information holding unit 112 accumulates the output of the speed sensor 5, calculates the average speed of the vehicle 2 for each shooting period of the camera 3, and outputs the average speed to the feature point series height calculation unit 110.
The traveling state holding unit 113 accumulates the outputs of the speed sensor 5 and the gyro sensor 6, and outputs the movement information of the vehicle 2, that is, the speed history and the traveling locus to the feature point tracking unit 105 and the feature point series registration determining unit 114.

The feature point series registration determination unit 114 determines whether or not the feature point series used for calculating the height output from the feature point series height calculation unit 110 can be used for calibration of external parameters. Judgment using the output of. The feature point series registration determination unit 114 outputs the feature point series determined to be usable for calibration to the feature point series storage unit 115.
When the two conditions are satisfied, the feature point series registration determination unit 114 determines that the feature point information of the input feature point series can be used. The first condition is that the height of the feature point series is approximately zero, that is, substantially matches the height of the ground. For example, the feature point series registration determination unit 114 determines that the first condition is satisfied when the difference between the feature point series and the height of the ground is within 10 cm. The second condition is that the vehicle 2 was in a constant velocity linear motion when the feature points collected as one feature point sequence by the feature point sequence creation unit 107 were photographed. Whether the speed of the vehicle 2 is constant or whether the vehicle 2 is traveling straight can be determined from the output of the traveling state holding unit 113.

The feature point series storage unit 115 stores the coordinates of each feature point constituting the feature point series output from the feature point series registration determination unit 114 in the RAM 13. The coordinates stored here are the feature point coordinates before coordinate conversion by the coordinate conversion unit 108. Note that the feature point series storage unit 115 determines the number of feature point series to be stored in advance, and restricts the necessary memory area by discarding the oldest feature point series when a new feature point series is registered. Also good. In addition, a feature point sequence that has been stored for a certain time after being stored may be discarded.
The calibration timing creation unit 116 determines the timing for performing calibration and outputs it to the calibration execution unit 117. The calibration timing creation unit 116 may determine that “next calibration is performed” when a predetermined time has elapsed from the previous calibration execution timing, or “next” when new registration of feature point series reaches a certain number. You may decide to “calibrate”. Further, when either one of the two conditions is satisfied, or when both of the two conditions are satisfied, it may be determined that “the next calibration is performed”.

The calibration execution unit 117 executes calibration by a known method according to the calibration timing created by the calibration timing creation unit 116, and outputs the execution result to the external parameter storage control unit 109 and the calibration result holding unit 118. The processing of the calibration execution unit 117 will be described in detail later.
The calibration result holding unit 118 holds the calibration result. This is for performing the output timing of the calibration result asynchronously with the calibration execution timing.
The output unit 119 outputs the calibration result at the requested timing. The output destination of the calibration result may be, for example, a program that uses a captured image of the camera 3 or a storage unit provided in the vehicle 2.

(Conceptual explanation of the feature point series height calculation unit)
The feature point series height calculation unit 110 calculates the height of the feature point series by comparing the moving speed of the feature points in the captured image with the moving speed of the vehicle 2. This method is closely related to the coordinate transformation performed by the coordinate transformation unit 108 on the assumption that the feature point exists on the road surface of zero height. Prior to the description of specific processing, this concept will be described with reference to FIG.

  FIG. 3A shows a vehicle 2 traveling on the road surface G, a camera 3 mounted on the vehicle 2, a feature point r on the road surface G, a feature point s not on the road surface G, and the physicality of the virtual point t. It is a figure which shows typical positional relationship. The height of the camera 3 from the road surface G is Hc, the height of the feature point s from the road surface G is Hfs, and the height of the feature point r on the road surface G from the road surface G is Hfr, that is, zero. The front side of the vehicle 2, that is, the traveling direction is the rear side in the drawing in FIG. The left-right direction in FIG. 3A is the left-right direction of the vehicle 2, that is, the width direction of the vehicle 2, and the up-down direction in FIG. The feature point r on the road surface is a feature point extracted from a broken line drawn on the road surface G, for example. The feature point s not on the road surface is a feature point extracted from the tip of the guard rail column S. The virtual point t is a virtual point where a straight line connecting the camera 3 and the feature point s exists at the position with the road surface G. The coordinate conversion unit 108 performs coordinate conversion on the assumption that all feature points are present on a road surface having a height of zero, so that the feature point s is given the three-dimensional coordinates of the position of the virtual point t. Since the feature point r exists on the road surface G, the coordinate conversion unit 108 attaches the three-dimensional coordinates of the correct position, that is, the position of the feature point r.

  3 (b) is a plan view of FIG. 3 (a), and the horizontal direction of FIG. 3 (b) is the horizontal direction of the vehicle 2 as in FIG. 3 (a), and the vertical direction of FIG. 3 (b). The direction is the front-rear direction of the vehicle 2. However, the vehicle 2 is omitted in FIG. FIG. 3B shows the positions of the feature point r and the virtual point t calculated by the coordinate conversion unit 108, and further shows the position of the feature point s for explanation. In FIG. 3B, the feature point r, the feature point s, and the virtual point t are indicated by the same markers as in FIG. That is, in FIG. 3B, the characteristic point r and the like are initially positioned in front of the vehicle 2 (r1 to r3), but when the vehicle 2 moves forward for a while, the characteristic point r is aligned with the vehicle 2 (r4). When the vehicle 2 further advances, the feature point r indicates that it is located behind the vehicle 2 (r5 to r7).

  Since the feature point r exists on the road surface, the coordinate conversion unit 108 correctly calculates the three-dimensional coordinate, that is, the relative position of the camera 3 and the feature point r. Therefore, the speed calculated from the three-dimensional coordinates of the feature point r, that is, the relative speed between the feature point r and the camera 3 is equal to the speed of the vehicle 2. On the other hand, since the feature point s does not exist on the road surface, when the coordinate conversion unit 108 assumes that the feature point s exists on the road surface and performs coordinate conversion, the coordinate of the virtual point t is attached to the feature point s. As described in FIG. 3A, the coordinates of the virtual point t exist on a straight line connecting the camera 3 and the feature point s. Further, since the relationship between the height of the camera 3 and the feature point s is constant, the movement amount of the virtual point t is larger than that of the feature point s as shown in FIG. Therefore, the feature point s is calculated so as to move faster than the vehicle 2.

  In FIG. 3B, Lr indicates a distance at which the feature point r has advanced to six shooting intervals at r1 to r7. In FIG. 3B, Ls indicates a virtual distance in which t1 to t7 of the virtual point t has advanced to the six shooting intervals in which the feature point s is s1 to s7. As can be seen from the fact that Ls is longer than Lr in FIG. 3B, the virtual point t is calculated as if it moved a longer distance in the same time than the feature point r. Based on the above idea, the feature point series height calculation unit 110 calculates the height of the feature point series.

(Feature point series height calculator)
The feature point series height calculation unit 110 calculates the height of the feature point series using the speed of the vehicle 2 obtained from the vehicle speed information holding unit 112 and the coordinates of the feature point series coordinate-converted by the coordinate conversion unit 108. . The processing of the feature point series height calculation unit 110 includes feature point speed calculation processing and speed comparison processing.

  In the feature point speed calculation process, which is the first process, the speed of the feature point is calculated by dividing the movement distance of the feature point by the movement time. The moving distance of the feature point is obtained from the three-dimensional coordinates of the feature point based on the camera 3 calculated by the coordinate conversion unit 108. The moving time is obtained from the known shooting period of the camera 3 and the number of shot images whose feature points are tracked. A certain feature point is tracked over N shot images except for the last detected shot image, the moving distance of the feature point is calculated as L from the difference in three-dimensional coordinates, and the shooting interval of the camera 3 is time T. In this case, the velocity Vf of the feature point series is calculated as in the following formula 1.

Vf = L / (T * N)... Equation 1
For example, in FIG. 3, when the distance Lr from r1 to r7 is 1 m and the shooting period T is 33.3 ms, the number of tracked sheets N is 6, so the speed Vfr of the feature point series r is calculated as 72 km / h. Similarly, if the distance Ls from t1 to t7 is 2 m, the speed Vfs of the feature point series s is calculated as 142 km / h.
In the speed comparison process that is the second process, the height Hf of the feature point series is the attachment height Hc of the camera 3, the speed Vf of the feature point series calculated by Equation 1, and the vehicle obtained from the vehicle speed information holding unit 112. Using the vehicle degree Vc of 2, the following equation 2 is calculated. The mounting height Hc of the camera 3 is included in the external parameter of the camera 3, that is, the parameter 14P.

Hf = Hc * (1-Vc / Vf) Equation 2
For example, in FIG. 3, when the mounting height Hc of the camera 3 is 1 m and the vehicle speed Vc is 72 km / h, if the speed Vfr of the feature point series r is also 72 km / h, the height Hfr of the feature point series r is 0 m. If the velocity Vfs of the feature point series s is 144 km / h, the height Hfs of the feature point series is calculated as 0.5 m.

(Calibration execution part)
First, the calibration execution unit 117 reads the coordinates of each feature point of the registered feature point series from the feature point series storage unit 115. As described above, this coordinate is a coordinate in the captured image of the camera 3. Next, the calibration execution unit 117 reads out the posture angle information of the camera 3 from the external parameter storage control unit 109. Hereinafter, the posture angle information read from the external parameter storage control unit 109 is referred to as a posture angle initial value. The calibration execution unit 117 performs coordinate axis conversion of the coordinates of the feature points of the feature point series read from the feature point series storage unit 115 using the attitude angle initial value. The coordinate axis conversion process at this time is the same as that performed by the coordinate conversion unit 108.

  The coordinates of the feature points of the feature point series that have undergone the coordinate axis conversion process are the positions on the road surface. If the posture angle information used for the coordinate axis conversion process is a correct value, each feature point of all feature point series moves straight from the front to the rear of the vehicle body at regular intervals. In addition, the feature point interval at this time is an interval corresponding to the vehicle speed. That is, assuming that the moving direction of the vehicle 2 is the Y-axis direction and the axis orthogonal thereto is the X-axis direction, the coordinates with respect to the X-axis take the same value, and the values with respect to the Y-axis are equally spaced.

  The posture angle information obtained from the external parameter storage control unit 109 is a design value or information obtained by the previous calibration, and may not match the current posture angle information of the camera 3. If they do not match, the coordinates of the feature points in the feature point series after coordinate conversion do not have the same X coordinate value and a constant Y coordinate value interval. The calibration execution unit 117 linearly approximates the line segment of the feature point series obtained by connecting the feature points of the coordinate point-converted feature point series with a straight line, calculates the angle formed by this straight line and the Y axis, Assume an evaluation error value. By adding the evaluation error values of all the feature point series, an evaluation value for the corresponding angle information is obtained.

  The calibration execution unit 117 changes the posture angle of the camera 3 by a minute amount from the posture angle initial value and performs coordinate conversion again to obtain an evaluation value for the angle. Since the camera 3 has a three-dimensional arrangement, there are three angles, a pitch angle, a roll angle, and a yaw angle. An evaluation error value is calculated by giving a small amount of change to all angles. When the evaluation error is smaller than before giving a minute change in angle, since the correction is performed in the correct direction, the same method is further changed. If the evaluation error increases, the correction amount is too large or the correction is applied in the reverse direction, so the evaluation error value is the same as the previous time by reducing the slight change in angle or adding the correction in the reverse direction. To calculate.

In this way, by continuously applying a minute amount change to all posture angles, a combination of angles that minimizes the evaluation error value can be obtained. The combination of angles obtained in this way is output as a calibration result to the calibration result holding unit 118 and the external parameter storage control unit 109.
Calibration of the mounting height information of the camera 3, which is another element constituting the external parameter, can be performed in the same procedure. When there is an error in the mounting height, the moving speed of the feature point series does not match the vehicle speed of the vehicle 2. In other words, the difference between the movement distance of the feature point after coordinate conversion and the movement distance calculated from the vehicle speed is used as an evaluation error value, and the sum of errors of all the feature point series can be set as an evaluation error value for the corresponding height. it can. The mounting height information obtained from the external parameter storage control unit 109 is used as an initial value, the height at which the evaluation error value is minimized while changing a minute amount in the same manner as the calibration of the attitude angle, and the calibration result holding unit 118 and the calibration result are obtained. Output to the external parameter storage control unit 109.

(flowchart)
The operation of each functional block described above will be described using a flowchart.
FIG. 4 is a flowchart showing the calibration processing operation of the calibration apparatus 1.
In step S <b> 201, the image acquisition unit 102 reads a captured image from the camera 3. In subsequent step S202, the feature point detection unit 103 extracts feature points from the captured image. In step S203, the feature point tracking unit 105 performs tracking processing on one of the feature points extracted in step S202 as a processing target. In subsequent step S204, the tracking completion determination unit 106 determines whether or not the feature point tracking is completed. If it is determined that the feature point tracking is completed, the process proceeds to step S205. If it is determined that the feature point tracking is not completed, the process proceeds to step S211. .

  In step S205, the feature point series creation unit 107 collects the feature points that have been tracked as a feature point series. In subsequent step S206, the coordinate conversion unit 108 converts each feature point constituting the feature point series into three-dimensional coordinates on the assumption that the feature point exists on a road surface having a height of zero. In subsequent step S207, the feature point series height calculation unit 110 calculates the height of the feature point series using the above-described Expression 1 and Expression 2. In subsequent step S208, the feature point series registration determining unit 114 determines whether or not the height of the feature point series is approximately zero. If it is determined that the feature point series is approximately zero, the process proceeds to step S209, and is not approximately zero. If so, the process proceeds to step S210.

In step S209, the feature point series storage unit 115 stores the coordinates of each feature point constituting the feature point series in the RAM, and the process proceeds to step S211. In step S210, the feature point series storage unit 115 discards the coordinates of each feature point in the feature point series and proceeds to step S211.
In step S211, the CPU 11 determines whether or not the processing has been completed for all the feature points extracted in step S202. If it is determined that the processing has been completed for all the feature points, the process proceeds to step S212, and the unprocessed processing is performed. If it is determined that there is a feature point, one of the unprocessed feature points is set as a processing target, and the process returns to step S203.

  In step S212, the calibration execution unit 117 determines whether or not it is the calibration timing created by the calibration timing creation unit 116. If the calibration execution unit 117 determines that it is not the calibration timing, the process returns to step S201. If the calibration execution unit 117 determines that it is the calibration timing, calibration is executed in step S213, the output unit 119 outputs the calibration result in step S214, and the flowchart of FIG.

According to the first embodiment described above, the following operational effects are obtained.
(1) The calibration device 1 includes an image acquisition unit 102 that acquires a captured image obtained by the camera 3 fixed to the vehicle 2, a feature point detection unit 103 that detects a feature point from the captured image, A feature point tracking unit 105 that tracks feature points in time series and generates a feature point sequence, a tracking completion determination unit 106, and a feature point sequence creation unit 107, and a feature point sequence height that calculates height information of the feature point sequence The calculation unit 110 and the quality determination unit that determines the quality of the feature point series based on the height information, that is, the feature point series registration determination unit 114, and the feature point series that is affirmed by the quality determination unit are external to the camera 3. A calibration execution unit 117 for calibrating the parameters.
The calibration apparatus 1 calculates the height of the extracted feature point series, determines whether or not each feature point series is suitable for calibration based on the height, and uses the feature point series that has been affirmed. 3 external parameters are configured. Therefore, the calibration apparatus 1 can perform calibration using appropriate feature points.

(2) The external parameter calibrated by the calibration execution unit 117 of the calibration apparatus 1 is at least one of the yaw angle, the roll angle, the pitch angle, and the mounting height of the camera 3 that are the mounting angles of the camera 3 to the vehicle 2. It is. Therefore, the calibration apparatus 1 can calibrate at least one of the yaw angle, the roll angle, the pitch angle, and the mounting height of the camera 3.

(3) The pass / fail determination unit, that is, the feature point series registration determination unit 114 makes a negative determination when the difference between the height of the feature point series and the height of the road surface on which the vehicle 2 travels is greater than a predetermined value. Therefore, the calibration execution unit 117 can execute the calibration by excluding the feature points present at a position higher than the road surface, for example, the feature points extracted from the top of the guard race support column. That is, the calibration apparatus 1 can perform calibration using appropriate feature points.

(4) The coordinate conversion for calculating the relative position of the feature point with respect to the camera 3 as a coordinate in the real space on the assumption that the detected feature point exists at the height of the road surface on which the vehicle 2 travels. The unit 108 is provided. The feature point series height calculation unit 110 calculates the speed of the feature point series based on the relative positions of the feature points constituting the feature point series, and the height of the feature point series based on the speed of the vehicle 2 and the speed of the feature point series. Estimate.
The feature point series height calculation unit 110 calculates the speed of the feature point series based on the shooting interval time of the shot image and the relative position of the feature point in each shot image with the vehicle 2 as shown in Equation 1 above. calculate. Then, the feature point series height calculation unit 110 calculates the height of the feature point series based on the speed of the feature point series and the speed of the vehicle 2 as shown in Equation 2 above. Since the formulas 1 and 2 only execute the four rule operations twice and three times, the feature point series height calculation unit 110 can easily execute the process. The coordinate conversion unit 108 performs coordinate conversion from the coordinates on the captured image to the three-dimensional coordinates, but the three-dimensional coordinates of the feature points are assumed to be zero as with the road surface, that is, a constraint condition is added. Coordinate conversion is easy.

(5) Information indicating the traveling state of the vehicle 2 is input to the quality determination unit 114. The pass / fail determination unit 114 makes a negative determination when determining that the vehicle 2 is not traveling straight at a constant speed. Therefore, the calibration is performed using only the feature points extracted from the photographed image photographed under conditions suitable for the calibration, so that the reliability of the calibration can be further improved.

(Modification)
The first embodiment described above may be modified as follows.
(1) The calibration execution unit 117 calibrated all of the yaw angle, roll angle, pitch angle, and mounting height of the camera 3. However, the calibration execution unit 117 may calibrate at least one of the external parameters of the camera 3.
(2) The feature point series registration determination unit 114 makes an affirmative determination when both the condition regarding the height of the feature point series and the condition regarding the motion of the vehicle 2 are satisfied. However, the feature point series registration determination unit 114 may make an affirmative determination when the feature point series satisfies the condition regarding the height regardless of the condition regarding the motion of the vehicle 2. In this case, since it is not necessary to determine whether or not the vehicle 2 has performed a constant velocity linear motion, the calibration device 1 may not include the traveling state holding unit 113.

(3) Instead of using the output of the gyro sensor 6, the traveling state holding unit 113 may determine whether the vehicle 2 is traveling straight by detecting the angle of the steering wheel.
(4) The flash memory 14 may be a volatile memory. In this case, the design value of the external parameter or the value of the external parameter calibrated at the time of shipment is stored in the ROM 12. Then, when the calibration apparatus 1 is activated, the design value of the external parameter is read from the ROM 12 to the flash memory 14 as the parameter 14P.

(5) The shooting by the camera 3 may not be periodic. In this case, the image acquisition unit 102 adds information indicating the difference between the shooting time of the captured image or the time when the immediately preceding captured image is input to the captured image. As a result, the feature point series height calculation unit 110 can calculate the moving speed of the feature points even when the camera 3 is not periodically photographed.
(6) The vehicle 2 may include a plurality of cameras 3, and the calibration apparatus 1 may calibrate external parameters for each of the plurality of cameras 3. Regardless of the shooting direction of the camera 3, the calibration method is as described in the first embodiment. However, when photographing the side of the vehicle 2, since there are many opportunities to photograph an object present at a position higher than the road surface compared to photographing the front and rear of the vehicle 2, selection of feature points used for calibration is performed. Is particularly effective.

(7) The feature point series height calculation unit 110 specifically calculated the feature point series height Hf based on Equation 2. However, R_Hf, which is a value representing the ratio of the height to the mounting height Hc of the camera 3, may be calculated without calculating the product of the mounting height Hc of the camera 3 in Equation 2. R_Hf is calculated by the following equation 3. R_Hf is information regarding the height of the feature point series.
R_Hf = 1−Vc / Vf Equation 3
In this case, the feature point series registration determination unit 114 determines whether the above-described value R_Hf is within a predetermined range, for example, 0.9 or more and less than 1.1.

(8) The feature point series height calculation unit 110 may output (Vc / Vf) or Vf. In these cases, the feature point series registration determination unit 114 determines whether the ratio between the vehicle speed Vc and the feature point series speed Vf is within a predetermined range, for example, 0.9 or more and less than 1.1. This is based on the idea that the judgment regarding the height of the feature point series in the feature point series registration determination unit 114 is approximated to the judgment as to whether or not the vehicle speed Vc and the speed Vf of the feature point series match. It is. That is, (Vc / Vf) and Vf are information regarding the height of the feature point series.

(9) The vehicle 2 may be an automatic driving vehicle that is controlled based on a captured image of the camera 3. In this case, in the vehicle 2, the braking unit that operates the brake, the engine control unit that controls the engine speed, and the steering unit that operates the steering wheel operate based on the operation command of the automatic driving program. The automatic driving program performs, for example, an avoiding operation for avoiding a collision with the vehicle 2 by detecting a moving body approaching from the captured image of the camera 3, or an automatic parking operation by detecting a parking frame from the captured image of the camera 3.

(10) A part or all of the processing in the calibration apparatus 1 may be executed by a server connected to the calibration apparatus 1. For example, the calibration apparatus 1 transmits information input from the camera 3, the vehicle speed sensor 5, and the gyro sensor 6 to the server, and the server performs all the processes performed by the calibration apparatus 1 in the first embodiment. Also good. Further, instead of transmitting the captured image of the camera 3 to the server, the calibration apparatus 1 may acquire an image and detect a feature point, and transmit the coordinates of the detected feature point to the server. .

(Second Embodiment)
A second embodiment of the calibration apparatus will be described with reference to FIGS. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and different points will be mainly described. Points that are not particularly described are the same as those in the first embodiment. In the present embodiment, it is mainly determined by using map information whether or not the feature point is inside the roadway, and is different from the first embodiment.

(Hardware configuration)
FIG. 5 is a diagram showing a hardware configuration of the calibration apparatus 1A. The calibration device 1A is mounted on the vehicle 2A. The vehicle 2A includes a calibration device 1A, a camera 3, a GPS receiver 4, a speed sensor 5, a gyro sensor 6, and a map database 7.
Since the camera 3, the speed sensor 5, and the gyro sensor 6 are the same as those in the first embodiment, description thereof is omitted.
The GPS receiver 4 receives signals from a plurality of satellites, and calculates the position of the GPS receiver 4 by calculation using the received signals.

  The map database 7 is a database in which map information including at least the position and shape of the road is stored. The map database 7 is composed of, for example, nodes that indicate intersections of roads and links that connect the nodes. The node contains latitude and longitude information. A link is specified by two nodes serving as end points thereof, and includes information on a road, that is, information on a road width and a road shape. The width of the road includes the width of the roadway and the width of the sidewalk. The shape of the road includes at least one of the latitude and longitude of a plurality of points constituting the node and the radius of curvature of the node.

  The calibration device 1 </ b> A includes a CPU 11, a ROM 12 </ b> A, a RAM 13, and a flash memory 14. Since the configurations of the CPU 11, RAM 13, and flash memory 14 are the same as those in the first embodiment, description thereof is omitted. The ROM 12A stores a program 12AP and internal parameters of the camera 3.

(Function block)
FIG. 6 is a diagram illustrating functions realized by the CPU 11 of the calibration apparatus 1 executing the program 12AP stored in the ROM 12A as function blocks. In the following, differences from FIG. 2 will be mainly described.
The functional blocks provided as functions by the calibration apparatus 1A are substantially the same as the functional blocks provided as functions by the calibration apparatus 1 according to the first embodiment. The difference between the two is that the calibration device 1A includes a traveling state holding unit 113A instead of the traveling state holding unit 113, and a feature point series registration determining unit 114A instead of the feature point series registration determining unit 114. is there.

  In addition to the speed sensor 5 and the gyro sensor 6, the traveling state holding unit 113 </ b> A is connected to the GPS receiver 4 and the map database 7. The traveling state holding unit 113A identifies the road on which the vehicle 2 is currently traveling based on the current position of the vehicle 2 output from the GPS receiver 4 and the map information output from the map database 7, and stores information on the road. Output to the feature point series registration determination unit 114A. The traveling state holding unit 113A also outputs the current position of the vehicle 2 to the feature point series registration determining unit 114A. When the position information cannot be acquired from the GPS receiver 4, the traveling state holding unit 113 </ b> A calculates the current position using the position information acquired immediately before, the output of the speed sensor 5 and the output of the gyro sensor 6.

  The feature point series registration determination unit 114A makes an affirmative determination when two conditions are satisfied. As in the first embodiment, the first condition is that the height of the feature point series is approximately zero, that is, substantially matches the height of the ground. The second condition is that it can be determined that the feature point exists in the area inside the roadway with reference to the map information. The area inside the carriageway is the area inside the carriageway that is demarcated by curb lines or fences or other similar structures or road markings for use in vehicle traffic, in other words, excluding sidewalks on the road It is an area. That is, the traveling lane of the vehicle that travels in the opposite direction to the vehicle 2 is also included inside the roadway. The feature point series registration determination unit 114A uses the road information input from the traveling state holding unit 113A, the current position of the vehicle 2, and the position of the feature point calculated by the coordinate conversion unit 108 to determine whether the feature point is a road. It is determined whether or not it exists inside.

According to the second embodiment described above, the following operational effects can be obtained.
(1) Map information including the position of the vehicle 2 and at least the position and shape of the roadway is input to the feature point series registration determination unit 114A. The feature point series registration determination unit 114A makes a negative determination when determining that the feature point series exists outside the roadway based on the position of the vehicle 2, the map information, and the relative positions of the feature points and the vehicle 2. Therefore, it is possible to reject the feature points existing outside the roadway such as a sidewalk or outside the road.

(Modification of the second embodiment)
The feature point series registration determining unit 114A may set the second condition that the roadway is a straight line. The feature point series registration determination unit 114 </ b> A identifies the road on which the vehicle 2 is traveling based on the road information input from the traveling state holding unit 113 </ b> A and the current position of the vehicle 2. Then, the feature point series registration determination unit 114A determines whether the traveling lane in which the vehicle 2 is traveling is a straight line based on road information. However, since a plurality of traveling lanes are generally parallel, whether or not the traveling lane is a straight line may be determined by whether or not the roadway is a straight line. Furthermore, roads with a sidewalk added to the roadway are generally straight if the road is straight, so whether or not the driving lane is a straight line is determined by whether or not the road is a straight line. Also good.

According to the modified example described above, the following operational effects can be obtained.
(1) Map information including the position of the vehicle 2 and at least the position and shape of the road is input to the feature point series registration determination unit 114A. The feature point series registration determination unit 114A makes a negative determination when determining that at least the road on which the vehicle 2 is traveling is not a straight line. Therefore, the calibration apparatus 1 </ b> A can execute calibration using feature points obtained when the vehicle travels straight, which is suitable for calibration.

  Although the program is stored in the ROM 12, the program may be read from the outside and stored in the nonvolatile memory. The calibration apparatus 1 may include an input / output interface (not shown), and the program may be read from another apparatus via the input / output interface and a medium that can be used by the calibration apparatus 1 when necessary. Here, the medium refers to, for example, a storage medium that can be attached to and detached from the input / output interface, or a communication medium, that is, a wired, wireless, or optical network, or a carrier wave or digital signal that propagates through the network. Also, part or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

  FIG. 7 is a diagram showing an example in which a program is read from the outside. The calibration apparatus 1 may include an input / output interface (not shown), and the program may be read from the medium 304. A program is stored in the hard disk device 303 of the server 302 connected to the communication line 301 and the program is provided to the calibration device 1 via the communication line 301. The program can be supplied as various forms of computer program products such as provision via the recording medium 304 or the communication line 301.

The above-described embodiments and modifications may be combined.
Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

DESCRIPTION OF SYMBOLS 1, 1A ... Calibration apparatus 2, 2A ... Vehicle
DESCRIPTION OF SYMBOLS 3 ... Camera 12P ... Program 14 ... Flash memory 102 ... Image acquisition part 103 ... Feature point detection part 104 ... Feature point position storage control part 105 ... Feature point tracking part 106 ... Tracking completion determination part 107 ... Feature point series creation part 108 ... Coordinate conversion unit 109 ... external parameter storage control unit 110 ... feature point series height calculation unit 114, 114A ... feature point series registration determination unit 117 ... calibration execution unit

Claims (9)

An image acquisition unit that acquires a captured image obtained by a camera fixed to the vehicle;
A feature point detector for detecting a feature point from the captured image;
A feature point sequence generation tracking unit for tracking the feature points in time series and generating a feature point sequence;
A feature point series height calculating unit for calculating height information of the generated feature point series;
A quality determination unit that determines quality of the feature point series based on the height information;
A calibration apparatus comprising: a calibration unit that calibrates external parameters of the camera using the feature point series that has been affirmed by the pass / fail determination unit. The calibration device according to claim 1,
An external parameter calibrated by the calibration unit is at least one of a yaw angle, a roll angle, a pitch angle, and a mounting height of the camera. The calibration device according to claim 1,
The quality determination unit is a calibration device that makes a negative determination when a difference between a height of the feature point series and a height of a road surface on which the vehicle travels is greater than a predetermined value. The calibration device according to claim 1,
Assuming that the feature point exists at the height of the road surface on which the vehicle travels, a coordinate conversion unit that calculates a relative position of the feature point with the camera as coordinates in real space is further provided,
The feature point series height calculation unit calculates the speed of the feature point series based on the relative position of the feature points constituting the feature point series, and based on the speed of the vehicle and the speed of the feature point series A calibration device that estimates the height of a feature point series. The calibration device according to claim 4,
Map information including the position of the vehicle and at least the position and shape of the roadway is input to the pass / fail judgment unit,
The calibration device that makes a negative determination when it is determined that the feature point series exists outside a roadway based on the position of the vehicle, the map information, and the relative position. The calibration device according to claim 1,
Information indicating the running state of the vehicle is input to the pass / fail judgment unit,
The quality determination unit is a calibration device that makes a negative determination when determining that the vehicle is not traveling straight at a constant speed. The calibration device according to claim 1,
Map information including the position of the vehicle and at least the position and shape of the roadway is input to the pass / fail judgment unit,
The quality determination unit is a calibration device that makes a negative determination when determining that at least the roadway on which the vehicle is traveling is not a straight line. A calibration method for a camera fixed to a vehicle,
Obtaining a photographed image obtained by photographing with the camera,
Detecting feature points from the captured image;
Tracking the feature points in time series to generate a feature point series,
Calculating height information of the generated feature point series;
Judge the quality of the feature point series based on the height information,
A calibration method for calibrating external parameters of the camera using a feature point series that has been affirmatively determined by the quality determination. A calibration program for calibrating external parameters of a camera fixed to a vehicle,
On the computer,
The photographed image obtained by photographing with the camera is acquired,
Feature points are detected from the captured image;
The feature points are tracked in time series to generate a feature point series,
Calculating height information of the generated feature point series;
Based on the height information, the quality of the feature point series is judged,
A calibration program for calibrating external parameters of the camera using a feature point series that has been affirmed by the pass / fail judgment.

Images