JP2017135680A - Camera calibration device and camera calibration method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of improving camera calibration accuracy.SOLUTION: A camera calibration device 20 comprises a feature point extraction part 24, an area calculation part 26, and an adjustment part 28. The feature point extraction part 24 extracts plural feature points from an image of a calibration marker captured with a camera 10. The area calculation part 26 calculates the area of a graphic regulated by the extracted feature points. The adjustment part 28 adjusts a camera parameter for calibrating the camera 10 on the basis of the coordinates of the extracted feature points and the calculated area of the graphic.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a camera calibration apparatus and a camera calibration method for calibrating a camera.

  2. Description of the Related Art A technique is known in which a camera attached to a vehicle captures an image of the surroundings of the vehicle, and driving assistance is performed using the captured image. In such a technique, a deviation from a design value occurs in a captured image due to an error in the mounting position of the camera and a manufacturing error in the camera itself. For this reason, it is necessary to calibrate the camera mounting position error in advance in a factory or the like. At the time of calibration, the calibration marker is imaged by the camera, the feature points are extracted from the captured calibration marker image, and the camera parameters for calibrating the camera are adjusted based on the coordinates of the extracted feature points. (For example, refer to Patent Document 1). The larger the occupation ratio of the calibration marker image in the captured image and the more uniformly and densely distributed feature points, the more the calibration accuracy can be improved.

JP 2011-155687 A

  However, it is desired to use a small calibration marker due to the limitation of installation space. The smaller the calibration marker, the smaller the number of feature points that can be extracted, and the feature points are concentrated in a part of the captured image. In this case, the movement of the feature point due to the error of the camera mounting position is not easily affected by the lens distortion and is close to the parallel movement, so that the calibration accuracy is lowered.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a technique capable of improving the calibration accuracy of a camera.

  In order to solve the above problems, a camera calibration apparatus according to an aspect of the present invention includes a feature point extraction unit that extracts a plurality of feature points from a calibration marker image captured by a camera, and the feature point extraction unit extracts the feature points. An area calculation unit for calculating the area of the figure defined by the feature points, the coordinates of the feature points extracted by the feature point extraction unit, and the area of the figure calculated by the area calculation unit And an adjustment unit that adjusts camera parameters for calibrating the camera.

  Another aspect of the present invention is a camera calibration method. The method includes a step of extracting a plurality of feature points from an image of a calibration marker imaged by a camera, a step of calculating an area of a figure defined by the extracted feature points, and the extracted feature points And adjusting the camera parameters for calibrating the camera based on the calculated coordinates and the calculated area of the figure.

  According to the present invention, the calibration accuracy of the camera can be improved.

Fig.1 (a) is a figure which shows the positional relationship of the vehicle and the calibration marker in the case of the calibration which concerns on one Embodiment, FIG.1 (b) is an example of the calibration marker of Fig.1 (a). FIG. It is a block diagram which shows the schematic structure of the camera calibration apparatus of FIG. FIG. 3A is a diagram illustrating an image of a calibration marker in an ideal case where there is no error in the camera attachment position, and FIG. 3B is a case where there is an error in the yaw angle of the camera attachment position. It is a figure which shows the image of the marker for calibration of this. 4A is a diagram in which the feature points of FIG. 3A and FIG. 3B are overlapped, and FIG. 4B is a feature point of FIG. 3A and FIG. 3B. FIG. It is a flowchart which shows the process of the camera calibration apparatus of FIG.

  FIG. 1A is a diagram showing a positional relationship between the vehicle C1 and the calibration markers M1 and M2 at the time of calibration according to an embodiment, and FIG. 1B is a diagram for calibration in FIG. It is a top view which shows an example of marker M1, M2. In FIG. 1A, the surroundings of the vehicle C1 when the camera 10 is calibrated in a production factory of the vehicle C1 are viewed from above. The vehicle C <b> 1 includes a camera 10 and a camera calibration device 20. The camera 10 is attached to a back door or the like at the rear of the vehicle C1, and images the rear of the vehicle C1. The camera 10 may be attached near the center axis of the vehicle, or may be attached offset from the center axis of the vehicle. The camera calibration device 20 adjusts camera parameters for calibrating the camera 10.

  The calibration markers M1 and M2 are disposed substantially perpendicular to the floor surface at predetermined positions behind the vehicle C1 within the imaging range of the camera 10. The calibration markers M1 and M2 are arranged substantially symmetrically on both sides of the center axis of the vehicle.

  As shown in FIG. 1B, each of the calibration markers M1 and M2 has a checkered pattern in which 16 squares are arranged in a matrix. The camera 10 images this checkerboard pattern.

  FIG. 2 is a block diagram showing a schematic configuration of the camera calibration device 20 of FIG. The camera calibration device 20 includes an image storage unit 22, a feature point extraction unit 24, an area calculation unit 26, and an adjustment unit 28.

  The image storage unit 22 stores images of the calibration markers M1 and M2 captured by the camera 10.

  The feature point extraction unit 24 extracts a plurality of feature points from the images of the calibration markers M1 and M2 stored in the image storage unit 22. The method for extracting the feature points is not particularly limited, but the feature point extraction unit 24 may extract the feature points using a pattern matching technique. When the pattern matching technique is used, the feature point extraction unit 24 scans the template for each feature point on the image, and extracts the feature point from the pattern on the image having a high degree of coincidence with the template.

  The area calculation unit 26 calculates the area of the graphic defined by the feature points extracted by the feature point extraction unit 24. Each figure is a polygon having three or more feature points as vertices.

The adjustment unit 28 adjusts the camera parameters based on the feature point coordinates extracted by the feature point extraction unit 24 and the figure area calculated by the area calculation unit 26.
Detailed functions of the area calculation unit 26 and the adjustment unit 28 will be described later.

  The configuration of the camera calibration device 20 can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer, and is realized by a program loaded in the memory in software. Describes functional blocks realized by collaboration. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms only by hardware, or by a combination of hardware and software.

The camera parameter adjustment will be described with reference to FIGS. 3A and 3B and FIGS. 4A and 4B.
FIG. 3A is a diagram illustrating an image I1 of calibration markers M1 and M2 in an ideal case where there is no error in the mounting position of the camera 10, and FIG. 3B is a yaw of the mounting position of the camera 10. It is a figure which shows the image I2 of the marker M1, M2 for calibration when there exists an angle | corner error. The images I1 and I2 are captured by the camera 10. Hereinafter, an example of adjusting camera parameters in the case of FIG. 3B will be described.

  FIG. 4A is a diagram in which the characteristic points P1C to P10C and P1 to P10 of FIG. 3A and FIG. 3B are overlapped, and FIG. 4B is the same as FIG. It is a figure which overlaps and shows the feature points P1C-P10C, P1-P10 of 3 (b), and the figures F1C-F8C, F1-F8. In order to clarify the explanation, the calibration markers M1 and M2 are removed in FIGS. 4 (a) and 4 (b).

  As shown in FIGS. 3A and 3B, there are ten feature points P1C to P10C in the image I1. In the image I2, there are ten feature points P1 to P10. The feature points P1 and P1C are located at the center of the calibration marker M1. The feature points P2 and P2C are located at the center of the four squares at the upper left of the calibration marker M1, and the feature points P3 and P3C are located at the center of the four squares at the upper right of the calibration marker M1. The feature points P4 and P4C are located at the center of the lower left four squares of the calibration marker M1, and the feature points P5 and P5C are located at the middle of the lower right four squares of the calibration marker M1. Since the relationship between the feature points P6 to P10, P6C to P10C and the calibration marker M2 is the same, the description thereof is omitted.

The figure F1 is a triangle having feature points P1 to P3 as vertices. The figure F2 is a triangle having feature points P1, P4, and P5 as vertices. The figure F3 is a triangle having the feature points P1, P2, and P4 as vertices. The figure F4 is a triangle having feature points P1, P3, and P5 as vertices.
The figure F5 is a triangle having feature points P6 to P8 as vertices. The figure F6 is a triangle having feature points P6, P9, and P10 as vertices. The figure F7 is a triangle having vertices at the feature points P6, P7, and P9. The figure F8 is a triangle having feature points P6, P8, and P10 as vertices.
Since the relationship between the figures F1C to F8C and the feature points P1C to P10C is the same, the description is omitted.

  As described above, the area calculation unit 26 in FIG. 2 includes the area of the pair of figures F1 and F2 arranged in the first direction (up and down direction) y and the second direction (left and right direction) in the image of the calibration marker M1. ) The area of a set of figures F3 and F4 arranged in x is calculated. Further, the area calculation unit 26 in the image of the calibration marker M2 has an area of a set of figures F5 and F6 arranged in the first direction y and an area of a set of figures F7 and F8 arranged in the second direction x. And are calculated. The second direction x intersects the first direction y.

  Here, since the calibration markers M1 and M2 are installed at predetermined positions as described above, each of the calibration markers M1 and M2 in the reference coordinate system (world coordinate system) with reference to the vehicle C1 is used. The coordinates of the feature points are known. By calculating the coordinates of the known feature points in the reference coordinate system and the camera parameters by a known method, the coordinates of the feature points on the image can be calculated. Here, it is assumed that the coordinates of the feature points on the image calculated using the initial camera parameters are equal to the coordinates of the feature points P1C to P10C in the ideal case of FIG.

  If the camera parameter is an optimal value, the calculated coordinates of the feature points P1C to P10C are equal to the coordinates of the corresponding feature points P1 to P10 extracted from the captured image. As the camera parameters differ from the optimum values, the difference between the coordinates of the feature points P1 to P10 corresponding to the feature points P1C to P10C increases. Therefore, the adjustment unit 28 in FIG. 2 repeatedly adjusts the camera parameter so that the evaluation value becomes small, and brings the camera parameter close to the optimum value. The evaluation value is the sum of the first evaluation value and the second evaluation value.

  The first evaluation value is the sum of distances L1 to L10 shown in FIG. The distance L [i] (i is an integer of 1 to 10) is a distance between the calculated coordinates of the feature point P [i] C and the coordinates of the feature point P [i] extracted from the captured image. is there. In FIG. 4A, the distances L1 and L6 are not shown.

  The second evaluation value is the sum of the area differences S1 to S8. The area difference S [j] (j is an integer of 1 to 8) is calculated from the area of the figure F [j] C calculated from the calculated feature points P1C to P10C and the feature points P1 to P10 extracted from the image. It is the difference from the calculated area of the figure F [j].

  As a method of adjusting the camera parameter so that the evaluation value becomes small, a known method such as a steepest descent method can be used. The adjustment unit 28 in FIG. 2 determines that the evaluation value has converged when, for example, the evaluation value becomes substantially constant or the evaluation value becomes smaller than a predetermined threshold value, and adjusts the camera parameter. finish.

  For example, compared with the distance L2 between the calculated feature point P2C and the extracted feature point P2 in FIG. 4A, the area difference S3 between the area of the figure F3C and the area of the figure F3 in FIG. Is a large numerical value. Therefore, compared with the case where only the feature points are used, a change in the repetitive calculation can be grasped greatly, and the calibration accuracy can be improved.

  In addition, the area difference S1 between the area of the figure F1C and the area of the figure F1 is smaller than the area difference S3 between the area of the figure F3C and the area of the figure F3. Compared to the area difference S4 between the area of the figure F4C and the area of the figure F4, the area difference S2 between the area of the figure F2C and the area of the figure F2 is small. That is, the areas of the figures F1 to F8 change with different tendencies due to the error in the yaw angle at the mounting position of the camera 10. Therefore, by adjusting the camera parameters so that the sum of the area differences S1 to S8 becomes small, the error of the yaw angle at the mounting position of the camera 10 can be calibrated with higher accuracy.

  Similarly, even when there is an error in the pitch angle and roll angle of the mounting position of the camera 10, the areas of the figures F1 to F8 change with different tendencies, so that calibration can be performed with higher accuracy.

  FIG. 5 is a flowchart showing processing of the camera calibration device 20 of FIG. First, the feature point extraction unit 24 extracts a plurality of feature points P1 to P10 from the images of the calibration markers M1 and M2 captured by the camera 10 (S1). Next, the area calculation unit 26 calculates the areas of the graphics F1 to F8 defined by the extracted feature points P1 to P10 (S2). Next, the adjustment unit 28 adjusts camera parameters based on the coordinates of the extracted feature points P1 to P10 and the calculated areas of the graphics F1 to F8 (S3). Next, the adjustment unit 28 determines whether or not the evaluation value has converged (S4). If the evaluation value has not converged (N in S4), the process returns to S3. When the evaluation value has converged (Y in S4), the adjustment unit 28 ends the process.

  The adjusted camera parameters are stored in a storage unit (not shown) in the vehicle C1. Then, an image processing device (not shown) in the vehicle C1 corrects the image captured by the camera 10 using the stored camera parameters, and generates an image in which distortion due to an error in the mounting position of the camera 10 is corrected. To do. The corrected image is used for the driver to confirm the rear of the vehicle C1.

  As described above, according to the present embodiment, the feature amount used for adjusting the camera parameter is increased as compared with the case where only the coordinates of the feature point are used. In addition, the area changes differently from the feature points according to the error in the mounting position of the camera 10. Further, the amount of change in the area due to the error in the mounting position of the camera 10 is larger than the amount of change in the coordinates of the feature points due to the error in the mounting position of the camera 10. Therefore, the accuracy of calibration can be improved as compared with the case where only feature points are used.

  Therefore, a smaller calibration marker can be used without lowering the calibration accuracy as compared with the case of using only feature points.

  The area of the set of figures F1 and F2 arranged in the first direction y and the set of figures arranged in the second direction x due to at least one of the pitch angle, the yaw angle, and the roll angle of the camera 10 are shifted. The areas of F3 and F4 change with different tendencies. Similarly, the area of the set of figures F5 and F6 and the area of the set of figures F7 and F8 also change with different tendencies. Therefore, it is possible to calibrate the influence of errors in the pitch angle, yaw angle, and roll angle at the mounting position of the camera 10 with higher accuracy.

  In the above, this invention was demonstrated based on embodiment. This embodiment is an exemplification, and it is understood by those skilled in the art that various modifications can be made to each of those constituent elements or combinations of the respective treatment processes, and such modifications are also within the scope of the present invention. is there.

For example, one calibration marker may be used. In this case, although the calibration accuracy is lower than that in the above-described embodiment, the installation space for the calibration marker can be reduced.
Also, three or more calibration markers may be used. In this case, although the calibration accuracy is improved as compared with the above-described embodiment, the installation space for the calibration marker is increased.

In addition, the specific pattern of the calibration marker is not particularly limited as long as three or more feature points that define at least one figure can be extracted from one calibration marker. When the camera parameter is adjusted based on the area of one figure, the calibration accuracy is lower than in the above-described embodiment, but the camera parameter adjustment time may be shorter.
In addition, the figure may be a polygon other than a triangle, or may have a different shape.

  Further, the first evaluation value may be a value based on the distances L1 to L10 illustrated in FIG. 4A, and may be an average of the distances L1 to L10. The second evaluation value may be a value based on the area differences S1 to S8, and may be an average of the area differences S1 to S8.

  Furthermore, the position where the camera 10 is attached may be the front part or the side part of the vehicle C1. Further, the camera 10 attached to a device other than the vehicle C1 may be calibrated by the camera calibration device 20.

One embodiment of the present invention is as follows.
[Item 1]
A feature point extraction unit that extracts a plurality of feature points from a calibration marker image captured by a camera;
An area calculation unit for calculating an area of a figure defined by the feature points extracted by the feature point extraction unit;
An adjustment unit that adjusts camera parameters for calibrating the camera based on the coordinates of the feature points extracted by the feature point extraction unit and the area of the figure calculated by the area calculation unit;
A camera calibration device comprising:
According to this aspect, the area changes differently from the feature point according to the camera mounting position error, and the amount of change in the area due to the camera mounting position error is the coordinate of the feature point due to the camera mounting position error. Since it is larger than the amount of change, the accuracy of calibration can be improved.

[Item 2]
Item 1 is characterized in that the area calculation unit calculates an area of a set of figures arranged in a first direction and an area of a set of figures arranged in a second direction intersecting the first direction. The camera calibration device described.
In this case, the area of the set of figures arranged in the first direction and the area of the set of figures arranged in the second direction due to the shift of at least one of the pitch angle, the yaw angle, and the roll angle of the camera, Since they change with different tendencies, the effects of camera mounting position errors can be calibrated with higher accuracy.

[Item 3]
Extracting a plurality of feature points from a calibration marker image captured by a camera;
Calculating the area of the figure defined by the extracted feature points;
Adjusting camera parameters for calibrating the camera based on the coordinates of the extracted feature points and the calculated area of the figure;
A camera calibration method comprising:
According to this aspect, the area changes differently from the feature point according to the camera mounting position error, and the amount of change in the area due to the camera mounting position error is the coordinate of the feature point due to the camera mounting position error. Since it is larger than the amount of change, the accuracy of calibration can be improved.

10 camera, 20 camera calibration device, 24 feature point extraction unit, 26 area calculation unit, 28 adjustment unit.

Claims (3)

A feature point extraction unit that extracts a plurality of feature points from a calibration marker image captured by a camera;
An area calculation unit for calculating an area of a figure defined by the feature points extracted by the feature point extraction unit;
An adjustment unit that adjusts camera parameters for calibrating the camera based on the coordinates of the feature points extracted by the feature point extraction unit and the area of the figure calculated by the area calculation unit;
A camera calibration device comprising:   2. The area calculation unit calculates an area of a set of figures arranged in a first direction and an area of a set of figures arranged in a second direction intersecting the first direction. The camera calibration device described in 1. Extracting a plurality of feature points from a calibration marker image captured by a camera;
Calculating the area of the figure defined by the extracted feature points;
Adjusting camera parameters for calibrating the camera based on the coordinates of the extracted feature points and the calculated area of the figure;
A camera calibration method comprising:

Images