WO2017130823A1

WO2017130823A1 - Camera calibration device and camera calibration method

Info

Publication number: WO2017130823A1
Application number: PCT/JP2017/001637
Authority: WO
Inventors: 将由道口
Original assignee: パナソニックＩｐマネジメント株式会社
Priority date: 2016-01-29
Filing date: 2017-01-19
Publication date: 2017-08-03
Also published as: CN108496354A; CN108496354B; JP2017135680A; JP6688989B2

Abstract

A camera calibration device provided with a feature point extraction unit, an area calculation unit, and an adjustment unit. The feature point extraction unit extracts a plurality of feature points from an image of a marker for calibration that is captured by a camera. The area calculation unit calculates the area of a diagram defined by the extracted feature points. The adjustment unit adjusts, on the basis of the coordinates of the extracted feature points and the calculated area of the diagram, a camera parameter for calibrating the camera.

Description

Camera calibration apparatus and camera calibration method

The present disclosure relates to a camera calibration apparatus and a camera calibration method for calibrating a camera.

A technique is known in which a camera attached to a vehicle is used to image the surroundings of the vehicle and driving assistance is performed using the captured images. In such a technique, a deviation from a design value occurs in a captured image due to an error in the mounting position of the camera and a manufacturing error in the camera itself. For this reason, it is necessary to calibrate the camera mounting position error in advance in a factory or the like. At the time of calibration, the calibration marker is imaged by the camera, the feature points are extracted from the captured calibration marker image, and the camera parameters for calibrating the camera are adjusted based on the coordinates of the extracted feature points. (For example, refer to Patent Document 1). The larger the occupation ratio of the calibration marker image in the captured image and the more uniformly and densely distributed feature points, the more the calibration accuracy can be improved.

JP 2011-155687 A

The camera calibration device according to an aspect of the present disclosure includes a feature point extraction unit, an area calculation unit, and an adjustment unit. The feature point extraction unit extracts a plurality of feature points from the calibration marker image captured by the camera. The area calculation unit calculates the area of the figure defined by the feature points extracted by the feature point extraction unit. The adjustment unit adjusts camera parameters for calibrating the camera based on the coordinates of the feature points extracted by the feature point extraction unit and the area of the graphic calculated by the area calculation unit.

Another aspect of the present disclosure is a camera calibration method. This method extracts a plurality of feature points from an image of a calibration marker imaged by a camera, calculates an area of a figure defined by the extracted feature points, and coordinates of the extracted feature points And adjusting camera parameters for calibrating the camera based on the calculated area of the figure.

According to the present disclosure, the calibration accuracy of the camera can be improved.

FIG. 1A is a diagram illustrating a positional relationship between a vehicle and a calibration marker at the time of calibration according to an embodiment. FIG. 1B is a plan view showing an example of the calibration marker of FIG. 1A. FIG. 2 is a block diagram showing a schematic configuration of the camera calibration apparatus of FIG. 1A. FIG. 3A is a diagram illustrating an image of a calibration marker in an ideal case where there is no error in the camera mounting position. FIG. 3B is a diagram illustrating an image of the calibration marker in the case where there is an error in the yaw angle at the camera mounting position. FIG. 4A is a diagram showing the feature points of FIG. 3A and FIG. 3B in an overlapping manner. FIG. 4B is a diagram in which the feature points and figures of FIGS. 3A and 3B are overlapped. FIG. 5 is a flowchart showing the processing of the camera calibration apparatus of FIG. 1A.

Prior to the description of the embodiment of the present disclosure, problems in the conventional camera calibration apparatus will be briefly described. In camera calibration, it is desired to use a small calibration marker because of the limited installation space. The smaller the calibration marker, the smaller the number of feature points that can be extracted, and the feature points are concentrated in a part of the captured image. In this case, the movement of the feature point due to the error of the camera mounting position is not easily affected by the lens distortion and is close to the parallel movement, so that the calibration accuracy is lowered.

The present disclosure has been made in view of such circumstances, and the present disclosure provides a technique that can improve the calibration accuracy of a camera.

FIG. 1A is a diagram illustrating a positional relationship between the vehicle C1 and calibration markers M1 and M2 during calibration according to an embodiment, and FIG. 1B is a plane illustrating an example of the calibration markers M1 and M2 in FIG. 1A. FIG. In FIG. 1A, the surroundings of the vehicle C1 when the camera 10 is calibrated in a production factory of the vehicle C1 are viewed from above. The vehicle C <b> 1 includes a camera 10 and a camera calibration device 20. The camera 10 is attached to a back door or the like at the rear of the vehicle C1, and images the rear of the vehicle C1. The camera 10 may be attached near the center axis of the vehicle, or may be attached offset from the center axis of the vehicle. The camera calibration device 20 adjusts camera parameters for calibrating the camera 10.

The calibration markers M1 and M2 are arranged substantially perpendicular to the floor surface at a predetermined position behind the vehicle C1 within the imaging range of the camera 10. The calibration markers M1 and M2 are arranged substantially symmetrically on both sides of the center axis of the vehicle.

As shown in FIG. 1B, each calibration marker M1, M2 has a checkered pattern in which 16 squares are arranged in a matrix. The camera 10 images this checkerboard pattern.

FIG. 2 is a block diagram showing a schematic configuration of the camera calibration device 20 of FIG. 1A. The camera calibration device 20 includes an image storage unit 22, a feature point extraction unit 24, an area calculation unit 26, and an adjustment unit 28.

The image storage unit 22 stores images of the calibration markers M1 and M2 captured by the camera 10.

The feature point extraction unit 24 extracts a plurality of feature points from the images of the calibration markers M1 and M2 stored in the image storage unit 22. The method for extracting the feature points is not particularly limited, but the feature point extraction unit 24 may extract the feature points using a pattern matching technique. When the pattern matching technique is used, the feature point extraction unit 24 scans the template for each feature point on the image, and extracts the feature point from the pattern on the image having a high degree of coincidence with the template.

The area calculation unit 26 calculates the area of the graphic defined by the feature points extracted by the feature point extraction unit 24. Each figure is a polygon having three or more feature points as vertices.

The adjustment unit 28 adjusts the camera parameters based on the feature point coordinates extracted by the feature point extraction unit 24 and the figure area calculated by the area calculation unit 26.

Detailed functions of the area calculation unit 26 and the adjustment unit 28 will be described later.

The configuration of the camera calibration device 20 can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer, and is realized by a program loaded in the memory in software. Describes functional blocks realized by collaboration. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms only by hardware, or by a combination of hardware and software.

Adjustment of camera parameters will be described with reference to FIGS. 3A and 3B and FIGS. 4A and 4B.

FIG. 3A is a diagram illustrating an image I1 of calibration markers M1 and M2 in an ideal case where there is no error in the attachment position of the camera 10, and FIG. 3B is a case where there is an error in the yaw angle of the attachment position of the camera 10. It is a figure which shows the image I2 of marker M1, M2 for calibration. The images I1 and I2 are captured by the camera 10. Hereinafter, an example of adjusting camera parameters in the case of FIG. 3B will be described.

4A is a diagram in which the feature points P1C to P10C and P1 to P10 of FIG. 3A and FIG. 3B are overlapped, and FIG. 4B is a diagram of the feature points P1C to P10C and P1 to P10 of FIG. 3A and FIG. It is a figure which shows F8C and F1-F8 in piles. In order to clarify the explanation, the calibration markers M1 and M2 are removed in FIGS. 4A and 4B.

As shown in FIGS. 3A and 3B, the image I1 has ten feature points P1C to P10C. In the image I2, there are ten feature points P1 to P10. The feature points P1 and P1C are located at the center of the calibration marker M1. The feature points P2 and P2C are located at the center of the four squares at the upper left of the calibration marker M1, and the feature points P3 and P3C are located at the center of the four squares at the upper right of the calibration marker M1. The feature points P4 and P4C are located at the center of the lower left four squares of the calibration marker M1, and the feature points P5 and P5C are located at the middle of the lower right four squares of the calibration marker M1. Since the relationship between the feature points P6 to P10, P6C to P10C and the calibration marker M2 is the same, the description thereof is omitted.

The figure F1 is a triangle having vertices at the feature points P1 to P3. The figure F2 is a triangle having feature points P1, P4, and P5 as vertices. The figure F3 is a triangle having the feature points P1, P2, and P4 as vertices. The figure F4 is a triangle having feature points P1, P3, and P5 as vertices.

The figure F5 is a triangle having vertices at the feature points P6 to P8. The figure F6 is a triangle having feature points P6, P9, and P10 as vertices. The figure F7 is a triangle having vertices at the feature points P6, P7, and P9. The figure F8 is a triangle having feature points P6, P8, and P10 as vertices.

Since the relationship between the figures F1C to F8C and the feature points P1C to P10C is the same, the description thereof is omitted.

As described above, the area calculation unit 26 in FIG. 2 includes the area of the pair of figures F1 and F2 arranged in the first direction (up and down direction) y and the second direction (left and right direction) in the image of the calibration marker M1. ) The area of a set of figures F3 and F4 arranged in x is calculated. Further, the area calculation unit 26 in the image of the calibration marker M2 has an area of a set of figures F5 and F6 arranged in the first direction y and an area of a set of figures F7 and F8 arranged in the second direction x. And are calculated. The second direction x intersects the first direction y.

Here, since the calibration markers M1 and M2 are installed at predetermined positions as described above, each of the calibration markers M1 and M2 in the reference coordinate system (world coordinate system) with reference to the vehicle C1 is used. The coordinates of the feature points are known. By calculating the coordinates of the known feature points in the reference coordinate system and the camera parameters by a known method, the coordinates of the feature points on the image can be calculated. Here, it is assumed that the coordinates of the feature points on the image calculated using the initial camera parameters are equal to the coordinates of the ideal feature points P1C to P10C in FIG. 3A.

If the camera parameter is an optimal value, the calculated coordinates of the feature points P1C to P10C are equal to the coordinates of the corresponding feature points P1 to P10 extracted from the captured image. As the camera parameter differs from the optimum value, the difference between the coordinates of the feature points P1 to P10 corresponding to the feature points P1C to P10C increases. Therefore, the adjustment unit 28 in FIG. 2 repeatedly adjusts the camera parameter so that the evaluation value becomes small, and brings the camera parameter close to the optimum value. The evaluation value is the sum of the first evaluation value and the second evaluation value.

The first evaluation value is the sum of the distances L1 to L10 shown in FIG. 4A. The distance L [i] (i is an integer of 1 to 10) is a distance between the calculated coordinates of the feature point P [i] C and the coordinates of the feature point P [i] extracted from the captured image. is there. In FIG. 4A, the distances L1 and L6 are not shown.

The second evaluation value is the sum of the area differences S1 to S8. The area difference S [j] (j is an integer of 1 to 8) is calculated from the area of the figure F [j] C calculated from the calculated feature points P1C to P10C and the feature points P1 to P10 extracted from the image. It is the difference from the calculated area of the figure F [j].

As a method for adjusting the camera parameters so that the evaluation value becomes small, a known method such as the steepest descent method can be used. 2 adjusts the camera parameters by determining that the evaluation value has converged, for example, when the evaluation value becomes substantially constant or when the evaluation value becomes smaller than a predetermined threshold value. finish.

For example, compared with the distance L2 between the calculated feature point P2C and the extracted feature point P2 in FIG. 4A, the area difference S3 between the area of the graphic F3C and the area of the graphic F3 in FIG. Therefore, compared with the case where only the feature points are used, a change in the repetitive calculation can be grasped greatly, and the calibration accuracy can be improved.

Also, the area difference S1 between the area of the figure F1C and the area of the figure F1 is smaller than the area difference S3 between the area of the figure F3C and the area of the figure F3. Compared to the area difference S4 between the area of the figure F4C and the area of the figure F4, the area difference S2 between the area of the figure F2C and the area of the figure F2 is small. That is, the areas of the figures F1 to F8 change with different tendencies due to the error of the yaw angle at the mounting position of the camera 10. Therefore, by adjusting the camera parameters so that the sum of the area differences S1 to S8 becomes small, the error of the yaw angle at the mounting position of the camera 10 can be calibrated with higher accuracy.

Similarly, even when there is an error in the pitch angle and roll angle of the mounting position of the camera 10, the areas of the figures F1 to F8 change with different tendencies, so that calibration can be performed with higher accuracy.

FIG. 5 is a flowchart showing processing of the camera calibration device 20 of FIG. 1A. First, the feature point extraction unit 24 extracts a plurality of feature points P1 to P10 from the images of the calibration markers M1 and M2 captured by the camera 10 (S1). Next, the area calculation unit 26 calculates the areas of the graphics F1 to F8 defined by the extracted feature points P1 to P10 (S2). Next, the adjusting unit 28 adjusts the camera parameters based on the coordinates of the extracted feature points P1 to P10 and the calculated areas of the graphics F1 to F8 (S3). Next, the adjustment unit 28 determines whether or not the evaluation value has converged (S4). If the evaluation value has not converged (N in S4), the process returns to S3. When the evaluation value has converged (Y in S4), the adjustment unit 28 ends the process.

The adjusted camera parameters are stored in a storage unit (not shown) in the vehicle C1. Then, an image processing device (not shown) in the vehicle C1 corrects the image captured by the camera 10 using the stored camera parameters, and generates an image in which distortion due to an error in the mounting position of the camera 10 is corrected. To do. The corrected image is used for the driver to confirm the rear of the vehicle C1.

Thus, according to the present embodiment, the feature amount used for adjusting the camera parameter is increased as compared with the case where only the coordinates of the feature point are used. In addition, the area changes differently from the feature points according to the error in the mounting position of the camera 10. Further, the amount of change in the area due to the error in the mounting position of the camera 10 is larger than the amount of change in the coordinates of the feature points due to the error in the mounting position of the camera 10. Therefore, the accuracy of calibration can be improved as compared with the case where only feature points are used.

Therefore, it is possible to use a smaller calibration marker without degrading the calibration accuracy as compared with the case where only the feature points are used.

The area of the set of figures F1 and F2 arranged in the first direction y and the set of figures arranged in the second direction x due to at least one of the pitch angle, the yaw angle, and the roll angle of the camera 10 are shifted. The areas of F3 and F4 change with different tendencies. Similarly, the area of the set of figures F5 and F6 and the area of the set of figures F7 and F8 also change with different tendencies. Therefore, it is possible to calibrate the influence of errors in the pitch angle, yaw angle, and roll angle at the mounting position of the camera 10 more accurately.

The present disclosure has been described based on the embodiments. This embodiment is an exemplification, and it is understood by those skilled in the art that various modifications can be made to each of those constituent elements or combinations of processing processes, and such modifications are also within the scope of the present disclosure. is there.

For example, one calibration marker may be used. In this case, although the calibration accuracy is lower than that in the above-described embodiment, the installation space for the calibration marker can be reduced.

Also, three or more calibration markers may be used. In this case, although the calibration accuracy is improved as compared with the above-described embodiment, the installation space for the calibration marker is increased.

Further, the specific pattern of the calibration marker is not particularly limited as long as three or more feature points that define at least one figure can be extracted from one calibration marker. When the camera parameter is adjusted based on the area of one figure, the calibration accuracy is lower than in the above-described embodiment, but the camera parameter adjustment time may be shorter.

In addition, the figure may be a polygon other than a triangle, or may have a different shape.

Furthermore, the first evaluation value may be a value based on the distances L1 to L10 shown in FIG. 4A, and may be an average of the distances L1 to L10. The second evaluation value may be a value based on the area differences S1 to S8, and may be an average of the area differences S1 to S8.

Furthermore, the position where the camera 10 is attached may be the front part or the side part of the vehicle C1. Further, the camera 10 attached to a device other than the vehicle C1 may be calibrated by the camera calibration device 20.

One aspect of the present disclosure is as follows.

[Item 1]
A feature point extraction unit that extracts a plurality of feature points from a calibration marker image captured by a camera;
An area calculation unit for calculating an area of a figure defined by the feature points extracted by the feature point extraction unit;
An adjustment unit that adjusts camera parameters for calibrating the camera based on the coordinates of the feature points extracted by the feature point extraction unit and the area of the figure calculated by the area calculation unit;
A camera calibration device.

According to this aspect, the area changes differently from the feature point according to the camera mounting position error, and the amount of change in the area due to the camera mounting position error is the coordinate of the feature point due to the camera mounting position error. Since it is larger than the amount of change, the accuracy of calibration can be improved.

[Item 2]
The camera according to item 1, wherein the area calculation unit calculates an area of a set of figures arranged in a first direction and an area of a set of figures arranged in a second direction intersecting the first direction. Calibration device.

In this case, the area of the set of figures arranged in the first direction and the area of the set of figures arranged in the second direction due to the shift of at least one of the pitch angle, the yaw angle, and the roll angle of the camera, Since they change with different tendencies, the effects of camera mounting position errors can be calibrated with higher accuracy.

[Item 3]
Extracting a plurality of feature points from a calibration marker image captured by a camera;
Calculating the area of the figure defined by the extracted feature points;
Adjusting camera parameters for calibrating the camera based on the coordinates of the extracted feature points and the calculated area of the figure;
A camera calibration method comprising:

The camera calibration apparatus and the camera calibration method according to the present disclosure can improve the calibration accuracy of the camera, and thus can be applied to the calibration of a camera mounted on a moving body such as an automobile.

DESCRIPTION OF SYMBOLS 10 Camera 20 Camera calibration apparatus 24 Feature point extraction part 26 Area calculation part 28 Adjustment part

Claims

A feature point extraction unit that extracts a plurality of feature points from a calibration marker image captured by a camera;
An area calculation unit for calculating an area of a figure defined by the feature points extracted by the feature point extraction unit;
An adjustment unit that adjusts camera parameters for calibrating the camera based on the coordinates of the feature points extracted by the feature point extraction unit and the area of the figure calculated by the area calculation unit;
A camera calibration device.
The area calculating unit calculates an area of the set of figures arranged in the first direction and an area of the set of figures arranged in the second direction intersecting the first direction. Camera calibration device.
Extracting a plurality of feature points from a calibration marker image captured by a camera;
Calculating the area of the figure defined by the extracted feature points;
Adjusting camera parameters for calibrating the camera based on the coordinates of the extracted feature points and the calculated area of the figure;
A camera calibration method comprising: