WO2013154085A1

WO2013154085A1 - Calibration method and device

Info

Publication number: WO2013154085A1
Application number: PCT/JP2013/060680
Authority: WO
Inventors: 盛彦坂野; 啓二佐藤
Original assignee: クラリオン株式会社
Priority date: 2012-04-09
Filing date: 2013-04-09
Publication date: 2013-10-17

Abstract

A calibration method used in a calibration device that converts around-vehicle images captured by a plurality of cameras into overhead images for synthesis, wherein one calibration index in which at least two parallel straight lines are drawn is installed in a common imaging area of two of the cameras, and at least three units of the calibration index installed on a flat surface around a vehicle are used to perform calibration. Therefore, the time and effort necessary for calibration such as the creation, transportation, and laying of a calibration chart are reduced, and high-precision camera calibration including the calibration of internal distortion and external parameters is performed.

Description

Calibration method and apparatus

The present invention relates to a calibration method and apparatus using a plurality of cameras.

By displaying an image of the rear of the vehicle captured by the in-vehicle camera on the in-vehicle monitor, the situation immediately behind the vehicle, which is a blind spot from the driver, is visually confirmed by the image displayed on the in-vehicle monitor, and visibility when the vehicle is moving backward is improved. Improvements are being made.

When displaying such an in-vehicle camera image on the in-vehicle monitor, in order to calibrate the mounting state of the in-vehicle camera on the vehicle, a calibration index is installed behind the vehicle, and the calibration index reflected on the in-vehicle monitor is displayed. While viewing the image, the mounting state of the in-vehicle camera is adjusted so that the image of the calibration index is appropriately reflected.

In addition, an image displayed on the in-vehicle monitor is appropriately calibrated by applying a predetermined arithmetic processing based on the image of the calibration index to the image obtained by the in-vehicle camera.

Furthermore, the entire periphery of the vehicle is photographed by a plurality of in-vehicle cameras, and the plurality of images obtained by each in-vehicle camera are converted into images (overhead images) that look down from directly above the vehicle, and between the images. A single viewpoint-converted composite image is also obtained by performing mapping with the position adjusted. In such a case, since it is necessary to perform alignment between two adjacent images with high accuracy, highly accurate calibration is required.

However, in the conventional calibration method, it is necessary to strictly determine the relative positional relationship between the calibration index and the vehicle. After the vehicle is installed, the calibration index is accurately set for the vehicle. Alternatively, after the calibration index is installed, it is necessary to accurately install the vehicle with respect to the calibration index.

For this reason, the vehicle production line has been devised to improve the alignment accuracy between the vehicle and the calibration index by modifying the equipment at high cost. Furthermore, if the calibration is performed again by the maintenance department of the sales / service company after it has been shipped from the production site (for repairs, retrofitting an in-vehicle camera, etc.), the calibration index must be set accurately each time. Since it is necessary to install it, it takes much time and labor.

Under such circumstances, there is a need for a calibration method that does not require relative installation accuracy between the vehicle and the calibration index.

JP 2012-015576 A JP 2008-187564 A

Patent Document 1 calibrates internal / distortion parameters and external parameters of a plurality of cameras by using features unrelated to the position of the vehicle such as linearity, parallelism, orthogonality, and spacing of white line grids. Since vehicle positioning is unnecessary and internal / distortion parameters can be calibrated, simple and highly accurate calibration is possible. However, the white line grid for calibration is used by drawing or laying directly on the plane under the vehicle, so if the vehicle is huge, such as a large construction machine, a large calibration chart is required. It takes time and effort to prepare for calibration such as chart creation, transportation and laying.

Patent Document 2 installs a calibration chart that is imaged in common between cameras, and calibrates the relative posture between the cameras via the calibration chart. Vehicle positioning is not required and simple calibration is possible. However, since calibration of internal / distortion parameters is not assumed, the calibration accuracy is insufficient.

In view of the above-described problems, the present invention provides a calibration method for a calibration apparatus that converts a vehicle surrounding image captured by a plurality of cameras into a bird's-eye view image and synthesizes the calibration index on which at least two parallel straight lines are drawn. One is provided in the common imaging area of the two cameras, and the calibration is performed using at least three or more calibration indices installed on a plane around the vehicle.

In addition, in a calibration device that converts a vehicle surrounding image captured by a plurality of cameras into a bird's-eye view image and combines them, a calibration index on which at least two parallel straight lines are drawn is placed in a common imaging area of the two cameras. And a calibration is performed using at least three or more calibration indicators installed on a plane around the vehicle.

This specification includes the contents described in the specification and / or drawings of Japanese Patent Application No. 2012-088598, which is the basis of the priority of the present application.

According to the present invention, ensuring high calibration accuracy derived from performing internal / strain parameter calibration, and realizing ease of preparation necessary for calibration, such as transportation, creation, and laying of a calibration chart, It is possible to provide a calibration method and apparatus that are compatible with each other.

1 is a block diagram illustrating a schematic configuration of an image calibration apparatus according to Example 1 which is an embodiment of the present invention. It is a schematic diagram which shows the vehicle carrying four cameras. It is a figure which shows an example of the parameter | index for calibration. It is a figure which shows an example of installation of the parameter | index for calibration shown in FIG. It is a figure which shows the image which image | photographed the peripheral area | region with each camera, S1 represents the imaging by a front camera, S2 imaged by the rear camera, S3 imaged by the right side camera, S4 represents the imaging area by the left side camera, respectively. It is a figure which shows the partial straight line extracted from each image. It is a figure which shows the image after distortion correction. It is a figure which shows the partial straight line after distortion correction. It is a figure which shows each image after overhead conversion. It is a figure which shows a bird's-eye view conversion synthetic | combination image. It is a figure which shows the example of the virtual straight line in the parameter | index for calibration. It is a figure which shows each image after bird's-eye view conversion by a default external parameter.

In the calibration method and apparatus according to the present invention, a partial straight line placed on a road surface as a calibration index is photographed by each of a plurality of vehicle-mounted cameras, and the obtained plurality of images are distorted using default internal parameters, respectively. Correction is performed by adjusting the internal parameters so that the partial straight line image of the calibration index forms a straight line shape in the distortion correction image (or a viewpoint conversion composite image described later) obtained by correcting the distortion. The new distortion-corrected image obtained using the internal parameters is converted into a single viewpoint conversion composite image using the default external parameters, and the partial straight line image of the calibration index in the viewpoint conversion composite image is As with the actual calibration index line, the calibration index is imaged between different images so that the distance between the lines is aligned to be parallel. As to, and corrects the external parameters. It does not require rigorous relative positional relationship between calibration indicators installed on the road surface and vehicles placed on the road surface, and can be transported by humans even when the vehicle is very large, such as construction machinery. A small calibration index can be used, and the burden of preparation necessary for calibration can be reduced.

That is, in the calibration method of the present invention, a peripheral area of the vehicle including a calibration index provided on a road surface on which the vehicle is installed is respectively photographed by a plurality of cameras installed on the vehicle, Optical characteristics of the camera with respect to the peripheral area image captured by each of them (for example, a shift between the optical axis of the image sensor of the vehicle-mounted camera and the optical axis of the lens, or an image by the lens (image formed on the image sensor)) Single viewpoint conversion obtained by looking down from above the vehicle with a plurality of distortion correction images obtained by performing distortion correction based on internal parameters corresponding to distortion (aberration, etc.) The camera is attached to the vehicle (a vehicle, for example, a bird's-eye view image that is an image when looking down vertically from an upper position of the vehicle). Viewpoint conversion processing of the plurality of distortion-corrected images based on external parameters corresponding to the relative positional relationship (three-dimensional spatial coordinate position) and posture (angle of the optical axis around each three-dimensional axis), In the viewpoint conversion process, in the image calibration method for calibrating the viewpoint conversion composite image by adjusting the external parameter based on the image of the calibration index in the viewpoint conversion composite image, the calibration is performed. The index for use includes a sequence of points arranged on at least two straight lines that are parallel to each other, and the calibration index includes two cameras (for example, a front camera and a right side camera) that are adjacent to each other in an imaging region among a plurality of cameras. The right side camera and the rear camera) are set in a common imaging area so that the calibration index is set. When the slope of the straight line formed by the point sequence of the calibration index is taken simultaneously in one camera (for example, the calibration index installed in the common area of the front camera and the right side camera, and the front camera And calibration indices installed in a common area of the left side camera) are set different from each other, and for each of the plurality of distortion correction images, a sequence of points included in each distortion correction image, Correction is performed to adjust the internal parameter so as to form a straight line, and a new distortion-converted image obtained from the corrected internal parameter is obtained using the new distortion-corrected image. The correction for adjusting the external parameter is performed so that the new viewpoint-converted composite image satisfies the following conditions (1) to (3).

(1) Images corresponding to the two parallel lines are parallel to each other.

(2) The distance between the images corresponding to the two parallel lines is a known distance between the point sequences of the calibration index.

(3) The images of the calibration index captured by the two cameras adjacent to the imaging area match.

Here, as the calibration index, specifically, one having two parallel straight lines having a certain width on a rectangular plate can be applied. For example, the rectangular plates are installed so that the inclinations of the straight lines are orthogonal to each other as much as possible between the calibration indices photographed simultaneously in one camera.

However, the calibration method according to the present invention is not limited to the calibration index of this form, and corresponds to at least a point sequence arranged on two lines parallel to each other in each vehicle peripheral image. It is sufficient that each of the images is included.

In the first image calibration method of the present invention, the distortion correction image obtained by using the default internal parameters is set so that the partial straight line image of the calibration index in the distortion correction image is a straight line. Since the parameters are corrected, distortion correction suitable for variations in optical characteristics of individual cameras and individual differences can be performed.

Furthermore, using a new distortion-corrected image obtained based on the internal parameters corrected in this way (the partial straight line image of the calibration index is a straight line), a viewpoint based on the default external parameters is used. A converted composite image is generated. Since the generated viewpoint-converted composite image has a high accuracy of the distortion-corrected image as a base, the accuracy of the viewpoint-converted composite image is inevitably high.

The above three conditions (1) to (3) do not depend on the relative positional relationship between the vehicle and the calibration index, but only the relationship between the partial straight line images of the calibration index in the viewpoint conversion composite image. (Parallelity, spacing, and image coincidence) are defined, and therefore, the vehicle position and vehicle attitude (orientation) with respect to the calibration index when the vehicle is installed with respect to the calibration index are not required.

That is, the relative positional relationship between the calibration index installed on the road surface and the vehicle placed on the road surface is the position of the vehicle obtained by a normal parking operation in a parking frame in a general parking lot or the like, The accuracy of the posture (a degree having a practical level deviation) is sufficient, and the strict degree of the relative positional relationship can be relaxed.

This means that even if the type of vehicle (vehicle size and shape) is different, a common calibration index can be used as it is. Therefore, each time a calibration is performed, a dedicated calibration index is provided for each type of vehicle. This eliminates the need for preparation and installation, reduces the work effort, and greatly reduces the work time.

Furthermore, since the calibration index with a partial straight line drawn on the rectangular plate can be used, the calibration index can be reduced to a size that is easy for an operator to carry even when the vehicle is huge like a construction machine. It is possible to reduce the labor required for preparing the calibration index.

Regarding internal / distortion parameter estimation, there are cases where there is not necessarily an internal / distortion parameter for linearizing all line images simultaneously, but in such a case, the linearity of all line images is comprehensive. The internal / distortion parameters may be corrected by performing general evaluation (a state in which linearity is ensured in the most balanced manner as a whole).

Similarly, with respect to external parameter estimation, it may not be possible to adjust to an external parameter that is in a completely parallel state (a state in which the slope values are exactly the same) between corresponding images. And external parameters may be adjusted by comprehensive evaluation of the orthogonality (as a whole, in a state where parallelism and orthogonality are ensured in the most balanced manner).

Hereinafter, embodiments of an image calibration apparatus according to the present invention will be described with reference to the drawings.

[Example 1]
FIG. 1 is a block diagram showing a configuration of an image calibration apparatus 100 according to the first embodiment which is an embodiment of the present invention.

The illustrated calibration device 100 is mounted on a vehicle 200 shown in FIG. For example, four calibration indexes 20 shown in FIG. 3 are arranged around the vehicle 200 as shown in FIG. A front camera (front camera) 11, a rear camera (rear camera) 12, a right side camera (right side camera) 13, and a left side camera (left side camera) 14 are mounted on a vehicle as shown in FIG.

The front camera 11, the rear camera 12, the right side camera 13, and the left side camera 14 photograph the surrounding areas R1, R2, R3, and R4 of the vehicle 200 shown in FIG. 2 as surrounding area images S1, S2, S3, and S4, respectively. . FIG. 5 shows S1, S2, S3, and S4. S1 to S4 are fish-eye cameras for photographing at a wide angle, and each image is distorted as shown in FIG.

In S1, a calibration index 33 and a calibration index 34 are imaged. In S2, the calibration index 23 and the calibration index 24 are imaged. In S3, the calibration index 33 and the calibration index 23 are imaged. In S4, the calibration index 34 and the calibration index 24 are imaged.

The partial straight line extraction unit 60 extracts a feature amount used for calibration from each of S1, S2, S3, and S4 from a calibration index. From S1, the partial straight line F1 is extracted as the feature amount of the calibration index 33 and the calibration index 34. From S2, the partial straight line F2 is extracted as the feature quantity of the calibration index 23 and the calibration index 24. From S3, the partial straight line F3 is extracted as the feature quantity of the calibration index 33 and the calibration index 23. From S4, the partial straight line F4 is extracted as the feature quantity of the calibration index 34 and the calibration index 24. FIG. 6 shows F1, F2, F3, and F4.

The internal / distortion parameter estimator 51 uses the partial straight lines F1, F2, F3, and F4 to determine internal parameters M (a plurality of cameras (front camera 11, rear camera 12, right side camera 13, left side camera 14)). The internal parameter M1 of the front camera 11, the internal parameter M2 of the rear camera 12, the internal parameter M3 of the right side camera 13, and the internal parameter M4 of the left side camera 14) are estimated. The internal parameter is a numerical value representing the optical properties of each camera, such as the optical axis position, focal length, and distortion of the camera.

The distortion correction processing unit 30 shoots images of a plurality of cameras (front camera 11, rear camera 12, right side camera 13, left side camera 14) with a camera without distortion using internal parameters M1, M2, M3, and M4. Convert to a video. By correcting the distortion, the curved partial straight lines F1, F2, F3, and F4 are converted into linear partial straight lines F1, F2, F3, and F4. FIG. 7 shows images T1, T2, T3, and T4 after converting S1, S2, S3, and S4, respectively, and FIG. 8 also shows corrected partial straight lines P1, P2, P3, and P4.

The external parameter estimator 52 uses the partial straight lines P1, P2, P3, and P4 converted into the straight line shape to external the plurality of cameras (front camera 11, rear camera 12, right side camera 13, left side camera 14). Parameters N (external parameter N1 of front camera 11, external parameter N2 of rear camera 12, external parameter N3 of right side camera 13, external parameter N4 of left side camera 14) are corrected. The external parameter is a numerical value that represents the position and orientation of the camera.

The calibration unit 50 includes an internal / distortion parameter estimation unit 51 and an external parameter estimation unit 52.

The viewpoint conversion processing unit 40 uses the external parameters N1, N2, N3, and N4 of the plurality of cameras (front camera 11, rear camera 12, right side camera 13, and left side camera 14) to use the plurality of cameras (front camera 11). The rear camera 12, the right side camera 13, and the left side camera 14) are converted into overhead images Q1, Q2, Q3, and Q4 looked down from above to generate a composite image Q10. FIG. 9 shows Q1, Q2, Q3, and Q4, and FIG. 10 shows Q10.

Here, the calibration index 20 includes, for example, a first straight line L1 and a second straight line L2 drawn on the rectangular plate 25 and parallel to each other, as shown in FIG. The first straight line L1 and the second straight line L2 are parallel to each other, and the distance between the first straight line L1 and the second straight line L2 is set to a known W.

The calibration index includes a common region R13 of the peripheral regions R1 and R3 of the vehicle 200, a common region R14 of the peripheral regions R1 and R4 of the vehicle 200, a common region R23 of the peripheral regions R2 and R3 of the vehicle 200, and the periphery of the vehicle 200 It is installed in the common area R24 of the areas R2 and R4. Note that the index set in the common area R13 is the calibration index 33, the index set in the common area R14 is the calibration index 34, and the index set in the common area R23 is set in the calibration index 23 and the common area R24. The index is the calibration index 24.

Here, the image of the partial straight lines L1 and L2 of the calibration index 33 and the images of the partial straight lines L1 and L2 of the calibration index 34 are shown in the peripheral area image S1. Here, when the calibration index is placed on the ground, for example, the inclinations are orthogonal so that the partial straight lines L1 and L2 of the calibration index 33 and the partial straight lines L1 and L2 of the calibration index 34 do not match. Install in.

Similarly, the peripheral area image S2 includes images of partial straight lines L1 and L2 of the calibration index 23 and images of partial straight lines L1 and L2 of the calibration index 24. Here, when the calibration index is placed on the ground, for example, the inclinations are orthogonal so that the partial straight lines L1 and L2 of the calibration index 23 and the partial straight lines L1 and L2 of the calibration index 24 do not match. Install in.

Similarly, the peripheral area image S3 includes the images of the partial straight lines L1 and L2 of the calibration index 33 and the images of the partial straight lines L1 and L2 of the calibration index 23. Here, when the calibration index is installed on the ground, for example, the inclinations are orthogonal so that the partial straight lines L1 and L2 of the calibration index 33 and the partial straight lines L1 and L2 of the calibration index 23 do not match. Install in.

Similarly, the peripheral area image S4 includes images of the partial straight lines L1 and L2 of the calibration index 34 and images of the partial straight lines L1 and L2 of the calibration index 24. Here, when the calibration index is placed on the ground, for example, the inclinations are orthogonal so that the partial straight lines L1 and L2 of the calibration index 34 and the partial straight lines L1 and L2 of the calibration index 24 do not match. Install in.

The calibration index only needs to include point sequences FP1, FP2, FP3 and FP4, FP5, FP6 arranged in parallel with each other as in the calibration index 120 shown in FIG. The virtual straight line L11 formed by the feature points FP1, FP2, and FP3 and L12 formed by the feature points FP4, FP5, and FP6 may be regarded as the same as the partial straight lines L1 and L2 and set as calibration indices. .

Further, the distance W1 between the center of the feature point FP1 and the center of the feature point FP2, the distance W2 between the center of the feature point FP2 and the center of the feature point FP3, the distance W3 between the center of the feature point FP4 and the center of the feature point FP5, and the feature point FP5. The interval W4 between the center of the image and the center of the feature point FP6 may be a known interval and may be a constraint condition during external parameter calibration described later.

In addition, the calibration unit 50 makes the images corresponding to the correction partial straight lines P1 to P4 extend linearly for each of the four distortion correction partial straight lines P1, P2, P3, and P4. An internal / distortion parameter correction unit 51 (internal parameter correction unit) that performs correction to adjust the internal parameter M, and a new distortion correction partial straight line obtained by the distortion correction processing unit 30 using the corrected internal parameter M When a new viewpoint transformation partial straight line Q10 is obtained using P1 to P4, the external parameter N is set so that the obtained new viewpoint transformation partial straight line Q10 satisfies the following conditions (1) to (3). An external parameter estimation unit 52 (external parameter correction unit) that performs correction to be adjusted is included.

(1) The images corresponding to the two partial straight lines L1 and L2 parallel to each other are parallel.

(2) The distance between the images corresponding to the two partial straight lines parallel to each other is a known distance W between the lines of the calibration index 20 (3) The partial straight line image corresponding to the calibration index is a viewpoint conversion composition. Match on partial line Q10.

Next, the operation of the calibration apparatus 100 of this embodiment will be described.

First, as shown in FIG. 2, in the vicinity of a vehicle 200 in which a front camera 11 is installed at the front, a rear camera 12 is installed at the rear, a right side camera 13 is installed on the right side, and a left side camera 14 is installed on the left side. 4 are arranged on a flat road surface as shown in FIG.

The arrangement state of the calibration indices 33, 34, 23, and 24 with respect to the arrangement state of the vehicle 200 is arranged as long as it is included in the common imaging regions R13, R14, R23, and R24 of the cameras shown in FIG. May be. However, the inclination of the partial straight lines L1 and L2 of the calibration index 33 included in the peripheral area image R1 captured by the front camera 11 of the peripheral area R1 is different from the inclination of the partial straight lines L1 and L2 of the calibration index 34. , Place these. Similarly, the inclination of the partial straight lines L1 and L2 of the calibration index 23 included in the peripheral area image R2 captured by the rear camera 12 of the peripheral area R2 and the inclination of the partial straight lines L1 and L2 of the calibration index 24 are different. , Place these. The inclination of the partial straight lines L1 and L2 of the calibration index 33 included in the peripheral area image R3 captured by the right side camera 13 of the peripheral area R3 and the inclination of the partial straight lines L1 and L2 of the calibration index 23 are different from each other. Arrange these. The inclinations of the partial straight lines L1 and L2 of the calibration index 34 included in the peripheral area image R4 captured by the left side camera 14 of the peripheral area R4 and the inclinations of the partial straight lines L1 and L2 of the calibration index 24 are different. Arrange these. Therefore, for example, the calibration indicators 34 and 23 are installed so that the partial straight line is parallel to the vehicle width direction, and the calibration indicators 33 and 23 are installed so that the partial straight line is parallel to the vehicle length direction. This is intended to improve the accuracy of internal parameter calibration by including straight lines in two directions in the image.

In the present embodiment, the case of the arrangement state shown in FIG. 4 will be described below. However, even in the arrangement state other than this arrangement state, the action and effect exhibited thereby are the case of the arrangement state of FIG. It is the same.

As shown in FIG. 4, each camera (front camera 11, rear camera 12, right side camera 13, left side camera 14) of the calibration apparatus 100 is in a state where the vehicle 200 is placed on the calibration index. By capturing each of the peripheral areas R1 to R4 including a part of the image and a part of the calibration index, each camera (front camera 11, rear camera 12, right side camera 13, left side camera 14) Peripheral area images S1, S2, S3, and S4 (FIG. 5) corresponding to R2, R3, and R4, respectively, are obtained. The peripheral area images S1, S2, S3, and S4 obtained by each camera (front camera 11, rear camera 12, right side camera 13, and left side camera 14) are images of calibration indices 33, 34, 23, and 24, respectively. It is included.

Specifically, the peripheral area image S1 includes images of partial straight lines L1 and L2 of the calibration index 33 and partial straight lines L1 and L2 of the calibration index 34, and the peripheral area image S2 includes The images of the partial straight lines L1 and L2 of the calibration index 23 and the images of the partial straight lines L1 and L2 of the calibration index 24 are reflected. Further, the peripheral area image S3 includes the images of the partial straight lines L1 and L2 of the calibration index 33. Then, the images of the partial straight lines L1 and L2 of the calibration index 23 are shown, and the peripheral area image S4 includes the images of the partial straight lines L1 and L2 of the calibration index 34 and the images of the partial straight lines L1 and L2 of the calibration index 24. Is reflected.

These peripheral region images S1 to S4 are input to the distortion correction processing unit 30, and the distortion correction processing unit 30 performs distortion correction on each of the input peripheral region images S1 to S4 using a default internal parameter M. As a result, each of the peripheral area images S1 to S4 becomes a distortion corrected image in which the distortion is corrected to some extent.

Here, the default internal parameter M is a design value that is uniformly set ignoring the individual differences in the optical characteristics of the camera, so that the ideal optical parameter of the camera is not different from the design value. In some cases, the distortion correction by the default internal parameter M will cause the distortion corrected image to be completely corrected for distortion.

However, actual cameras have individual differences in their optical characteristics due to design tolerances that take into account manufacturing errors. Therefore, the internal parameter M used for distortion correction should be different for each individual camera. However, since the distortion correction processing unit 30 initially performs distortion correction using the default internal parameter M, the obtained distortion-corrected image is not one in which distortion is completely corrected.

Therefore, in the calibration device 100 of the present embodiment, the internal / distortion parameter correction unit 51 of the calibration unit 50 corrects the default internal parameter M of the distortion correction processing unit 30.

The correction of the internal parameter M by the internal / distortion parameter correction unit 51 is performed by sequentially changing the value of the internal parameter M to adjust the calibration index lines H1, H2, V1, and V2 included in each distortion corrected image S. This is a process for converging the images H1, H2, V1, and V2 into linearly extending images.

This process is technically realized by designing an error function and performing optimization so as to minimize the error. The error function shall evaluate the linearity of the line. For example, it can be realized by using a spherical projection method. In the spherical projection method, a captured straight line is projected onto a virtual sphere. If the straight line has an original linear shape, it forms part of a great circle on the spherical surface. If a straight line is not formed due to distortion or the like, the projected straight line will not be part of the great circle. Therefore, the degree of linearity is determined by using the degree to which each straight line on the spherical surface forms part of a great circle as an error function. In this case, the error function is the orthogonality of the vector of feature points on the straight line and the normal vector of the surface formed by the feature vector group composed of other feature points on the straight line. By partial differentiation of this error function with respect to each internal parameter, the amount of change of the error function for each internal parameter can be obtained, and the value of the internal parameter is updated based on this amount of change (for example, the steepest descent method, etc. ). By repeating this process until the error function does not decrease, the internal parameter M is obtained.

Then, each camera when the calibration index lines H1, H2, V1, and V2 included in each distortion-corrected image S are converged to linearly extending images H1, H2, V1, and V2, respectively. The internal parameters M1 to M4 respectively corresponding to 11 to 14 are corrected internal parameters.

That is, the internal parameters M1 to M4 are internal parameters for each of the cameras 11 to 14 having individual differences in optical characteristics, and the distortion correction processing unit 30 performs distortion correction by the internal parameters M1 to M4 for each of the cameras 11 to 14. By performing the above, each of the peripheral area images S1 to S4 becomes the distortion corrected images T1 to T4 in which the distortion is accurately corrected as shown in FIG.

Next, the obtained distortion correction images T1 to T4 are input to the viewpoint conversion processing unit 40, and the viewpoint conversion processing unit 40 performs viewpoint conversion processing on the input distortion correction image according to the default external parameter N. The single viewpoint conversion partial images Q1 to Q4 are generated (see FIG. 12).

Here, in the distortion corrected images T1 to T4, the correction index lines H1, H2, V1, and V2 have already been corrected so that the images H1, H2, V1, and V2 are linear. The calibration index lines H1, H2, V1, and V2 images H1, H2, V1, and V2 in the respective viewpoint conversion partial images Q1 to Q4 generated based on the distortion corrected images T1 to T4 are also linear.

However, since the external parameter N used in the viewpoint conversion processing by the viewpoint conversion processing unit 40 is a design value that is uniformly set ignoring individual differences in the mounting state of each camera with respect to the vehicle 200, When the mounting state is ideal so as not to differ from the design value, the calibration index lines H1, H2, V1, V2 in the viewpoint conversion composite image Q10 obtained by the viewpoint conversion processing with the default external parameter N are used. Like the lines H1, H2, V1, and V2 in the actual calibration index, the images H1, H2, V1, and V2 are parallel to each other and the distance between the lines is equal to a known value as shown in FIG. In addition, in the viewpoint conversion images Q1 to Q4, the line image that is included in common in the two images should be at the same position on the image.

However, since the actual camera and the vehicle 200 are formed with design tolerances and the like that take into account manufacturing errors, the mounting state such as the mounting position and mounting posture of the camera with respect to the vehicle 200 is naturally determined. While there are individual differences.

Therefore, the external parameter N used for the viewpoint conversion / combination processing should be different for each individual camera.

However, since the viewpoint conversion processing unit 40 initially performs the viewpoint conversion processing with the default external parameter N, the obtained viewpoint conversion images Q1 to Q4 are included therein as shown in FIG. The images H1, H2, V1, and V2 of the calibration index lines H1, H2, V1, and V2 are not parallel to each other, the distance between the images corresponding to the lines is inappropriate, or are originally in the same position. The image of the line that should be may not be at the same position.

Therefore, in the calibration apparatus 100 of the present embodiment, the external parameter estimation unit 52 of the calibration unit 50 corrects the default external parameter N of the viewpoint conversion processing unit 40.

The correction of the external parameter N by the external parameter estimator 52 is performed by sequentially changing the value of the external parameter N, and images of the calibration index lines H1, H2, V1, and V2 included in the viewpoint conversion images Q1 to Q4. H1, H2, V1, V2 parallelism (condition (1)), appropriateness of distance between lines (condition (2)), and processing to converge to satisfy the same position (condition (3)) It is.

That is, the external parameter estimation unit 52 specifically sets the condition (2) as the condition (2) so as to satisfy the conditions (1-1) to (1-8). Each external parameter N is corrected so as to satisfy the above conditions (3), specifically, the conditions (3-1) to (3-4) so as to satisfy 1) to (2-8).

That is, the external parameter estimation unit 52
(1-1) The images V1 and V2 respectively corresponding to the two parallel lines V1 and V2 in the viewpoint converted image Q1 are parallel to each other.
(1-2) The images H1 and H2 respectively corresponding to the two parallel lines H1 and H2 in the viewpoint-converted image Q1 are parallel to each other.
(1-3) The images V1 and V2 respectively corresponding to the two parallel lines V1 and V2 in the viewpoint converted image Q2 are parallel to each other.
(1-4) The images H1 and H2 respectively corresponding to the two parallel lines H1 and H2 in the viewpoint-converted image Q2 are parallel to each other.
(1-5) The images V1 and V2 respectively corresponding to the two parallel lines V1 and V2 in the viewpoint converted image Q3 are parallel to each other.
(1-6) The images H1 and H2 respectively corresponding to the two parallel lines H1 and H2 in the viewpoint converted image Q3 are parallel to each other.
(1-7) The images V1 and V2 respectively corresponding to the two parallel lines V1 and V2 in the viewpoint-converted image Q4 are parallel to each other.
(1-8) The images H1 and H2 corresponding to the two parallel lines H1 and H2 in the viewpoint-converted image Q4 are parallel to each other.
(2-1) The distance between the images V1 and V2 corresponding respectively to the two parallel lines V1 and V2 in the viewpoint conversion image Q1 is the known distance W between the lines L1 and L2 in the actual calibration index 20. So that
(2-2) The distance between the images H1 and H2 respectively corresponding to the two parallel lines H1 and H2 in the viewpoint conversion image Q1 is the known distance W between the lines L1 and L2 in the actual calibration index 20 So that
(2-3) The distance between the images V1 and V2 respectively corresponding to the two parallel lines V1 and V2 in the viewpoint-converted image Q2 is the known distance W between the lines L1 and L2 in the actual calibration index 20. So that
(2-4) The distance between the images H1 and H2 respectively corresponding to the two parallel lines H1 and H2 in the viewpoint converted image Q2 is the known distance W between the lines L1 and L2 in the actual calibration index 20. So that
(2-5) The distance between the images V1 and V2 respectively corresponding to the two parallel lines V1 and V2 in the viewpoint conversion image Q3 is the known distance W between the lines L1 and L2 in the actual calibration index 20. So that
(2-6) The distance between the images H1 and H2 respectively corresponding to the two parallel lines H1 and H2 in the viewpoint converted image Q3 is the known distance W between the lines L1 and L2 in the actual calibration index 20. So that
(2-7) The distance between the images V1 and V2 respectively corresponding to the two parallel lines V1 and V2 in the viewpoint-converted image Q4 is the known distance W between the lines L1 and L2 in the actual calibration index 20. So that
(2-8) The distance between the images H1 and H2 respectively corresponding to the two parallel lines H1 and H2 in the viewpoint conversion image Q4 is the known distance W between the lines L1 and L2 in the actual calibration index 20. So that
(3-1) In such a manner that the positions of the straight images V1 and V2 captured in common to the viewpoint converted image Q1 and the viewpoint converted image Q3 are the same as each other.
(3-2) The linear images H1 and H2 captured in common to the viewpoint converted image Q1 and the viewpoint converted image Q4 are positioned at the same position.
(3-3) In such a manner that the positions of the straight images H1 and H2 captured in common in the viewpoint converted image Q3 and the viewpoint converted image Q2 are the same position.
(3-4) In order that the positions of the straight images V1 and V2 captured in common to the viewpoint converted image Q4 and the viewpoint converted image Q2 are the same,
Correction is performed to adjust the external parameter N1 corresponding to the front camera 11, the external parameter N2 corresponding to the rear camera 12, the external parameter N3 corresponding to the right side camera 13, and the external parameter N4 corresponding to the left side camera 14, respectively.

The external parameter N is adjusted by the external parameter estimation unit 52 in the above conditions (1-1) to (1-8), conditions (2-1) to (2-8), and conditions (3-1) to ( 3-4) is preferably completely satisfied, but not all external parameters N may be uniquely determined. Therefore, the above conditions (1-1) to (1-8), It is also possible to converge to an external parameter N that satisfies the conditions (2-1) to (2-8) and the conditions (3-1) to (3-4) in a well-balanced manner.

Then, the external parameter N when the above-described conditions are converged becomes the corrected parameter N, and the viewpoint conversion processing unit 40 performs viewpoint conversion processing on each of the distortion corrected images T1 to T4 with the corrected external parameter N. The obtained new viewpoint conversion composite image Q10 is as shown in FIG.

This processing can also be realized by designing an error function and applying an optimization method, as in the case of internal parameter estimation. The error function is designed to measure the parallelism of straight lines, the distance between straight lines, and the degree of coincidence of straight line coordinates. Each error function can be calculated from a straight line equation formed by feature points.

Whether or not the line images are parallel to each other is determined by approximating each line image with a straight line expression in the two-dimensional orthogonal coordinate system with the x-axis and y-axis in FIG. ) Are the same.

Further, for example, when H1 and H2 of the calibration index 33 and V1 and V2 of the calibration index 34 are arranged orthogonally, whether or not the line images are orthogonal to each other is determined by the inclination in the above linear equation. Whether the product value is “−1” (when each image is represented by a vector, the inner product value is “0”) can be determined.

This error function is partially differentiated with respect to each external parameter, so that the change amount of the error function with respect to each external parameter can be obtained, and the value of the external parameter is updated based on this change amount (for example, steepest descent method). . By repeating this process until the error function does not decrease, the external parameter is obtained.

The external parameter N expressed as one code corresponding to each camera is actually a position parameter (three variables) for specifying a position in a three-dimensional space and a direction parameter for specifying a direction in a three-dimensional space. Since each of the external parameters N1, N2, N3, and N4 is composed of 6 variables, the external parameter estimation unit 52 adjusts a total of 24 (= 6 × 4) variables. It will be.

As described above, according to the first image calibration apparatus 100 of the present embodiment, the calibration index in the distortion corrected image P is obtained for the distortion corrected image P obtained using the internal parameter M having the default value. Since the internal parameter M is corrected so that the images H and V of the 20 lines H and V are straight lines, it is possible to perform distortion correction suitable for variations in optical characteristics of individual cameras and individual differences, and high accuracy. A distortion-corrected image can be obtained.

Furthermore, a new distortion correction image obtained based on the internal parameters corrected in this way (the lines H and V images H and V of the calibration index 20 are straight lines) is used as a default value. The viewpoint conversion composite image Q10 based on the external parameter N is generated. Since the generated viewpoint conversion composite image Q10 has a high accuracy of the distortion correction image P as a base, the accuracy of the viewpoint conversion composite image Q10 is high. Inevitably high.

Further, by correcting the external parameter N so that the viewpoint conversion composite image Q10 satisfies the above three conditions (1) to (3), the viewpoint conversion composite image Q10 with higher accuracy can be obtained.

In addition, the above three conditions (1) to (3) do not depend on the relative positional relationship between the vehicle 200 and the calibration index 20, and are merely a line H of the calibration index 20 in the viewpoint conversion composite image Q10. And the relationship between only the images H and V (parallel, line spacing, same position), the vehicle 200 with respect to the calibration index 20 when the vehicle 200 is installed with respect to the calibration index 20. The strictness of the position and the attitude (orientation) of the vehicle 200 is not required.

That is, the relative positional relationship between the calibration index 20 installed on the road surface and the vehicle 200 arranged on the road surface is obtained by a normal parking operation on a parking frame in a general parking lot or the like. The accuracy of the position and orientation (the extent of having a practical level deviation) is sufficient, and the strict degree of the relative positional relationship can be relaxed.

This means that even if the type of vehicle 200 (the size and shape of the vehicle) is different, the common calibration index 20 can be used as it is. It is no longer necessary to prepare and install a dedicated calibration index 20, and labor and time can be greatly reduced.

As described above, according to the image calibration apparatus 100 of the present embodiment, the images H and V in the distortion corrected image P corresponding to the lines H and V of the calibration index 20 extend linearly. By adjusting the internal parameter M, it is possible to realize calibration with higher accuracy than the conventional calibration using the fixed internal parameter and adjusting the external parameter N.

In addition, according to the image calibration apparatus 100 of the present embodiment, since the line is used as the calibration index 20 as compared with the conventional calibration method performed only with points, distortion between points is not considered. The accuracy can be improved over the conventional calibration.

Then, at least two calibration markers with at least two parallel straight lines are provided in the common imaging area of the two cameras, and at least three calibration markers are installed on the plane around the vehicle. Use to calibrate. Specifically, the degree of parallelism of the images H and V in the viewpoint conversion composite image Q10, the distance between the images (lines), and the images H and V corresponding to the lines H and V captured across the two viewpoint conversion images. By adjusting the external parameter N so that they are located at the same position, it is possible to perform calibration with high accuracy over the entire viewpoint conversion composite image Q10.

The image calibration apparatus 100 according to the present embodiment uses a simple pattern in which the distance between the lines is known and parallel, and the calibration index 20 having such a simple configuration is available. In addition, there is an advantage that handling is very easy because it may be drawn on the road surface and can be drawn easily.

In the calibration apparatus 100 of the present embodiment, the external parameter estimation unit 52 repeatedly performs the calculation under the above conditions (1) to (3), so that the viewpoint conversion composite image Q10 becomes a new viewpoint conversion composite image Q10. Although the external parameter N is corrected so as to converge, the image calibration apparatus of the present invention is not limited to this form, and the external parameter estimation unit 52 performs the above (1) to (3). ) May be added to the condition (4) (reliable line image is specified using the default value of the external parameter), and correction for adjusting the external parameter N may be performed.

In this way, not only the above conditions (1) to (4) but also the default values of the external parameters are used to identify a reliable line image, so that the image is converged to a new viewpoint conversion composite image Q10. Therefore, it is possible to reduce the number of repetitive computations and shorten the elapsed time until convergence.

DESCRIPTION OF SYMBOLS 11 ... Front camera, 12 ... Rear camera, 13 ... Right side camera, 14 ... Left side camera 30 ... Distortion correction processing part, 40 ... Viewpoint conversion processing part, 60 ...・ Partial line extraction unit,
50 ... Calibration unit, 51 ... Internal / distortion parameter estimation unit, 52 ... External parameter estimation unit 100 ... Calibration device All publications, patents and patent applications cited in this specification It shall be taken into this specification as it is for reference.

Claims

In a calibration method of a calibration device that converts a vehicle surrounding image captured by a plurality of cameras into a bird's-eye view image and combines them,
One calibration index on which at least two parallel straight lines are drawn is provided in the common imaging area of the two cameras,
A calibration method, wherein calibration is performed by using at least three or more calibration indexes installed on a plane around the vehicle.
The calibration method according to claim 1, wherein the plurality of cameras are arranged on the front, rear, left and right of the vehicle.
The calibration method according to claim 1, wherein the plurality of cameras are wide-angle fisheye cameras.
The calibration method according to claim 1, wherein the calibration index is a pattern arranged in a parallel straight line.
The calibration method according to claim 1, wherein internal parameters and external parameters of all the plurality of cameras are calibrated.
6. The calibration method according to claim 5, wherein the calibration method of the internal parameter is performed by optimizing so as to minimize an error, using an error function related to linearity of the parallel straight line of the calibration index. .
The calibration method of the external parameter is calibrated by optimizing so as to minimize the error by using an error function regarding any of parallelism, interval, coordinates, and coincidence of the parallel straight lines of the calibration index. The calibration method according to claim 5.
In a calibration device that converts a vehicle surrounding image captured by a plurality of cameras into an overhead image and synthesizes it,
One calibration index on which at least two parallel straight lines are drawn is provided in the common imaging area of the two cameras,
A calibration apparatus characterized in that calibration is performed using at least three or more calibration indices installed on a plane around the vehicle.
The calibration device according to claim 8, wherein the cameras are attached to the front, rear, left and right of the vehicle.
The calibration device according to claim 8, wherein the camera uses a wide-angle fisheye camera.
The calibration apparatus according to claim 8, wherein the calibration index is a pattern arranged in a parallel straight line.
The calibration device according to claim 8, wherein internal parameters and external parameters of all the plurality of cameras are calibrated.
13. The calibration apparatus according to claim 12, wherein the calibration of the internal parameter is performed by using an error function related to linearity of the parallel straight line of the calibration index so as to minimize the error.
The external parameter calibration is performed by optimizing to minimize an error by using an error function related to any of parallelism, spacing, coordinates, and coincidence of the parallel straight lines of the calibration index. 12. The calibration apparatus described in 12.