JP2010258897A - Determination program and calibration apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow a user to easily determine whether or not a parameter of a calibration result is appropriate. <P>SOLUTION: A calibration apparatus 100 obtains an image photographed by a camera, a parameter of the camera and coordinates of a jig on a user designated image in the state where jigs are disposed on regulated coordinates so as to include a plurality of jigs in a field of view of the camera installed in a vehicle. The calibration apparatus generates a bird's-eye view image of a mobile based on the image and the parameter and transforms the coordinates of the jig on the image and the regulated coordinates into coordinates on the bird's eye view image. The calibration apparatus then determines whether or not the parameter is appropriate based on the coordinates of the jig on the bird's eye view image and a transformation result. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a calibration apparatus that calibrates a camera installed on a moving body.

  In recent years, an image taken by a plurality of cameras installed around a moving body is used to display an overhead view (bird's-eye view) as if the moving body was taken from above, and a 360-degree panoramic image is displayed by combining the images. Technology is widespread. Thus, in order to convert an image captured by the camera into an overhead view or to display a panoramic image by combining the images, the camera position (x coordinate, y coordinate, z coordinate) and angle ( Various parameters including a roll angle, pitch angle, yaw angle), distortion of the lens system, horizontal, vertical angle of view, and the like are used.

  When a camera is installed on a moving body, parameters unique to the camera such as distortion of the lens system, horizontal, and vertical angle of view may be measured for each camera in advance. However, the camera position and angle may include errors depending on the mounting conditions. Among these, the error relating to the camera position can be relatively ignored, but the error relating to the camera angle has a great influence on the overhead view image and the panoramic image and cannot be ignored. Therefore, it is necessary to calculate the position and angle of the camera by performing some kind of calibration.

  In the conventional calibration technology, one calibration jig with a known shape is placed on the road surface at an arbitrary location in the overlapping part of the field of view of each camera, and the position of the camera is determined by the shape conditions of the calibration jig. A technique for calculating the angle is proposed.

  In addition, two small calibration jigs consisting of one feature point are placed in the overlapping part of the camera field of view, and a part of the parameters is calculated by photographing four calibration jigs per camera. There are also techniques to do this.

JP 2008-187466 A JP 2008-245326 A

  However, the above-described conventional technique has a problem in that when the camera parameter is calculated by performing calibration, the user cannot easily check whether the calculated parameter is an appropriate parameter. It was.

  In general, the operation of confirming whether or not the parameter calculated by calibration is appropriate is complicated, and therefore, the operation of confirming the parameter at the time of parameter calculation is often not performed. For this reason, there is a case in which the calibration is re-executed from the beginning only when an overhead view or a panoramic image is generated from the parameters obtained by the calibration and the parameter is noticed.

  The disclosed technology has been made in view of the above, and provides a determination program and a calibration device that allow a user to easily determine whether a parameter of a calibration result is appropriate or not. With the goal.

  The determination program includes an image captured by the camera and a parameter of the camera in a state where the jig is arranged at a predetermined coordinate so that a plurality of jigs are included in the field of view of the camera installed on the moving body. Obtaining the coordinates of the jig on the image; obtaining the bird's-eye view image of the moving object generated based on the image and the parameter; and the coordinates of the jig on the image and the rule A step of converting coordinates to coordinates on the bird's-eye view image, a step of determining whether or not the parameter is appropriate based on the coordinates of the jig on the bird's-eye view image and the conversion result of the converting step It is a requirement to execute.

  By causing the computer to execute this determination program, the user can easily determine whether or not the parameter of the calibration result is appropriate.

FIG. 1 is a diagram illustrating an example of a jig according to the first embodiment. FIG. 2 is a diagram illustrating an example of another jig. FIG. 3 is a diagram illustrating a relationship between a camera and a jig installed in the vehicle. FIG. 4 is a diagram illustrating an example of an image captured by each camera. FIG. 5 is a diagram for explaining each parameter of the camera. FIG. 6 is a diagram for explaining each parameter of the camera. FIG. 7 is a diagram illustrating a result of specifying the image coordinates of the jig. FIG. 8 is a diagram illustrating the relationship between the jig and the camera in the camera coordinate system. FIG. 9 is a diagram illustrating a relationship between a jig and a camera in the vehicle coordinate system. FIG. 10 is a diagram showing the relationship between the depression angle ψ and the optical axis vector _VWA . FIG. 11 is a diagram showing the relationship between the vertical vector and the rotation angle φ. FIG. 12 is a diagram illustrating a relationship between vertical vectors in the vehicle coordinate system and the camera coordinate system. FIG. 13 is a diagram illustrating the configuration of the calibration apparatus according to the present embodiment. FIG. 14 is a diagram illustrating an example of image data stored in the marker image storage unit 110a. FIG. 15 is a diagram illustrating an example of image data stored in the marker image storage unit 110b. FIG. 16 is a diagram for explaining image composition. Figure 17 is a diagram showing the relationship between the coordinates R _W of a point on the road surface and a line-of-sight vector V _W. FIG. 18 is a diagram illustrating an example of an overhead image. FIG. 19 is a diagram illustrating a display example when a cross is drawn at the overhead coordinates on the overhead image. FIG. 20 is a flowchart illustrating a processing procedure of the determination unit when a user is present. FIG. 21 is a flowchart showing the procedure of the pointing error confirmation process. FIG. 22 is a flowchart illustrating a processing procedure of a jig setting position confirmation process. FIG. 23 is a flowchart illustrating the processing procedure of the determination unit when the user does not intervene. FIG. 24 is a flowchart illustrating a processing procedure of the pointing error determination process. FIG. 25 is a flowchart illustrating a processing procedure of jig installation error determination processing. FIG. 26 is a diagram illustrating a hardware configuration of a computer configuring the calibration apparatus according to the embodiment.

  Embodiments of a determination program and a calibration device disclosed in the present application will be described below in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

  The calibration device according to the first embodiment includes an image captured by the camera, camera parameters, and a camera parameter in a state where the jig is arranged at the specified coordinates so that the field of view of the camera installed in the vehicle includes a plurality of jigs. Get the coordinates of the jig on the image. The calibration device generates a bird's-eye view image of the moving body based on the image and the parameters, and converts the coordinates of the jig and the specified coordinates on the camera image into coordinates on the bird's-eye view image. Then, the calibration device determines whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result.

  As described above, the calibration apparatus according to the first embodiment generates an overhead image based on the image and parameters of the camera that captured the plurality of jigs in the field of view, and the coordinates of the jig on the overhead image, Compare the conversion results to determine whether the parameters are appropriate, so you can easily and quickly confirm whether the parameters of the calibration results are appropriate, and reduce the burden on the user I can do it.

  Next, a jig used by the calibration device will be described. FIG. 1 is a diagram illustrating an example of a jig according to the first embodiment. As shown in FIG. 1, this jig is a jig having a vertical width of 297 mm and a horizontal width of 210 mm, and a line having a width of 30 mm is painted in a cross shape on the surface of the jig. In addition, as long as the jig can identify the jig itself, any jig may be used. FIG. 2 is a diagram illustrating an example of another jig.

Next, the relationship between the camera installed in the vehicle and the jig will be described. FIG. 3 is a diagram illustrating a relationship between a camera and a jig installed in the vehicle. As shown in FIG. 3, the vehicle has cameras 20 to 23. Place the origin to the road surface of the vehicle center, the vehicle traveling direction and Y _W-axis, a horizontal direction orthogonal to the Y _W-axis and X _W axis, the vertical upward direction in the coordinate system with the Z _W-axis, the jig 10 and the jig 11 It is installed 0.5m ahead the vehicle front end, and the positions of the left and right 1m to control the Y _W-axis. Jig 12 and the jig 13 is placed 0.5m behind the rear end of the vehicle, and the positions of the left and right 1m to control the _{Y W-axis.}

  FIG. 4 is a diagram illustrating an example of an image captured by each camera. The field of view of the camera 20 includes the jigs 10 and 11, and the field of view of the camera 21 includes the jigs 11 and 12. The field of view of the camera 22 includes the jigs 12 and 13, and the field of view of the camera 23 includes the jigs 13 and 10. In the following description, the coordinate axis system based on the vehicle is referred to as a vehicle coordinate system, and the coordinate axis system based on the camera is referred to as a camera coordinate system.

  Next, camera parameters will be described. 5 and 6 are diagrams for explaining each parameter of the camera. As shown in FIGS. 5 and 6, the camera parameters include a camera depression angle ψ, a rotation angle φ, a pan angle θ in the camera coordinate system, and a three-dimensional coordinate (x, y, z) in the vehicle coordinate system. In the following description, the camera depression angle ψ, rotation angle φ, and pan angle θ are referred to as camera angle parameters. Three-dimensional coordinates (x, y, z) in the vehicle coordinate system are expressed as camera position parameters.

In the first embodiment, it is assumed that the camera position parameter is known and the camera angle parameter is calculated. Next, a camera angle parameter calculation method will be described. The calibration device acquires an image captured by each camera, and specifies the position of the jig included in the acquired image. Hereinafter, the position of the jig in the image is referred to as image coordinates. FIG. 7 is a diagram illustrating a result of specifying the image coordinates of the jig. The process of specifying the image coordinates in the image may be performed by well-known image processing, or may be performed interactively by the user using a pointing device. As shown in FIG. 7, each image coordinate is set to P ₁ and P ₂ .

After specifying the image coordinates of the jig, the calibration device converts the image coordinates into a line-of-sight vector of camera coordinates. Here, the line-of-sight vector is a direction vector of a line segment connecting the optical center of the camera and the jig. Regarding the direction of the line-of-sight vector, the direction from the optical center to the marker is a positive direction. The user creates a conversion table indicating the relationship between the image coordinates and the line-of-sight vector in advance, and the calibration apparatus converts the image coordinates into line-of-sight vectors V _C1 and V _C2 using the conversion table.

FIG. 8 is a diagram illustrating the relationship between the jig and the camera in the camera coordinate system. As shown in FIG. 8, it is assumed that the optical axis vector in the camera coordinate system is V _ca = (0, 0, −1), and the line-of-sight vectors of the jigs are V _C1 and V _C2 , respectively. At this time, the incident angle _{a i} form the line of sight vector and the optical axis vector _{V CA} of the image coordinates _{P i} is
a _i = arccos (V _ca · V _ci ) (1)
Calculate by

FIG. 9 is a diagram illustrating a relationship between a jig and a camera in the vehicle coordinate system. Jig in the vehicle coordinate system is on the road plane (z = 0), since the position as shown in FIG. 3 are determined, the coordinates M _W1, M _W2 of the jig in the vehicle coordinate system is known. If the camera coordinates C _W1 are known, the line-of-sight vectors V _W1 and V _W2 in the vehicle coordinate system are
V _Wi = (M _Wi −C _W1 ) / | M _Wi −C _W1 | (2)
Can be calculated. The camera coordinates _CW1 correspond to camera position parameters.

The incident angle a _i calculated by the equation (1) is the same in both the camera coordinate system shown in FIG. 8 and the vehicle coordinate system shown in FIG. Therefore, the optical axis vector V _WA in the vehicle coordinate system is calculated by obtaining a vector that forms the line-of-sight vectors V _W1 and V _W2 derived from the equation (2) and the incident angles a ₁ and a ₂ .

Here, it is assumed that the line-of-sight vectors V _W1 and V _W2 are V _W1 = (V _W1x , V _W1y , V _W1z ), V ₂ = (V _W2x , V _W2y , V _W2z ), and the obtained optical axis vector V _WA is Let V _WA = (V _WAx , V _WAy , V _WAz ). Assuming that the optical axis vector V _WA is a unit vector,
^{_{V WAx 2 + V WAy 2 +}} V WAz 2 = 1 ··· (3)
Holds. Further, from the inner product condition, V _W1x V _WAx + V _W1y V _WAy + V _W1z V _WAz = cosa ₁ (4)
V _W2x V _WAx + V _W2y V _WAy + V _W2z V _WAz = cosa ₂ (5)
Holds. The calibration apparatus calculates the optical axis vector V _WA in the vehicle coordinate system by solving the simultaneous equations of Expressions (3) to (5).

The calibration device calculates the pan angle θ and the depression angle ψ with reference to the optical axis vector V _WA in the vehicle coordinate system. The pan angle θ is
θ = arctan (V _WAx / V _WAY ) (6)
Can be calculated.

FIG. 10 is a diagram showing the relationship between the depression angle ψ and the optical axis vector _VWA . Since the depression angle and the optical axis vector V _WA there is relationship of FIG. 10, the depression angle [psi,
ψ = π / 2-arccos (V _CA · V _WN ) (7)
Can be calculated. In Expression (7), V _WN is a vertical vector indicating a vertically downward direction in the vehicle coordinate system. V _WN is V _WN = (0, 0, −1).

11 is a diagram showing the relationship between the vertical vector and the rotation angle φ, and FIG. 12 is a diagram showing the relationship between the vertical vector in the vehicle coordinate system and the camera coordinate system. Calibration apparatus calculates the vertical vector V _CN in the camera coordinate system, calculates a rotation angle φ from the relationship of FIG. 11.

As shown on the left side of FIG. 12, the vertical vector V _WN is V _WN = (0, 0, 1) in the vehicle coordinate system, and the line-of-sight vectors V _W1 and V _W2 have been calculated by the equation (2). _Therefore , the incident angles c ₁ and c ₂ of the vertical vector V _WN and the line-of-sight vectors V _W1 and V _W2 are
c _i = arccos (V _WN · V _wi ) (8)
Calculated by

The incident angle c _i calculated by the equation (8) is the same in both the vehicle coordinate system and the camera coordinate system. Accordingly, the line-of-sight vector _{V C1, V C2,} by obtaining the vector at an angle of incidence angles _c 1, _{c 2,} and calculates the vertical vector _{V CN} of the camera coordinate system.

Here, it is assumed that the line-of-sight vectors V _c1 and V _c2 are V _c1 = (V _c1x , V _c1y , V _c1z ), V ₂ = (V _c2x , V _c2y , V _c2z ), and the obtained optical axis vector V _CN is Let V _CN = (V _CNx , V _CNy , V _CNz ). Assuming that the optical axis vector V _CN is a unit vector,
V _CNx ² + V _CNy ² + V _CNz ² = 1 (9)
Holds. In addition, _V C1x from the inner product of the conditions _{V CNx + V C1y V CNy +} V C1z V CNz = cosc 1 ··· (10)
V _C2x V _CNx + V _C2y V _CNy + V _C2z V _CNz = cosc ₂ (11)
Holds. The calibration device calculates the vertical vector V _CN in the camera coordinate system by solving the simultaneous equations of Equations (9) to (11). The calibration device uses the rotation angle φ,
φ = arctan (−V _CNz / V _CNy ) (12)
Calculated by

  Next, the configuration of the calibration apparatus according to the first embodiment will be described. FIG. 13 is a diagram illustrating the configuration of the calibration apparatus according to the present embodiment. As illustrated in FIG. 13, the calibration apparatus 100 includes a frame buffer 110, a marker extraction unit 120, a camera angle parameter estimation unit 130, an overhead image creation unit 140, a determination unit 150, a parameter storage unit 160, and a display 170. The calibration apparatus 100 is connected to the cameras 20-23.

  The frame buffer 110 is a buffer for storing image data captured by the cameras 20 to 23. The frame buffer 110 stores image data for each camera. The frame buffer 110 includes marker image storage units 110a and 110b.

  The marker extraction unit 120 is means for acquiring image data from the frame buffer 110 and specifying the image coordinates of the jig on the image data using a known image processing technique. Alternatively, the marker extraction unit 120 may output the image data to the display 170, cause the user to specify the position of the jig, and specify the specified coordinates as the image coordinates. The marker extraction unit 120 classifies the specified image coordinate data for each image data of each camera and registers the data in the designated position parameter of the parameter storage unit 160. The marker extraction unit 120 notifies the camera angle parameter estimation unit of completion when the identification of the image coordinates is completed.

  When the marker extraction unit 120 specifies a jig, the marker extraction unit 120 stores image data including the jig in the marker image storage units 110a and 110b. The marker extraction unit 120 may divide the image data into a plurality of pieces and store them in the marker image storage units 110a and 110b.

  For example, the camera 20, 21, 23 takes an image in a state where the user places the jigs 10, 11 at a predetermined position in front of the vehicle and does not place the jigs 12, 13 behind. The image data of the camera 20 includes the jigs 10 and 11, the camera 21 includes the jig 11, and the camera 23 includes the jig 10.

  The marker extraction unit 120 detects two jigs from the image data of the camera 20 and one jig from the image data of the cameras 21 and 23. In this case, the marker extraction unit 120 stores all image data of the camera 20 and half of the image data of the cameras 21 and 23 in the marker image storage unit 110a. FIG. 14 is a diagram illustrating an example of image data stored in the marker image storage unit 110a.

  Subsequently, the camera 21, 22, 23 shoots images in a state where the user places the jigs 12, 13 at a predetermined position behind the vehicle and does not place the jigs 10, 11 in front. The image data of the camera 21 includes the jig 12, the image data of the camera 22 includes the jigs 12 and 13, and the image data of the camera 23 includes the jig 13.

  The marker extraction unit 120 detects two jigs from the image data of the camera 22 and one jig from the image data of the cameras 21 and 23. In this case, the marker extraction unit 120 stores all image data of the camera 22 and half of the image data of the cameras 21 and 23 in the marker image storage unit 110b. FIG. 15 is a diagram illustrating an example of image data stored in the marker image storage unit 110b.

Returning to the description of FIG. 13, the camera angle parameter estimation unit 130 is a processing unit that calculates camera angle parameters. When the camera angle parameter estimation unit 130 receives a completion notification from the marker extraction unit 120, the camera angle parameter estimation unit 130 acquires the designated position parameter 161 and converts the image coordinates into the line-of-sight vectors V _C1 and V _C2 using the conversion table.

  Then, the camera angle estimation unit 130 acquires the marker position parameter 162 and the camera position parameter 163, and calculates the camera angle parameter using the above-described equations (1) to (12). The camera angle estimation unit 130 registers the camera angle parameter 163 in the parameter storage unit 160. Here, the marker position parameter 162 includes the coordinates of the jigs 10 to 13 when the jigs are arranged at the predetermined positions shown in FIG. 3 as the coordinates of the vehicle coordinate system, and the camera position parameters are the coordinates of the vehicle coordinate system. Includes the coordinates of the cameras 20-23. The camera angle parameter estimation unit 130 outputs a completion notification to the overhead image creation unit 140 when the calculation of the camera angle parameters of the cameras 20 to 23 is completed.

  The overhead image creation unit 140 is a processing unit that generates an overhead image based on the image data stored in the frame buffer 110 and the camera angle parameter 164. The overhead image creation unit 140 synthesizes the image data stored in the marker image storage unit 110a and the image data stored in the marker image storage unit 110b when a completion notification is acquired from the camera angle parameter estimation unit 130. FIG. 16 is a diagram for explaining image composition.

  As shown in FIG. 16, the overhead image creation unit 140 creates the image data of the camera 21 by combining the left half and the right half of the image data of the camera 21, and the left of the image data of the camera 23. The image data of the camera 23 is created by combining the image data of the half and the right half.

  Subsequently, the bird's-eye view image creation unit 140 converts the image data into a bird's-eye view image based on each image data and the camera angle parameter. The overhead image creation unit 140 obtains the relationship between the coordinates on the image data and the coordinates on the overhead image. In the following description, coordinates on image data are referred to as coordinates on the image, and coordinates on the overhead image are referred to as overhead coordinates.

First, the overhead image creating unit 140 uses the conversion table, the image on the coordinate P for calculating the sight line vector V _C of the camera coordinate system. Bird's-eye view image creation unit 140, a sight line vector V _C of the calculated camera coordinate system, into a line-of-sight vector V _W of the vehicle coordinate system. The conversion formula is shown in the following formula (13).

RotX (ψ), RotY (θ), and RotZ (φ) in Expression (13) are represented by the following Expressions (14) to (16).

Then, the bird's-eye view coordinate creation unit 140, based on the camera position parameter 163 (camera position C _W), the line-of-sight vector V _W of the vehicle coordinate system, the coordinates R _W of a point on the road surface in the direction of the sight line vector V _W Is calculated. Figure 17 is a diagram showing the relationship between the coordinates R _W of a point on the road surface and a line-of-sight vector V _W. The relationship between the line-of-sight vector and coordinates is
R _W = C _W + kV _W (17)
It becomes. However, k = C _WZ / V _WZ .

Then, the bird's-eye view coordinate creation unit 140 converts the overhead on the image T on the overhead view image coordinates R _W. Since a linear relationship is established between the bird's-eye view image and the (x, y) coordinates of the vehicle coordinate system in a plane at Z = 0, conversion can be performed by the offset amount O (pixel) of the origin and the scale (pixel / mm). . Relationship between the bird's-eye view on the coordinate T on the bird's-eye view image coordinates R _W is,
T = S × (R _WX , R _WY ) + O (18)
It becomes. In Expression (18), when the origin of the overhead image coincides with the origin of the vehicle coordinates, O = (0, 0).

  Since the relationship between the image coordinates P and the overhead view coordinates T is calculated based on the equations (13) to (18), the overhead image creation unit 140 converts the image data of the image coordinates P into the corresponding overhead image. A bird's-eye view image is created by pasting to the top-of-view coordinates T. FIG. 18 is a diagram illustrating an example of an overhead image. The overhead image creation unit 140 outputs the overhead image to the determination unit 150.

  The determination unit 150 is a processing unit that determines whether calibration is successful based on the overhead image, the specified position parameter 161, and the marker position parameter 162. In the following, a case where a user is involved in the processing of the determination unit 150 and a case where the determination unit 150 automatically performs processing will be described.

  First, the case where a user intervenes in the process of the determination part 150 is demonstrated. The determination unit 150 acquires the image coordinates of the jig in the image captured by the camera 20 from the specified position parameter 161. In addition, the determination unit 150 acquires camera position parameters and camera angle parameters of the camera 20. The determination unit 150 calculates overhead coordinates corresponding to the image coordinates based on the equations (13) to (18). Hereinafter, the overhead view coordinates corresponding to the image coordinates are referred to as first overhead coordinates.

  The determination unit 150 draws a cross at the first overhead view coordinates on the overhead view image and displays the cross on the display. FIG. 19 is a diagram illustrating a display example when a cross is drawn at the overhead coordinates on the overhead image. The user visually checks the image displayed on the display 170 and determines whether the center of the jig matches the cross. Then, the user inputs a determination result as to whether the center of the jig and the position of the cross are within the allowable range via the input device to the determination unit 150. The determination unit 150 also performs the above process for the cameras 21 to 23.

  If even one determination result by the user for all cameras is not within the allowable range, the determination unit 150 notifies the user of the jig extraction failure and restarts the marker extraction unit 120. The extraction of the jig has failed, that is, the center position of the cross is deviated from the center of the jig displayed on the overhead image, in the coordinate specification of the jig on the image data by the marker extraction unit 120, This means that the position of the jig on the image was not correctly specified. For example, when the user designates a position shifted from the position of the jig on the image as the position of the jig, the determination result here is an extraction failure.

  On the other hand, when the determination result of the user for all the cameras is within the allowable range, the determination unit 150 acquires the position of the jig imaged by the camera 20 from the marker position parameter 162.

  The determination unit 150 converts the position of the jig into coordinates on the bird's-eye view image using Expression (18) described above. Hereinafter, the overhead view coordinates corresponding to the position of the jig acquired from the marker position parameter 162 will be referred to as second overhead view coordinates.

  The determination unit 150 draws a cross at the second overhead coordinate on the overhead image and displays the cross on the display 170. The user visually checks the image displayed on the display 170 and determines whether the center of the jig matches the cross. Then, the user inputs a determination result as to whether the center of the jig and the position of the cross are within the allowable range via the input device to the determination unit 150. The determination unit 150 also performs the above process for the cameras 21 to 23. Here, the marker position parameter represents the coordinates of each jig on the image when the jig is correctly installed at the predetermined position shown in FIG. Therefore, that is, that the center position of the cross is shifted from the center of the jig displayed on the overhead image, it means that the place where the jig was installed at the time of shooting the image was not the correct position. means.

  The determination unit 150 notifies the user of the failure of the jig placement and notifies the user of the redo of the jig placement when one or more judgment results of the user for all the cameras are not within the allowable range. On the other hand, when the determination result of the user for all the cameras is within the allowable range, the determination unit 150 notifies that the calibration is successful.

  Next, a case where the determination unit 150 performs processing automatically will be described. The determination unit 150 acquires the image coordinates of the jig in the image captured by the camera 20 from the specified position parameter 161. In addition, the determination unit 150 acquires camera position parameters and camera angle parameters of the camera 20. The determination unit 150 calculates overhead coordinates corresponding to the image coordinates based on the equations (13) to (18). Hereinafter, the overhead view coordinates corresponding to the image coordinates are referred to as first overhead coordinates.

The determination unit 150 performs image processing on the overhead image and extracts the coordinates of the jig on the overhead image. Hereinafter, the coordinates of the jig directly extracted from the bird's-eye view image are referred to as third bird's-eye coordinates. The determination unit 150 compares the first overhead coordinates (T _N1 , T _N2 ) with the third overhead coordinates (T _M1 , T _M2 ),
| T _N1 −T _M1 | ≦ Threshold C
And | T _N2 −T _M2 | ≦ Threshold C
It is determined whether or not the condition is satisfied. When such a condition is satisfied, the determination unit 150 determines that the error with respect to the camera 20 is within an allowable range. If the condition is not satisfied, it is determined that the error with respect to the camera 20 is outside the allowable range. The determination unit 150 also performs the above process for the cameras 21 to 23.

  If at least one determination result for all the cameras is not within the allowable range, the determination unit 150 notifies the user of the jig extraction failure and restarts the marker extraction unit 120.

  On the other hand, when the determination results for all the cameras are within the allowable range, the determination unit 150 acquires the position of the jig photographed by the camera 20 from the marker position parameter 162.

The determination unit 150 converts the position of the jig into coordinates on the bird's-eye view image using Expression (18) described above. Hereinafter, the overhead view coordinates corresponding to the position of the jig acquired from the marker position parameter 162 are referred to as second overhead view coordinates (M _T1 , M _T2 ).

The determination unit 150 compares the second overhead view coordinates with the third overhead view coordinates,
| M _T1 −T _M1 | ≦ Threshold D
And | M _T2 −T _M2 | ≦ Threshold D
It is determined whether or not the condition is satisfied. When such a condition is satisfied, the determination unit 150 determines that the error with respect to the camera 20 is within an allowable range. If the condition is not satisfied, it is determined that the error with respect to the camera 20 is outside the allowable range. The determination unit 150 also performs the above process for the cameras 21 to 23.

  The determination unit 150 notifies the user of the failure of the jig placement and notifies the user of the redo of the jig placement when one or more judgment results of the user for all the cameras are not within the allowable range. On the other hand, when the determination result of the user for all the cameras is within the allowable range, the determination unit 150 notifies that the calibration is successful.

  The parameter storage unit 160 is a storage unit that stores a specified position parameter 161, a marker position parameter 162, a camera position parameter 163, and a camera angle parameter 164. The display 170 is a device that displays various types of information.

  Next, the processing procedure of the determination unit 150 will be described. First, the processing procedure of the determination unit 150 when a user is present will be described. FIG. 20 is a flowchart illustrating a processing procedure of the determination unit when a user is present. As illustrated in FIG. 20, the determination unit 150 performs a pointing error confirmation process on an image captured by the camera 20 (step S101), and performs a pointing error confirmation process on an image captured by the camera 21 (step S102).

  The determination unit 150 performs a pointing error confirmation process on the image captured by the camera 22 (step S103), and performs a pointing error confirmation process on the image captured by the camera 23 (step S104).

  The determination unit 150 determines whether or not the pointing error with respect to the images of all the cameras is within an allowable range (step S105). When the pointing error with respect to the image of any camera is outside the allowable range (No at Step S106), the determination unit 150 outputs that the jig detection has failed to the display 170 (Step S107), and extracts the marker. The unit 120 is restarted (step S108).

  On the other hand, when the pointing error with respect to the images of all the cameras is within the allowable range (step S106, Yes), the determination unit 150 performs an installation error confirmation process for the jig included in the field of view of the camera 20 (step S109). An installation error confirmation process for the jig included in the field of view of the camera 21 is performed (step S110).

  The determination unit 150 performs an installation error confirmation process for the jig included in the visual field of the camera 22 (step S111), and performs an installation error confirmation process for the jig included in the visual field of the camera 23 (step S112).

  The determination unit 150 determines whether or not the installation errors of all the jigs are within an allowable range (step S113). If the jig installation error with respect to any camera image is outside the allowable range (step S114, No), the determination unit 150 outputs to the display 170 that the jig installation has failed (step S115). On the other hand, when the installation error of all the jigs is within the allowable range (step S114, Yes), the determination unit 150 outputs that the calibration is successful to the display 170 (step S116).

  Next, the processing procedure of the pointing error confirmation process shown in steps S101 to S104 in FIG. 20 will be described. FIG. 21 is a flowchart showing the procedure of the pointing error confirmation process.

  As illustrated in FIG. 21, the determination unit 150 acquires the image coordinates of the jig specified based on the image captured by the camera n (n is 20, 21, 22, or 23) (step S201), and the camera of the camera n A position parameter and a camera angle parameter are acquired (step S202). The determination unit 150 converts the image coordinates to first overhead coordinates (step S203).

  The determination unit 150 draws a cross at the first overhead view coordinates on the overhead view image and displays the cross on the display 170 (step S204). The determination unit 150 determines whether information indicating that the cross of the jig matches the drawn cross is acquired from the input device (step S205).

  When the determination unit 150 acquires information indicating that they match (Yes in Step S206), the determination unit 150 sets the return value for the camera n to “within an allowable range” (Step S207). On the other hand, when the determination unit 150 acquires information indicating that they do not match (No at Step S206), the determination unit 150 sets the return value for the camera n to “out of allowable range” (Step S208).

  Next, the processing procedure of the jig setting position confirmation process shown in steps S109 to S112 of FIG. 20 will be described. FIG. 22 is a flowchart illustrating a processing procedure of a jig setting position confirmation process.

  As shown in FIG. 21, the marker position parameter of the jig included in the image captured by the camera n (n is 20, 21, 22, or 23) is acquired (step S301), and the marker position parameter is set to the second overhead coordinate. Conversion is performed (step S302).

  The determination unit 150 draws a cross at the second overhead coordinate on the overhead image and displays it on the display 170 (step S303). The determination unit 150 determines whether information indicating that the cross of the jig matches the drawn cross is acquired from the input device (step S304).

  When the determination unit 150 acquires information indicating that they match (Yes in Step S305), the determination unit 150 sets the return value for the camera n to “within allowable range” (Step S306). On the other hand, when the determination unit 150 acquires information indicating that they do not match (No in step S305), the determination unit 150 sets the return value for the camera n to “out of allowable range” (step S307).

  Next, a processing procedure of the determination unit 150 when no user is present will be described. FIG. 23 is a flowchart illustrating a processing procedure of the determination unit 150 when the user does not intervene. As illustrated in FIG. 23, the determination unit 150 performs a pointing error determination process on an image captured by the camera 20 (step S401), and performs a pointing error determination process on an image captured by the camera 21 (step S402).

  The determination unit 150 performs a pointing error determination process on the image captured by the camera 22 (step S403), and performs a pointing error determination process on the image captured by the camera 23 (step S404).

  The determination unit 150 determines whether or not the pointing error with respect to the images of all the cameras is within an allowable range (step S405). When the pointing error with respect to the image of any camera is outside the allowable range (No at Step S406), the determination unit 150 outputs that the jig detection has failed to the display 170 (Step S407), and extracts the marker. The unit 120 is restarted (step S408).

  On the other hand, when the pointing error with respect to the images of all the cameras is within the allowable range (step S406, Yes), the determination unit 150 performs an installation error determination process for the jig included in the field of view of the camera 20 (step S409). An installation error determination process for the jig included in the field of view of the camera 21 is performed (step S410).

  The determination unit 150 performs an installation error confirmation process for the jig included in the visual field of the camera 22 (step S411), and performs an installation error confirmation process for the jig included in the visual field of the camera 23 (step S412).

  The determination unit 150 determines whether or not the installation errors of all the jigs are within the allowable range (step S413). When the jig installation error with respect to any camera image is outside the allowable range (step S414, No), the determination unit 150 outputs to the display 170 that the jig installation has failed (step S415). On the other hand, when the installation error of all the jigs is within the allowable range (step S414, Yes), the determination unit 150 outputs that the calibration is successful to the display 170 (step S416).

  Next, the processing procedure of the pointing error determination process shown in steps S401 to S404 in FIG. 23 will be described. FIG. 24 is a flowchart illustrating a processing procedure of the pointing error determination process.

  As shown in FIG. 24, the determination unit 150 acquires the image coordinates of the jig specified based on the image captured by the camera n (n is 20, 21, 22, or 23) (step S501), and the camera of the camera n A position parameter and a camera angle parameter are acquired (step S502).

  The determination unit 150 converts the image coordinates to the first overhead coordinates (step S503), and extracts the third overhead coordinates from the overhead image (step S504). The determination unit 150 compares the first overhead coordinate and the third overhead coordinate to determine whether or not the difference between the first overhead coordinate and the third overhead coordinate is equal to or less than a threshold value (step S505).

  When the difference between the first bird's-eye view coordinates and the third bird's-eye view coordinates is equal to or smaller than the threshold (step S506, Yes), the determination unit 150 sets the return value for the camera n to “within allowable range” (step 507). On the other hand, when the difference between the first bird's-eye view coordinates and the third bird's-eye view coordinates is larger than the threshold value (step S506, No), the determination unit 150 sets the return value for the camera n to “out of allowable range” (step S506). 508).

  Next, the processing procedure of the jig installation error determination process shown in steps S409 to S412 of FIG. 23 will be described. FIG. 25 is a flowchart illustrating a processing procedure of jig installation error determination processing.

  As shown in FIG. 25, the determination unit 150 acquires the marker position parameter of the jig included in the image captured by the camera n (n is 20, 21, 22, or 23) (step S601), and determines the marker position parameter. 2 is converted into overhead coordinates (step S602).

  The determination unit 150 extracts the third overhead coordinate from the overhead image (step S603), compares the second overhead coordinate with the third overhead coordinate, and compares the second overhead coordinate with the third overhead coordinate. Is less than or equal to a threshold value (step S604).

  When the difference between the second bird's-eye view coordinates and the third bird's-eye view coordinates is equal to or smaller than the threshold value (step S605, Yes), the determination unit 150 sets the return value for the camera n to “within allowable range” (step 606). On the other hand, when the difference between the second bird's-eye view coordinates and the third bird's-eye view coordinates is larger than the threshold (No at Step S605), the determination unit 150 sets the return value for the camera n to “outside the allowable range” (Step S605). 607).

  As described above, the calibration apparatus 100 according to the first embodiment was photographed by the camera in a state where the jigs are arranged at the specified coordinates so that a plurality of jigs are included in the field of view of the camera installed in the vehicle. The image, camera parameters, and jig coordinates on the camera image are acquired. The calibration apparatus 100 generates a bird's-eye view image of the moving body based on the image and the parameters, and converts the coordinates of the jig and the specified coordinates on the camera image into coordinates on the bird's-eye view image. Then, the calibration device determines whether the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result.

  As described above, the calibration apparatus according to the first embodiment generates an overhead image based on the image and parameters of the camera that captured the plurality of jigs in the field of view, and the coordinates of the jig on the overhead image, Compare the coordinates of the actual jig, etc., and determine whether the parameter is appropriate, so it is easy and early to check whether the parameter of the calibration result is appropriate, and it takes the user The burden can be reduced.

  Although the embodiments of the present invention have been described so far, the present invention may be implemented in various different forms other than the first embodiment described above. Therefore, another embodiment included in the present invention will be described below as a second embodiment.

(1) Imaging of a jig In the first embodiment, the timing at which the cameras 20 to 23 capture images is not mentioned, but it is not necessary to simultaneously capture all the camera images necessary for calibration. For example, the calibration apparatus 100 may calculate the camera angle parameter in the same manner as in the first embodiment by synthesizing each image after dividing the photographing range and photographing each jig. In this way, by dividing the imaging range and imaging each jig, the space required for calibration can be further reduced. The number of cameras is not limited to four.

(2) System configuration, etc. Of the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or can be performed manually. All or part of the described processing can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-mentioned document and drawings can be arbitrarily changed unless otherwise specified.

  Each component of each illustrated device is functionally conceptual and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  For example, the server has functions corresponding to the marker extraction unit 120, the camera angle parameter estimation unit 130, the overhead image creation unit 140, and the determination unit 150 described in FIG. The server may determine whether the calibration is successful by transmitting each data stored in the parameter storage unit 160, and notify the user terminal of the determination result. Alternatively, the mobile terminal or the like may have functions corresponding to the camera angle parameter estimation unit 130, the overhead image creation unit 140, and the determination unit 150.

  Further, all or any part of each processing function performed in each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

  FIG. 26 is a diagram illustrating a hardware configuration of a computer constituting the calibration apparatus 100 according to the embodiment. As shown in FIG. 26, the computer (calibration device) 200 communicates with an input device 201, a monitor 202, a camera 203, a RAM (Random Access Memory) 204, a ROM (Read Only Memory) 205, and other devices. A communication control device 206, a medium reading device 207 that reads data from a storage medium, a CPU (Central Processing Unit) 208, and an HDD (Hard Disk Drive) 209 are connected by a bus 210. Assume that the computer 200 has a camera in addition to the camera 203.

  The HDD 209 stores a calibration program 209b that exhibits the same function as that of the calibration apparatus 100 described above. When the CPU 208 reads out and executes the calibration program 209b, the calibration process 208a is activated. Here, the calibration process 208a corresponds to the marker extraction unit 120, the camera angle parameter estimation unit 130, the overhead image creation unit 140, and the determination unit 150 illustrated in FIG.

  The HDD 209 stores various data 209 a corresponding to information stored in the parameter storage unit 160. The CPU 208 reads out the various data 209a stored in the HDD 209, stores it in the RAM 204, and determines whether the calibration is successful using the various data 204a stored in the RAM 204 and an image taken by the camera 203. To do.

  By the way, the calibration program 209b shown in FIG. 26 is not necessarily stored in the HDD 209 from the beginning. For example, a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into a computer, or a hard disk drive (HDD) provided inside or outside the computer. The calibration program 209b is stored in the “fixed physical medium” of the computer, and in addition to the “other computer (or server)” connected to the computer via a public line, the Internet, a LAN, a WAN, or the like. The computer may read and execute the calibration program 209b from these.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary note 1)
An image captured by the camera, a parameter of the camera, and a jig on the image in a state where the jig is arranged at a predetermined coordinate so that a plurality of jigs are included in the field of view of the camera installed on the moving body Obtaining the coordinates of
Obtaining an overhead image of the moving body generated based on the image and the parameter;
Converting the coordinates of the jig on the image and the specified coordinates into coordinates on the overhead image;
A determination program that executes the step of determining whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step.

(Additional remark 2) When the said determination step determines that the said parameter is not appropriate, the coordinate of the jig | tool on the said bird's-eye view image, the distance of the said specified coordinate converted by the said step of converting, and the said bird's-eye view The reason why the parameter is not appropriate is notified based on the distance between the coordinates of the jig on the image and the coordinates of the jig on the image converted in the converting step. Judgment program.

(Appendix 3)
Obtaining an image captured by the camera and parameters of the camera in a state in which the jig is arranged at specified coordinates so that a plurality of jigs are included in the field of view of the camera installed on the moving body;
Obtaining an overhead image of the moving body generated based on the image and the parameter;
Converting the prescribed coordinates to coordinates on the overhead image;
A determination program that executes the step of determining whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step.

(Additional remark 4) When it is determined that the parameter is not appropriate, the determining step is based on the distance between the coordinates of the jig on the overhead image and the specified coordinates converted in the converting step. The determination program according to attachment 3, wherein the reason why the parameter is not appropriate is notified.

(Appendix 5)
An image captured by the camera, a parameter of the camera, and a jig on the image in a state where the jig is arranged at a predetermined coordinate so that a plurality of jigs are included in the field of view of the camera installed on the moving body Obtaining the coordinates of
Obtaining an overhead image of the moving body generated based on the image and the parameter;
Converting the coordinates of the jig on the image into coordinates on the overhead image;
Determining whether the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step;
The determination program characterized by making it execute.

(Additional remark 6) When the said determination step determines that the said parameter is not appropriate, it is based on the distance of the coordinate of the jig on the said bird's-eye view image, and the coordinate of the jig on the said image converted at the said conversion step. The determination program according to appendix 5, wherein the reason why the parameter is not appropriate is notified.

(Appendix 7) An image photographed by the camera in a state where the jig is arranged at a specified coordinate so that a plurality of jigs are included in the field of view of the camera installed on the moving body, the specified coordinate, A parameter calculation unit for calculating the parameters of the camera based on the coordinates of the camera;
Based on the image and the parameter, an overhead image generation unit that generates an overhead image of the moving body;
A conversion unit that converts the coordinates of the jig on the image and the specified coordinates into coordinates on the overhead image;
Whether the parameter is appropriate based on the conversion result obtained by converting the position of the jig on the overhead image, the coordinates of the jig on the image, and the specified coordinates into the coordinates on the overhead image, respectively. A determination unit for determining;
A calibration apparatus comprising:

(Supplementary note 8)
An image photographed by the camera, parameters of the camera, and a jig on the image in a state where the jig is arranged at a predetermined coordinate so that a plurality of jigs are included in the field of view of the camera installed on the moving body An acquisition step of acquiring the coordinates of
Based on the image and the parameter, an overhead image generation step of generating an overhead image of the moving body;
A conversion step of converting the coordinates of the jig on the image and the specified coordinates into coordinates on the overhead image;
A determination step of determining whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the conversion step;
The determination method characterized by including.

(Additional remark 9) The image which the said camera image | photographed in the state which has arrange | positioned the said jig | tool at the predetermined coordinate so that a several jig may be included in the visual field of the camera installed in the mobile body, the parameter of the said camera, and the said image An acquisition unit for acquiring coordinates of the upper jig, and an overhead image of the moving body generated based on the image and the parameter;
A conversion unit that converts the coordinates of the jig on the image and the specified coordinates into coordinates on the overhead image;
A determination apparatus comprising: a determination unit configured to determine whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step.

(Additional remark 10) The image which the said camera image | photographed in the state which has arrange | positioned the said jig | tool at a predetermined coordinate so that a some jig | tool may be included in the visual field of the camera installed in the moving body, the parameter of the said camera, and the said image And an acquisition unit for acquiring an overhead image of the moving object generated based on the parameters,
A conversion unit that converts the specified coordinates into coordinates on the overhead image;
A determination apparatus comprising: a determination unit configured to determine whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step.

(Additional remark 11) The image which the said camera image | photographed in the state which has arrange | positioned the said jig | tool at a predetermined coordinate so that a several jig | tool may be included in the visual field of the camera installed in the mobile body, the parameter of the said camera, and the said image An acquisition unit for acquiring coordinates of the upper jig, and an overhead image of the moving body generated based on the image and the parameter;
A converter that converts the coordinates of the jig on the image into coordinates on the overhead image;
A determination unit that determines whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step;
The determination apparatus characterized by having.

10, 11, 12, 13 Jig 20, 21, 22, 23 Camera 100 Calibration device 110 Frame buffer 110a, 110b Marker image storage unit 120 Marker extraction unit 130 Camera angle parameter estimation unit 140 Overhead image creation unit 150 Determination unit 160 Parameter Storage unit 161 Specified position parameter 162 Marker position parameter 163 Camera position parameter 164 Camera angle parameter 170 Display 200 Computer 201 Input device 202 Monitor 203 Camera 204 RAM
204a Various data 205 ROM
206 Communication control device 207 Medium reading device 208 CPU
208a Calibration process 209 HDD
209a Various data 209b Calibration program 210 Bus

Claims (5)

On the computer,
An image captured by the camera, a parameter of the camera, and a jig on the image in a state where the jig is arranged at a predetermined coordinate so that a plurality of jigs are included in the field of view of the camera installed on the moving body Obtaining the coordinates of
Obtaining an overhead image of the moving body generated based on the image and the parameter;
Converting the coordinates of the jig on the image and the specified coordinates into coordinates on the overhead image;
A determination program that executes the step of determining whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step.   In the determining step, when it is determined that the parameter is not appropriate, the distance between the coordinates of the jig on the overhead image and the specified coordinate converted in the converting step, and the correction on the overhead image. The determination program according to claim 1, wherein a reason why the parameter is not appropriate is notified based on a distance between the coordinates of the tool and the coordinates of the jig on the image converted in the converting step. On the computer,
Obtaining an image captured by the camera and parameters of the camera in a state in which the jig is arranged at specified coordinates so that a plurality of jigs are included in the field of view of the camera installed on the moving body;
Obtaining an overhead image of the moving body generated based on the image and the parameter;
Converting the prescribed coordinates to coordinates on the overhead image;
A determination program that executes the step of determining whether or not the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step. On the computer,
An image captured by the camera, a parameter of the camera, and a jig on the image in a state where the jig is arranged at a predetermined coordinate so that a plurality of jigs are included in the field of view of the camera installed on the moving body Obtaining the coordinates of
Obtaining an overhead image of the moving body generated based on the image and the parameter;
Converting the coordinates of the jig on the image into coordinates on the overhead image;
Determining whether the parameter is appropriate based on the coordinates of the jig on the overhead image and the conversion result of the converting step;
The determination program characterized by making it execute. An image captured by the camera, the specified coordinates, and the coordinates of the camera in a state where the jig is arranged at specified coordinates so that a plurality of jigs are included in the field of view of the camera installed on the moving body. Based on the parameter calculation unit for calculating the parameters of the camera,
Based on the image and the parameter, an overhead image generation unit that generates an overhead image of the moving body;
A conversion unit that converts the coordinates of the jig on the image and the specified coordinates into coordinates on the overhead image;
Whether the parameter is appropriate based on the conversion result obtained by converting the position of the jig on the overhead image, the coordinates of the jig on the image, and the specified coordinates into the coordinates on the overhead image, respectively. A determination unit for determining;
A calibration apparatus comprising:

Images