JP2018205008A - Camera calibration device and camera calibration method - Google Patents

Abstract

To realize highly accurate calibration over a wide depth range on a camera using a wide viewing angle optical system.SOLUTION: The present invention comprises a compound eye camera 101 for capturing an image, a range sensor 102 for acquiring distance information pertaining to substantially the same scene as the camera, a range data evaluation unit 103 for evaluating the distribution of the distance information, and a calibration unit 104 for executing calibration of the camera using the camera image, the distance information and the range data evaluation result and outputting a camera parameter, the calibration being executed by a camera model whose pupil position changes with each angle of incidence when the distance information contains information pertaining to a prescribed range of distance. The present constitution makes it possible to provide a camera parameter for stereo cameras for which calibration accuracy in a necessary range of distance is secured.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a calibration apparatus, method, and program for a compound eye camera such as a stereo camera.

  As a conventional stereo camera calibration device, there is one that uses a dedicated subject having a checkered pattern, detects an intersection of the checkered pattern as a feature point, and calibrates each camera (see, for example, Patent Document 1). . FIG. 8 shows a conventional stereo camera calibration apparatus described in Patent Document 1. In FIG.

  In FIG. 8, reference numerals 801A and 801B denote cameras. The video imaged by the camera is processed by a monocular stereo processing unit and a binocular stereo processing unit. In the monocular stereo processing unit, a flow between frames (movement in the time direction) is estimated by the flow estimation unit 802, and the amount of movement of the own vehicle (that is, the amount of movement of the camera) from information such as a wheel speed sensor by the own vehicle momentum estimation unit 803. In the three-dimensional coordinate estimation apparatus 804, the three-dimensional coordinates of the feature points are publicly known from the feature points detected by the flow estimation unit 802, the flow information thereof, and the own vehicle momentum estimated by the own vehicle momentum estimation unit. Obtained by the method of In the binocular stereo processing unit, the corresponding point estimation unit 806 performs feature point association between the stereo images, and the temporary calibration unit 807 has a provisional translational displacement amount (size) between the two cameras 801A and 801B. The positional deviation amount normalized to 1) and the rotational deviation amount are estimated, and the three-dimensional feature points associated with the corresponding point estimation unit 806 in the temporary three-dimensional coordinate estimation unit 808 using the calibration result. Estimate coordinate values. In the estimation of the three-dimensional coordinate value in the provisional three-dimensional coordinate estimation unit 808, since the calculation is performed by normalizing the amount of the provisional translational displacement between the two cameras 801A and 801B to 1, A three-dimensional coordinate value having a different scale from the three-dimensional coordinate value is calculated. In order to correct this, in the inter-camera distance scale estimation unit 805, the inter-camera distance scale (that is, the inter-camera distance scale estimation unit 805 estimated by the monocular stereo processing unit and the binocular stereo processing unit respectively for the same feature point) A non-normalized inter-camera distance) is estimated, and a three-dimensional coordinate estimation unit 809 estimates a three-dimensional coordinate value in consideration of the inter-camera distance scale.

  In-vehicle cameras use an optical system with a wide viewing angle, such as a fisheye lens, to capture the entire area around the vehicle. In such an optical system with a wide viewing angle, since a distortion of an image to be taken becomes large, a dedicated calibration method has been proposed (for example, see Non-Patent Document 1).

Japanese Patent No. 4814669

Ryota Moriyasu, Shinpei Nakamura, Kenichi Kanaya, “Highly accurate calibration of fisheye lens camera with belt pattern”, Information Processing Society of Japan Report, Computer Vision and Image Media (CVIM), Vol. 2011-CVIM-176, 2011

  However, the conventional fisheye lens calibration apparatus does not consider that the pupil position moves in accordance with the incident angle. For this reason, there is a problem that the calibration accuracy is lowered for a subject that is far from the distance for which calibration has been performed.

  The present invention solves the above-described conventional problems. When the distribution of distance information in the field of view of a stereo camera is evaluated, and the distance information includes information of a predetermined distance range, the pupil for each incident angle. An object of the present invention is to provide a camera parameter of a stereo camera that ensures calibration accuracy in a necessary distance range by performing calibration using a camera model whose position changes.

  In order to solve the conventional problems, a stereo camera calibration apparatus according to the present invention includes a compound-eye camera that captures an image, a range sensor that acquires distance information of a scene that is substantially the same as the camera, and the distance information A range data evaluation unit that evaluates the distribution; a calibration unit that calibrates the camera and outputs camera parameters using the camera image, the distance information, and the range data evaluation result; and , Calibration is performed with a camera model in which the pupil position changes for each incident angle.

  With this configuration, it is possible to provide a camera parameter of a stereo camera that ensures calibration accuracy in a necessary distance range.

  According to the stereo camera calibration apparatus of the present invention, when the distance information includes information of a predetermined distance range, it is necessary to perform calibration using a camera model in which the pupil position changes for each incident angle. It is possible to provide camera parameters of a stereo camera that ensures calibration accuracy in a wide distance range.

1 is a configuration diagram of a stereo camera calibration apparatus according to Embodiment 1 of the present invention. Illustration of the phenomenon that the pupil position moves according to the incident angle Illustration of camera model with fixed pupil position regardless of incident angle Configuration diagram of a monocular camera in a modification of the first embodiment of the present invention Configuration diagram of compound eye camera according to a modification of Embodiment 1 of the present invention Configuration diagram of a calibration apparatus for a stereo camera according to Embodiment 2 of the present invention Configuration diagram of stereo camera calibration apparatus according to Embodiment 3 of the present invention Configuration diagram of a conventional stereo calibration device

  Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment 1)
FIG. 1 is a configuration diagram of a stereo camera calibration apparatus according to the first embodiment. The stereo camera calibration apparatus shown in FIG. 1 includes a compound eye camera 101, a range sensor 102, a range data evaluation unit 103, and a calibration unit 104. The compound eye camera 101 captures a subject from different viewpoint positions. An example of the compound eye camera 101 has a wide-angle lens. The range sensor 102 measures three-dimensional shape information in a range including the visual field of the compound eye camera 101. The range data evaluation unit 103 evaluates whether or not the range data measured by the range sensor 102 includes a plurality of distance ranges. The calibration unit 104 calibrates the camera parameters of the compound eye camera 101 with reference to a predetermined standard recorded in the memory. For example, the range data evaluation unit 103 and the calibration unit 104 are realized by a control circuit.

  The necessity for evaluation by the range data evaluation unit 103 will be described below. In-vehicle cameras use an optical system with a wide viewing angle such as a fisheye lens in order to photograph obstacles over a wide range. FIG. 2 shows a situation of photographing using the optical system.

  As shown in FIG. 2, it is an objective-side curved surface 201 of the top lens (the most objective-side lens), and a central axis 202 of the optical system and an imaging surface 203 are defined. In FIG. 2, the three-dimensional coordinate values ((X1n, Y1n, Z1n), (X1f, Y1f, Z1f), (X2n, Y2n, Z2n), and (X2f, Y2f, Z2f)) in the subject space on the imaging surface Of the two-dimensional coordinates ((x1, y1), (x2, y2)). Here, (X1n, Y1n, Z1n) and (X1f, Y1f, Z1f) are photographed at (x1, y1). (X2n, Y2n, Z2n) and (X2f, Y2f, Z2f) are photographed at (x2, y2). A wide-angle lens such as a fisheye lens is configured by combining a plurality of lenses. The light from the subject reaches the imaging surface after being refracted a plurality of times on each surface of the plurality of lenses. Here, in order to explain the operation of the entire optical system, the position where the light traveling straight on the objective-side curved surface of the top lens intersects the optical axis is considered as the entrance pupil position, and the same position is considered as the exit pupil position.

  As shown in FIG. 2, the position of the pupil changes depending on whether the incident angle θ is small (θ = θ1) or large (θ = θ2). However, there is no existing camera model used for calibration that takes this pupil position variation into consideration.

  FIG. 3 is an explanatory diagram of calibration using such an existing camera model. 3, the same components as those in FIG. 2 are denoted by the same reference numerals as those in FIG. In FIG. 3, the pupil position 301 does not change regardless of the incident angle θ. Therefore, the incident angles θ′1 and θ′2 with respect to the optical axes of (X1n, Y1n, Z1n) and (X2n, Y2n, Z2n) are different from the incident angles θ1 and θ2 in FIG. However, the imaging positions of (X1n, Y1n, Z1n) and (X2n, Y2n, Z2n) are (x1, y1) and (x2, y2), respectively, as in the case of FIG. However, (X1f, Y1f, Z1f) and (X2f, Y2f, Z2f) that are further from the lens than (X1n, Y1n, Z1n), (X2n, Y2n, Z2n) are the pupil position 301 and (X1n, Y1n, Z1n). ), (X2n, Y2n, Z2n) away from the straight line, which means that the imaging positions of (X1f, Y1f, Z1f), (X2f, Y2f, Z2f) are (x1, y1), (x2, It shows that the position is different from y2).

  Based on the above, with existing calibration methods that do not consider pupil position fluctuations, calibration can be performed correctly near the depth of the subject used for calibration (calibration plane), but at positions away from the calibration plane. For the subject, the calibration accuracy decreases. Therefore, in order to increase the calibration accuracy, it is most desirable to have a subject with different distances in all directions. An example of this is a scene where snow, fallen leaves, etc. dance. Further, as a preferable scene according to such a scene, a situation where there are subjects at different distances in a specific direction (angle range) is desirable. Such examples include scenes that include street trees and high-rise buildings.

  Whether or not a plurality of distance ranges are included can be evaluated by the distribution or frequency distribution of distance data for each angle range. Specifically, these numerical values can be obtained by evaluating the dispersion or frequency distribution of distance data for each block in the image or for each window region set around the pixel of interest. In other words, the range data evaluation unit 103 determines that the angle is greater when the variance of the distance data for each angle range within the viewing angle is greater than a predetermined threshold, or when the variation in the frequency data frequency distribution is greater than the predetermined threshold. Output a high evaluation value for the range.

  Returning to FIG. 1, when the range data evaluation unit 103 determines that the range data evaluation unit 103 includes a plurality of distance ranges, the calibration unit 104 performs calibration using a camera model in which the pupil position changes for each incident angle. The calibration in the calibration unit 104 will be described below.

  First, a three-dimensional coordinate value (Xw, Yw, Zw) in the world coordinate system and a three-dimensional coordinate value (Xc, Y) in a coordinate system based on each camera (a camera coordinate system in which the optical axis of each camera is the Z axis). Yc, Zc) is represented by (Equation 1).

  In (Equation 1), r11 to r33 are elements of a rotation matrix representing rotation in the three-dimensional space, and Tx, Ty, and Tz represent translational movement in the three-dimensional space. These parameters are called external parameters.

  Next, the relationship between the incident angle θ to the optical axis of the camera and the three-dimensional coordinate values (Xc, Yc, Zc) in the camera coordinate system is expressed by (Equation 2).

  As the relationship between the incident angle θ and the image height r in the wide viewing angle camera that is the subject of the present invention, there are a plurality of relations such as equidistant projection, stereoscopic projection, and equisolid angle projection. The case will be described. The relationship between the incident angle θ and the image height r in equidistant projection is expressed by (Equation 3).

  The pixel coordinate value of the image that is not photographed is expressed in the form of (Formula 4) by the image height r and the azimuth angle γ.

  In (Expression 4), k represents the size of one pixel (here, a square pixel having the same size in the x direction and the y direction is considered). F and k in (Equation 3) and (Equation 4) are called internal parameters.

As described above, the three-dimensional coordinate values (Xw, Yw, Zw) in the world coordinate system are associated with the pixel coordinate values (x, y) by the relationship of (Equation 1) to (Equation 4). In the calibration, a plurality of points with known three-dimensional coordinate values (Xw, Yw, Zw) in the world coordinate system are photographed, and a set (Xw, Yw, Zw, x) with their imaging positions (x _true , y _true ). _true , y _true ) are prepared as calibration data, and the camera parameters (internal parameters and external parameters) are expressed so as to most accurately represent the relationship between (Xw, Yw, Zw) and (x _true , y _true ) in these sets. Parameter).

  The calculation will be described below.

と表すと、ｇｘ（），ｇｙ（）を用いて関係式から計算した画素座標（ｘ，ｙ）と実際の撮像位置（ｘ_ｔｒｕｅ，ｙ_ｔｒｕｅ）との差異は以下の再投影誤差として評価できる。 The relationship between (Xw, Yw, Zw) and (x, y) shown in (Equation 1) to (Equation 4) is

In other words, the difference between the pixel coordinates (x, y) calculated from the relational expression using gx () and gy () and the actual imaging position (x _true , y _true ) can be evaluated as the following reprojection error. .

The camera parameters are calculated by obtaining camera parameters (r11 to r33, T _X , T _Y , T _Z , f, k) that minimize the evaluation value of (Equation 6) as shown in (Equation 7). .

  The above calculation becomes a non-linear optimization problem, and is performed by a known optimization calculation method such as the Levenberg-Merquardt method or grid search.

  Here, in the calibration of the existing wide-angle camera and fish-eye camera, the focal length f is constant regardless of the incident angle θ as shown in FIG. 3, but in the present invention, for example, an LUT (Look Up Table) or the like is used. Is allowed to change according to the incident angle. Thereby, it is possible to reduce the occurrence of calibration errors caused by the existing calibration method at a position away from the calibration plane shown in FIG.

FIG. 4 is a configuration diagram for explaining the operation of the calibration unit 104. In FIG. 4, a 3D coordinate-2D coordinate association unit 401 associates a 3D coordinate value with a 2D coordinate value from the association between reflectance data included in range data and a compound-eye camera image. Xw, Yw, Zw, x _true , y _true ). The evaluation value minimizing unit 402 calculates and outputs camera parameters by calculating (Equation 7) for a camera model having a focal length f that changes according to the incident angle.

  In the present embodiment, the case of equidistant projection has been described as the projection model. However, the present invention is not limited to this, and other projection models such as stereoscopic projection and equisolid angle projection may be used.

(Modification of Embodiment 1)
In the first embodiment, the calibration of the stereo camera has been described. However, the number of cameras is not necessarily two. FIG. 5 shows the configuration of the calibration apparatus when there are three or more cameras. In FIG. 5, the same components as those in FIG. 1 are denoted by the same reference numerals as those in FIG. In FIG. 5, a camera 501 is a compound eye camera having three or more eyes, and a corresponding point estimation unit 502 performs corresponding point estimation for a plurality of stereo pairs (two-eye pairs) selected from the compound eye camera. A three-dimensional coordinate estimation unit 503 estimates a three-dimensional coordinate value from the plurality of corresponding point estimation results using initial camera parameters for each camera of the compound eye camera. The three-dimensional coordinate evaluation unit 504 evaluates the camera parameters of each stereo pair from the distribution of three-dimensional coordinate values estimated from the plurality of corresponding point estimation results. That is, if the camera parameters are correct for all cameras, the three-dimensional coordinate values estimated from each stereo pair are concentrated at the same position, but the three-dimensional coordinate values cannot be estimated correctly in a stereo pair including a camera with incorrect camera parameters. Therefore, the three-dimensional coordinate value is estimated at a position away from the estimated value by the other stereo pair. From this, when the camera parameters of most cameras are accurate, a camera whose camera parameters are not accurate can be determined from the distribution of the three-dimensional coordinate values.

  The calibration unit 505 calibrates the parameters of the camera whose reliability is low by the three-dimensional coordinate evaluation unit 504 using the three-dimensional coordinate estimation value obtained by the camera whose reliability is low by the three-dimensional coordinate evaluation unit 504. Perform Similarly to the first embodiment, when the estimated three-dimensional coordinate value is not uniform and includes a plurality of distance ranges, the focal length is determined according to the viewing angle by performing calibration preferentially. Calibration is performed using a camera model in which f varies.

  With the above configuration, if all the cameras of the compound eye camera having three or more eyes are calibrated in the initial state, the compound eye camera can be calibrated without using the range sensor.

(Embodiment 2)
FIG. 6 is a configuration diagram of the stereo camera according to the second embodiment of the present invention. In FIG. 6, the same components as those in FIG.

  In FIG. 6, a still area extraction unit 601 extracts a still area from the scene. The engine operation sensor senses the engine operation of a vehicle equipped with a compound eye camera. The calibration unit 603 calibrates the compound eye camera using information on the still area when the engine is stopped.

  With the above configuration, according to the present embodiment, calibration of a compound eye camera can be performed even when the frame rates of the compound camera and the range sensor are different.

(Embodiment 3)
FIG. 7 is a configuration diagram of the stereo camera according to the third embodiment of the present invention. In FIG. 7, the same components as those in FIGS. 1 and 6 are denoted by the same reference numerals, and description thereof is omitted.

  In FIG. 7, the route recommendation unit 701 refers to the real world shape database 702, and if the arrival times are approximately the same, the route recommender includes a route suitable for calibration (a plurality of non-uniform distance information such as street trees and concept buildings). Course) is given priority.

  With the above configuration, according to the present embodiment, calibration of the in-vehicle compound eye camera can be performed with high accuracy even while traveling.

  The stereo camera calibration apparatus according to the present invention has a camera model in which the pupil position changes for each incident angle, and a range data evaluation unit that evaluates the distribution or frequency distribution of distance data for each angle range, and for all directions. Wide angle of fish-eye lens etc. in scenes with snow and fallen leaves with different distance subjects, and street scenes with different distance subjects and skyscrapers in a specific direction (angle range) It is useful as a device for camera calibration.

DESCRIPTION OF SYMBOLS 101 Compound eye camera 102 Range sensor 103 Range data evaluation part 104 Calibration part 502 Corresponding point estimation part 503 3D coordinate estimation part 504 3D coordinate evaluation part

Claims (6)

A camera for acquiring a camera image including a plurality of objects;
A range sensor for acquiring distances to the plurality of objects;
A control circuit,
The control circuit includes:
(A1) calculating a distribution of distance values of the plurality of objects,
(A2) obtaining a plurality of camera parameters of the camera;
(A3) When the distribution of the distance values satisfies the first condition, the camera parameter of the camera is calibrated with reference to a predetermined criterion, where the first condition is the plurality of the conditions The distance value of the object is distributed in a distance range equal to or greater than the first threshold value.
Camera calibration device. The control circuit includes:
In (a1), for each of a plurality of viewing directions of the camera, a distance distribution of the plurality of objects is calculated,
In (a3), when the first condition is satisfied for each visual field direction, the camera parameters of the camera are calibrated with reference to a predetermined standard.
The camera calibration device according to claim 1. The first condition is that a variance value of distance values of the plurality of objects is greater than or equal to a second threshold value, or a variance value of frequency for each section related to the distance values of the plurality of objects is greater than or equal to a third threshold value. Is,
The camera calibration device according to claim 1. (B1) obtaining a camera image including a plurality of objects imaged by the camera;
(B2) acquiring distances to the plurality of objects acquired by the range sensor;
(B3) calculating a distribution of distance values of the plurality of objects;
(B4) obtaining a plurality of camera parameters of the camera;
(B5) When the distribution of the distance values satisfies the first condition, the camera parameter of the camera is calibrated with reference to a predetermined criterion, where the first condition is the plural The distance value of the object of is distributed over a distance range equal to or greater than the first threshold,
Executing at least one of (b1) to (b5) by a control circuit;
Camera calibration method. In (b3), for each of a plurality of viewing directions of the camera, a distance distribution of the plurality of objects is calculated,
In (b5), when the first condition is satisfied for each visual field direction, the camera parameters of the camera are calibrated with reference to a predetermined standard.
The camera calibration method according to claim 4. The first condition is that a variance value of distance values of the plurality of objects is greater than or equal to a second threshold value, or a variance value of frequency for each section related to the distance values of the plurality of objects is greater than or equal to a third threshold value. Is,
The camera calibration method according to claim 4.

Images