JP4803449B2 - On-vehicle camera calibration device, calibration method, and vehicle production method using this calibration method - Google Patents

Description

  The present invention relates to an in-vehicle camera calibration apparatus and a calibration method for calibrating an in-vehicle camera attached to a vehicle.

  In recent years, an increasing number of vehicles are equipped with cameras so that the driver of the vehicle can visually recognize a scene such as the side or the rear of the vehicle via a monitor device in the vehicle. Furthermore, an apparatus that supports driving such as parking by performing image processing using a photographed image of the camera has been developed. In particular, when a camera that captures an image with such image processing is mounted on a vehicle, a relatively high assembly accuracy is required. Accordingly, various efforts have been made to improve the accuracy of mounting the camera on the vehicle.

  Japanese Patent Application Laid-Open Publication No. 2005-151867, which is cited below, describes a technique for calibrating such a camera. This is to perform calibration by matching the calibration index photographed by the camera with the adjustment window superimposed on the screen on the display screen by operating the adjustment button. There are three types of adjustment windows corresponding to pan, tilt, and roll corresponding to three axes of three-dimensional orthogonal coordinates. Sequentially, the camera parameters are changed by operating the adjustment buttons so that the calibration index is within these adjustment windows. When the adjustment for the three axes is completed, the changed camera parameters are written to the EEPROM.

Japanese Patent Laid-Open No. 2001-245326 (see paragraphs 16, 34-44, FIG. 3, FIG. 6, etc.)

  The technique described in Patent Document 1 is an excellent technique that enables camera calibration with a simple configuration. However, since it is necessary to manually operate the adjustment buttons while looking at the display screen, it takes time for calibration. In addition, since adjustment with respect to the three axes is necessary corresponding to the rotation of the camera, time is required for calibration. In addition, in manual calibration, the calibration accuracy is not stabilized by the operator, and a product with insufficient calibration may be sent to a subsequent process.

  The present invention has been made in view of such a problem, and an object thereof is to provide an in-vehicle camera calibration device capable of calibrating an in-vehicle camera with a simple configuration and in a short time with high accuracy.

In order to achieve the above object, an in-vehicle camera calibration device according to the present invention for calibrating an in-vehicle camera attached to a vehicle has the following configuration,
An image receiving unit that receives a photographed image of the in-vehicle camera that includes at least two calibration indices arranged in different places in the field of view;
A calibration point specifying unit for specifying each calibration point in the calibration index in the captured image;
A first matrix calculation for calculating a first matrix indicating a vector related to the calibration point viewed from the optical center of the vehicle-mounted camera and a straight line passing through the calibration point based on the coordinates of the calibration point of the calibration index in a reference coordinate system And
Based on the coordinates of the calibration point on the captured image identified by the calibration point identifying unit, a second matrix indicating a vector related to the calibration point viewed from the optical center of the in-vehicle camera and a straight line passing through the calibration point. A second matrix calculation unit for calculating;
And a third matrix calculation unit that calculates a rotation matrix indicating a rotation state of the in-vehicle camera in the reference coordinate system based on the first matrix and the second matrix.

A rotation matrix representing the rotation state of the camera in a reference coordinate system (generally the world coordinate system) can be obtained from two images before and after the rotation. However, an in-vehicle camera attached to a vehicle in a certain rotation state cannot handle images before and after the rotation because the rotation state is fixed. On the other hand, in this configuration, the first matrix is calculated based on the coordinates of the calibration point of the calibration index in the reference coordinate system. This first matrix shows the camera state when the in-vehicle camera only translates without rotation. On the other hand, the second matrix indicates the camera state of the in-vehicle camera attached to the vehicle and having some rotation state. Assuming that the first matrix is before rotation and the second matrix is after rotation, a rotation matrix indicating the rotation state of the in-vehicle camera in the reference coordinate system can be obtained based on the first matrix and the second matrix.
The first matrix can be obtained by geometric calculation from the arrangement of calibration indices. Further, the second matrix can be obtained by performing image processing on a captured image of the in-vehicle camera. Therefore, according to this configuration, it is possible to automatically obtain a rotation matrix indicating the rotation state of the in-vehicle camera without manually operating the adjustment button while looking at the display screen. As a result, it is possible to provide an in-vehicle camera calibration apparatus capable of calibrating the in-vehicle camera with a simple configuration and in a short time with high accuracy.

  The in-vehicle camera calibration apparatus according to the present invention may further include a rotation angle calculation unit that decomposes the rotation matrix into three rotation angle components with respect to three axes orthogonal to each other in a three-dimensional space.

  The rotation matrix does not represent the rotation amount for each axis of the three-dimensional orthogonal coordinates as it is. When the rotation angle calculation unit is decomposed into rotation angle components for the respective axes, it can be favorably reflected in the projection parameters of the in-vehicle camera.

  In addition to the above configuration, the in-vehicle camera calibration device according to the present invention can include a region setting unit that sets a region including the calibration point on the captured image. The calibration point specifying unit specifies the calibration point in the region.

  Since the region including the calibration point is set by the region setting unit, the calibration point specifying unit can specify the calibration point from the captured image with a small amount of calculation. In addition, the calibration point can be specified with high accuracy.

In addition, the in-vehicle camera calibration device according to the present invention has the above configuration,
Based on the rotation matrix, a projection parameter calculation unit that calculates the projection parameters of the in-vehicle camera,
A coordinate conversion unit that converts the coordinates of the calibration point in the reference coordinate system into coordinates on the captured image using the projection parameters;
A limit area calculation unit that calculates a limit area of the calibration error on the captured image based on the coordinates of the calibration point that has undergone coordinate conversion and an allowable value of the calibration error with respect to the rotation of the in-vehicle camera. Can do.

  According to this configuration, the accuracy of the calibration result of the in-vehicle camera can be verified based on the calculated limit region. In manual calibration, the accuracy of calibration is not stabilized by the operator, and a product with insufficient calibration may be sent to a subsequent process. However, according to this configuration, since the accuracy of the calibration result can be verified quantitatively, the calibration accuracy is stabilized.

  Furthermore, the in-vehicle camera calibration device according to the present invention may include a superimposing unit that superimposes the highlighted display of the center coordinates of the calibration point specified by the calibration point specifying unit and the limit region on the captured image. it can.

  According to this configuration, the calibration point specified on the captured image and the limit region are superimposed on the captured image, so that the calibration result of the in-vehicle camera can be easily verified.

  Further, the in-vehicle camera calibration device according to the present invention is configured to determine whether the center coordinates of the calibration point specified by the calibration point specifying unit are within the limit region on the captured image. Can have a part.

  According to this configuration, it is accurately determined whether or not the calibration point specified on the captured image exists within the limit region. When the determination is made only by the visual recognition by the worker, there may be a variation in the determination for each worker when the marginal region is at the limit. However, according to this configuration, accurate determination with high reproducibility is possible.

  Furthermore, the in-vehicle camera calibration device according to the present invention may include a notification unit that notifies the determination result by the determination unit at least visually or audibly.

  According to this configuration, the determination result is surely notified to the worker, so that the worker can quickly implement measures such as re-calibration, re-installation and replacement of the in-vehicle camera.

  In the on-vehicle camera calibration device according to the present invention, the calibration point of the calibration index may be represented as an intersection of two straight lines on a plane.

  A straight line on a plane can be detected more accurately than a curved line. If two straight lines can be detected accurately, the intersection of the two straight lines can also be detected accurately. Since the calibration point of the calibration index can be accurately detected, the in-vehicle camera can be calibrated with high accuracy.

  In the on-vehicle camera calibration device according to the present invention, the calibration index may be arranged on any surface including a floor surface and a wall surface.

  Since the calibration index can be arranged on an arbitrary surface, the vehicle-mounted camera can be calibrated regardless of conditions unique to each factory such as the size of the production factory or repair factory.

Moreover, the calibration method of the in-vehicle camera according to the present invention for calibrating the in-vehicle camera attached to the vehicle includes the following steps,
A calibration index placement step for placing calibration indices in at least two different locations, including in the field of view of the in-vehicle camera;
Based on the coordinates of the respective calibration points in the calibration index in the reference coordinate system, a first matrix indicating a vector relating to the calibration point viewed from the optical center of the in-vehicle camera and a straight line passing through the calibration point is calculated. One matrix operation step;
An image receiving step for receiving a photographed image of the in-vehicle camera including the calibration index in a field of view;
In the captured image, a calibration point specifying step for specifying each of the calibration points;
Based on the coordinates of the calibration point specified on the photographed image, a second matrix calculation that calculates a second matrix indicating the calibration point viewed from the optical center of the in-vehicle camera and a vector relating to a straight line passing through the calibration point. Process,
And a third matrix calculation step of calculating a rotation matrix indicating a rotation state of the in-vehicle camera in the reference coordinate system based on the first matrix and the second matrix.

  According to this vehicle-mounted camera calibration method, the vehicle-mounted camera can be calibrated in a short time in the same manner as the vehicle-mounted camera calibration device described above. In addition, the method according to the present invention can be provided with the operational effects related to the above-described apparatus, and all the additional features and operational effects.

A vehicle production method according to the present invention for sequentially performing a plurality of assembly adjustment steps including an assembly adjustment step of calibrating the in-vehicle camera using the in-vehicle camera calibration method according to the present invention includes the following features: Have
The in-vehicle camera calibration method further includes the following steps:
A projection parameter calculation step of calculating a projection parameter of the in-vehicle camera based on the rotation matrix;
A coordinate conversion step of converting the coordinates of the calibration point in the reference coordinate system into coordinates on the captured image using the projection parameters;
A limit area calculation step of calculating a limit area of the calibration error on the captured image based on the coordinates of the calibration point that has been coordinate-transformed and an allowable value of the calibration error with respect to the rotation of the in-vehicle camera;
A determination step of determining whether or not the center coordinates of the calibration point specified in the calibration point specifying step exist in the limit region on the captured image;
And a limiting step of limiting the shift to the next assembly / adjustment operation after the calibration of the in-vehicle camera when the center coordinates of the calibration point do not exist within the limit region.

  According to this configuration, even if the operator misses the calibration when the calibration of the in-vehicle camera is not achieved, the production process cannot proceed. Therefore, it is possible to suppress the possibility that a product with insufficient calibration is sent to a subsequent process, and the operator can quickly implement measures such as re-calibration.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is an explanatory diagram showing an example of an arrangement relationship between the calibration marker 80 (calibration index) and the vehicle 90. The camera 1 (on-vehicle camera) is installed at a position offset laterally from the axle above the license plate behind the vehicle 90 with the optical axis facing downward (for example, 30 ° downward from the horizontal). The camera 1 is, for example, a wide-angle camera with a horizontal viewing angle of 110 to 120 °, and can capture an area up to about 8 m behind. The camera 1 is calibrated to absorb mounting errors when mounted on the vehicle 90 at a vehicle production factory or the like. Further, in a repair shop or the like, calibration is performed to correct the displacement of the camera 1 due to the accumulation of vibration caused by running or impact. Hereinafter, calibration in a production factory will be described as an example.

[Calibration index placement process]
As shown in FIG. 1, the calibration of the camera 1 is performed in a state where the vehicle 90 is stopped at a predetermined position. For example, if the vehicle 90 is moved back or forward and the wheels are stopped by a tire groove or a tire stop provided at a predetermined position, the vehicle 90 can be stopped at an accurate position. In the example shown in FIG. 1, two markers 80 (80a, 80b) are arranged on the floor surface. In FIG. 1, the vehicle 90 is stopped so that the floor surface at the center of the rear end is the origin O _{W of the} world coordinate system (reference coordinate system, X _W , Y _W , Z _W ). In the camera coordinate system (X _C , Y _C , Z _C ) centered on the optical center O _C of the camera 1, each axis is not always parallel to the world coordinate system. World coordinate system, a camera coordinate system are both right-handed coordinate system, in the figure, X _W axis in a direction perpendicular to the paper surface, and is not shown X _C-axis of the substantially vertical direction. The three-dimensional relationship between the world coordinate system and the camera coordinate system is shown in FIG.

  The markers 80 are arranged in at least two places within the visual field range of the camera 1. Further, the marker 80 is arranged so that its coordinates are known in the world coordinate system. In this example, the marker 80 has a black and white checkered pattern as shown in FIG. A point Q at the center of the pattern is a calibration point, which is a reference for calibration of the camera 1. That is, the coordinates of the calibration point Q are arranged so that the coordinates are known in the world coordinate system. Here, an example of a total of four rectangles including two white rectangles and two black rectangles is shown, but a total of four or more rectangles may be used, and the number is not limited.

In the example shown in FIG. 1, the two markers 80 are arranged on the floor symmetrically with respect to the main axis of the vehicle (the Z _W axis of the world coordinate system) (D ₁ = D ₂ , W ₁ = W ₂ ). However, it is not always necessary to be left-right symmetric. If the coordinate values are known within the field of view of the camera 1, the arrangement is free. That is, the arrangement of the markers 80 can be arbitrarily set depending on the area that can be secured in the factory and the relationship with other equipment.

  The dimension of the marker 80 is appropriately determined so that the calibration point Q can be detected with high accuracy in accordance with the resolution of the camera 1, the performance of the image processing function provided in the calibration apparatus, the arrangement position of the marker, and the like. As an example, when Di is 1 to 2 m and Wi is about 1.5 m, a marker 80 of 10 to 15 cm square for each black and white and 20 to 30 cm square for the whole is used.

As shown in FIGS. 3 and 4, the marker 80 may be installed on the wall surface or on the partition 85. In this case, the world coordinates (X _W , Y _W , Z _W ) of the calibration point Q are, for example, (H _3, W ₃ , D ₃ ) and can be accurately grasped. That is, the marker 80 can be placed on any surface including the floor surface and the wall surface within the field of view of the camera 1. In this embodiment, since the camera 1 is mounted on the vehicle facing downward, it is not appropriate to place the marker 80 on the ceiling surface. However, when the ceiling surface falls within the field of view of the camera 1, it may be arranged on the ceiling surface.

The marker 80 does not need to be perpendicular to the central axis (Z _W axis) of the vehicle 90. For example, you may arrange | position so that it may face with respect to the camera 1 as shown in FIG. Of course, it is not necessary to be perpendicular to the other axes (Y _W axis and X _W axis). Of course, one of the two markers 80 may be arranged on a different surface, such as one on the floor and the other on the partition. What is important is the calibration point Q, and the marker 80 may be arranged on any surface. That is, if the calibration point Q is accurately arranged in the world coordinate system and the calibration point Q can be detected satisfactorily from the captured image of the camera 1, the installation direction of the marker 80 is arbitrary.

〔System configuration〕
FIG. 5 is a block diagram schematically showing the configuration of the on-vehicle camera calibration device of the present invention. As shown in the figure, the calibration device includes an image receiving unit 2, an image output unit 3, a system control unit 5, an input I / F unit 6, an output I / F unit 7, an image processing unit 10, have.
The image receiving unit 2 receives an image captured by the camera 1 which is a digital video camera via a known image I / F such as a buffer 2a, a synchronization separation unit 2b, a clock generation unit 2c, an A / D converter 2c, and the like. 2e (image receiving step). An image data control unit 11 provided in the image processing unit 10 controls the image receiving unit 2. For example, it controls the storage of the captured image in the frame memory 2e and the reading of the captured image from the frame memory 2e.

The image processing unit 10 further includes a distortion correction unit 12, a region setting unit 13, a calibration point specifying unit 15, a rotation amount calculation unit 17, and an applied function processing unit 19. FIG. 6 is a block diagram schematically showing the configuration of the rotation amount calculation unit 17 of FIG. The rotation amount calculation unit 17 includes a first matrix calculation unit 21, a second matrix calculation unit 22, a third matrix calculation unit 23, and a rotation angle calculation unit 25. Although detailed functions of each part will be described later, the rotation amount of the camera 1 is calculated by the function of each part.
FIG. 7 is a block diagram schematically showing the configuration of the applied function processing unit 19 in FIG. The applied function processing unit 19 includes a projection parameter calculation unit 31, a coordinate conversion unit 33, a limit area calculation unit 35, and a success / failure determination unit 37 (determination unit). Although detailed functions of each unit will be described later, the projection parameters of the camera 1 are calculated by the operation of these units, and the accuracy of the calibration result is verified.

  The image output unit 3 generates an image signal for displaying the captured image on a display or the like based on the obtained rotation amount of the camera 1 and projection parameters. The image output unit 3 includes a display controller 3a and a drawing unit 3b. The drawing unit 3b draws a superimposition such as a verification result of calibration accuracy based on the result of image processing. The display controller 3a superimposes this superimpose on the captured image and outputs an image signal. The image output unit 3 also functions as a superimposing unit and a notification unit of the present invention.

  The input I / F unit 6 receives an input signal such as an instruction to start calibration of the camera 1 from a system higher than the calibration device of the camera 1 and transmits it to the system control unit 5. Here, the host system is an assembly adjustment system of the vehicle 90, a back monitor system of the vehicle 90, or the like. The system control unit 5 controls the entire calibration apparatus of the camera 1 and controls the image processing unit 10 and the like based on an instruction to start calibration. The output control unit 7 receives an image processing result or the like via the system control unit 5 and outputs an output signal to a host system, a notifying unit of the calibration apparatus, or the like.

Note that each unit of the image processing unit 10 indicates a sharing as a function, and is not necessarily provided independently. Of course, each function may be realized by cooperation of hardware such as a microcomputer and software such as a program executed on the hardware.
Hereinafter, a procedure of image processing by the image processing unit 10 will be described.

(Distortion correction process)
As described above, the camera 1 is a wide-angle camera. Therefore, the image taken by the camera 1 is an image U1 having distortion as shown in FIG. In the image U1, the bumper portion 91 of the vehicle 90 is shown below, and the wall surface of the room where the vehicle 90 is stopped is shown above and to the side. Since the internal parameters (focal length, etc.) of the camera 1 are known, the distortion correction unit 12 corrects the distortion. FIG. 9 shows the corrected image U2.

[Region setting process]
The marker 80 is arranged at a known coordinate in the world coordinate system. Also, detailed rotational state (external parameters) in the world coordinate system of the camera 1 is of course unknown, rough optical axis direction (Z _C direction in FIG. 1) are known. That is, since the camera 1 is attached to the vehicle 90 within a predetermined tolerance range, the approximate direction of the optical axis direction of the camera 1 is known. Therefore, the position of the marker 80 in the captured image U1 and the captured image U2 whose distortion has been corrected can be predicted. Therefore, in the image U2 (or U1), a region where the marker 80 including the calibration point Q is set is set as an ORI (Region of Interest). The ORI is set based on a position and size determined in consideration of a tolerance range and a margin for image processing with reference to a position on the image based on an average value of the tolerance. Therefore, ideally, the calibration point Q falls within the center of the ORI.
FIG. 10 shows an example of the ORI. 10A shows the ORI set for the marker 80a in FIG. 1, and FIG. 10B shows the ORI set for the marker 80b in FIG.

[Calibration point identification process]
Next, the calibration point specifying unit 15 detects the coordinates on the captured image of the calibration point Q, which is the center point of the marker 80. When a checkered pattern is used as the marker 80 as in this embodiment, the calibration point Q can be detected using a known corner detector. As a corner detector, a Tomasi-Kanade detector or a Harris detector can be used.
However, if the camera 1 is an inexpensive camera, the image information may be modified by aperture or coring techniques. In this case, there is a possibility that the correct intersection position (calibration point Q) cannot be detected even if the above corner detector is used due to low image quality. It is desirable that the in-vehicle camera calibration device is compatible with various types of vehicles and various cameras. Therefore, in this embodiment, the calibration point Q is detected by a more general method. Hereinafter, the calibration point specifying step for detecting the horizontal / vertical boundary line of the marker 80 and obtaining the coordinates of the calibration point Q, which is the intersection point, is divided into an outline (edge) detection step, a straight line fitting step, and an intersection point calculation step. explain.

[Calibration point identification process / contour line (edge) detection process]
In this embodiment, a Canny edge detector incorporating a Gaussian filter is used. The Canny edge detector has a function of preserving edge features and removing noise contained in an image, and enables stable edge detection. For Canny edge detectors, JFCanny "Finding edge and lines in images". Master's thesis, AITR-720. MIT, Cambridge, USA, 1983 and JFCanny "A computational approach to edge detection". IEEE Trans. On Pattern Analysis and Since it is familiar with Machine Intelligence, 8 (6): 679-698, November 1986, detailed explanation is omitted here. Needless to say, the edge detection may be performed not only using the Canny edge detector but also using other methods.
When edge detection processing is performed on the image in the ROI area shown in FIG. 10, an edge point group E (contour pixel) corresponding to the horizontal / vertical boundary line of the marker 80 is detected as shown in FIG. Is done.

[Calibration point identification process / Line fitting process]
Next, a straight line fitting process is performed on the edge point group E. For this straight line fitting process, it is possible to use the least square method or the Hough transform, but the calculation load is heavy. In the present embodiment, straight-line fitting is performed using a RANSAC (RANdom SAmple Consensus) method with a much smaller calculation amount than these. In the RANSAC method, the minimum number of points necessary for fitting a straight line or curve is selected from the edge point group E, and a fitting model is set. In the case of a straight line, the minimum number of points is two, and a straight line connecting two randomly selected points is set as a straight line model. Then, how much the other points of the edge point group E are fitted to this linear model is evaluated. Sequentially, two points are selected and evaluation is repeated, and the most fitted straight line model is determined as a straight line. FIG. 12 shows an example in which a straight line G is applied to the edge point group E shown in FIG.

[Calibration point identification process / intersection calculation process]
When the straight line G is fitted, it is first confirmed that the straight line G is not significantly different from the assumption as the horizontal / vertical boundary line of the marker 80. The straight line G is represented by a linear function in the coordinate system of the captured image, that is, the camera coordinate system. Therefore, the fitted straight line can be evaluated by a simple calculation.
Next, the coordinates of the intersection point of the fitted straight line G are calculated. Since the straight line G is a linear function, the intersection coordinates (calibration point Q) in the camera coordinate system can be obtained by simple calculation (see FIG. 13).

[Modification]
In the above description, the distortion correction process, the area setting process, and the contour line (edge) detection process are performed in this order, but the area setting process and the contour line (edge) detection process may be performed first. In other words, the image may be distorted before the straight line fitting, so there is no problem even if the order is changed in consideration of the calculation load.

  The coordinate value of the calibration point Q in the world coordinate system is known when the marker 80 is placed. Also, the coordinate values in the camera coordinate system were obtained by calculation as described above. Hereinafter, a procedure for obtaining an external parameter of the camera 1 indicating the rotation state of the camera 1 using these coordinate values will be described.

〔principle〕
In a device that requires conversion between a two-dimensional image and a three-dimensional space, such as a vehicle back monitor device or a driving support device, it is necessary to know all of the camera parameters for the conversion. As described above, in Patent Document 1, of the camera parameters, the internal parameters and the translational components of the external parameters are known, and only the rotational component having a large assembly tolerance is manually calibrated. In the present invention, this rotational component is automatically detected. First, the principle will be described, and then a specific procedure will be described.

FIG. 14 is an explanatory diagram showing the relationship between the world coordinate system and the camera coordinate system. As shown in FIGS. 1 and 3, the vehicle 90 and the marker 80 are arranged at coordinates defined by a world coordinate system (X _W , Y _W , Z _W ) with the origin being O _W. Hereinafter, the calibration points Q of the two markers 80 are represented by _two points P ₁ and P ₂ in space. The camera 1 is arranged to have a translation component T (T _X , T _Y , T _Z ) in the world coordinate system. The camera 1 is arranged with a rotation component R with respect to the world coordinate system. That is, there is a camera coordinate system (X _C , Y _C , Z _C ) with a rotation component R, with the position with the translation component T as the origin O _C in the world coordinate system.

  Here, both the world coordinate system and the camera coordinate system use a right-handed orthogonal coordinate system. The right-hand coordinate system is a method for determining the X, Y, and Z axes in the order in which the thumb, index finger, and middle finger of the right hand are opened. In the right-handed system, as shown in FIG. 15, the rotation direction when the axis advances in the direction of the right-hand thread is defined as the positive direction of rotation. Here, rotation around the X axis is referred to as pan, rotation around the Y axis is referred to as tilt, and rotation around Z is referred to as roll.

  FIG. 16 is an explanatory diagram showing the N vector. First, the N vector will be described. When an image plane having an orthogonal coordinate system is regarded as a two-dimensional projection space (projection plane), points and straight lines are represented by homogeneous coordinates composed of three real numbers that are not all zero. The homogeneous coordinates are suitable for solving calculation problems in image processing. The simplest means for solving is to make sure that the three elements of homogeneous coordinates are always unit vectors. A normalized vector set of homogeneous coordinates expressed as a unit vector is referred to as an N vector.

  The point P is a point on the projection plane, and f indicates the distance from the viewpoint O to the projection plane (substantially equivalent to the focal length). The Z axis of the three-dimensional orthogonal coordinates coincides with f, the Z axis is orthogonal to the projection plane, and the XY plane is parallel to the projection plane. The N vector m is a unit vector that points to the point P starting from the viewpoint O. The N vector n is a unit normal vector perpendicular to a plane determined by the straight line L passing through the point P and the viewpoint O. That is, the N vector n is a vector representing the straight line L. The vector l is not an N vector but a unit vector representing the direction of the straight line L.

  Here, it is assumed that the camera is rotated with the optical center of the camera (viewpoint O in FIG. 16) fixed. As the camera rotates, the coordinates of the point P and the straight line L on the projection plane change, and the N vector also changes. That is, a straight line L passing through the point P and the point P on the projection surface before rotation and a straight line L ′ passing through the point P ′ and the point P ′ on the projection surface after rotation are given. At this time, the camera rotation matrix R that matches the point P ′ with the point P and matches the straight line L ′ with the straight line L including the direction is uniquely determined as the following equation (1).

R = (l'm'n ') (lmn) ^T (1)

Here, l ′ is a unit vector representing the direction of the straight line L ′, m ′ is an N vector representing the point P ′, and n ′ is an N vector representing the straight line L ′.
Formula (1) can be expressed as the following formula (2) when R ₁ = (lmn) and R ₂ = (l′ m′n ′).

R = R ₂ R ₁ ^T (2)

  That is, as shown in FIG. 17, the rotation vector R of the camera 1 mounted on the vehicle can be obtained from the two images before and after the rotation based on the world coordinate system. However, in the calibration apparatus of the present invention, since the vehicle 90 and the camera 1 are fixed, the number of images input from the camera 1 is one. Moving the vehicle 90 and the camera 1 to obtain a plurality of images increases the calibration time and is not preferable from the viewpoint of calibration accuracy.

However, the essence of the above principle is that if a rotation matrix of a coordinate system determined by {l ′, m ′, n ′} is obtained from a coordinate system determined by {l, m, n}, this is the rotation matrix of the camera 1. It is in agreement. That is, it is sufficient to obtain {l, m, n} from the coordinate system before rotation and {l ′, m ′, n ′} from the coordinate system after rotation.
Therefore, the coordinate system before rotation is considered to be a camera coordinate system (first camera coordinate system) that is purely translated to the optical center O _C of the camera 1 in the world coordinate system. Furthermore, the coordinate system after the rotation is considered the first camera coordinate system is a camera coordinate system that is rotated around the optical center O _C (second camera coordinate system).

[First matrix calculation step]
The first camera coordinate system is a pure translation and rotation components with no coordinate system relative to the origin O _W of the world coordinate system. As shown in FIG. 18, N vectors from the optical center O _C to points P ₁ and P ₂ indicating the calibration point Q of the marker 80 are obtained. Needless to say, in the first matrix calculation step, a photographed image by the camera 1 and various information obtained from the photographed image are unnecessary. This step is executed by the first matrix calculation unit 21 shown in FIG. 6, but the image taken by the camera 1 is not used.

In the present embodiment, since it is disposed in the marker 80 the floor surface, although the X _W axis direction of the coordinate is substantially zero, the coordinates of the point _{P 1 P 1 (X 1,} Y 1, Z 1), The coordinates of the point P ₂ are P ₂ (X ₂ , Y ₂ , Z ₂ ). The straight line L is a straight line from the point P ₁ to the point P ₂ . There is an N vector m _{1 in} the direction from the optical center O _C to the point P ₁ . There is an N vector m _{2 in} the direction from the optical center O _C to the point P ₂ . For ease of explanation, FIG. 18 represents an N vector m ₁ . As a result, the N vector m is expressed by the following equation (3).

The N vector n is a unit normal vector of the plane O _C P ₁ P _{2 and} is expressed by the following equation (4).

  The direction l of the straight line L is obtained as an outer product of m and n as shown in the following formula (5).

      l = m × n (5)

As a result, the rotation matrix R ₁ is expressed by the following equation (6).

R ₁ = (lmn) (6)

In this way, the first matrix calculation unit 21 is viewed from the optical center O _{C of the} camera 1 based on the coordinates of the calibration points Q (P ₁ , P ₂ ) of the marker 80 in the world coordinate system (reference coordinate system). A first matrix R ₁ indicating a vector related to the straight line L passing through the calibration points P ₁ and P ₂ and the calibration points P ₁ and P ₂ is calculated.

[Second matrix calculation step]
The second camera coordinate system is a coordinate system in which the first camera coordinate system is a rotating component rotates about the optical center O _C. Unlike the first matrix calculation process, an image captured by the camera 1 is used. As shown in FIG. 19, calibration points Q (P ₁ , P ₂ ) arranged on the floor correspond to points p ₁ , p ₂ on the projection plane. As described above, the coordinate values of the points p ₁ and p _{2 on} the projection plane are specified by the calibration point specifying unit 15. Therefore, as shown in FIG. 6, this process is executed in the second matrix calculation unit 22 in response to the result of the calibration point specifying unit 15.

Here, consider an N vector from the optical center O _C to points p ₁ (x ₁ , y ₁ ) and p ₂ (x ₂ , y ₂ ) on the projection plane. Note that (x _i , y _i ) are coordinates on the projection plane (on the captured image). A straight line l corresponding to the straight line L is a straight line from the point p ₁ to the point p ₂ . There is an N vector m ₁ ′ in the direction from the optical center O _C to the point p ₁ . There is an N vector m ₂ ′ in the direction from the optical center O _C to the point p ₂ . For ease of explanation, FIG. 19 represents an N vector m ₁ ′. As a result, the N vector m ′ is expressed by the following equation (7).

Further, the N vector n ′ is expressed as the following equation (8) as a unit normal vector of the plane O _C p ₁ p ₂ .

  The direction l 'of the straight line L is obtained as an outer product of m' and n 'as shown in the following formula (9).

      l ′ = m ′ × n ′ (9)

As a result, the rotation matrix R ₂ is expressed by the following equation (10).

R ₂ = (l'm'n ') (10)

In this way, the second matrix calculation unit 22 uses the optical coordinates of the in-vehicle camera based on the coordinates of the calibration point Q (p ₁ , p ₂ ) on the projection plane (captured image) specified by the calibration point specifying unit 15. A second matrix R ₂ indicating a vector relating to the straight line 1 passing through the calibration points p ₁ and p ₂ and the calibration points p ₁ and p ₂ viewed from the center O _C is calculated.

[Third matrix calculation step]
The third matrix calculator 23 calculates a rotation matrix R indicating the rotation state of the camera 1 in the world coordinate system based on the first matrix R ₁ and the second matrix R ₂ obtained as described above. That is, the rotation matrix R is calculated according to the following equation (11) as in the above equation (2).

R = R ₂ R ₁ ^T = (l′ m′n ′) (lmn) ^T (11)

[Rotation angle calculation process]
The camera 1 is a perspective camera model represented by a pinhole camera. The camera matrix of the perspective camera model is composed of an internal parameter matrix of the camera and an external parameter matrix. The internal parameter matrix is a parameter matrix inside the camera such as the focal length. The external parameter matrix is the parallel progression T and the rotation matrix R obtained above. A projective camera model is a generalized representation of a camera matrix of a perspective camera model. In the projection camera model, if the rotation matrix R obtained above is obtained, projection from a three-dimensional space is possible. However, in the perspective camera model, it is necessary to further decompose the rotation matrix R into the rotation angle pan / tilt / roll of each axis of three-dimensional orthogonal coordinates. The rotation angle calculation unit 25 calculates the rotation angle of each axis in the following procedure.

  The rotation matrix R obtained by the equation (11) can be written as the following equation (12). Here, θ is a tilt angle, φ is a roll angle, and ψ is a pan angle.

r ₁₁ ² + r ₂₁ ² = cosφcosθ ² + sinφcosθ ² = cos ² θ (13)

  Since φ can be eliminated by equation (13), equation (14) is obtained.

There are two solutions in equation (14). Since “−r ₃₁ ” is “−sin θ”, θ is as shown in Expression (15).

  Further, when k is a positive constant, the following equation (16) is obtained.

tan ^-1 (a / b) = tan ^-1 (ka / kb) (16)

  Here, when cos θ ≠ 0 can be set, the roll angle φ and the pan angle ψ are as shown in Expression (17) and Expression (18), respectively.

  Here, based on the mounting method of the camera 1 (downward approximately 30 degrees = −30 degrees), the constraint condition of Expression (19) is applied.

    -90 [deg] <Tilt angle θ <+90 [deg] (19)

Since the case where the cos value of the tilt angle θ is zero can be ignored from the constraint condition of the equation (19), the unique rotation angles θ, φ, and ψ can be obtained.
First, from equation (19), since “cos θ> 0”, equation (15) becomes the following equation (20) without considering negative values, and the tilt angle θ is determined.

  Similarly, since “cos θ> 0”, the roll angle φ and the pan angle ψ can be easily obtained from the equation (12) as shown in the equations (21) and (22).

[Projection parameter calculation process]
As described above, the rotation component with respect to each axis of the three-dimensional orthogonal coordinates as the external parameter of the camera 1 is obtained. The translation component T (T _X , T _Y , T _Z ) among the external parameters is known in the world coordinate system as the coordinates of the optical axis center O _C. The internal parameters are known as the characteristics of the camera 1. Therefore, the projection parameter calculation unit 31 of the applied function processing unit 19 can adjust all the projection parameters of the camera 1.

[Coordinate transformation process]
When all the projection parameters of the camera 1 are set, the coordinate conversion unit 33 converts the three-dimensional coordinate value of the calibration point Q (P ₁ , P ₂ ) in the world coordinate system into a coordinate value on a two-dimensional projection plane. That is, the coordinate value of the theoretical calibration point V as shown in FIG. 20 is calculated.

[Limit area calculation process]
As shown in FIG. 20, the limit area calculation unit 35 calculates a limit area C on the projection plane that takes into account the allowable error of each rotation angle with reference to the theoretical calibration point V obtained by coordinate transformation. On the projection surface, a marker 80 actually captured by the camera 1 is shown. Further, the coordinates on the projection plane of the calibration point Q (P ₁ , P ₂ ) specified by the calibration point specifying unit 15 are also clear. If the calibration point Q is within the limit area C, the camera 1 has been calibrated with a predetermined accuracy.

The drawing unit 3 b draws the theoretical calibration point V calculated by the coordinate conversion unit 33 so that it can be superimposed on the captured image of the camera 1. In addition, the limit area C calculated by the limit area calculation unit 35 is drawn so as to be superimposed on the captured image of the camera 1. Furthermore, a highlight display for emphasizing the center coordinates of the calibration point Q specified by the calibration point specifying unit 15 is drawn. These are superimposed on the captured image via the display controller 3a and output as an image signal. The output image signal is displayed on a display device. Thus, the drawing unit 3b and the display controller 3a correspond to the superimposing unit of the present invention.
In practice, the limit area C and the calibration points V and Q are superimposed on the photographed image as shown in FIG. 8, for example, by color-coded, but in consideration of the visibility shown in FIG. Omitted.

[Success / failure determination step (determination step)]
FIG. 21 is an example of a display screen of the display device of the assembly adjustment control device of the vehicle 90 including the calibration device of the camera 1. The theoretical calibration point V, the limit area C, and the calibration point Q obtained from the actually captured image are displayed. In this example, one calibration point Q2 is within the limit region C, but the other calibration point Q1 is outside the limit region C. For example, an operator who performs assembly adjustment of the vehicle 90 can determine the success or failure of the calibration from this screen.
Also, success / failure can be automatically determined without depending on the visual observation of the operator. Since each coordinate value on the projection plane has already been calculated as described above, the success / failure determination unit (determination unit) 37 automatically determines the success or failure of the calibration.

[Notification process]
The determination result by the success / failure determination unit 37 is displayed on the display device via the drawing unit 3b and the display controller 3a. In the example shown in FIG. 21, a superimpose with “optical axis adjustment NG (S1)” is displayed. When the display device has a function of accepting input, such as a touch panel, operation buttons such as “readjustment (S2)” and “return (S3)” are provided. These inputs are transmitted as input signals from the input I / F unit 6 to the system control unit 5.
Moreover, not only such a visual display but you may alert | report auditorily by a buzzer, a chime, a sound, etc. The calibration result may be output to the outside via the system control unit 5 and the output I / F unit 7.

[Restriction process]
On the other hand, the “Next (S4)” operation button is displayed lighter than other superimposes, and its function is restricted so that it cannot be selected. Calibration of the camera 1 is performed during assembly adjustment work of the vehicle 90. If the calibration of the camera 1 is unsuccessful, if the assembly and adjustment work is continued, a product with insufficient calibration may be sent to a subsequent process. However, if the transition to the next assembly calibration work is restricted, such a problem can be prevented in advance. In addition, the operator can quickly take measures such as redo calibration.
Such restriction processing is executed by the application function processing unit 19 and the system control unit 5 and transmitted to the image output unit 3. Alternatively, it may be executed by an assembly adjustment system or the like that has received the calibration result via the output I / F circuit 7. In that case, a restriction instruction is given to the image output unit 3 via the input I / F circuit 6.

  As described above, according to the present invention, it is possible to provide an in-vehicle camera calibration apparatus capable of calibrating an in-vehicle camera with a simple configuration and in a short time with high accuracy. In addition, it is possible to provide an in-vehicle camera calibration device capable of quickly determining whether calibration is successful.

Explanatory drawing which shows an example of arrangement | positioning relationship between the marker for calibration and a vehicle Explanatory drawing showing an example of a calibration marker Explanatory drawing which shows the other example of arrangement | positioning relationship between the marker for calibration and a vehicle Explanatory drawing which shows three-dimensional arrangement | positioning of the calibration marker of FIG. The block diagram which shows typically the structure of the calibration apparatus of the vehicle-mounted camera of this invention The block diagram which shows typically the structure of the rotation amount calculating part of FIG. Blocks schematically showing the configuration of the application function processing unit of FIG. Explanatory drawing which shows an example of the image before distortion correction Explanatory drawing which shows an example of the image after distortion correction Explanatory drawing which shows an example of the image after area | region setting Explanatory drawing which shows an example of the image after edge detection Explanatory drawing which shows an example of the image after a straight line fitting Explanatory drawing showing an example of an image after calibration point detection Explanatory diagram showing the relationship between the world coordinate system and the camera coordinate system Explanatory drawing explaining camera rotation in right hand system Explanatory drawing showing N vector Explanatory drawing explaining the basic theorem about camera rotation Explanatory drawing which shows N vector used as the origin of the 1st matrix FIG. 2 is an explanatory diagram showing the N vector that is the basis of the matrix. Explanatory diagram showing the limit area Explanatory drawing showing an example of assembly adjustment screen

Explanation of symbols

1: Camera (on-vehicle camera)
2: Image receiving unit 3: Image output unit (superimposition unit)
DESCRIPTION OF SYMBOLS 10: Image processing part 13: Area | region setting part 15: Calibration point specific | specification part 21: 1st matrix calculating part 22: 2nd matrix calculating part 23: 3rd matrix calculating part 25: Rotation angle calculating part 31: Projection parameter calculating part 33 : Coordinate conversion unit 35: Limit area setting unit 37: Correctness determination unit (determination unit)
80: Marker (calibration index)
90: Vehicle C: Limit region E: Edge point (contour line pixel)
Q: Calibration point

Claims (11)

An in-vehicle camera calibration device for calibrating an in-vehicle camera attached to a vehicle,
An image receiving unit that receives a photographed image of the in-vehicle camera that includes at least two calibration indices arranged in different places in the field of view;
A calibration point specifying unit for specifying each calibration point in the calibration index in the captured image;
A first matrix calculation for calculating a first matrix indicating a vector related to the calibration point viewed from the optical center of the vehicle-mounted camera and a straight line passing through the calibration point based on the coordinates of the calibration point of the calibration index in a reference coordinate system And
Based on the coordinates of the calibration point on the captured image identified by the calibration point identifying unit, a second matrix indicating a vector related to the calibration point viewed from the optical center of the in-vehicle camera and a straight line passing through the calibration point. A second matrix calculation unit for calculating;
A vehicle-mounted camera calibration apparatus comprising: a third matrix calculation unit that calculates a rotation matrix indicating a rotation state of the vehicle-mounted camera in the reference coordinate system based on the first matrix and the second matrix.   The in-vehicle camera calibration device according to claim 1, further comprising a rotation angle calculation unit that decomposes the rotation matrix into three rotation angle components with respect to three axes orthogonal to each other in a three-dimensional space. An area setting unit for setting an area including the calibration point on the captured image;
The in-vehicle camera calibration device according to claim 1, wherein the calibration point specifying unit specifies the calibration point in the region. Based on the rotation matrix, a projection parameter calculation unit that calculates the projection parameters of the in-vehicle camera,
A coordinate conversion unit that converts the coordinates of the calibration point in the reference coordinate system into coordinates on the captured image using the projection parameters;
A limit area calculation unit that calculates a limit area of the calibration error on the photographed image based on the coordinates of the calibration point that has undergone coordinate conversion and an allowable value of the calibration error with respect to the rotation of the in-vehicle camera. The in-vehicle camera calibration device according to any one of Items 1 to 3.   The in-vehicle camera calibration device according to claim 4, further comprising: a superimposing unit that superimposes a highlighted display of a center coordinate of the calibration point specified by the calibration point specifying unit and the limit region on the captured image.   The on-vehicle camera calibration according to claim 4, further comprising: a determination unit that determines whether or not a center coordinate of the calibration point specified by the calibration point specifying unit exists in the limit region on the captured image. apparatus.   The in-vehicle camera calibration device according to claim 6, further comprising a notifying unit that at least visually or audibly notifies the determination result by the determining unit.   The in-vehicle camera calibration device according to any one of claims 1 to 3, wherein a calibration point of the calibration index is represented as an intersection of two straight lines on a plane.   The on-vehicle camera calibration device according to any one of claims 1 to 4, wherein the calibration index is arranged on an arbitrary surface including a floor surface and a wall surface. An in-vehicle camera calibration method for calibrating an in-vehicle camera attached to a vehicle,
A calibration index placement step for placing calibration indices in at least two different locations, including in the field of view of the in-vehicle camera;
Based on the coordinates of the respective calibration points in the calibration index in the reference coordinate system, a first matrix indicating a vector relating to the calibration point viewed from the optical center of the in-vehicle camera and a straight line passing through the calibration point is calculated. One matrix operation step;
An image receiving step for receiving a photographed image of the in-vehicle camera including the calibration index in a field of view;
In the captured image, a calibration point specifying step for specifying each of the calibration points;
Based on the coordinates of the calibration point specified on the photographed image, a second matrix calculation that calculates a second matrix indicating the calibration point viewed from the optical center of the in-vehicle camera and a vector relating to a straight line passing through the calibration point. Process,
A vehicle-mounted camera calibration method comprising: a third matrix calculation step of calculating a rotation matrix indicating a rotation state of the vehicle-mounted camera in the reference coordinate system based on the first matrix and the second matrix. A vehicle production method for sequentially performing a plurality of assembly adjustment steps including an assembly adjustment step of calibrating the in-vehicle camera using the in-vehicle camera calibration method according to claim 10,
The in-vehicle camera calibration method is further
A projection parameter calculation step of calculating a projection parameter of the in-vehicle camera based on the rotation matrix;
A coordinate conversion step of converting the coordinates of the calibration point in the reference coordinate system into coordinates on the captured image using the projection parameters;
A limit area calculation step of calculating a limit area of the calibration error on the captured image based on the coordinates of the calibration point that has been coordinate-transformed and an allowable value of the calibration error with respect to the rotation of the in-vehicle camera;
A determination step of determining whether or not the center coordinates of the calibration point specified in the calibration point specifying step exist in the limit region on the captured image;
And a restricting step of restricting a shift to the next assembly / adjustment operation of the calibration of the in-vehicle camera when the center coordinates of the calibration point do not exist within the limit region.

Images