JP2018044942A - Camera parameter calculation device, camera parameter calculation method, program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a camera parameter calculation device that can use a stereo camera or multi-eye camera without using a calibration index having a three-dimensional structure known.SOLUTION: A camera parameter calculation device comprises: a point group acquisition unit 107 that acquires three-dimensional point group data representing a three-dimensional position of each of a plurality of three-dimensional points included in shooting space of one or more cameras 102 and 103; a camera parameter calculator 108 that obtains a corresponding point in an individual image shot by the one or more cameras about each of the plurality of three-dimensional points on the basis of the three-dimensional point group data and an initial camera parameter of each camera, and calculates a camera parameter of each camera on the basis of the initial camera parameter of each camera and a pixel value in the corresponding point in the individual image; and a camera parameter output unit 109 that outputs the calculated camera parameter of each camera.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for calculating camera parameters.

  In order to calculate camera parameters, that is, to calibrate the camera, it is necessary to associate the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image. For this purpose, conventionally, a three-dimensional coordinate and a pixel position in a two-dimensional image are associated with each other by photographing a calibration index such as a checker pattern with a known shape and detecting an intersection of the checker patterns. (For example, see Patent Documents 1 and 2 and Non-Patent Documents 1 and 2.)

Japanese Patent No. 4681856 Japanese Patent No. 5580164

Roger Y. Tsai.A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses.IEEE Journal of Robotics and Automation.Vol. 3, pp.323-344, 1987 Zhengyou Zhang.A flexible new technique for camera calibration.IEEE Transactions on Pattern Analysis and Machine Intelligence.Vol. 22, pp.1330-1334, 2000

  In the conventional camera calibration, a calibration index whose three-dimensional structure is known is used. For this reason, a calibration index with high processing accuracy and a wide photographing space are required. In such a wide-angle camera calibration, a large-scale calibration facility is required in order to arrange the calibration index in the entire field of view.

  The present invention has been made in view of the above problems, and an object thereof is to calibrate a monocular camera, a stereo camera, and a multi-lens camera without using a calibration index with a known three-dimensional structure.

  In order to achieve the above object, a camera parameter calculation device according to an aspect disclosed is three-dimensional point group data representing three-dimensional coordinates of a plurality of three-dimensional points included in an imaging space common to one or a plurality of cameras. The one or more cameras for each of the plurality of three-dimensional points on the basis of the point cloud acquisition unit for acquiring the three-dimensional points, and the one or more initial camera parameters of the one or more cameras The image coordinates of corresponding points in one or a plurality of images photographed in Step 1 are calculated, and the one or more camera parameters of the one or more cameras are calculated based on the pixel values in the image coordinates in the one or more images. A camera parameter calculator that calculates the one or more camera parameters, and a camera parameter output device that outputs the one or more camera parameters. The initial camera parameters, the one or more individual images, the one or more camera parameters are one-to-one correspondence.

  According to the camera calibration technique according to the present disclosure, a calibration index with a known three-dimensional structure is used by evaluating camera parameters based on pixel values of pixel coordinates obtained by projecting three-dimensional coordinates onto an image with camera parameters. Without calculating the camera parameters, the camera can be calibrated. For this reason, the camera can be calibrated without associating the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image.

FIG. 3 is a block diagram illustrating an example of a configuration of a camera parameter calculation apparatus according to the first embodiment. 5 is a flowchart showing an example of the operation of the camera parameter calculation device according to the first embodiment. 3 is a diagram illustrating an example of a CG (computer graphics) image according to Embodiment 1. FIG. The figure which shows the evaluation value in the camera parameter vicinity of the correct Cx. The figure which shows the evaluation value in the camera parameter vicinity with the correct Cy. The figure which shows the evaluation value in the right camera parameter vicinity of f. The figure which shows the evaluation value in the camera parameter vicinity of d'x correct. The figure which shows the evaluation value in the right camera parameter vicinity of Rx. The figure which shows the evaluation value in the camera parameter vicinity with the correct Ry. The figure which shows the evaluation value in the camera parameter vicinity with the correct Rz. The figure which shows the evaluation value in the camera parameter vicinity of the correct Tx. The figure which shows the evaluation value in the camera parameter vicinity with the correct Ty. The figure which shows the evaluation value in the camera parameter vicinity with the correct Tz. 9 is a flowchart showing an example of the operation of the camera parameter calculation device according to the second embodiment. 10 is a flowchart illustrating an example of camera parameter calculation processing according to Embodiment 3. 10 is a flowchart showing an example of camera parameter calculation processing according to Embodiment 4. 10 is a flowchart illustrating an example of camera parameter calculation processing according to the fifth embodiment. 18 is a flowchart showing an example of camera parameter calculation processing according to Embodiment 6. 18 is a flowchart showing an example of camera parameter calculation processing according to Embodiment 7. 18 is a flowchart illustrating an example of camera parameter calculation processing according to Embodiment 8. 10 is a flowchart showing an example of camera parameter calculation processing according to Embodiment 9. FIG. 18 is a block diagram of a camera parameter calculation device according to Embodiment 10. 18 is a flowchart showing an example of camera parameter calculation processing according to Embodiment 10. 18 is a flowchart showing an example of camera parameter calculation processing according to Embodiment 11. 18 is a flowchart showing an example of a point group selection mask creation process according to the eleventh embodiment. An image of a checker pattern taken with a fisheye camera.

(Knowledge that became the basis of this disclosure)
The present inventor has found that the following problems occur with respect to the camera calibration described in the background art section.

  In order to perform camera calibration, the pixel on which the three-dimensional coordinates of the point of interest in the three-dimensional space where the camera exists and the point of interest in the two-dimensional image obtained by photographing the three-dimensional space with the camera are projected It is necessary to associate positions (hereinafter referred to as corresponding points). For this purpose, conventionally, a three-dimensional coordinate and a pixel position in a two-dimensional image are associated with each other by photographing a calibration index such as a checker pattern with a known shape and detecting an intersection of the checker patterns. Yes. Here, three-dimensional coordinates in the three-dimensional space are referred to as world coordinates, and two-dimensional coordinates in the two-dimensional image are referred to as image coordinates.

  For example, FIG. 26 shows a calibration index in which checker patterns are drawn at regular intervals inside a box-shaped subject. As for the pixel position in the two-dimensional image, as shown in FIG. 26, for example, the operator reads the intersection position of the image coordinate system with the upper left corner of the image as the origin. The corresponding three-dimensional coordinates are obtained from the checker pattern used for photographing. Specifically, the origin of the world coordinates and three axes of X, Y, and Z are defined at a specific position, and the three-dimensional coordinates can be identified by the number of checker pattern intersections from the origin.

  A point on the world coordinates can be projected on the image coordinates by coordinate transformation based on the camera parameters. In other words, the calculation corresponding point on the camera image corresponding to the point on the world coordinate can be obtained using the camera parameter. Conversely, camera parameters can be calculated from a set of world coordinates (X, Y, Z) and image coordinates (x, y) that currently correspond. Taking a pinhole camera as an example, the projection from the world coordinates to the image coordinates by the camera parameters is shown in Equation 1.

  The camera parameters of this pinhole camera model are as follows: x and y components at the center of the image are Cx and Cy, focal length is f, and the length of one pixel of the image sensor in the x and y directions is d′ x and d′ y, respectively. , R is a 3 × 3 rotation matrix with respect to the camera world coordinate reference (where the 10's place in the lower right subscript represents a row and the 1's place represents a column), x of the translation relative to the camera world coordinate reference, Let y and z components be Tx, Ty, and Tz, respectively, and a parameter with no degree of freedom be h. Further, distortion such as distortion aberration can be expressed by using Φ (xd, yd) representing conversion from image coordinates (xu, yu) without distortion to image coordinates (x, y) with distortion (formulas). 2).

  A plurality of conventional camera calibration techniques and their problems will be described in order.

  In Non-Patent Document 1, for a set of world coordinates and image coordinates, the sum of distances (reprojection errors) between the points on the world coordinates projected on the image with camera parameters and the corresponding points on the image is minimized. To calibrate the camera. For this reason, it is necessary to associate the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image.

  In Non-Patent Document 2, calibration indices are installed at a plurality of different depths, and each is photographed with one camera. By minimizing the sum of the squares of the distances between the points where the world coordinates are projected on the image with camera parameters and the corresponding points on the image, for the set of world coordinates and image coordinates on the plane obtained in this way Calibrate the camera. For this reason, it is necessary to associate the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image.

  In Patent Document 1, information on a calibration point on a plane in which three-dimensional coordinates and pixel values in a two-dimensional image are associated across a plurality of frames is input. Camera parameters are determined so that a calibration point existing on a plane in world coordinates and a calibration point on image coordinates are subjected to plane projective transformation. For this reason, it is necessary to associate the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image.

  In Patent Document 2, two sets of three-dimensional positions are acquired using a three-dimensional laser scanner and a stereo camera. The two sets of three-dimensional positions are associated with each other, and the stereo camera is calibrated on the basis of the three-dimensional position of the three-dimensional laser scanner having higher ranging accuracy than the stereo camera. It is complicated to manually associate the three-dimensional points with each other, and a dedicated calibration index is required for automation. In addition, in a small stereo camera with a short baseline length, when the camera parameter includes an error, the distance measurement accuracy is greatly reduced as compared with a stereo camera with a long baseline length. For this reason, when the baseline length is shorter than the subject distance, there is a possibility of erroneous correspondence with the three-dimensional position of the three-dimensional laser scanner.

  Stereo cameras are used for surrounding monitoring and driving support in moving bodies such as cars and drones. For such purposes, a wide-angle camera is preferred. A general stereo camera for the purpose of ranging requires calibration at the time of manufacture. In addition, the camera needs to be recalibrated against aging and distortion due to impact.

  Usually, calibration at the time of manufacture and recalibration are performed by a dedicated calibration facility. In these calibrations, a calibration index whose three-dimensional structure is known is used. For this reason, a calibration index with high processing accuracy and a wide photographing space are required. In such a wide-angle camera calibration, a large-scale calibration facility is required in order to arrange the calibration index in the entire field of view.

  The present disclosure has been devised in view of the above problems, and aims to calibrate various cameras without using a calibration index whose three-dimensional structure is known.

  A camera parameter calculation device according to an aspect of the present disclosure includes a point cloud acquisition unit that acquires 3D point cloud data representing 3D coordinates of a plurality of 3D points included in a shooting space common to one or a plurality of cameras. One or a plurality of images taken by the one or a plurality of cameras for each of the plurality of the three-dimensional points based on the three-dimensional point cloud data and one or a plurality of initial camera parameters of the one or a plurality of cameras. A camera parameter calculator that calculates the image coordinates of the corresponding point in the image and calculates the one or more camera parameters of the one or more cameras based on pixel values in the image coordinates in the one or more images A camera parameter output device for outputting the one or more camera parameters, the one or more cameras, the one or more initial camera parameters, One or more individual images, the one or more camera parameters are one-to-one correspondence.

  According to this configuration, since the camera parameter is evaluated based on the pixel value of the pixel coordinate obtained by projecting the three-dimensional coordinate on the image with the camera parameter, the camera parameter can be set without using a calibration index with a known three-dimensional structure. It is possible to calculate, ie calibrate the camera. For this reason, the camera can be calibrated without associating the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image.

  In addition, the calculation of the image coordinates is performed by the camera parameter calculator using the one or more cameras based on the one or more initial camera parameters corresponding to the one or more cameras. Conversion to image coordinates in the corresponding image may be included.

  According to this configuration, for example, the image coordinates can be calculated from the three-dimensional coordinates by a routine procedure by coordinate transformation using a camera parameter corresponding to the camera model such as a pinhole camera model.

  The one or more cameras include two cameras, and the camera parameter calculator calculates the camera parameters based on a plurality of differences, and each of the plurality of differences includes the plurality of three-dimensional points. Pixel values at corresponding points in the first image taken by the first camera included in the two cameras for one of the three-dimensional points, and the second camera included in the two cameras for the one three-dimensional point It may be a difference in pixel values at corresponding points in the second image photographed in FIG. The difference may be an absolute value of the pixel value difference or a square value of the pixel value difference.

  According to this configuration, if the camera parameter is correct, one three-dimensional point is correctly projected on the corresponding point in the individual image, and thus the plurality of differences are all close to zero. That is, a large cumulative total of the plurality of differences means that the error of corresponding points in the individual images is large, that is, the error of the camera parameter is large. Therefore, for example, an evaluation function defined by the accumulated difference is introduced, and the camera parameters are updated in a direction of decreasing the evaluation function by a known procedure such as a gradient descent method. As a result, the camera parameter error is reduced, and camera calibration can be performed.

  Further, the one or more cameras are two or more cameras, and the camera parameter calculator calculates the camera parameters based on a plurality of differences, and each of the plurality of differences is the plurality of three-dimensional It may be a difference between one pixel value among pixel values at corresponding points in individual images taken by the two or more cameras with respect to one three-dimensional point among the points and an average value of the pixel values. The difference may be an absolute value of the pixel value difference or a square value of the pixel value difference.

  According to this configuration, if the camera parameter is correct, one three-dimensional point is correctly projected on the corresponding point in the individual image, and thus the plurality of differences are all close to zero. That is, a large cumulative total of the plurality of differences means that the error of corresponding points in the individual images is large, that is, the error of the camera parameter is large. Therefore, for example, an evaluation function defined by the accumulated difference is introduced, and the camera parameters are updated in a direction of decreasing the evaluation function by a known procedure such as a gradient descent method. As a result, the camera parameter error is reduced, and camera calibration can be performed.

  Further, the one or more cameras are two or more cameras, and the camera parameter calculator is based on the similarity of pixel value patterns in pixels near the corresponding point in the individual images corresponding to the same three-dimensional point. The camera parameters may be calculated. Specifically, the camera parameter calculator calculates camera parameters based on a plurality of normalized cross-correlations, and each of the plurality of normalized cross-correlations is a three-dimensional one of the plurality of three-dimensional points. It may be a correlation between pixel values included in the vicinity of corresponding points in individual images taken by the two or more cameras.

  According to this configuration, even when there is a gain difference (for example, an overall luminance difference due to a mismatch in exposure) in the individual images, the gain difference is eliminated based on the similarity of the pixel value pattern, and the corresponding points You can know the magnitude of the error.

  Further, the camera parameter calculator calculates, from the calculation of the camera parameter, a three-dimensional point in which a plurality of corresponding points whose separation distances are equal to or smaller than a threshold in at least one of the plurality of three-dimensional points is obtained. It may be excluded.

  In this configuration, the plurality of corresponding points whose separation distances are equal to or smaller than the threshold value are corresponding points of a plurality of three-dimensional points existing in a region extending in approximately one direction from the camera. Therefore, there is a possibility that the rear three-dimensional object is shielded by the front three-dimensional object. Therefore, in the calculation of the camera parameters, the three-dimensional points for which a plurality of corresponding points whose separation distances are equal to or smaller than the threshold are obtained are excluded from the calculation of the camera parameters. Thereby, the pixel value of the corresponding point that may represent a pixel value different from the original value can be excluded from the calculation of the evaluation value, and the evaluation value can be calculated without the influence of the shielding.

  Further, the camera parameter calculator excludes one or more three-dimensional points having a luminance gradient at a corresponding point of at least one image of the one or more images smaller than a threshold value from the plurality of three-dimensional points. The camera parameters may be calculated from the three-dimensional points.

  According to this configuration, by calculating the camera parameters by excluding the three-dimensional point located in the region where the luminance gradient is small in the camera image, the calculation by the three-dimensional point having a small contribution to the evaluation value is omitted, and the calculation amount is reduced. Can be reduced.

  The three-dimensional point cloud data represents a three-dimensional position of the plurality of three-dimensional points based on reflection of measurement light emitted from a predetermined distance measuring viewpoint, and the camera parameter calculator Two three-dimensional points included in the point, the distance between the two three-dimensional points being larger than a threshold, and the angle formed by the direction vector of the measurement light to the two three-dimensional points being smaller than the threshold 2 Two three-dimensional points may be excluded from the calculation of the camera parameters.

  In this configuration, the two three-dimensional points are gap regions located in the boundary of the object of the three-dimensional point cloud data or a region having a large surface unevenness, and a gap is also likely to occur in the pixel values at the corresponding points of the image. . When the pixel value of such a corresponding point is used for calculation of the evaluation value, there is a concern that the evaluation value changes too sharply with respect to the error of the corresponding point, thereby degrading the convergence of the camera parameter. With respect to such a problem, in the above configuration, an evaluation value can be calculated using pixel values in a suitable region while avoiding a region having a large object boundary or surface unevenness.

  The three-dimensional point group data includes the three-dimensional positions of the plurality of three-dimensional points and the return light of the measurement light from each three-dimensional point based on the reflection of the measurement light emitted from a predetermined distance measuring viewpoint. The camera parameter calculator, of the plurality of three-dimensional points, can be seen within a predetermined vicinity from the distance measurement viewpoint, and the difference in intensity is a first threshold value and a second threshold value. The camera parameter may be calculated using only a plurality of three-dimensional points within the range determined by.

  According to this configuration, the camera parameters can be calculated using only a plurality of three-dimensional points within a range in which the intensity of the measurement light, that is, the reflectance of the object changes smoothly. For example, in a region where there is a gap in the reflectance of the object such as at the boundary of the object, a gap is also likely to occur in the pixel value at the corresponding point of the image. When the pixel value of such a corresponding point is used for calculation of the evaluation value, there is a concern that the evaluation value changes too sharply with respect to the error of the corresponding point, thereby degrading the convergence of the camera parameter. With respect to such a problem, in the above configuration, an evaluation value can be calculated using a pixel value in a suitable region while avoiding a region where the reflectance of the object is excessively different. In addition, for example, a region where the reflectance does not change excessively, such as a monotonous continuous surface, is excluded from the calculation of the evaluation value in order to make it difficult to converge the camera parameters.

  The three-dimensional point group data may be data obtained by measuring the plurality of three-dimensional points with a three-dimensional laser scanner.

  According to this configuration, the three-dimensional point cloud data can be acquired from a three-dimensional laser scanner that is generally easy to use.

  The three-dimensional point cloud data includes three-dimensional positions of the plurality of three-dimensional points and three for each of the three-dimensional points based on reflection of measurement light emitted from a predetermined distance measurement viewpoint and color photographing from the distance measurement viewpoint. Representing the color of an object at a three-dimensional point, and the camera parameter calculator is located within a predetermined vicinity from the distance measurement viewpoint among the plurality of three-dimensional points, and the color difference of the object is a first threshold value. And the camera parameter may be calculated using only a plurality of three-dimensional points within a range determined by the second threshold.

  According to this configuration, the camera parameters can be calculated using only a plurality of three-dimensional points within a range where the color of the object changes smoothly. For example, in a region where there is a gap in the color of the object at the boundary of the object, a gap also occurs in the color at the corresponding point of the image. When the pixel value of such a corresponding point is used for calculation of the evaluation value, there is a concern that the evaluation value changes too sharply with respect to the error of the corresponding point, thereby degrading the convergence of the camera parameter. With respect to such a problem, in the above configuration, an evaluation value can be calculated using a pixel value in a suitable region while avoiding a region where there is an excessive difference in the color of the object. Further, for example, a region where the color change is excessive, such as a monotonous continuous surface, is excluded from the calculation of the evaluation value in order to make it difficult to converge the camera parameters.

  The three-dimensional point cloud data includes three-dimensional positions of the plurality of three-dimensional points and three for each of the three-dimensional points based on reflection of measurement light emitted from a predetermined distance measurement viewpoint and color photographing from the distance measurement viewpoint. Each of the one or more cameras is a color camera, and the camera parameter calculator is one camera of the one or more cameras based on a plurality of differences. Each of the plurality of differences is a color represented by a pixel value at a corresponding point in an image photographed by the one camera for one of the plurality of three-dimensional points. And the difference between the color of the object at the one three-dimensional point represented by the three-dimensional point group data.

  According to this configuration, it is possible to calibrate a single camera and to calibrate a plurality of cameras one by one by using the color information of the object represented by the three-dimensional point cloud data.

  Further, the point group acquisition unit may acquire the three-dimensional point group data from a three-dimensional laser scanner with a color camera that performs color imaging of the distance measurement space.

  According to this configuration, the three-dimensional point cloud data can be acquired with color information from the three-dimensional laser scanner.

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. The system, method, integrated circuit, computer program And any combination of recording media.

(Embodiment 1)
Hereinafter, embodiments of the present invention will be described with reference to the drawings. Although the present invention can be implemented in one or more cameras, for the sake of simplicity, the present invention will be described using an example of a stereo camera including a left camera and a right camera.

  FIG. 1 is a block diagram illustrating an example of a functional configuration of the camera parameter calculation apparatus according to the first embodiment. In FIG. 1, the camera parameter calculation apparatus 101 includes a frame memory 104, an initial camera parameter storage 105, a point group acquisition unit 107, a camera parameter calculation unit 108, and a camera parameter output unit 109. The camera parameter calculation apparatus 101 uses the three-dimensional laser scanner 106 to calibrate the left camera 102 and the right camera 103. In this specification, to calibrate the camera means to obtain the camera parameters of the actual camera. Therefore, the camera parameter calculation device of this specification is synonymous with the camera calibration device.

  The operation of each component shown in FIG. 1 will be described below.

  The left camera 102 captures a first space, which is a capturing space, and sends the captured first image to the frame memory 104.

  The right camera 103 captures a second space, which is a capturing space, and sends the captured second image to the frame memory 104. The first space and the second space include a third space that is a common space. The third space may include one or more objects and / or one or more organisms. A subject that can be approximated by Lambertian reflection is preferable from the viewpoint that the appearance of the subject surface (color at a certain point) does not change depending on the camera position.

  The frame memory 104 receives and stores the first image and the second image.

  The initial camera parameter storage 105 holds initial camera parameters in advance. The initial camera parameter is, for example, a camera parameter included in (Equation 1), and may be a design value set and / or used when designing a stereo camera. Or, according to the method of Tsai (Non-Patent Document 1) or Zhang (Non-Patent Document 2) described above, the necessary number of pairs of world coordinates and image coordinates that are in a correspondence relationship is manually acquired using a calibration index. By doing so, the camera parameters may be estimated with coarse accuracy. Here, the Tsai method requires at least 13 sets of world coordinates and image coordinates. The Zhang method requires at least three shootings and at least thirteen world coordinate and image coordinate pairs.

  The three-dimensional laser scanner 106 scans the laser beam in the space including the third space described above, and measures each three-dimensional position of a plurality of points included in the space including the third space, that is, a plurality of three-dimensional points. To do. Data representing the measured three-dimensional position of the N point in world coordinates (X, Y, Z) is referred to as three-dimensional point group data. The three-dimensional laser scanner 106 sends it to the point cloud acquisition unit 107.

  The point cloud acquisition unit 107 receives 3D point cloud data from the 3D laser scanner 106.

  The camera parameter calculator 108 receives the first image and the second image from the frame memory 104, the initial camera parameter from the initial camera parameter storage 105, and the three-dimensional point group from the point group acquisition unit 107, respectively. The procedure shown in FIG. Obtain the camera parameters.

  The camera parameter output unit 109 outputs the camera parameter calculated by the camera parameter calculator 108.

  The camera parameter calculation device 101 is realized by, for example, a computer device (not shown) including a processor, a memory, an interface circuit, and the like, and the constituent elements of the camera parameter calculation device 101 execute a program recorded in the memory in advance. It may be a software function achieved by this. The camera parameter calculation device 101 may be realized by a dedicated hardware circuit (not shown) that performs the above-described operation.

  Further, the camera parameter calculation device 101 is not necessarily realized by a single computer device, and may be realized by a distributed processing system (not shown) including a terminal device and a server. As an example, the frame memory 104, the initial camera parameter storage 105, the point cloud acquisition unit 107, and the camera parameter output unit 109 are provided in the terminal device, and a part or all of the functions of the camera parameter calculation unit 108 are executed by the server. Also good. In this case, data exchange between the components is performed via a communication line connected to the terminal device and the server.

  FIG. 2 is a flowchart illustrating an example of the camera parameter calculation process S300.

  The camera parameter calculator 108 receives the first image captured by the left camera 102 stored in the frame memory 104 and the second image captured by the right camera 103 stored in the frame memory 104 (S301). In the following, an image captured by a camera and received from the camera is also referred to as a captured image or a camera image.

  The camera parameter calculator 108 acquires initial camera parameters of the left camera 102 and the right camera 103 from the initial camera parameter storage 105 (S302).

  The point cloud input unit 107 receives the 3D point cloud data measured by the 3D laser scanner 106, the point cloud input unit 107 sends the 3D point cloud data to the camera parameter calculator 108, and the camera parameter calculator 108 The dimension point cloud data is received (S303).

The camera parameter calculator 108 calculates the image coordinates (x _Lk , y _Lk ) and (x _Rk , y _Rk ) from the world coordinates (X, Y, Z) according to Equations 1 and 2. The (x _Lk , y _Lk ) is the image coordinates of the k-th corresponding point included in the first image corresponding to the k-th three-dimensional point, and the (x _Rk , y _Rk ) is the k-th 3 The image coordinates of the kth corresponding point included in the second image corresponding to the dimension point.

  Points included in the first image corresponding to the kth three-dimensional point included in the plurality of three-dimensional points obtained by the three-dimensional laser scanner 106, that is, k included in the first image corresponding to the kth three-dimensional point. The pixel value at the th corresponding point and the point included in the second image corresponding to the k th three dimensional point, that is, the k th corresponding point included in the second image corresponding to the k th three dimensional point. An evaluation function J (formula 3) defined by the sum of absolute values of pixel value differences is calculated (S304).

  Here, N is the number of three-dimensional points with which pixel values are compared, and all or some of the points acquired by the three-dimensional laser scanner are selected. “Selecting a portion” may mean, for example, excluding an area that does not affect the calculation of the evaluation value J and in which pixel values on the image continuously have the same value.

I _L (x, y) is a pixel value at the image coordinates (x, y) of the first image, and I _R (x, y) is a pixel at the image coordinates (x, y) of the second image. Value. When the first image and the second image are monochrome images, the pixel value is a luminance value, and when the first image and the second image are color images, the pixel value is a color vector. The absolute value of the color vector difference means the distance between the color vectors. In order to eliminate the influence of chromatic aberration, only a specific color component of the color components constituting the color vector may be used.

  Each of a plurality of three-dimensional points included in the three-dimensional point cloud data is projected onto the first image using camera parameters of the left camera 102, and is projected onto the second image using camera parameters of the right camera 103. .

  Here, projecting means obtaining a corresponding point on the camera image corresponding to the three-dimensional point, specifically, the world of each of a plurality of three-dimensional points included in the three-dimensional point group data. A coordinate transformation based on the camera parameters of the left camera 102 is performed on the coordinates to calculate image coordinates on the first image, and a coordinate transformation based on the camera parameters of the right camera 103 is performed to obtain image coordinates on the second image. Say to calculate. For the coordinate conversion, for example, projection based on the matrix calculation described in Expression 1 may be used. The image coordinates may be calculated with subpixel accuracy, and the pixel values may be calculated with decimal accuracy from bilinear or bicubic interpolation for the image coordinates with subpixel accuracy.

  Further, the absolute value of the difference between the pixel values may be weighted with respect to the addition of N points in the calculation of the evaluation value J. For example, the weight of a point group in which the subject color continuously changes is increased, or the weight of a point group with a large unevenness on the surface of the object is decreased. These weightings smooth the change of the evaluation value J with respect to the continuous change of the camera parameter, and make it easy to minimize the evaluation value J.

When the calculation of the evaluation value J within the camera parameter search range is completed, or when the evaluation value J is smaller than the threshold value, the iterative calculation ends (ends in S305). On the other hand, when the iterative calculation is continued (continue in S305), the camera parameter is changed within the search range (S306). As the camera parameter search range, a range that each camera parameter can take is set in advance. For example, the image center positions Cx and Cy, the focal length f, and the imaging element lengths d ′ _x and d ′ _y may be set to ± 5% of the design values, respectively. Further, the rotational component of the camera position, R _x , R _y , R _z , and the translational movement component are T _x , T _y , T _z , and the positional relationship between the three-dimensional laser scanner 106 and the cameras 102, 103 is scaled, etc. The angle may be ± 10 degrees of the measured value, and the translational movement may be ± 0.2 m.

  In order to reduce the calculation time of the iterative processing in steps S304 to S306, the range may be limited to the vicinity of the initial camera parameter, or the steepest descent method may be applied using the gradient of the evaluation function. .

  Finally, the camera parameter having the smallest evaluation value is selected from the set of the camera parameter and the evaluation value calculated by the iterative calculation in steps S304 to S306 described above, and the selected camera parameter is output (S307).

  It will be sequentially described that the camera can be calibrated according to such a procedure. The evaluation function for calibrating the camera needs to have the minimum evaluation value with the correct camera parameters. This property of the evaluation function makes it possible to search for the correct camera parameter by minimizing the evaluation value.

  Change the value of only one camera parameter from all correct camera parameters. At this time, if the evaluation value J is all the camera parameters and is a downward convex function form with the correct camera parameter being the minimum, the evaluation function J is convex downward with respect to all the camera parameters. This is because changing the value of only one camera parameter λ and calculating the extreme value is equivalent to partial differentiation ∂J / ∂λ, and that all partial differentiations are zero at a certain point. This corresponds to dJ / dΩ being 0 (Formula 4).

Here, m represents the number of camera parameters, λ _{1 to} λ _m represent individual camera parameters such as a focal length, and Ω represents an entire camera parameter obtained by collecting the individual camera parameters.

Since the camera parameters differ depending on the camera model, description will be made by taking ten camera parameters used in the Tsai method shown in Non-Patent Document 1 as an example. The camera parameters of this pinhole camera model are as follows: x and y components at the center of the image are C _x and C _y , the focal length is f, the length of one image sensor in the x direction is d′ x, and the world coordinate of the camera is The x, y, and z components of rotation relative to the reference are R _x , R _y , and R _z , respectively, and the x, y, and z components of translation relative to the camera world coordinate reference are T _x , T _y , and T _z , respectively. At this time, the evaluation value J is a function having these 10 camera parameters as variables.

  This method can be performed by photographing an actual space or by using CG data. Hereinafter, camera calibration performed using CG data will be described as an example.

  A CG simulating the parking lot shown in FIG. 3 is created. The right camera is horizontally spaced 15 mm from the left camera, and the optical axis is parallel to the left camera. The camera is a fisheye camera.

With respect to the CG model, an evaluation value J is calculated when only one camera parameter of the right camera is changed from a state in which all camera parameter values are correct. These evaluation values J are set as camera parameter values on the horizontal axis and evaluation values J on the vertical axis, and 10 camera parameters are shown in FIGS. The camera parameters to be changed are C _x in FIG. 4, C _y in FIG. 5, f in FIG. 6, d ′ _x in FIG. 7, R _x in FIG. 8, R _y in FIG. 9, R _z in FIG. 11 is T _x , FIG. 12 is T _y , and FIG. 13 is T _z . The correct values for each camera parameter are as follows. That is, C _x is 640 pixels, C _y is 480 pixels, f is 1.12 mm, d ′ _x is 2.75 μm, R _x is 90 °, R _y is 0 °, R _z is 0 °, T _x is 0 mm, T _y is 0 mm and T _z is 0 mm.

In any of FIGS. 4 to 13, the evaluation value J is convex downward in the vicinity of the correct camera parameter, and the camera parameter at which the evaluation value J is minimal matches the correct camera parameter.
From the above, camera calibration is possible with this method.

  Note that when there are three or more cameras, the number of cameras is n, and the evaluation function J of Equation 3 may be expanded to Equation 5.

Here, the image coordinates of the k-th corresponding point included in the i-th camera image and the j-th camera image are (x _ik , y _ik ) and (x _jk , y _jk ), respectively. I _i (x, y) represents an RGB value or luminance pixel value in the image coordinates (x, y) of the i-th camera image, and I _j (x, y) represents the image coordinates of the j-th camera image. The RGB value or luminance pixel value at (x, y) is represented. Since the other variables are the same as those included in Equation 3, the description thereof is omitted.

  As described above, the camera parameters are evaluated based on the pixel values of the pixel coordinates obtained by projecting the three-dimensional coordinates onto the image with the camera parameters. Therefore, the camera parameters are calculated without using the calibration index with the known three-dimensional structure. That is, the camera can be calibrated. For this reason, the camera can be calibrated without associating the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image.

(Embodiment 2)
The evaluation function in Embodiment 1 is not necessarily limited to the absolute value sum of the pixel value differences shown in Equation 3. In the second embodiment, a case will be described in which the evaluation function is a square sum of pixel value differences.

  The second embodiment differs from the first embodiment only in the evaluation function used for calculating the evaluation value. Since the functional configuration of the camera parameter calculation apparatus according to the second embodiment is substantially the same as that of the camera parameter calculation apparatus 101 described in the first embodiment, the description thereof is omitted.

  FIG. 14 is a flowchart showing an example of the camera parameter calculation process S310 according to the second embodiment. In FIG. 14, the same steps as those in the first embodiment are given the same step names as those in FIG. 2, and the description thereof is omitted. In step S314, an evaluation function J (Equation 6) defined by the sum of squares of the difference between corresponding pixel values of the teleo camera is calculated.

  Here, since the variable of Formula 6 is the same as the variable contained in Formula 3, the description is abbreviate | omitted.

  When there are three or more cameras, the evaluation function J in Expression 6 may be expanded to Expression 7.

  Here, since the variable of Formula 7 is the same as the variable contained in Formula 5, the description is abbreviate | omitted.

  As described above, as in Embodiment 1, the camera can be calibrated without associating the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image.

(Embodiment 3)
The evaluation functions in Embodiments 1 and 2 do not have to be limited to the sum of absolute values or the sum of squares of the pixel value differences shown in Equation 3 and Equation 6. In the third embodiment, a case where the evaluation function is the sum of absolute values of differences from the average pixel value will be described.

  The third embodiment differs from the first embodiment only in the evaluation function for calculating the evaluation value. Since the functional configuration of the camera parameter calculation apparatus according to the third embodiment is substantially the same as that of the camera parameter calculation apparatus 101 described in the first embodiment, description thereof is omitted.

  FIG. 15 is a flowchart illustrating an example of the camera parameter calculation process S320 according to the third embodiment. In FIG. 15, the same steps as those in the first embodiment are given the same step names as those in FIG. 2, and the description thereof is omitted. In step S324, an evaluation function J (Equation 8) defined by the sum of absolute values of the difference between the corresponding pixel value of the tele camera and the average pixel value is calculated.

Here, I ′ _k represents the average of the pixel values of the k-th corresponding point in all camera images. Since the other variables are the same as those included in Equation 3, the description thereof is omitted.

  When there are three or more cameras, the evaluation function J in Expression 8 may be expanded to Expression 9.

  Here, since the variable of Formula 9 is the same as the variable contained in Formula 3 and Formula 5, the description is abbreviate | omitted.

  As described above, as in Embodiment 1, the camera can be calibrated without associating the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image.

(Embodiment 4)
The evaluation functions in the first to third embodiments need to be limited to the absolute value sum of the difference between the pixel values shown in Equation 3, Equation 6, and Equation 8, the sum of squares, or the absolute value sum of the difference from the average pixel value. Absent. In the fourth embodiment, a case where the evaluation function is the sum of squares of the difference between the average pixel values will be described.

  The fourth embodiment differs from the first embodiment only in the evaluation function for calculating the evaluation value. Since the functional configuration of the camera parameter calculation apparatus according to the fourth embodiment is substantially the same as that of the camera parameter calculation apparatus 101 described in the first embodiment, the description thereof is omitted.

  FIG. 16 is a flowchart illustrating an example of the camera parameter calculation process 330 according to the fourth embodiment. In FIG. 16, the same steps as those in the first embodiment are given the same step names as those in FIG. 2, and the description thereof is omitted. In step S334, an evaluation function J (Equation 10) defined by the sum of squares of the difference between the corresponding pixel value of the teleo camera and the average pixel value is calculated.

  Here, since the variable of Expression 10 is the same as the variable included in Expression 8, the description thereof is omitted.

  When there are three or more cameras, the evaluation function J in Expression 10 may be expanded to Expression 11.

  Here, since the variable of Formula 11 is the same as the variable contained in Formula 3 and Formula 8, the description is abbreviate | omitted.

As described above, as in Embodiment 1, the camera can be calibrated without associating the three-dimensional coordinates in the three-dimensional space with the pixel positions in the two-dimensional image. (Embodiment 5)
In the fifth embodiment, the camera parameter is calculated based on the pattern similarity in the neighboring pixels of the image coordinates of each camera corresponding to the three-dimensional point cloud data, so that the pixel value I based on the gain difference between the cameras is obtained. The influence of the difference between _L (x, y) and I _R (x, y) is reduced.

  Hereinafter, as a specific example, a case where the evaluation function is normalized cross-correlation will be described.

  The fifth embodiment differs from the first embodiment only in the evaluation function for calculating the evaluation value. Since the functional configuration of the camera parameter calculation apparatus according to the fifth embodiment is substantially the same as the camera parameter calculation apparatus 101 described in the first embodiment, the description thereof is omitted.

  FIG. 17 is a flowchart showing an example of the camera parameter calculation process S340 according to the fifth embodiment. In FIG. 17, the same steps as those in the first embodiment are given the same step names as those in FIG. 2, and the description thereof is omitted. In step S344, an evaluation function J (Equation 12) defined by the normalized cross-correlation of the corresponding pixel value of the stereo camera is calculated.

Here, the three-dimensional point group to be compared is projected on the camera images photographed by the left camera and the right camera according to the respective camera parameters, and image coordinates are obtained. M pixels are selected from the vicinity of these image coordinates. For example, a pixel inside a circle centered on the projected pixel is selected. The i-th pixel values of the selected m pixels of the left camera and the right camera are M _Lki and M _Rki , respectively. Since the other variables are the same as those included in Equation 3, the description thereof is omitted.

As described above, by using the normalized cross-correlation of pixel values, camera calibration can be performed without being affected by the gain difference between the left camera and the right camera.
When there are three or more cameras, the evaluation function J in Expression 12 may be expanded to Expression 13.

Here, N is the number of three-dimensional points for comparing pixel values, n is the number of cameras, M _jki is a normalized cross-correlation for pixel positions in the camera image by the j-th camera corresponding to the k-th three-dimensional point. Represents the i-th pixel value in the neighboring pixel for calculating.

  Instead of normalized cross-correlation, a ternary expression or a pixel value difference between adjacent pixels may be used.

(Embodiment 6)
In the first embodiment, the example in which the three-dimensional point group data used for the evaluation is selected based on the pixel value has been described. However, the selection criterion is not necessarily limited to the pixel value. In the sixth embodiment, a case where three-dimensional point cloud data is selected based on a shielding area will be described.

  Since the functional configuration of the camera parameter calculation apparatus according to the sixth embodiment is substantially the same as that of the camera parameter calculation apparatus 101 described in the first embodiment, the description thereof is omitted.

  FIG. 18 is a flowchart illustrating an example of the camera parameter calculation process 350 according to the sixth embodiment. In FIG. 18, the same steps as those in the first embodiment are given the same step names as those in FIG. 2, and the description thereof is omitted. In step S354, the 3D point data existing in at least one occluded region of the left camera 102 and the right camera 103 is excluded from the 3D point cloud data acquired in step S303. The shielding area is an area in which the front three-dimensional object in the shielding area shields the rear three-dimensional object and extends in one direction from the camera. The determination of the shielding area will be described.

  One point in the three-dimensional space corresponds to one optical path of light incident on the camera. Even when a plurality of three-dimensional objects exist side by side in the shielding area, the three-dimensional laser scanner 106 acquires the world coordinates of the plurality of three-dimensional objects from a viewpoint different from that of the camera. When the world coordinates are projected on the image based on the camera parameters, a plurality of image coordinates that are the same or have almost no difference are calculated. That is, it can be seen that a plurality of corresponding points whose separation distances are equal to or less than the threshold are obtained in one image, and thus a plurality of three-dimensional points are located in the shielding area of the camera that has captured the image. Such a corresponding point of the three-dimensional point may represent a pixel value that does not correspond to the original three-dimensional point, and thus is not suitable for calculating the evaluation value J.

  Therefore, it is determined that a set of three-dimensional points whose distance between image coordinates projected on the image is smaller than the threshold is a three-dimensional point existing in the shielding area. The evaluation value J is calculated except for.

  As described above, camera calibration can be performed without being affected by the shielding area by removing the three-dimensional points existing in the shielding area from the calculation of the evaluation value.

(Embodiment 7)
In the first and sixth embodiments, the example in which the three-dimensional point cloud data used for the evaluation is selected based on the pixel value or the shielding area is shown, but the selection criterion is not necessarily limited to these. In the seventh embodiment, a case will be described in which three-dimensional point group data is selected based on the distance between two three-dimensional points where the angle formed by the measurement light of the three-dimensional laser scanner is smaller than a threshold value. That the angle formed by the measurement light of the three-dimensional laser scanner is smaller than the threshold is synonymous with the fact that it is visible within a predetermined vicinity from the distance measurement viewpoint. For example, in a three-dimensional laser scanner that rotates two axes of pan angle and tilt angle, It is defined that the pan angle and the tilt angle are close (that is, both the pan angle and the tilt angle are smaller than the threshold value).

  Since the functional configuration of the camera parameter calculation apparatus according to the seventh embodiment is substantially the same as that of the camera parameter calculation apparatus 101 described in the first embodiment, the description thereof is omitted.

  FIG. 19 is a flowchart illustrating an example of the camera parameter calculation process S360. In FIG. 19, the same steps as those in the first embodiment are given the same step names as those in FIG. 2, and the description thereof is omitted. In step S364, an area having a large spatial distance change is excluded from the three-dimensional point cloud data acquired in step S303. The determination of an area where the spatial distance change is large will be described.

  A three-dimensional laser scanner rotates a laser irradiation port with a plurality of rotation axes, and acquires three-dimensional point cloud data at a certain angular interval. In order to simplify the explanation, the rotation center of the three-dimensional laser scanner is the origin, and the laser irradiation port is a two-axis rotation of the pan angle and the tilt angle about the origin. The distance between two three-dimensional points having a close pan angle and tilt angle is approximated by, for example, a difference in depth value measured by a three-dimensional laser scanner. A region including a set of points having a maximum distance between two three-dimensional points having close pan angles and tilt angles larger than a threshold is determined as a region having a large spatial distance change. Then, the evaluation value J is calculated by excluding the three-dimensional point group existing in the region where the spatial distance change is large. By the exclusion, the outline of the function of the evaluation value J becomes smooth, and it is easy to minimize the evaluation value J.

  As described above, by excluding the 3D point cloud data in the region having a large spatial change, the camera calibration is performed without being affected by the 3D point cloud data existing on the boundary between objects or on the surface with large unevenness. be able to.

(Embodiment 8)
In the first, sixth, and seventh embodiments, the three-dimensional point cloud data used for evaluation is a pixel value, a shielding area, or a distance between two points of a three-dimensional point cloud that is close to the pan angle and tilt angle of the three-dimensional laser scanner. Although the example which selects based on a spatial change was shown, it does not need to restrict to these. In the eighth embodiment, a case where three-dimensional point cloud data is selected based on an intensity value will be described. Since the functional configuration of the camera parameter calculation apparatus according to the eighth embodiment is substantially the same as the camera parameter calculation apparatus 101 described in the first embodiment, the description thereof is omitted.

  FIG. 20 is a flowchart illustrating an example of the camera parameter calculation process S370. In FIG. 20, the same steps as those in the first embodiment are given the same step names as those in FIG. In step S374, the 3D point cloud data is selected from the 3D point cloud data including the intensity value acquired in step S373 based on the intensity value. The selection method will be described.

  The three-dimensional laser scanner irradiates an object with a laser, and calculates the depth of the object based on the received reflected light. For this reason, the intensity value (light reception intensity or object reflectance) associated with the three-dimensional point cloud data can be acquired. The intensity value varies depending on the shape and material of the object surface and has a correlation with the color of the object surface. The correlation will be described below. White and black objects that do not emit light in the visible light region correspond to cases where the reflectance of light is high and low, respectively. For this reason, the reflection characteristic of an object with respect to visible or near-infrared laser light is close to the reflection characteristic of visible light. Therefore, there is a correlation between the color of the object surface and the intensity value.

  Similar to the description of the seventh embodiment, in a three-dimensional laser scanner that rotates biaxially with a pan angle and a tilt angle, a three-dimensional point group that is close in pan angle and tilt angle (that is, visible within a predetermined vicinity from a distance measurement viewpoint) A set of three-dimensional points whose intensity value difference between any two points is smaller than the first threshold or larger than the second threshold is excluded from the three-dimensional point cloud data acquired in S303. By excluding the three-dimensional point group in which the difference in intensity value is small and the evaluation value J does not change and the three-dimensional point group in which the difference in intensity value is large and the change in the evaluation value J is large from the calculation of the evaluation value, The outline of the function of the evaluation value J becomes smooth, and it is easy to minimize the evaluation value J.

  As described above, the camera calibration can be performed without being affected by the three-dimensional point group data existing on the surface of the object having a large color change by excluding the three-dimensional point group data having a large difference in intensity value.

(Embodiment 9)
In Embodiments 1 and 6 to 8, the three-dimensional point cloud data used for evaluation is represented by a pixel value, a shielding area, and a spatial distance between two points of a three-dimensional point cloud having a pan angle and a tilt angle close to each other. Although an example in which selection is made based on various changes or intensity values has been shown, it is not necessary to be limited thereto. In the ninth embodiment, a case will be described in which three-dimensional point cloud data is selected based on color information of a three-dimensional point cloud.

  Since the functional configuration of the camera parameter calculation apparatus according to the ninth embodiment is substantially the same as that of the camera parameter calculation apparatus 101 described in the first embodiment, the description thereof is omitted.

  FIG. 21 is a flowchart illustrating an example of the camera parameter calculation process S380. In FIG. 21, the same steps as those in the first embodiment are given the same step names as those in FIG. 2, and the description thereof is omitted. In step S384, 3D point cloud data is selected based on the color information of the 3D point cloud from the 3D point cloud data including the color information acquired in step S383. The selection method will be described.

When visible light imaging can be performed coaxially with the laser irradiation axis of the three-dimensional laser scanner, color information (for example, RGB value or luminance value) associated with the three-dimensional point cloud data can be acquired. Similar to the description of the seventh embodiment, in a three-dimensional laser scanner that rotates biaxially with a pan angle and a tilt angle, a three-dimensional point group that is close in pan angle and tilt angle (that is, visible within a predetermined vicinity from a distance measurement viewpoint) A set of three-dimensional points in which the difference in color information (for example, RGB value or luminance value) between any two points is smaller than the first threshold value or larger than the second threshold value is acquired in step S383. Exclude from dimension point cloud data. By excluding the three-dimensional point group in which the difference in color information value is small and the evaluation value J does not change and the three-dimensional point group in which the color information difference is large and the evaluation value J is large are excluded from the calculation of the evaluation value. The outline of the function of the evaluation value J becomes smooth, and it becomes easy to minimize the evaluation value J.
As described above, the camera calibration can be performed without being influenced by the three-dimensional point group data existing on the surface of the object having a large color change by excluding the three-dimensional point group data having a large difference in color information.

(Embodiment 10)
In the first and sixth embodiments, the camera calibration using the compound eye camera and the three-dimensional laser scanner has been described. However, as described in the ninth embodiment, when the three-dimensional laser scanner can acquire color information, The camera can be calibrated with a camera and a 3D laser scanner. In the tenth embodiment, camera calibration using one camera and a three-dimensional laser scanner will be described.

  FIG. 22 is a block diagram of the camera parameter calculation device 201. In FIG. 22, the same components as those in the first embodiment are denoted by the same reference numerals as those in FIG. 1, and the description thereof is omitted. Compared to the camera parameter calculation apparatus 101 of the first embodiment, the camera parameter calculation apparatus 201 has only one camera 202 to be corrected, and an evaluation used for camera parameter calculation processing in the camera parameter calculator 108. Function is different.

  FIG. 23 is a flowchart showing an operation of camera parameter calculation processing S400 parameter calculation according to the tenth embodiment. In FIG. 23, the same steps as those in the first and ninth embodiments are given the same step names as those in FIGS. 3 and 21, and the description thereof is omitted. The three-dimensional point cloud data acquired in step S383 is added with color information obtained by the three-dimensional laser scanner and its coaxial photographing. Equation 14 shows the evaluation function J used in the evaluation value calculation in step S404.

Here, A _k represents the color information of the three-dimensional point used for calculating the k-th image coordinate of the camera. The color information is the same color information as the pixel value of the camera. For example, the pixel value of the camera is an RGB value, the color information of the _{A k} uses RGB values. Since the other variables are the same as those included in Equation 3, the description thereof is omitted.

  As described above, in the camera parameter calculation process 400 according to the tenth embodiment, camera calibration can be performed with one camera and a three-dimensional laser scanner by using the color information of the three-dimensional point.

  The absolute value sum of the difference between the pixel value and the color information in Expression 14 may be the square sum of the difference. Two or more cameras may be used.

(Embodiment 11)
In the eleventh embodiment, by calculating the camera parameters except for an area where there is no image luminance gradient or a sufficiently small area, the calculation of the area having a small contribution to the evaluation value is omitted, and the calculation amount is reduced.

  This is based on the following concept. In other words, when the luminance gradient is 0 around the pixels on the two camera images to which a certain three-dimensional point corresponds, even if the camera parameter is slightly changed, the luminance difference between the two pixels with respect to the three-dimensional point is almost constant. Become. In other words, there is no influence on the camera parameter calculation for minimizing the evaluation value J. Thus, by removing such three-dimensional points from the calculation of the evaluation value J, the amount of calculation can be reduced.

  The eleventh embodiment is different from the first embodiment only in the operation of the camera parameter calculator 108 for calculating camera parameters. Since the functional configuration of the camera parameter calculation apparatus 101 according to Embodiment 11 is substantially the same as that of the camera parameter calculation apparatus 101 described in Embodiment 1, description thereof is omitted.

  FIG. 24 is a flowchart showing an example of the camera parameter calculation process S500 according to the eleventh embodiment. In FIG. 24, the same steps as those in the first embodiment are given the same step names as those in FIG. 2, and the description thereof is omitted.

  In step S501 of FIG. 24, the camera parameter calculator 108 uses each of the N three-dimensional points acquired in step S303 for calculating the evaluation value based on the luminance gradient of the camera image acquired in step S301. A point group selection mask for determining whether or not is generated. The point group selection mask indicates, for each pixel in the camera image, whether the pixel is an effective pixel used for calculating the evaluation value or an invalid pixel not using the luminance value of the pixel for calculating the evaluation value. Take the binary value shown. The point group selection mask is created for individual camera images taken by one or a plurality of cameras.

  FIG. 25 is a flowchart illustrating a detailed example of the point group selection mask creation process S501.

  In the point group selection mask creation processing S501, a loop processing is performed to specify whether the selected pixel i is a valid pixel or an invalid pixel while sequentially selecting the pixel i in the camera image with the pixel index i. (S5011 to S5016) ).

  A luminance gradient Gi at the pixel i is calculated from neighboring pixels centered on the pixel i (S5012). As an example of the luminance gradient, Equation 15 shows the luminance gradient Gi due to adjacent pixels centered on the pixel i.

  Here, I (x, y) is a luminance value at the image coordinates (x, y).

  The luminance gradient Gi is compared with the threshold value (S5013). If the luminance gradient Gi is larger than the threshold value, the pixel i is set as an effective pixel in the point group selection mask (S5014). If the luminance gradient Gi is equal to or smaller than the threshold value, the pixel i is set as an invalid pixel in the point group selection mask (S5015). For example, the threshold value may be a constant multiple of the average luminance gradient of the entire image.

  Referring to FIG. 24 again, in step S502, the camera parameter calculator 108 is selected based on the point group selection mask created in step S501 from the three-dimensional point group data acquired in step S303. The evaluation function J (for example, Expression 3) is calculated using only the coordinates.

  In an example of a stereo camera, when a pixel located at a corresponding point of a certain three-dimensional point is indicated as an invalid pixel by a point group selection mask in at least one of the left and right camera images, the camera image corresponding to the three-dimensional point May be excluded from the calculation of the evaluation function.

  Also, in both the left and right camera images, when a pixel located at a corresponding point of a certain three-dimensional point is indicated as an invalid pixel by the point group selection mask, the pixel value on the camera image corresponding to the three-dimensional point is evaluated. It may be excluded from the calculation of the function. As described above, from the calculation of the evaluation function J, only the three-dimensional point having a small luminance gradient at the corresponding point of the camera image is excluded from the calculation of the evaluation function J, and the camera calibration is performed. Can do.

  Note that the evaluation function J need not be limited to Expression 3, and Expressions 4 to 14 may be used. In addition, the point group selection mask does not need to indicate whether the pixel is a valid pixel or an invalid pixel for all pixels of each camera image. May be indicated.

(Modification)
While the camera parameter calculation device according to one or more aspects of the present invention has been described based on the embodiment, the present invention is not limited to this embodiment. Unless it deviates from the gist of the present invention, one or more of the present invention may be applied to various modifications that can be conceived by those skilled in the art, or forms constructed by combining components in different embodiments. It may be included within the scope of the embodiments.

  The camera parameter calculation apparatus of the present invention is useful as a camera parameter calculation apparatus that calibrates various cameras without using a calibration index whose three-dimensional structure is known.

101, 201 Camera parameter calculation device 102 Left camera 103 Right camera 104 Frame memory 105 Initial camera parameter storage 106 Three-dimensional laser scanner 107 Point cloud input device 108 Camera parameter calculation device 109 Camera parameter output device 202 Camera

Claims (21)

A point cloud acquirer that acquires 3D point cloud data representing the 3D coordinates of a plurality of 3D points included in an imaging space common to one or a plurality of cameras;
Based on the 3D point cloud data and one or more initial camera parameters of the one or more cameras, each of the plurality of 3D points in one or more images taken by the one or more cameras A camera parameter calculator that calculates image coordinates of corresponding points, and calculates the one or more camera parameters of the one or more cameras based on pixel values in the image coordinates in the one or more images;
A camera parameter output device for outputting the one or more camera parameters;
The one or more cameras, the one or more initial camera parameters, the one or more images, and the one or more camera parameters correspond one-to-one. In calculating the image coordinates, the camera parameter calculator corresponds to the one or more cameras based on the one or more initial camera parameters corresponding to the one or more cameras. Including conversion to image coordinates in the image,
The camera parameter calculation apparatus according to claim 1. The one or more cameras include two cameras;
The camera parameter calculator calculates the camera parameter based on a plurality of differences;
Each of the plurality of differences is
A pixel value at a corresponding point in a first image captured by a first camera included in the two cameras with respect to one of the plurality of three-dimensional points;
The difference between the one three-dimensional point and the pixel value at the corresponding point in the second image captured by the second camera included in the two cameras.
The camera parameter calculation apparatus according to claim 1 or 2. The difference is an absolute value of the difference between the pixel values.
The camera parameter calculation apparatus according to claim 3. The difference is a square value of the difference between the pixel values.
The camera parameter calculation apparatus according to claim 3. The one or more cameras are two or more cameras;
The camera parameter calculator calculates the camera parameter based on a plurality of differences;
Each of the plurality of differences is
The difference between the pixel value of one pixel value at the corresponding point in each image captured by the two or more cameras and the average value of the pixel values for one of the plurality of three-dimensional points. Is,
The camera parameter calculation apparatus according to claim 1 or 2. The difference is an absolute value of a difference between the pixel value and the average value.
The camera parameter calculation apparatus according to claim 6. The difference is a square value of a difference between the pixel value and the average value.
The camera parameter calculation apparatus according to claim 6. The one or more cameras are two or more cameras;
The camera parameter calculator calculates the camera parameter based on a similarity of a pixel value pattern in a neighboring pixel of the corresponding point in the individual images corresponding to the same three-dimensional point;
The camera parameter calculation apparatus according to claim 1 or 2. The camera parameter calculator calculates camera parameters based on a plurality of normalized cross-correlations, and each of the plurality of normalized cross-correlations includes the 2 for one 3D point of the plurality of 3D points. Correlation of pixel values included in the vicinity of corresponding points in individual images taken with the above cameras.
The camera parameter calculation apparatus according to claim 9. The camera parameter calculator excludes, from the calculation of the camera parameters, a three-dimensional point in which a plurality of corresponding points whose separation distances are equal to or less than a threshold in at least one of the plurality of three-dimensional points is obtained. ,
The camera parameter calculation device according to claim 1. The camera parameter calculator excludes one or a plurality of three-dimensional points having a luminance gradient at a corresponding point of at least one of the one or more images smaller than a threshold value from the plurality of three-dimensional points. Calculating the camera parameters from the dimension points;
The camera parameter calculation device according to claim 1. The three-dimensional point cloud data represents a three-dimensional position of the plurality of three-dimensional points based on reflection of measurement light emitted from a predetermined distance measuring viewpoint,
The camera parameter calculator is two three-dimensional points included in the plurality of three-dimensional points, and a distance between the two three-dimensional points is larger than a threshold value, and the measurement to the two three-dimensional points is performed. Excluding from the calculation of the camera parameters two three-dimensional points whose angle formed by the light direction vector is smaller than a threshold value;
The camera parameter calculation device according to claim 1. The three-dimensional point group data includes the three-dimensional positions of the plurality of three-dimensional points and the intensity of the return light of the measurement light from each three-dimensional point based on the reflection of the measurement light emitted from a predetermined distance measurement viewpoint. Value and
The camera parameter calculator includes a plurality of three-dimensional points that are visible within a predetermined vicinity from the distance measurement viewpoint, and that the difference in intensity is within a range determined by a first threshold value and a second threshold value. The camera parameters are calculated using only the three-dimensional points of
The camera parameter calculation device according to claim 1. The three-dimensional point group data is data obtained by measuring the plurality of three-dimensional points with a three-dimensional laser scanner.
The camera parameter calculation apparatus according to any one of claims 1 to 14. The three-dimensional point cloud data is based on the reflection of measurement light emitted from a predetermined distance measurement viewpoint and color photographing from the distance measurement viewpoint, and the three-dimensional positions of the plurality of three-dimensional points and the respective three-dimensional points. Represents the color of the object in
The camera parameter calculator is within a range of the plurality of three-dimensional points that can be seen within a predetermined vicinity from the distance measurement viewpoint and that the color difference of the object is determined by a first threshold value and a second threshold value. Calculating the camera parameters using only a plurality of three-dimensional points;
The camera parameter calculation apparatus according to any one of claims 2 to 10. The three-dimensional point cloud data is based on the reflection of measurement light emitted from a predetermined distance measurement viewpoint and color photographing from the distance measurement viewpoint, and the three-dimensional positions of the plurality of three-dimensional points and the respective three-dimensional points. Represents the color of the object in
Each of the one or more cameras is a color camera;
The camera parameter calculator calculates camera parameters of one of the one or more cameras based on a plurality of differences, and each of the plurality of differences is one of the plurality of three-dimensional points. A color represented by a pixel value at a corresponding point in an image photographed by the one camera with respect to two three-dimensional points, and a color of an object at the one three-dimensional point represented by the three-dimensional point group data Is the difference,
The camera parameter calculation apparatus according to claim 1. The point group acquisition unit acquires the three-dimensional point group data from a three-dimensional laser scanner with a color camera that color-shoots a ranging space;
The camera parameter calculation apparatus according to claim 16 or 17. A point cloud acquisition step of acquiring three-dimensional point cloud data representing a three-dimensional position of each of a plurality of three-dimensional points included in the imaging space of one or a plurality of cameras;
Based on the three-dimensional point cloud data and the initial camera parameters of each of the cameras, a corresponding point in each image captured by the one or more cameras is obtained for each of the plurality of three-dimensional points, Camera parameter calculation step for calculating camera parameters of each camera based on initial camera parameters and pixel values at the corresponding points in the individual images;
A camera parameter output step of outputting the calculated camera parameter of each camera;
Camera parameter calculation method. A point cloud acquisition step of acquiring three-dimensional point cloud data representing a three-dimensional position of each of a plurality of three-dimensional points included in the imaging space of one or a plurality of cameras;
Based on the three-dimensional point cloud data and the initial camera parameters of each of the cameras, a corresponding point in each image captured by the one or more cameras is obtained for each of the plurality of three-dimensional points, Camera parameter calculation step for calculating camera parameters of each camera based on initial camera parameters and pixel values at the corresponding points in the individual images;
A camera parameter output step of outputting the calculated camera parameter of each camera;
A program that causes a computer to execute. A point cloud acquisition step of acquiring three-dimensional point cloud data representing a three-dimensional position of each of a plurality of three-dimensional points included in the imaging space of one or a plurality of cameras;
Based on the three-dimensional point cloud data and the initial camera parameters of each of the cameras, a corresponding point in each image captured by the one or more cameras is obtained for each of the plurality of three-dimensional points, Camera parameter calculation step for calculating camera parameters of each camera based on initial camera parameters and pixel values at the corresponding points in the individual images;
A camera parameter output step of outputting the calculated camera parameter of each camera;
A computer-readable recording medium in which a program for causing a computer to execute is recorded.