JP7429667B2 - Camera calibration device, method and program - Google Patents

Description

The present invention relates to an apparatus, method, and program for calibrating the relative positions of a plurality of cameras with high precision.

Free viewpoint video technology is a technology that inputs video from multiple cameras and enables viewing from a viewpoint where no camera exists. As a free viewpoint video generation method, the visual volume intersection method described in Non-Patent Document 1 is known. The visual volume intersection method reconstructs the intersection of multiple silhouette images A, B, C, and D taken from different camera positions into a 3D space as a 3D model, as shown in Figure 10. It's technology.

The projection relationship between the three-dimensional space and the silhouette image is determined by internal parameters such as focal length and external parameters indicating the installation position and orientation of the camera. At this time, if errors are included in the estimation of the internal and external parameters of each camera, the intersection set when generating the 3D model will be restored in a shape different from the original shape, and the resulting 3D model will be Defects may occur.

Patent Document 1 discloses a technology that prevents defects from occurring in a 3D model by expanding or contracting the outline of a silhouette according to the size of the error in the camera parameters, assuming that the camera parameters include errors. Disclosed.

Non-Patent Document 2 describes a camera parameter estimation method in which multiple points of interest whose three-dimensional spatial coordinates are known are photographed with a camera, and the three-dimensional spatial coordinates of each point of interest and each point of interest reflected in the camera image are calculated. Corresponding points (corresponding points) are correlated using the method disclosed in Non-Patent Document 3, for example, and each point of interest is projected from 3D space coordinates to image coordinates using reprojection errors based on multiple correspondences, that is, camera parameters. A technique has been disclosed for optimizing camera parameters so that the image coordinate error between a given point and a corresponding point on a camera image is reduced.

Furthermore, as a method for increasing the accuracy of relative positions between cameras, Loop Closing is disclosed in Non-Patent Document 4. Loop Closing evaluates image similarity (BoW), which can be calculated based on many points of interest on the images, for multiple images taken surrounding the subject by multiple cameras or moving cameras, and then The relative positional relationship is optimized based on the constraint that the cameras surrounding the frame form a loop.

Patent application No. 2020-212906 Patent application No. 2020-012676 Patent application No. 2020-127288

Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). Z. Zhang. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(11):1330-1334, 2000. Yuki Tsurusaki, Keisuke Nonaka, Ryosuke Watanabe, Sei Naito, "Study on improving the accuracy of camera calibration using Line Segment Detector," 2020-AVM-108, 7, pp. 1 - 6, 2020. R Mur-Artal, JMM Montiel, JD Tardos. ORB-SLAM: a Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics, 2015.

Patent Document 1 prevents defects from occurring in the 3D model by expanding or contracting the silhouette image, but since the projection relationship between the 3D space and the silhouette image cannot be correctly corrected by only modifying the silhouette image, the 3D model It is difficult to approximate the 3D shape of the original object.

The camera calibration methods disclosed in Non-Patent Documents 2 and 3 do not take into account the relative positions between the cameras, and therefore cannot prevent the loss of the 3D model caused by the camera parameter estimation error pointed out in Patent Document 1.

Non-Patent Document 4 evaluates the similarity of images based on a large number of points of interest on the image and optimizes the relative position, but the image coordinates of the detected points of interest include errors. is not taken into account.

An object of the present invention is to solve the above technical problems and provide a camera calibration device, method, and program for estimating the relative position between each camera with high accuracy by focusing on the points that appear in common between multiple cameras. There is a particular thing.

In order to achieve the above object, the present invention provides an apparatus for relatively calibrating each camera based on camera images taken by a plurality of cameras of a calibration coordinate model with known three-dimensional coordinates, which has the following configuration. It is characterized by having the following.

(1) means for associating a plurality of points of interest commonly seen in all camera images of the calibration coordinate model with corresponding points on the camera images; and means for determining one of the plurality of cameras as a reference camera; , a means for estimating the relative position of each camera based on a projection relationship between the cameras of each corresponding point of each camera image; and a means for projecting the corresponding point of each camera onto another camera based on the projection relationship; means for correcting the image coordinates of the corresponding points of the cameras based on the image coordinates of the projected points; and means for calculating the extrinsic parameters of each camera based on the extrinsic parameters of the reference camera and the corrected corresponding points of each camera. Equipped with.

(2) The correcting means determines some cameras with high accuracy of correspondence with three-dimensional space coordinates as selected cameras, and the correcting means includes each corresponding point of the selected camera including other selected cameras based on the projective relationship. Projected to each camera.

(3) comprising means for calculating an error in the image coordinates of each corresponding point after correction of each camera, and means for determining whether the error has converged, and selecting a reference camera until the error has converged; Estimation of relative positions and correction of image coordinates of corresponding points are repeated.

According to the present invention, the following effects are achieved.

(1) The positions of multiple cameras that photograph the subject from different viewpoints are mutually corrected based on the relative positional relationship with other cameras, and external parameters are calculated based on the corrected relative positions. The relative relationships of external parameters are highly accurate. Therefore, it becomes possible to improve the quality of 3D models created based on multiple camera images.

(2) Since each corresponding point of the selected camera with relatively high accuracy is projected to each camera including other selected cameras, even if there is a camera with poor matching accuracy, the selected camera can improve the matching. Accuracy (coordinate correction) will be possible.

(3) Since coordinate correction is repeated until the error converges, the relative relationships of all cameras can be highly accurate.

FIG. 1 is a functional block diagram of a camera calibration device according to a first embodiment of the present invention. FIG. 3 is a diagram showing an example of arrangement of multiple cameras. FIG. 7 is a diagram showing a method of associating a calibration coordinate model with image coordinates of a camera. FIG. 3 is a diagram for explaining a method of calculating relative positions between cameras. FIG. 2 is a diagram (part 1) showing a method for correcting image coordinates of corresponding points; FIG. 7 is a diagram (Part 2) showing a method for correcting image coordinates of corresponding points. FIG. 3 is a diagram for explaining the effects of the present invention. FIG. 3 is a functional block diagram of a camera calibration device according to a second embodiment of the present invention. FIG. 6 is a diagram showing a method for determining an error in image coordinates of corresponding points. FIG. 2 is a diagram showing a method of generating a 3D model using a visual volume intersection method.

Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of main parts of a camera calibration device 1 according to a first embodiment of the present invention, including a camera image input section 10, a distortion correction section 20, a matching section 30, and a camera determination section 10. The main components include a section 40, a relative position calculation section 50, an image coordinate correction section 60, and a camera parameter calculation section 70.

Such a camera calibration device 1 can be configured by installing an application (program) for realizing each function on at least one general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or single-function machine in which part of the application is converted into hardware or software.

This embodiment assumes application to a free-viewpoint video production system targeting sports scenes, and as shown in Figure 2, 16 cameras facing the soccer court are installed around the soccer court to create a 3D model quality image. An example will be explained in which camera parameters of each camera are estimated for the purpose of improvement.

Images captured by a plurality of cameras (16 in this embodiment) from different viewpoints are input to the image input unit 10. The distortion correction unit 20 performs distortion correction on the input camera image. In this example, the parameters (distortion parameters) of the distortion correction model represented by the following equations (1) and (2) are calculated for each camera from the camera setting information at the time of shooting using the method disclosed in Non-Patent Document 2. Distortion correction is performed using the distortion parameters calculated in advance.

Here, (x', y') and (x, y) are the coordinates on each camera image before and after distortion correction (hereinafter sometimes expressed as image coordinates), and k _i , P _i are distortion correction amounts for radial distortion and circumferential distortion, respectively.

The association unit 30 is given a calibration coordinate model (in this embodiment, a soccer field model) whose three-dimensional coordinates are known, and calculates the three-dimensional spatial coordinates (X , Y, Z) and the camera's image coordinates (x, y).

In this embodiment, as an example shown in FIG. 3, attention is paid to at least four points on the field line whose three-dimensional spatial coordinates are known from the standards of soccer courts, etc., and the three-dimensional spatial coordinates (Xi, Yi, Zi) and the image coordinates (xi, yi) of the corresponding point seen by the camera.

This kind of correspondence is achieved by displaying each image frame input from all 16 cameras on a computer, and using a pointing device that controls the computer to identify each point of interest on the field line and each corresponding point on the image. It may be carried out by associating with the above, or it may be automatically estimated by the conventional method disclosed in Non-Patent Document 3.

Furthermore, when shooting at a close distance to the subject, such as in an indoor competition, a known pattern such as an AR marker may be used as a calibration coordinate model, and correspondence may be automatically performed by pattern matching or the like. In addition, in both manual and automatic mapping, errors are inevitably included in the image coordinates of each corresponding point due to deviations in pointing operations and problems with estimation accuracy.

The camera determining section 40 includes a reference camera determining section 41 and a selected camera determining section 42. The reference camera determination unit 41 determines one camera, which serves as a reference for calculating the relative positions of each camera, as the reference camera. The selected camera determining unit 42 determines, as selected cameras, a plurality of cameras to be used when an image coordinate correcting unit 60 (described later) relatively corrects the image coordinates of each camera.

The reference camera determination unit 41 can determine any one camera from a total of 16 cameras as the reference camera. The selected camera determining unit 42 projects points of each camera image corresponding to the same point in the three-dimensional space onto three-dimensional space coordinates, and determines a plurality of selected cameras based on the degree of convergence with other cameras.

In this embodiment, from the combination of the three-dimensional spatial coordinates of each point of interest on the soccer court, which are matched by the matching unit 30, and the image coordinates of each corresponding point on the camera image, the method disclosed in Non-Patent Document 2 is used. A homography matrix is estimated using the method, and points are projected from each camera onto the soccer court plane using the estimated homography matrix.

Here, the homography matrix is a projective transformation matrix of points on two different planes, and it transforms a point (X _i , Y _i , 1) on the soccer court plane into a point (x _i , y _i , The determinant projected onto 1) is expressed by the following equation (3).

In this embodiment, in order to exclude cameras whose projected points are significantly different from other cameras and determine the remaining cameras as selected cameras, a threshold is set based on the Mahalanobis distance of the projected points from all cameras, and if the projected point exceeds the threshold. I'm trying to exclude cameras.

Note that the method for determining the selected camera is not limited to the above method, and for each camera to be evaluated, a partial camera 3D model created based on images of all cameras other than the camera in question, and a 3D model of all cameras. The quality is compared with all camera 3D models created based on the image, and when the quality of some camera 3D models is higher than the quality of all camera 3D models, the camera other than the relevant camera is determined as the selected camera. It's okay.

Alternatively, the same selection may be repeated for only the cameras remaining in the above selection, and only the cameras that ultimately remain under predetermined conditions may be determined as the selected cameras. The quality of a 3D model can be automatically evaluated based on how well the silhouette of the created model when projected at the camera position matches the silhouette of the subject as seen by the camera.

As for the reference camera, in the same way as the selected camera, if the corresponding points of each camera image corresponding to the same point in the 3D space are projected into the 3D space and the one with the minimum reprojection error is selected, the camera Not only the relative position but also the absolute position can be made more precise.

As shown in FIG. 4, the relative position calculation unit 50 calculates the projection relationship of corresponding points corresponding to the same point of interest with each other camera (including other selected cameras and a reference camera) for each selected camera. Estimate the homography matrix using

In this embodiment, the association unit 30 focuses on each corresponding point of each camera image corresponding to each point of interest of the calibration coordinate model used for association, and based on the relationship between the image coordinates of each corresponding point, A homography matrix is calculated.

The image coordinate correction unit 60 relatively corrects the image coordinates of each corresponding point of each camera based on the projection relationship between each selected camera and each other camera calculated by the relative position calculation unit 50. In this embodiment, first, as shown in FIG. 5, each corresponding point is projected onto each other camera using a homography matrix indicating the relative positional relationship with each other camera for each selected camera. Therefore, if there are M selected cameras, M points will be projected for each corresponding point to each projection destination camera.

Next, for each projection destination camera, the average value and median value of the image coordinates of each point are calculated as the representative values of the multiple points projected from each selected camera regarding the same corresponding point, as shown in FIG. The representative value is provisionally determined as the corrected camera image coordinates for each corresponding point. At this time, if the projection destination is another selected camera, the image coordinates of its own corresponding point are included in the representative value calculation, but if the projection destination is other than the selected camera, the representative value is calculated only from the image coordinates of the projected point. Implement.

The camera parameter calculation unit 70 calculates camera parameters of all cameras and determines camera positions. The camera parameters are for converting a point (X _i , Y _i , Z _i ) on three-dimensional space coordinates into a point (x _i , y _i ) on image coordinates, and are expressed by the following equation (4).

Here, (c _x , c _y ) is the principal point in the camera image, usually at the center of the image. f indicates the focal length, which in this embodiment is a value to be calculated instead of the zoom value. These are called internal parameters and remain constant values as long as the focal length is fixed. Further, r ₁₁ to r ₃₃ are rotation matrices indicating the amount of rotation of the camera, t ₁ to t ₃ are translation matrices indicating the position of the camera, and s is a coefficient used for scaling. A matrix composed of rotation matrices r ₁₁ to r ₃₃ and translation matrices t ₁ to t ₃ is called an extrinsic parameter.

In the present embodiment, the homography matrix is estimated based on the relationship between the image coordinates of each corresponding point after correction and the corresponding point of interest on the soccer court for the reference camera determined by the camera determining unit 40, and The extrinsic parameters of the reference camera are calculated using the determined intrinsic parameters.

Next, using homography matrices between the reference camera and each other camera and internal parameters of each camera prepared in advance, external parameters indicating the relative position from the reference camera are determined. Then, the external parameters of all cameras based on the soccer court are calculated based on the external parameters of the reference camera calculated based on the soccer court in 3D space and the external parameters indicating the relative position of each camera, and the camera parameters are output. do.

According to the present invention, the positions of multiple cameras that photograph a subject from different viewpoints are mutually corrected based on the relative positional relationship with other cameras, and external parameters are calculated based on the corrected relative positions. Therefore, as shown in FIG. 7, although each camera parameter may take a parameter that is slightly different from the actual camera parameter, the error in the relative position of each camera is reduced. As a result, the relative relationships between external parameters become more precise, making it possible to improve the quality of 3D models created based on multiple camera images.

FIG. 8 is a functional block diagram showing the configuration of the main parts of the camera calibration device 1 according to the second embodiment of the present invention, and the same reference numerals as above represent the same or equivalent parts. is omitted. The present embodiment is characterized in that it includes an error determination section 80 and repeats image coordinate correction of corresponding points of each camera until the reprojection error converges.

The error determination unit 80 includes an error calculation unit 81 and a convergence determination unit 82, and by correcting the image coordinates of the corresponding points of each camera in the image coordinate correction unit 60, the relative positions between the cameras except the reference camera are improved in accuracy. Determine whether or not it has been done.

As shown in FIG. 9, the error calculation unit 81 calculates the uncorrected and Each corrected image coordinate is reprojected based on the projection relationship, and the distance between the pixel position of each corresponding point of interest photographed by the reference camera and the reprojection position from each camera is calculated as a reprojection error.

If the reprojection error of the pixel position after correction is smaller than the reprojection error of the pixel position before correction in all the remaining cameras except the reference camera, the convergence determination unit 82 determines that the error correction has converged and performs the correction. Each subsequent pixel position is determined to be the pixel position of each corresponding point.

On the other hand, if there is a camera with a larger reprojection error of the pixel position after correction, the image coordinates of each corresponding point of that camera will be returned to the image coordinates before correction, and the image coordinates after correction will be reprojected. For cameras with smaller projection errors, the process returns to the camera determining unit 40 while maintaining the corrected image coordinates.

Then, the processes of determining the reference camera and selected camera by the camera determining unit 40, determining the relative position by the relative position estimating unit 50, and correcting the image coordinates by the image coordinate correcting unit 60 are repeated. This repetition is repeated until the number of cameras in which the reprojection error of the image coordinates after the current correction is larger than the reprojection error of the image coordinates after the previous correction or correction becomes zero or less than a predetermined threshold. Alternatively, the process is repeated for all cameras until the absolute value of the difference between the reprojection error of the image coordinates after the current correction and the reprojection error of the image coordinates after or before the previous correction falls below a predetermined threshold. Also good.

According to this embodiment, the relative position of each other camera with respect to the reference camera that serves as a reference for external parameters is corrected, so the relative relationship between the external parameters of the reference camera and the external parameters of each other camera is improved. This makes it possible to further improve the quality of 3D models.

According to each of the above embodiments, it is possible to provide high-quality 3D models of objects in real time even via communication infrastructure, so it is possible to provide a variety of entertainment to many people regardless of geographic or economic disparity. be able to provide it. As a result, Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations: ``Build resilient infrastructure and promote inclusive and sustainable industrialization'' and Goal 11: ``Make cities inclusive, safe and resilient.'' It will be possible to contribute to "making the world more sustainable and more sustainable."

Note that in this embodiment, the relative position error was evaluated as a reprojection error, but the present invention is not limited to this, and a 3D model is generated from each image coordinate information before and after correction, Evaluation may be performed based on the quality of the 3D model.

1... Camera calibration device, 10... Camera image input section, 20... Distortion correction section, 30... Corresponding section, 40... Camera determining section, 41... Reference camera determining section, 42... Selected camera determining section, 50... Relative position Calculation section, 60... Image coordinate correction section, 70... Camera parameter calculation section, 80... Error judgment section, 81... Error calculation section, 82... Convergence judgment section

Claims (15)

In a device that relatively calibrates each camera based on camera images taken by multiple cameras of a calibration coordinate model whose three-dimensional coordinates are known,
means for associating a plurality of points of interest common to all camera images of the calibration coordinate model with corresponding points on the camera images;
means for determining one of the plurality of cameras as a reference camera;
means for estimating the relative position of each camera based on the projection relationship between the cameras of each corresponding point of each camera image;
means for projecting the corresponding point of each camera onto another camera based on the projection relationship, and correcting the image coordinates of the corresponding point of the projection destination camera based on the image coordinate of the projected point;
A camera calibration device comprising means for calculating an extrinsic parameter of each camera based on an extrinsic parameter of a reference camera and corrected corresponding points of each camera. comprising means for determining a plurality of selected cameras from all the cameras as selected cameras;
2. The camera calibration apparatus according to claim 1, wherein the correcting means projects each corresponding point of the selected camera onto other cameras including another selected camera based on the projection relationship. The means for determining the selected camera determines the selected camera based on the degree of convergence of errors when the corresponding points of each camera are reprojected onto the 3D space of the calibration coordinate model based on the projection relationship. The camera calibration device according to claim 2. The means for determining the selected camera is based on the quality of a partial camera 3D model created based on the images of all the remaining cameras excluding the camera and the quality of all camera 3D models created based on the images of all cameras for each camera. 3. The camera calibration device according to claim 2, wherein when the quality of some camera 3D models is superior to the quality of all camera 3D models, a camera other than the excluded camera is determined as the selected camera. . The correcting means corrects the image coordinates of each corresponding point based on each corresponding point of the other selected camera and the projected point when the projection destination is another selected camera, and when the projection destination is other than the selected camera. 5. The camera calibration device according to claim 2, wherein the image coordinates of each corresponding point are corrected based on the projected point, if any. means for calculating an error in the image coordinates of each corresponding point after the correction of each camera;
and means for determining whether the error has converged,
6. The means for determining the convergence of the error repeats the selection of the reference camera, the estimation of the relative position, and the correction of the image coordinates of the corresponding points until the error converges. Camera calibration device as described. The means for calculating the error projects each corresponding point before and after correction of each camera image onto a reference camera, and calculates a reprojection error of each corresponding point before and after correction,
The means for determining the convergence of the errors returns each corresponding point of the camera to the state before correction, in which the reprojection error of the corresponding point after correction is larger than the reprojection error of the corresponding point before correction, and selects the reference camera, and 7. The camera calibration device according to claim 6, wherein position estimation and correction of image coordinates of corresponding points are repeated. The means for determining the convergence of the error includes selecting the reference camera, and controlling the relative 8. The camera calibration device according to claim 7, wherein position estimation and correction of image coordinates of corresponding points are repeated. The means for calculating the external parameters include:
calculating a first extrinsic parameter based on the internal parameter and a projection relationship with respect to the calibration coordinate model of the reference camera;
calculating a second external parameter indicating the relative position of the reference camera and the other cameras based on the projective relationship between the reference camera and the other cameras and the internal parameters of each camera;
9. The camera calibration apparatus according to claim 1, wherein an external parameter for a calibration coordinate model of each camera is calculated based on the first and second external parameters. 10. The camera calibration device according to claim 1, further comprising a distortion correction section that removes lens distortion components from each camera image. Camera calibration according to any one of claims 1 to 10, characterized in that the means for determining the reference camera determines, as the reference camera, a camera with a minimum reprojection error of corresponding points with respect to the calibration coordinate model. Device. 12. The camera calibration device according to claim 1, wherein the calibration coordinate model is an AR marker. A method in which a computer relatively calibrates each camera based on camera images taken by a plurality of cameras of a calibration coordinate model whose three-dimensional coordinates are known,
Associating multiple points of interest commonly seen in all camera images of the calibration coordinate model with corresponding points on the camera images,
determining one of the plurality of cameras as a reference camera;
Estimate the relative position of each camera based on the projection relationship between the cameras of each corresponding point in each camera image,
After projecting the corresponding points of each camera to another camera based on the projection relationship, correcting the image coordinates of the corresponding points of the projection destination camera based on the image coordinates of the projected point,
A camera calibration method characterized in that an extrinsic parameter of each camera is calculated based on an extrinsic parameter of a reference camera and corresponding points after correction of each camera. Decide some of the selected cameras from all the cameras as the selected cameras,
14. The camera calibrator according to claim 13, wherein when correcting the image coordinates of the corresponding points, each corresponding point of the selected camera is projected onto other cameras including another selected camera based on the projection relationship. tion method. In a program that relatively calibrates each camera based on camera images taken by multiple cameras of a calibration coordinate model with known 3D coordinates,
a procedure for associating a plurality of points of interest commonly seen in all camera images of the calibration coordinate model with corresponding points on the camera images;
a procedure for determining one of the plurality of cameras as a reference camera;
a step of estimating the relative position of each camera based on the projective relationship between the cameras of each corresponding point of each camera image;
a step of projecting the corresponding point of each camera onto another camera based on the projection relationship, and then correcting the image coordinates of the corresponding point of the projection destination camera based on the image coordinate of the projected point;
A camera calibration program that causes a computer to execute steps for calculating external parameters of each camera based on external parameters of a reference camera and corresponding points after correction of each camera.

Images