JP2022146988A - Camera calibration device, method, and program - Google Patents

Abstract

To provide a camera calibration device configured to accurately estimate relative positions of cameras by focusing on a point captured in common by the cameras, a method, and a program.SOLUTION: A camera calibration device includes: an association unit 30 which associates a plurality of points of interest captured in common in all camera images of a calibration coordinate model with corresponding points on the camera images; a reference camera determination unit 41 which determines one of the cameras as a reference camera; a relative position calculation unit 50 which calculates relative positions of the cameras on the basis of projection relationship, between the cameras, of the corresponding points of the camera images; an image coordinate correction unit 60 which projects the corresponding points of the cameras to the other cameras on the basis of the projection relationship, and corrects image coordinates of the corresponding points of the projected camera on the basis of image coordinates of the projected points; and a camera parameter calculation unit 70 which calculates external parameters of the cameras on the basis of external parameter of the reference camera and the corrected corresponding points of the cameras.SELECTED DRAWING: Figure 1

Description

The present invention relates to an apparatus, method, and program for calibrating the relative positions of a plurality of cameras with high accuracy.

Free-viewpoint video technology is a technology that enables viewing from viewpoints where there are no cameras by inputting video from multiple cameras. As a method for generating a free-viewpoint video, the visual volume intersection method disclosed in Non-Patent Document 1 is known. In the visual volume intersection method, as shown in Fig. 10, multiple silhouette images A, B, C, and D taken from different camera positions are projected onto a three-dimensional space, and the intersection of these images is restored as a 3D model. Technology.

The projection relationship between the three-dimensional space and the silhouette image is determined by internal parameters such as the focal length and external parameters indicating the installation position and orientation of the camera. At this time, if there is an error in the estimation of the internal and external parameters of each camera, the intersection when generating the 3D model will be restored in a different form from the original shape, and the resulting 3D model will be Defects may occur.

Patent Document 1 discloses a technique for preventing defects in a 3D model by expanding or contracting the outline of a silhouette according to the magnitude of the camera parameter error, on the premise that the camera parameter contains an error. disclosed.

In Non-Patent Document 2, as a method of estimating camera parameters, a plurality of points of interest whose 3D space coordinates are known are photographed with a camera, and the 3D space coordinates of each point of interest and each point of interest reflected in the camera image are Corresponding points (corresponding points) are associated, for example, by the method disclosed in Non-Patent Document 3, and each point of interest is projected from the three-dimensional spatial coordinates to the image coordinates by the reprojection error based on the plurality of correspondence relationships, that is, the camera parameters. There is disclosed a technique for optimizing camera parameters so that the image coordinate error between a point and a corresponding point on the camera image is small.

In addition, Non-Patent Document 4 discloses Loop Closing as a technique for increasing the accuracy of relative positions between cameras. Loop Closing evaluates the image similarity (BoW), which can be calculated based on a large number of points of interest on the images, for multiple images captured by surrounding the subject with a large number of cameras or by moving the cameras. The relative positional relationship is optimized based on the constraint that the cameras surrounding the form a loop.

Patent application No. 2020-212906 Patent application No. 2020-012676 Patent application No. 2020-127288

Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). Z. Zhang. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(11):1330-1334, 2000. Yuki Tsurusaki, Keisuke Nonaka, Ryosuke Watanabe, Tadashi Naito, "Study on Improving Accuracy of Camera Calibration Using Line Segment Detector," 2020-AVM-108, 7, pp. 1 - 6, 2020. R Mur-Artal, JMM Montiel, JD Tardos. ORB-SLAM: a Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics, 2015.

In Patent Document 1, the silhouette image is expanded or contracted to prevent loss in the 3D model. is difficult to approximate the original 3D shape of the subject.

Since the camera calibration methods disclosed in Non-Patent Documents 2 and 3 do not consider the relative position between cameras, it is not possible to prevent the loss of the 3D model caused by the camera parameter estimation error pointed out in Patent Document 1.

In Non-Patent Document 4, image similarity is evaluated based on a large number of points of interest on the image, and relative positions are optimized, but there is an error in the image coordinates of the detected points of interest. is not considered.

An object of the present invention is to solve the above technical problems and to provide a camera calibration device, method, and program for estimating the relative position between cameras with high accuracy by paying attention to common points captured by multiple cameras. That's what it is.

In order to achieve the above object, the present invention provides an apparatus for relatively calibrating each camera based on camera images of a calibration coordinate model with known three-dimensional coordinates taken by a plurality of cameras, and has the following configuration: It is characterized by having

(1) means for associating a plurality of points of interest that appear in common on all camera images of the calibration coordinate model with corresponding points on the camera images; and means for determining one of the plurality of cameras as a reference camera. , means for estimating the relative position of each camera based on the projection relationship between the corresponding points of each camera image, and projecting the corresponding points of each camera to another camera based on the projection relationship, Means for correcting the image coordinates of the corresponding points of the camera based on the image coordinates of the projected points, and Means for calculating the extrinsic parameters of each camera based on the extrinsic parameters of the reference camera and the corrected corresponding points of each camera. Equipped with.

(2) A part of cameras having a high degree of matching accuracy with the three-dimensional space coordinates is determined as the selected camera, and the means for correcting each corresponding point of the selected camera includes other selected cameras based on the projection relationship. Projected to each camera.

(3) Selecting a reference camera until the error converges, comprising means for calculating the error of the image coordinates of each corresponding point after correction for each camera and means for determining whether the error has converged; The estimation of the relative position and the correction of the image coordinates of the corresponding points are repeated.

According to the present invention, the following effects are achieved.

(1) Since the positions of a plurality of cameras that photograph the subject from different viewpoints are mutually corrected based on the relative positional relationship with other cameras, and the extrinsic parameters are calculated based on the corrected relative positions, The relative relationship of external parameters is highly accurate. Therefore, it becomes possible to improve the quality of a 3D model produced based on multiple camera images.

(2) Since each corresponding point of a selected camera with relatively high accuracy is projected onto each camera including other selected cameras, even if there are cameras with poor matching accuracy, the selected camera can improve the matching. Accuracy improvement (coordinate correction) becomes possible.

(3) Since the coordinate correction is repeated until the error converges, it becomes possible to improve the accuracy of the relative relationships of all cameras.

1 is a functional block diagram of a camera calibration device according to a first embodiment of the present invention; FIG. FIG. 4 is a diagram showing an example of arrangement of multiple cameras; FIG. 4 is a diagram showing a method of associating a calibration coordinate model with image coordinates of a camera; FIG. 4 is a diagram for explaining a method of calculating relative positions between cameras; FIG. 10 is a diagram (part 1) showing a method of correcting image coordinates of corresponding points; FIG. 10 is a diagram (part 2) showing a method of correcting image coordinates of corresponding points; It is a figure for demonstrating the effect of this invention. FIG. 5 is a functional block diagram of a camera calibration device according to a second embodiment of the present invention; FIG. 10 is a diagram showing a method of determining an error in image coordinates of corresponding points; FIG. 4 is a diagram showing a method of generating a 3D model by the visual volume intersection method;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a camera calibration device 1 according to the first embodiment of the present invention. A section 40, a relative position calculation section 50, an image coordinate correction section 60, and a camera parameter calculation section 70 are main components.

Such a camera calibration device 1 can be configured by installing an application (program) that implements each function in at least one general-purpose computer or server. Alternatively, a part of the application can be configured as a dedicated machine or a single-function machine that is made into hardware or software.

This embodiment assumes application to a free-viewpoint video production system for sports scenes. A case of estimating the camera parameters of each camera for the purpose of improvement will be described as an example.

The image input unit 10 receives images captured by a plurality of (16 in this embodiment) cameras with different viewpoints. The distortion correction unit 20 performs distortion correction on the input camera image. In this embodiment, from the setting information of the camera at the time of shooting, the parameters (distortion parameters) of the distortion correction model represented by the following equations (1) and (2) are calculated for each camera by the method disclosed in Non-Patent Document 2. Pre-calculate and perform distortion correction using the distortion parameters.

Here, (x', y') and (x, y) are coordinates on each camera image before distortion correction and after distortion correction (hereinafter sometimes referred to as image coordinates), and k _i , P _i are distortion correction amounts for radial distortion and circumferential distortion, respectively.

The associating unit 30 is given a calibration coordinate model (soccer field model in this embodiment) whose three-dimensional coordinates are known, and converts the distortion-corrected camera image into three-dimensional spatial coordinates (X , Y, Z) and the camera image coordinates (x, y).

In this embodiment, as an example is shown in FIG. 3, attention is paid to at least four points on the field line whose three-dimensional spatial coordinates are known from the standards of soccer courts, etc., and the three-dimensional spatial coordinates (Xi, Yi, Zi) is associated with the image coordinates (xi, yi) of the corresponding point captured by the camera.

Such correspondence is performed by displaying each image frame input from all 16 cameras on a computer, and using a pointing device that controls the computer to point each point of interest on the field line and each corresponding point on the image. , or may be automatically estimated by the conventional method disclosed in Non-Patent Document 3.

Furthermore, when photographing an object at a short distance, such as an indoor sport, a known pattern such as an AR marker may be used as a coordinate model for calibration, and the correlation may be automatically performed by pattern matching or the like. In such association, whether manually or automatically, it is unavoidable that the image coordinates of each corresponding point contain errors due to deviations in pointing operations and problems with estimation accuracy.

Camera determination section 40 includes reference camera determination section 41 and selected camera determination section 42 . The reference camera determining unit 41 determines one camera that serves as a reference for relative position calculation of each camera as a reference camera. The selected camera determination unit 42 determines a plurality of cameras to be used when the image coordinate correction unit 60, which will be described later, relatively corrects the image coordinates of each camera.

The reference camera determination unit 41 can determine any one of the 16 cameras as the reference camera. A selected camera determination unit 42 projects the points of each camera image corresponding to the same point in the three-dimensional space onto the three-dimensional space coordinates, and determines a plurality of selected cameras based on the degree of convergence with the other cameras.

In the present embodiment, the method disclosed in Non-Patent Document 2 or the like is calculated from the combination of the three-dimensional space coordinates of each point of interest on the soccer court and the image coordinates of each corresponding point on the camera image, which are associated by the association unit 30. Then, the estimated homography matrix is used to project the points from each camera onto the soccer court plane.

Here, the homography matrix is the projective transformation matrix of points on two different planes, and the point (X _i , Y _i , 1) on the soccer court plane is transformed into the point (x _i , y _i , The determinant projected onto 1) is expressed by the following equation (3).

In this embodiment, a threshold is set according to the Mahalanobis distance of the points projected from all the cameras, and the threshold exceeds the threshold in order to exclude cameras whose projected points are greatly deviated from the other cameras and to determine the remaining cameras as the selected cameras. I'm trying to exclude cameras.

In addition, the method of determining the selected camera is not limited to the above method. Compare the quality with all camera 3D models created based on the image, and when the quality of some camera 3D models is better than the quality of all camera 3D models, select cameras other than the excluded camera. can be

Alternatively, the same selection may be repeated for only the cameras remaining in the above selection, and only the cameras finally remaining under predetermined conditions may be determined as the selected cameras. The quality of the 3D model can be automatically evaluated by the degree of matching between the silhouette of the produced model projected onto the camera position and the silhouette of the photographed subject captured by the camera.

As for the reference camera, as with the selected camera, by projecting the corresponding point of each camera image corresponding to the same point in the 3D space into the 3D space and selecting the one that minimizes the reprojection error, the camera Not only the relative position of , but also the absolute position can be made more precise.

As shown in FIG. 4, the relative position calculation unit 50 calculates the projection relationship of corresponding points corresponding to the same point of interest for each selected camera and each other camera (including other selected cameras and reference cameras). to estimate the homography matrix.

In this embodiment, attention is paid to each corresponding point of each camera image corresponding to each attention point of the calibration coordinate model used for the correspondence by the associating unit 30, and based on the relationship of the image coordinates of each corresponding point, A homography matrix is computed.

The image coordinate correction unit 60 relatively corrects the image coordinates of each corresponding point of each camera based on the projection relationship between each selected camera calculated by the relative position calculation unit 50 and each other camera. In this embodiment, first, as shown in FIG. 5, each corresponding point is projected onto each other camera using a homography matrix that indicates the relative positional relationship between each selected camera and each other camera. Therefore, if there are M cameras to be selected, M points will be projected for each corresponding point on each camera of the projection destination.

Next, as shown in FIG. 6, as representative values of a plurality of points projected with respect to the same corresponding point from each selected camera for each projection destination camera, the average value and median value of the image coordinates of each point are calculated, The representative value is tentatively determined as the corrected camera image coordinates for each corresponding point. At this time, if the projection destination is another selected camera, the image coordinates of its own corresponding points are included in the representative value calculation, but if the projection destination is other than the selected camera, the representative value is calculated only from the image coordinates of the projected points. to implement.

A camera parameter calculator 70 calculates the camera parameters of all cameras to determine camera positions. Camera parameters are used to transform a point (X _i , Y _i , Z _i ) on three-dimensional space coordinates into a point (x _i , y _i ) on image coordinates, and are expressed by the following equation (4).

where (c _x , c _y ) is the principal point in the camera image, usually the image center. f indicates the focal length, which is a value to be calculated instead of the zoom value in this embodiment. These are called internal parameters, and are constant values as long as the focal length is fixed. Also, r ₁₁ to r ₃₃ are rotation matrices indicating the amount of camera rotation, t ₁ to t ₃ are translation matrices indicating camera positions, and s is a coefficient used for scaling. The matrices composed of rotation matrices r ₁₁ to r ₃₃ and translation matrices t ₁ to t ₃ are called extrinsic parameters.

In this embodiment, the homography matrix is estimated based on the relationship between the image coordinates of each corresponding point after correction and the corresponding point of interest on the soccer court for the reference camera determined by the camera determination unit 40, and the Using the obtained intrinsic parameters, the extrinsic parameters of the reference camera are calculated.

Next, using the homography matrix of the reference camera and each of the other cameras and the previously prepared intrinsic parameters of each camera, the extrinsic parameters indicating the relative position from the reference camera are determined. Then, based on the extrinsic parameters of the reference camera calculated based on the soccer court in 3D space and the extrinsic parameters indicating the relative position of each camera, the extrinsic parameters of the soccer court are calculated for all cameras, and the camera parameters are output. do.

According to the present invention, the positions of a plurality of cameras that photograph an object from different viewpoints are mutually corrected based on the relative positional relationship with other cameras, and the extrinsic parameters are calculated based on the corrected relative positions. Therefore, as shown in FIG. 7, although each camera parameter can take a parameter slightly deviated from the actual camera parameter, the error in the relative position of each camera is reduced. As a result, since the relative relationship of the external parameters is highly accurate, it is possible to improve the quality of the 3D model produced based on the multiple camera images.

FIG. 8 is a functional block diagram showing the configuration of the main parts of the camera calibration device 1 according to the second embodiment of the present invention. are omitted. This embodiment is characterized by having an error determination unit 80 and repeating the image coordinate correction of the corresponding points of each camera until the reprojection error converges.

The error determination unit 80 includes an error calculation unit 81 and a convergence determination unit 82. By correcting the image coordinates of the corresponding points of each camera in the image coordinate correction unit 60, the relative positions of the cameras other than the reference camera are made highly accurate. It is determined whether or not

As shown in FIG. 9, the error calculation unit 81 calculates the pre-correction and the pre-correction and Each corrected image coordinate is reprojected based on the projection relationship, and the distance between the pixel position of each corresponding target point captured by the reference camera and the reprojection position from each camera is calculated as a reprojection error.

If the reprojection error of the pixel position after correction is smaller than the reprojection error of the pixel position before correction in all cameras other than the reference camera, the convergence determination unit 82 determines that the error correction has converged, and performs the correction. Each subsequent pixel position is determined to be the pixel position of each corresponding point.

On the other hand, if there is a camera with a larger re-projection error of the pixel position after correction, the image coordinates of each corresponding point of that camera are returned to the image coordinates before correction, while the image coordinates after correction are re-projected. For the camera with the smaller projection error, the process returns to the camera determination unit 40 while maintaining the corrected image coordinates.

Then, the determination of the reference camera and the selected camera by the camera determination unit 40, the determination of the relative position by the relative position estimation unit 50, and the image coordinate correction by the image coordinate correction unit 60 are repeated. This repetition is repeated until the number of cameras for which the reprojection error of the image coordinates after the current correction is greater than the reprojection error of the image coordinates after or before the previous correction becomes zero or falls below a predetermined threshold. Alternatively, in all cameras, it is repeated until the absolute value of the difference between the reprojection error of the image coordinates after this correction and the reprojection error of the image coordinates after or before the previous correction falls below a predetermined threshold. Also good.

According to this embodiment, since the relative positions of the other cameras with respect to the reference camera serving as the reference of the external parameters are corrected, the relative relationship between the external parameters of the reference camera and the external parameters of the other cameras is improved. It will be possible to further improve the quality of the 3D model.

Furthermore, according to each of the above embodiments, it is possible to provide high-quality 3D models of subjects in real time even via communication infrastructure, thereby providing diverse entertainment to many people beyond geographical or economic disparities. be able to provide. As a result, the UN-led Sustainable Development Goals (SDGs) include Goal 9 “Build resilient infrastructure and promote inclusive and sustainable industrialization” and Goal 11 “Make cities inclusive, safe and resilient.” and make it sustainable.

In this embodiment, the relative position error is evaluated as a reprojection error, but the present invention is not limited to this. An evaluation may be made based on the quality of the 3D model.

1 Camera calibration device 10 Camera image input unit 20 Distortion correction unit 30 Association unit 40 Camera determination unit 41 Reference camera determination unit 42 Selected camera determination unit 50 Relative position Calculation unit 60 Image coordinate correction unit 70 Camera parameter calculation unit 80 Error determination unit 81 Error calculation unit 82 Convergence determination unit

Claims (15)

In a device for relatively calibrating each camera based on camera images of a calibration coordinate model with known three-dimensional coordinates taken by a plurality of cameras,
means for associating a plurality of points of interest that appear in common on all camera images of the calibration coordinate model with corresponding points on the camera images;
means for determining one of the plurality of cameras as a reference camera;
means for estimating the relative position of each camera based on the projective relationship between the corresponding points of each camera image;
means for projecting the corresponding points of each camera onto another camera based on the projection relationship, and correcting the image coordinates of the corresponding points of the projection destination camera based on the image coordinates of the projected points;
A camera calibration device comprising means for calculating the extrinsic parameters of each camera based on the extrinsic parameters of a reference camera and the corrected corresponding points of each camera. Equipped with means for determining a selected plurality of selected cameras from all cameras,
2. The camera calibration apparatus according to claim 1, wherein said correcting means projects each corresponding point of the selected camera onto other cameras including other selected cameras based on said projection relationship. The means for determining the selected camera determines the selected camera based on the degree of convergence of errors when reprojecting the corresponding points of each camera onto the 3D space of the coordinate model for calibration based on the projection relationship. 3. The camera calibration device according to claim 2. The means for determining the selected camera is the quality of the partial camera 3D model produced based on the images of all the remaining cameras excluding the camera concerned and the all camera 3D model produced based on the images of all cameras for each camera 3. The camera calibration device according to claim 2, wherein when the quality of the 3D models of some cameras is superior to the quality of the 3D models of all cameras, the cameras other than the excluded cameras are determined as the selected cameras. . If the projection destination is another selected camera, the correction means corrects the image coordinates of each corresponding point based on each corresponding point of the other selected camera and the projected point, and if the projection destination is other than the selected camera, 5. The camera calibration device according to claim 2, wherein the image coordinates of each corresponding point are corrected based on the projected point, if any. means for calculating the error of the image coordinates of each corresponding point after the correction for each camera;
and means for determining whether the error has converged,
6. The means for determining convergence of said error repeats selection of said reference camera, estimation of relative position, and correction of image coordinates of corresponding points until said error converges. A camera calibration device as described. the means for calculating the error projects the corresponding points before and after the correction of each camera image onto a reference camera to calculate reprojection errors of the corresponding points before and after the correction;
The means for determining convergence of the error restores each corresponding point of the camera whose reprojection error of the corresponding point after correction is larger than the reprojection error of the corresponding point before correction to the state before correction, selects the reference camera, selects the relative 7. The camera calibration device according to claim 6, wherein estimation of positions and correction of image coordinates of corresponding points are repeated. The means for judging convergence of the error continues to select the reference camera and perform relative 8. The camera calibration device according to claim 7, wherein estimation of positions and correction of image coordinates of corresponding points are repeated. The means for calculating the extrinsic parameter comprises:
calculating a first extrinsic parameter based on the projection relation of the reference camera to the calibrating coordinate model and the extrinsic parameter;
calculating a second extrinsic parameter indicating the relative position of the reference camera and the other camera based on the projective relationship between the reference camera and the other camera and the intrinsic parameters of each camera;
9. The camera calibration apparatus according to any one of claims 1 to 8, wherein extrinsic parameters for calibration coordinate models of each camera are calculated based on said first and second extrinsic parameters. 10. The camera calibration device according to any one of claims 1 to 9, further comprising a distortion corrector that removes a lens distortion component from each camera image. 11. The camera calibration according to any one of claims 1 to 10, wherein the means for determining the reference camera determines a camera having a minimum reprojection error of corresponding points with respect to the calibration coordinate model as the reference camera. Device. 12. The camera calibration device according to any one of claims 1 to 11, wherein the calibration coordinate model is an AR marker. In a method in which a computer relatively calibrates each camera based on camera images of a calibration coordinate model with known three-dimensional coordinates taken by a plurality of cameras,
Correlating a plurality of points of interest commonly seen in all camera images of the calibration coordinate model with corresponding points on the camera images,
determining one of the plurality of cameras as a reference camera;
estimating the relative position of each camera based on the projection relationship between each corresponding point of each camera image,
After projecting the corresponding points of each camera to another camera based on the projection relationship, correcting the image coordinates of the corresponding points of the projection destination camera based on the image coordinates of the projected points,
A camera calibration method, wherein the extrinsic parameters of each camera are calculated based on the extrinsic parameters of a reference camera and the corrected corresponding points of each camera. Decide a part of multiple selected from all cameras as the selected camera,
14. The camera calibration according to claim 13, wherein when correcting the image coordinates of the corresponding points, each corresponding point of the selected camera is projected onto other cameras including other selected cameras based on the projection relationship. application method. In a program that relatively calibrates each camera based on camera images of a calibration coordinate model with known three-dimensional coordinates taken by multiple cameras,
a procedure for associating a plurality of points of interest that appear in common on all camera images of the calibration coordinate model with corresponding points on the camera images;
determining one of the plurality of cameras as a reference camera;
a step of estimating the relative position of each camera based on the projective relationship between the corresponding points of each camera image;
a step of projecting the corresponding points of each camera onto another camera based on the projection relationship, and then correcting the image coordinates of the corresponding points of the projection destination camera based on the image coordinates of the projected points;
A camera calibration program for causing a computer to execute a procedure for calculating the extrinsic parameters of each camera based on the extrinsic parameters of the reference camera and the corrected corresponding points of each camera.

Images