JP2024021218A - Camera calibration device, camera calibration method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To stably calibrate a plurality of cameras regardless of lens characteristics.

SOLUTION: A camera calibration device 10 includes a 3-dimensional posture estimation unit 11, an association processing unit 12, and an optimization unit 13. The 3-dimensional posture estimation unit 11 estimates 3-dimensional posture of a person from 2-dimensional posture showing coordinates of joint positions of the person on each frame image of video generated by a plurality of cameras installed at different locations which photograph a common person. The association processing unit 12 searches a similarity conversion parameter associated with the 3-dimensional posture estimated for each camera, associates the same person in a frame image generated by different cameras with each other, and determines an external parameter of each camera from the similarity conversion parameter. The optimization unit 13 optimizes the 3-dimensional posture and a camera parameter, such that a re-projection error between 2-dimensional coordinates obtained by re-projecting the associated 3-dimensional posture to the frame image of each camera and the 2-dimensional posture becomes minimum.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present disclosure relates to a camera calibration device, a camera calibration method, and a program.

The problem of restoring camera parameters and three-dimensional information of an object from an image sequence containing multiple images of the same object (subject) taken using multiple cameras can be solved using Structure-from-Motion (SfM) or This is called a multi-view geometry problem. Camera parameters include two types: "internal parameters" and "external parameters." Intrinsic parameters are lens-specific parameters such as focal length, lens distortion, and optical center coordinates, and extrinsic parameters are three-dimensional rotation matrices and translation vectors between cameras. The internal parameters can be measured in advance if the lens is fixed. A camera for which only internal parameters or both internal and external parameters are known is called a calibrated camera. On the other hand, a camera in which these are unknown is called an uncalibrated camera.

Zhang's method, which uses a flat chessboard pattern, is widely known as a method for calculating the internal parameters of a single camera. In addition, as a method for calculating the external parameters of a single calibrated camera, the three-dimensional coordinates of an object in space are measured using a laser range finder, and the three-dimensional point is observed as a two-dimensional point in the image using a projection relationship. Methods for solving Perspective-n-point problems are also widely known.

A method called SfM is widely known as a method for simultaneously calibrating internal parameters and external parameters of a plurality of cameras. In SfM, two cameras first photograph a common object and generate a set of two-dimensional points (hereinafter referred to as corresponding points) in which the same three-dimensional points on the object surface are projected onto an image. Next, using a plurality of corresponding points (generally several tens to several thousand), a constraint expression called an epipolar constraint that expresses the geometric relationship between the two cameras and the corresponding points is solved. Then, two cameras are selected from all the cameras, corresponding points are generated and epipolar constraint equations are solved, and the camera parameters of all the cameras and provisional values of the three-dimensional points are obtained.

Finally, optimization is performed so that the reprojection error between the observed corresponding points and the three-dimensional points projected onto the image using all the estimated camera parameters is minimized (hereinafter referred to as bundle adjustment). In order to perform SfM stably, it is desirable that one corresponding point be observed by multiple cameras. However, when cameras are arranged to surround an object, the corresponding points between opposing cameras are hidden by the object itself, and can only be observed with adjacent cameras. When SfM is executed in such an environment, there are problems in that some camera parameters cannot be estimated, and even if all camera parameters are determined, the accuracy is low.

In order to generate corresponding points when viewed from any camera, a method has been proposed that uses joint positions of a human body on an image as corresponding points. For example, joints such as the neck, elbows, and knees can be observed whether the human body is viewed from the front or from the back. Patent Document 1 describes a method of performing SfM by regarding a pedestrian's ankle joint as a corresponding point on the ground, and estimating camera parameters and three-dimensional coordinates (hereinafter referred to as three-dimensional posture) of a human body joint. Patent Document 2 describes a method of performing SfM by regarding joint positions of multiple people's whole bodies as corresponding points, and calibrating a camera including a telephoto lens and a wide-angle lens and estimating a three-dimensional posture.

Furthermore, when using human body joint information, it is not necessarily necessary to use multiple cameras. Non-Patent Document 1 describes a method of calibrating one camera and estimating the height of a pedestrian by regarding the spinal joints of the pedestrian as a perpendicular line to the ground. Non-Patent Document 2 describes a method of estimating the three-dimensional posture of a subject as seen from one camera using a neural network.

International Publication No. 2020/194486 Patent No. 6816058

Nakano, Gaku. "Camera Calibration Using Parallel Line Segments." 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021. Yang, Wei, et al. "3d human pose estimation in the wild by adversarial learning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

However, the above method has the following problems.
First, in the method described in Patent Document 1, the ankle joint is easily hidden by the upper body or other pedestrians, and is not always observed by a camera. Next, since the method described in Patent Document 2 requires the simultaneous use of a telephoto lens and a wide-angle lens, it is not possible to calibrate a plurality of cameras configured with the same lens characteristics, such as a surveillance camera system in a city. The method described in Non-Patent Document 1 calculates the parameters of one camera independently, and therefore cannot calibrate multiple cameras at the same time. In addition, the method described in Non-Patent Document 2 estimates only the three-dimensional coordinates of human body joints, and the camera parameters are implicitly determined in the neural network depending on learning data, so the camera is not calibrated. It is not possible.

As described above, in the conventional method using the joint positions of the human body as corresponding points, it is difficult to stably calibrate multiple cameras regardless of lens characteristics.

An example of an object of the present invention is to provide a camera calibration device, a camera calibration method, and a program that can stably calibrate a plurality of cameras regardless of lens characteristics, in view of the above-mentioned problems.

A camera calibration device according to one aspect of the present disclosure includes a three-dimensional pose estimation section, a correspondence processing section, and an optimization section. The 3D pose estimating unit calculates the coordinates of joint positions of a person on each frame image of a video generated by multiple cameras installed at different locations that capture a common location including one or more common people. Get dimensional pose. Further, the three-dimensional posture estimating unit estimates a three-dimensional posture representing three-dimensional joint positions of the person from the two-dimensional posture. The association processing unit searches for similarity transformation parameters that associate the three-dimensional poses estimated for each camera, associates the same person in frame images generated by different cameras, and extracts external parameters of each camera from the similarity transformation parameters. decide. The optimization unit calculates the 3D orientation, internal parameters of the camera, and Optimize at least one of the external parameters.

In a camera calibration method according to one aspect of the present disclosure, a computer executes the following processing.
Obtain the two-dimensional posture representing the coordinates of the joint positions of the person on each frame image of the video generated by multiple cameras installed at different locations that photograph a common place containing one or more common people, and A process of estimating a 3D posture representing the 3D joint positions of a person from a 2D posture.
A process of searching for a similarity transformation parameter that associates the three-dimensional posture estimated for each camera, associating the same person in frame images generated by different cameras, and determining an extrinsic parameter of each camera from the similarity transformation parameter.
The three-dimensional pose, the internal parameters of the camera, and the external A process that optimizes at least one of the parameters.

A program according to one aspect of the present disclosure causes a computer to execute the following processing.
Obtain the two-dimensional posture representing the coordinates of the joint positions of the person on each frame image of the video generated by multiple cameras installed at different locations that photograph a common place containing one or more common people, and A process of estimating a 3D posture representing the 3D joint positions of a person from a 2D posture.
A process of searching for a similarity transformation parameter that associates the three-dimensional posture estimated for each camera, associating the same person in frame images generated by different cameras, and determining an extrinsic parameter of each camera from the similarity transformation parameter.
The three-dimensional pose, the internal parameters of the camera, and the external A process that optimizes at least one of the parameters.

According to the present disclosure, it is possible to provide a camera calibration device, a camera calibration method, and a program that can stably calibrate a plurality of cameras regardless of lens characteristics.

FIG. 2 is a diagram illustrating an example of the arrangement of a plurality of cameras in a photographing system to which processing by a camera calibration device according to an embodiment is applied. FIG. 1 is a block diagram showing an example of the configuration of a camera calibration device according to an embodiment. It is a flow diagram showing an example of processing operation of the camera calibration device concerning an embodiment. It is a diagram showing an example of the hardware configuration of a camera calibration device.

Hereinafter, embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements are denoted by the same reference numerals, and redundant explanations will be omitted.

<Photography system>
Embodiments relate to a camera calibration device, a camera calibration method, and a camera calibration program that calibrate a plurality of cameras. FIG. 1 is a diagram illustrating an example of the arrangement of a plurality of cameras in a photographing system to which processing by a camera calibration device according to an embodiment is applied. As shown in Fig. 1, N cameras (N is an integer of 2 or more) C1, C2, ..., Cn are installed at different locations, and they share a common location (hereinafter referred to as "capture camera") containing one or more common people. area). In the example shown in FIG. 1, there are three people T1, T2, T3 in the shooting area, and the camera images taken by each camera C1, C2, ..., Cn include the people T1, T2, T3. This will include images of people.

<Camera calibration device>
FIG. 2 is a block diagram showing an example of the configuration of the camera calibration device 10 according to the embodiment. The camera calibration device 10 shown in FIG. 2 uses the camera parameters of each camera C1, C2, . This is a device that estimates the three-dimensional posture of In this embodiment, it is assumed that N cameras are time synchronized. That is, the i-th frame of one camera and the i-th frame of another camera have the same time stamp.

The camera calibration device 10 includes a three-dimensional pose estimation section 11, a correspondence processing section 12, and an optimization section 13. The three-dimensional posture estimating unit 11 estimates two-dimensional postures representing two-dimensional coordinates on the frame images of the respective joint positions of the persons T1, T2, and T3 in each frame image of the camera images generated by the cameras C1, C2, ..., Cn. is acquired and the three-dimensional posture of each person is estimated. Here, the three-dimensional posture is three-dimensional coordinates representing three-dimensional joint positions of a person.

Specifically, the three-dimensional posture estimating unit 11 acquires camera images captured by a plurality of cameras C1, C2, . The three-dimensional posture estimation unit 11 first obtains the two-dimensional coordinates of the joint positions of each person in each frame image of the obtained camera video, and then estimates the three-dimensional posture.

Note that the three-dimensional posture estimated by each camera may be composed of different joint positions depending on the viewing angle of each camera. For example, if the whole body of the person T1 is captured by the camera C1, the three-dimensional coordinates of the joint positions of the whole body are estimated. Furthermore, if only the upper body of the person T2 is captured by the camera C2, the three-dimensional coordinates of the joint positions of the upper body are estimated. Similarly, the two-dimensional posture obtained from each camera image may be configured with different joint positions depending on the viewing angle of each camera.

Using the three-dimensional pose of each camera image estimated by the three-dimensional pose estimation unit 11, the correspondence processing unit 12 first searches for a three-dimensional similarity transformation whose three-dimensional coordinates match, and The three-dimensional postures of the same person inside are correlated. Next, the association processing unit 12 determines the external parameters of each camera from the matched similarity transformation parameters. The external parameters are a three-dimensional rotation matrix and translation vector between cameras. Here, in order to uniquely determine the scale of the extrinsic parameters of each camera, any one camera may be set as the origin of the world coordinate system.

The optimization unit 13 performs bundle adjustment using the three-dimensional poses associated by the association processing unit 12 and the two-dimensional poses acquired by the three-dimensional pose estimating unit 11. In the bundle adjustment, first, the 3D postures associated with each other in the association processing unit 12 are regenerated into the original frame images using the camera parameters corresponding to the frame images (2D images) of each camera from which they are extracted. Two-dimensional coordinates are calculated by projecting. Then, the two-dimensional coordinates of the projected position and the two-dimensional posture acquired by the three-dimensional posture estimating section 11 are compared, and the difference (reprojection error) between the two-dimensional coordinates and the two-dimensional posture is calculated. The optimization unit 13 optimizes the three-dimensional posture and at least one of the internal and external parameters of the camera so that this reprojection error is minimized.

<Processing operation of camera calibration device>
An example of the processing operation of the camera calibration device 10 having the above configuration will be described below with reference to FIG. 3. FIG. 3 is a flow diagram illustrating an example of the processing operation of the camera calibration device according to the embodiment. In the following description, FIGS. 1 and 2 will be referred to as appropriate. Furthermore, in this embodiment, the camera calibration method is implemented by operating the camera calibration device. Therefore, the explanation of the camera calibration method in this embodiment is replaced by the explanation of the processing operation of the camera calibration apparatus below.

First, in a situation as shown in FIG. 1, a common photographing area is photographed by N cameras (N is an integer of 2 or more) C1, C2, . . . , Cn. That is, the cameras C1, C2, . . . , Cn are photographing a place where they share a field of view. Such a photographing system is expected to be installed in a place where people come and go, such as an intersection or inside a station. The photographing system generates camera images by photographing a photographing area with a plurality of cameras.

The internal parameters of these cameras may be calibrated in advance or may be uncalibrated. Although each camera is time-synchronized, it is not possible to correlate the multiple objects shown in each frame image of the camera video. That is, it is unclear whether the p-th person shown in the i-th frame of one camera and the q-th person shown in the i-th frame of another camera are the same person.

In the situation shown in Fig. 1, there are three people T1, T2, T3 in the shooting area, and the camera images taken by each camera C1, C2, ..., Cn include the people T1, T2, T3. Contains images of people. When such N camera images from each of the cameras C1 to Cn are input to the camera calibration device 10, the processing flow shown in FIG. 3 starts.

As shown in FIG. 3, when N camera images are input, the 3D posture estimating unit 11 acquires the 2D posture of the person for each frame of each camera image and estimates the 3D posture (step S11). Here, the three-dimensional posture in each camera image does not need to be defined using a certain common world coordinate system. That is, each camera may be defined by an independent coordinate system. Various existing methods can be used to obtain the two-dimensional and three-dimensional postures. For example, the method described in Non-Patent Document 1 based on geometry may be used, or the method described in Non-Patent Document 2 based on machine learning may be used.

Next, when the 3D postures of each camera image are input, the correlation processing unit 12 searches for a 3D similarity transformation that matches the 3D postures, and searches for a 3D similarity transformation that matches the 3D postures of the same person in different camera images. map. Furthermore, the association processing unit 12 determines the matching similarity transformation parameters by using the external parameters of each camera (step S12).

For example, the association processing unit 12 searches whether the three-dimensional posture of the p-th person reflected in the i-th frame of a certain camera is observed in the i-th frame of another camera, and determines the three-dimensional posture. Perform the mapping.

As the search method, for example, a method obtained by extending the widely known ICP (Iterative Closest Point) method to similarity transformation may be used. Specifically, the search may be performed as follows. First, if the three-dimensional posture of all frames is regarded as a group of three-dimensional points, N three-dimensional point groups can be defined. Next, two point groups are selected from the N points, and the ICP method is applied to estimate a similarity transformation in which the two point groups match. Next, one of the two point groups and one of the remaining N-2 point groups are selected and the ICP method is applied. By performing this operation on all point groups, all point groups can be finally integrated into one. Furthermore, the similarity transformation parameters match the external parameters of each camera.

As another search method, point correspondence matching using a widely known three-dimensional point feature amount may be used. For example, a set of corresponding points is calculated using 3D SIFT, which is a three-dimensional extension of ISS (Intrinsic Shape Signatures) and SIFT (Scale-Invariant Feature Transform), which is known for image features, and then RANSAC (Random Sample Consensus). The similarity transformation parameters are estimated while removing erroneous corresponding points.

In the above example, the ICP method or RANSAC is applied by treating the three-dimensional posture of all frames as one point group, but since time synchronization is achieved, the three-dimensional posture of one frame may be treated as one point group. This reduces the number of three-dimensional coordinates included in one point group, so the ICP method or RANSAC can be executed faster. Note that once the three-dimensional postures of the cameras have been correlated, the two-dimensional postures, which are the observed values on the images of the three-dimensional postures, are also correlated at the same time.

Finally, the optimization unit 13 uses the correlated three-dimensional posture and two-dimensional posture to perform bundle adjustment so that the reprojection error is minimized (step S13). Specifically, the optimization unit 13 uses the three-dimensional posture associated with the camera parameter as a variable, and determines whether the difference between the two-dimensional coordinates obtained by reprojecting the associated three-dimensional posture and the two-dimensional posture is the minimum. Optimize each camera parameter and three-dimensional pose so that Thereby, the three-dimensional posture and camera parameters can be estimated with higher accuracy. Then, the optimization unit 13 outputs the camera parameters and the three-dimensional orientation, and ends the operation.

Note that the output from the association processing unit 12 can be used as the initial value for the external parameter. If the internal parameters are unknown, initial values can be given as follows. That is, since the three-dimensional posture, two-dimensional posture, and external parameters of each camera are known, it is sufficient to solve the least squares method with the internal parameters unknown. Furthermore, as a simpler method, it is also possible to assume that half of the image size is the center of the lens, or to give the focal length based on the specifications of the camera.

As described above, according to the embodiment, multiple cameras can be stably calibrated regardless of lens characteristics. The reason is as follows.
The first reason is that the three-dimensional pose estimation unit 11 independently calculates the three-dimensional pose for each camera image, and the association processing unit 12 associates them. This is because this eliminates the need for a combination of a telephoto lens and a wide-angle lens.
The second reason is that the three-dimensional posture is not limited to the ankle in the association processing unit 12, and all detected joint points can be used for association.
The technique according to the embodiment is suitable for use in three-dimensional shape restoration from images (Structure-from-Motion).

Various modifications that can be understood by those skilled in the art can be applied to the example described above. The embodiment can also be implemented by the forms shown in the following modified examples.

In the above description, it is assumed that the N cameras are time synchronized, but the time synchronization may not be achieved. That is, it may be unknown whether the i-th frame of one camera and the i-th frame of another camera have the same timestamp. In this case, the correspondence processing unit 12 cannot limit the search range by treating the 3D posture of one frame as one point cloud, but if the 3D posture of all frames is regarded as a group of 3D points, Can be matched. If the same movement of a person can be captured between cameras, it is also possible to adjust the time lag in camera images.

The program in this embodiment may be any program that causes the computer to execute steps S11 to S13 shown in FIG. By installing and executing this program on a computer, the camera parameter estimation device and camera parameter estimation method according to the present embodiment can be realized.

FIG. 4 is a diagram showing an example of the hardware configuration of the camera calibration device 10. As shown in FIG. 4, the camera calibration device 10 includes a processor 101 and a memory 102. The processor 101 is, for example, an MPU (Micro Processing Unit) or a CPU (Central Processing Unit). Processor 101 may include multiple processors. Memory 102 is configured by a combination of volatile memory and nonvolatile memory.

Memory 102 may include storage located remotely from processor 101. In this case, processor 101 may access memory 102 via an I/O interface (not shown). The three-dimensional pose estimation unit 11, the association processing unit 12, and the optimization unit 13 of the camera calibration device 10 of the first embodiment may be realized by the processor 101 reading and executing a program stored in the memory 102. good.

The program can be stored and provided to the camera parameter estimator using various types of non-transitory computer readable media. Examples of non-transitory computer-readable media include magnetic recording media (eg, floppy disks, magnetic tape, hard disk drives), magneto-optical recording media (eg, magneto-optical disks). Furthermore, examples of non-transitory computer-readable media include CD-ROM (Read Only Memory), CD-R, and CD-R/W. Further examples of non-transitory computer readable media include semiconductor memory. Semiconductor memories include, for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory). The program may also be provided to the camera parameter estimation device by various types of transitory computer readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer readable medium can supply the program to the camera calibration device 10 via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. The configuration and details of the present invention can be modified in various ways that can be understood by those skilled in the art within the scope of the invention.

10 Camera Calibration Device 11 3D Posture Estimation Unit 12 Correlation Processing Unit 13 Optimization Unit 101 Processor 102 Memory

Claims (7)

Obtain the two-dimensional posture representing the coordinates of the joint positions of the person on each frame image of the video generated by multiple cameras installed at different locations that photograph a common place containing one or more common people, and a 3D posture estimation unit that estimates a 3D posture representing a 3D joint position of a person from a 2D posture;
Matching that searches for a similarity transformation parameter that associates the three-dimensional pose estimated for each camera, matches the same person in frame images generated by different cameras, and determines external parameters of each camera from the similarity transformation parameter. a processing section;
The three-dimensional pose, the internal parameters of the camera, and the external an optimization unit that optimizes at least one of the parameters;
A camera calibration device. Multiple cameras are time-synchronized,
The association processing unit limits the search range of the similarity transformation parameter to three-dimensional postures in which time stamps recorded in each frame image of the video generated by each camera are the same.
The camera calibration device according to claim 1. When the plurality of cameras are not time-synchronized, the association processing unit adjusts the time shift of the video based on the same movement of a common person photographed by the plurality of cameras.
The camera calibration device according to claim 1. The association processing unit sets any one camera among the plurality of cameras as the origin of the world coordinate system, and determines external parameters of the camera.
The camera calibration device according to claim 1. The three-dimensional pose estimated from images generated by multiple cameras is composed of different joint positions depending on the viewing angle of each camera.
The camera calibration device according to claim 1. The computer is
Obtain the two-dimensional posture representing the coordinates of the joint positions of the person on each frame image of the video generated by multiple cameras installed at different locations that photograph a common place containing one or more common people, and A process of estimating a 3D posture representing a 3D joint position of a person from a 2D posture;
A process of searching for a similarity transformation parameter that associates the three-dimensional pose estimated for each camera, associating the same person in frame images generated by different cameras, and determining external parameters of each camera from the similarity transformation parameter. ,
The three-dimensional pose, the internal parameters of the camera, and the external a process of optimizing at least one of the parameters;
Perform the camera calibration method. Obtain the two-dimensional posture representing the coordinates of the joint positions of the person on each frame image of the video generated by multiple cameras installed at different locations that photograph a common place containing one or more common people, and A process of estimating a 3D posture representing a 3D joint position of a person from a 2D posture;
A process of searching for a similarity transformation parameter that associates the three-dimensional pose estimated for each camera, associating the same person in frame images generated by different cameras, and determining external parameters of each camera from the similarity transformation parameter. ,
The three-dimensional pose, the internal parameters of the camera, and the external a process of optimizing at least one of the parameters;
A program that causes a computer to execute.

Images