WO2020105697A1

WO2020105697A1 - Motion capture camera system and calibration method

Info

Publication number: WO2020105697A1
Application number: PCT/JP2019/045552
Authority: WO
Inventors: 中村　仁彦; 洋介池上; 文香山田; 大輝小原; 智行堀川; 亮矢鈴木
Original assignee: 国立大学法人東京大学
Priority date: 2018-11-22
Filing date: 2019-11-21
Publication date: 2020-05-28
Also published as: JP7395189B2; JPWO2020105697A1

Abstract

The present invention simply performs a calibration on a camera with high accuracy even in a large image-capturing space for motion capture. Provided is a calibration method for acquiring camera parameters for three-dimensionally reconfiguring a plurality of camera images, the camera parameters including external parameters indicating the position and orientation of each camera in a global coordinate system, wherein the calibration method comprises: acquiring the position and orientation of each camera mounter in the global coordinate system by using one or a plurality of surveying instruments having an angle measurement function; and acquiring the external parameters by using the position and orientation of each camera mounter in the global coordinate system.

Description

Motion capture camera system and calibration method

The present invention relates to a camera system for motion capture.

Motion capture technology is widely used for various motion measurements. However, in team sports, it is not performed to capture the motion of each player during the game and analyze the motion. For example, moving images of sports such as soccer, rugby, baseball, volleyball, and handball are not acquired, and motion analysis of each player is not performed based on the acquired moving images.

In team sports, in order to capture the movement of each player during a match and analyze the movement, (a) motion capture does not reduce the performance of the player, (b) motion capture is a supporting operation (TV broadcast, team Staff, spectators, etc.), (c) large storage capacity, (d) synchronization between high-speed cameras over a long distance, (e) ability to receive large amounts of data in real time, (f) long time It is necessary to satisfy the conditions such as stable system with operable performance, and (g) camera calibration for accurate 3D reconstruction.

Optical motion capture and motion capture using an inertial sensor are known as conventional typical motion capture techniques. These motion captures have the drawback that it takes time to prepare a measurement for attaching a reflective marker or an inertial sensor to a target body, and further, the marker or the sensor may hinder the smooth movement of the target. Cannot be used to acquire the action of each player during a match. Moreover, these motion captures have a limited measurement space. In optical motion capture using an infrared light source, it is necessary to perform measurement in a space surrounded by a camera that is not affected by external light, and when receiving sensor data wirelessly, wireless signals are limited. It was necessary to measure in space.

Recently, so-called markerless motion capture technology has appeared. For example, the video motion capture technology (Non-Patent Document 1) performs motion capture from the images of a plurality of RGB cameras in a completely unconstrained manner. From the indoor living space to the wide outdoor sports field principle Specifically, this is a technology that enables motion measurement if images can be acquired.

In motion capture using a plurality of cameras, it is necessary to acquire camera parameters for three-dimensionally reconstructing a plurality of camera images. The camera parameters include optical parameters such as lens distortion, internal parameters such as focal length and optical center, and external parameters representing the position / orientation of the camera in the space where the camera is installed. Conventional camera calibration is performed by calibrating all of these parameters by photographing a calibration instrument (checker board, calibration wand, etc.) of known shape and dimensions with one or more cameras. ..

However, in a wide space such as a sports field, in order to perform three-dimensional reconstruction for motion measurement, highly accurate calibration of a camera far away from the target is required, which is different from the conventional motion capture system. A calibration method different from the system configuration and camera calibration at a relatively short distance is required. Of the camera parameters, internal parameters such as optical parameters including lens distortion and focal length can be calibrated in advance if the camera and lens are fixed, but external parameters related to the position and orientation in the space of the camera. The calibration must be done at the site where the camera is installed.

The measurement of a person or robot in a sports field or outdoors has a feature that the measurement area becomes a large space. It is possible in principle to calibrate using a calibration device (checker board, calibration wand, etc.) as in the past, but since the accuracy of calibration depends on the number of pixels of the image sensor image, If the distance to the object to be photographed is several tens of meters, the calibration accuracy will decrease and it is not suitable for three-dimensional reconstruction in motion capture. It is possible to prepare a larger calibration device (for example, a larger checkerboard or a large calibration wand) than before, and use this, but the shooting range of the camera in the horizontal and height directions When covering a wide range, it is not realistic to move a large-sized calibration instrument in different positions and postures so as to cover such a wide range.

U.S. Patent No. 8625086 WO2005 / 059473 Patent No. 3965781

The present invention has an object to easily and accurately calibrate a camera even if the imaging space for motion capture is a large space.

The motion capture camera system according to the present invention is
A plurality of camera units consisting of one or a plurality of cameras,
Multiple camera mounters equipped with each camera unit,
Calibration means for acquiring camera parameters for three-dimensionally reconstructing each camera image;
One or more surveying instruments with angle measuring function,
Equipped with
The camera parameters include external parameters that represent the position and orientation of each camera in the global coordinate system,
The calibration means obtains the external parameter using the position and orientation of each camera mounter in the global coordinate system, and the position and orientation of the camera mounter in the global coordinate system is the one or more surveying points. It is acquired using the machine.
In one aspect, one or more surveying instruments of the one or more surveying instruments have a distance measuring function.
Examples of the surveying instrument having the angle measuring function include a theodolite and a total station.
A total station can be exemplified as the surveying instrument having the angle measuring function and the distance measuring function.

In one aspect, a part or all of the plurality of camera mounters is the surveying instrument.
In one aspect, one camera mounter is composed of total stations and the remaining camera mounters are composed of theodolites.
The surveying instruments that make up the camera mounter are leveled.

In one aspect, a part or all of the plurality of camera mounters includes a plurality of markers (characteristic points) that can be measured by the surveying instrument.
In one aspect, all camera mounters are camera mounters with markers.
In one aspect, the camera mounter has a plane with three or more markers (eg, four survey markers).

In one aspect, the calibration means includes position information (three-dimensional position information) in a global coordinate system of a plurality of feature points located in the imaging space of each camera, and position information of each feature point in each camera image. (Two-dimensional position information) and
Position information of the plurality of feature points in the global coordinate system is acquired using the surveying instrument.
In one aspect, the feature point is a fixed point located in the imaging space of each camera, and includes, for example, an intersection (including a corner) of a line on the court, an intersection, a point on the goal post (a post and a post). The intersection of the crossbar or the bottom of the post) can be used as a feature point.
In one aspect, the three-dimensional position information and the two-dimensional position information of the feature points are used for optimization calculation when acquiring an external parameter.

In one aspect, the calibration means uses the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system (for example, when the position / orientation of the camera with respect to the camera mount is known).
In one aspect, the calibration means includes means for acquiring the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system.
In one aspect, the camera mounter comprises the surveying instrument,
The surveying instrument is used in acquiring the position and orientation of the camera in the camera mounter coordinate system.
In one aspect, the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is known or previously acquired.

In one aspect, the calibration means acquires the extrinsic parameter by an optimization calculation.
In one aspect, the position and orientation of the camera mounter in the global coordinate system, and the three-dimensional position information and the two-dimensional position information of the feature points are used in the optimization calculation.
In one aspect, in the optimization calculation, position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.

In one aspect, the camera parameter includes an optical parameter and an internal parameter, and the calibration unit acquires a part or all of the external parameter, the optical parameter, and the internal parameter by an optimization calculation.
The optical parameters and internal parameters may be obtained by a conventional method (using a calibration instrument).
For example, when the camera mounter is composed of the surveying instrument, the surveying instrument may be used in acquiring the optical parameter and the internal parameter.
In the acquisition of external parameters, some or all of the acquired optical parameters and internal parameters may be used.

The present invention includes a plurality of camera units including one or a plurality of cameras,
Multiple camera mounters equipped with each camera unit,
A motion-capture camera system comprising: a calibration method for acquiring camera parameters for three-dimensionally reconstructing the plurality of camera images,
The camera parameters include external parameters that represent the position and orientation of each camera in the global coordinate system,
Obtaining the position and orientation of each camera mounter in the global coordinate system using one or more surveying instruments equipped with an angle measuring function,
Obtaining the extrinsic parameters using the position and orientation of each camera mounter in the global coordinate system,
including.

In one aspect, a part or all of the plurality of camera mounters is the surveying instrument, and the position and orientation of each camera mounter in the global coordinate system is acquired using surveying data of a plurality of measurement points (feature points). To do.
In one aspect, the plurality of measurement points are two characteristic points A and B that can be surveyed from each surveying instrument (the intervals are known, for example, points A and B on a vertical surveying pole) and each camera. And a feature point on the mounter (for example, the lower end of the pendulum weight).
By measuring the two characteristic points A and B with a surveying instrument, a vector from the origin of each camera mounter coordinate system to the points A and B can be acquired. For example, the horizontal axis having the same horizontal angle as that of the vector is the z axis and the vertical axis is the x axis.
By measuring the characteristic points on the other camera mounters with each surveying instrument, the angle between the mounter coordinate systems can be acquired.
By matching the z-axis and x-axis of the global coordinate system with the z-axis and x-axis of the coordinate system of one mounter and using the distance between point A and point B and the above setting information, each camera is trigonometrically Obtain the position and orientation of the mounter in the global coordinate system.

In one aspect, a part or all of the plurality of camera mounters includes a plurality of markers (feature points) that can be surveyed by the surveying instrument, and camera in a global coordinate system is used by using surveying data of the plurality of markers. Get the position and orientation of the mounter.
In one aspect, the camera mounter has a plane with three or more markers (eg, four survey markers).
In one aspect, the distances between the three or more markers are known, and these markers are provided on a vertical plane, and are measured by a surveying instrument (for example, having an angle measuring function and a distance measuring function). By measuring the marker and setting the global coordinate system with the center of the surveying instrument as the origin, the position and orientation of each camera mounter in the global coordinate system is acquired by using the surveying information and trigonometry.

In one aspect, position information in a global coordinate system of a plurality of feature points located in the imaging space of each camera and position information of each feature point in each camera image are used in the acquisition of the external parameter. ,
Position information of the plurality of feature points in the global coordinate system is acquired using the surveying instrument.
In one aspect, the feature point is a fixed point located in the imaging space of each camera, and includes, for example, an intersection (including a corner) of a line on the court, an intersection, a point on the goal post (a post and a post). The intersection of the crossbar or the bottom of the post) can be used as a feature point.
In one aspect, the three-dimensional position information and the two-dimensional position information of the feature points are used for optimization calculation.

In one aspect, the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is used in the acquisition of the external parameter.
In one aspect, the calibration means includes means for acquiring the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system.
In one aspect, the camera mounter comprises the surveying instrument,
The position and orientation of the camera in the camera mounter coordinate system are acquired using the surveying instrument.
In one aspect, a plurality of feature points whose positions in the camera mounter coordinate system are known are set using the surveying instrument (see FIG. 15),
Obtaining the position information of each feature point in each camera image,
The position and orientation of the camera in the camera mounter coordinate system is acquired using the position information of the feature point in the camera mounter coordinate system and the position information in the camera image.
In one aspect, the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is known or previously acquired.

In one aspect, the extrinsic parameters are obtained by an optimization calculation.
In one aspect, the position and orientation of the camera mounter in the global coordinate system, and the three-dimensional position information and two-dimensional position information of the feature points are used in the optimization calculation.
In one aspect, in the optimization calculation, position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.
In one aspect, the cameras are synchronized, each camera captures images of calibration instruments in different positions and orientations, and the position information of each of the plurality of feature points on the calibration instrument in each camera image is used.

INDUSTRIAL APPLICABILITY The present invention can easily and highly accurately calibrate a camera by using highly accurate survey data measured by a surveying instrument in calibrating the camera even if the imaging space for motion capture is a large space. it can.
Therefore, by shooting sports games that are played in indoor / outdoor sports fields, indoor / outdoor courts, arenas, domes, gymnasiums, etc., the motion of each player during the game is reconstructed in three dimensions to perform motion capture, and motion analysis is performed. It can be performed.

It is a conceptual diagram which shows the aspect which arranged the motion capture camera system which concerns on this embodiment so that a soccer court may be imaged. It is a conceptual diagram which shows the aspect which arranged the motion capture camera system which concerns on this embodiment so that a volleyball court may be imaged. It is a schematic diagram of a camera network concerning this embodiment. It is a figure which shows an example of the camera mounter which concerns on this embodiment, and has shown the theodolite as a camera mounter, and three cameras mounted on the theodolite. It is a figure which shows an example of the camera mounter which concerns on this embodiment, and shows the flat plate as a camera mounter, the four markers provided in the four corners of a flat plate, and the one camera mounted in the width direction center part lower end of a flat plate. ing. It is a figure which shows the total station illustrated as a surveying instrument which concerns on this embodiment. It is a figure which shows several cameras arrange | positioned around the volleyball court, those FOVs, and the wiring of a cable. It shows the trajectory of the target ball in the space above the volleyball court and is used to acquire camera image data for bundle adjustment. It is a figure which shows 18 feature points on a volleyball court. It is a figure which shows the characteristic point on the court measured, and the position of the camera mounter. It is a figure which shows the relationship between a camera coordinate system and the image surface on OpenCV. It is a figure which shows the triangulation using theodolite. It is a figure which shows the triangular side amount using theodolite. It is a figure explaining the measurement of the position and the posture of the camera mounter with a marker using a total station. It is a figure explaining the calibration of a relative external parameter. It is a general view of a motion capture system. It is a flow chart which shows a process of motion analysis using motion capture.

[A] Motion capture / camera system [A-1] Overall configuration The motion capture / camera system according to the present embodiment captures a team sport performed in a large space such as a sports field or an arena with a plurality of cameras. The motion capture of each player is performed by using the captured image acquired by each camera. The motion capture used in this embodiment is a video motion capture technique (Non-Patent Document 1), and its specific content will be described later.

The motion capture camera system includes a plurality of camera units including one or a plurality of cameras, and a plurality of camera mounters in which the camera units are mounted. For example, FIG. 1 is a conceptual diagram of a motion capture camera system that captures a soccer match, in which four camera units including four cameras are arranged so as to surround a soccer court. The distance from each camera to the target athlete is several tens of meters.

The motion capture camera system shown in FIG. 1 includes four camera units CU ₁ , CU ₂ , CU ₃ , and CU ₄ , and the camera unit CU ₁ has four cameras C ₁₁ , C ₁₂ , C ₁₃ , C ₁₄ , and cameras. The unit CU ₂ has four cameras C ₂₁ , C ₂₂ , C ₂₃ , C ₂₄ , the camera unit CU ₃ has four cameras C ₃₁ , C ₃₂ , C ₃₃ , C ₃₄ , and the camera unit CU ₄ has four cameras C ₄₁ , It is equipped with C ₄₂ , C ₄₃ , and C ₄₄ . The camera unit CU ₁ is mounted on the camera mounter CM ₁ , the camera unit CU ₂ is mounted on the camera mounter CM ₂ , the camera unit CU ₃ is mounted on the camera mounter CM ₃ , and the camera unit CU ₄ is mounted on the camera unit CU _4. It is mounted on the camera mounter CM ₄ . In FIG. 1, four camera units are shown for convenience, but in reality, more camera units could be arranged. Note that the number of cameras installed in one camera unit is not limited.

FIG. 2 is a conceptual diagram of a motion capture camera system for shooting a volleyball game, in which four camera units having one camera are arranged so as to surround a volleyball court. As shown in FIG. 2, the motion capture camera system includes four camera units CU ₁ , CU ₂ , CU ₃ , and CU ₄ , and the camera unit CU ₁ is one camera C ₁₁ and the camera unit CU ₂ is one. The camera C ₂₁ and the camera unit CU ₃ are equipped with one camera C ₃₁ , and the camera unit CU ₄ is equipped with one camera C ₄₁ . The camera unit CU ₁ is mounted on the camera mounter CM ₁ , the camera unit CU ₂ is mounted on the camera mounter CM ₂ , the camera unit CU ₃ is mounted on the camera mounter CM ₃ , and the camera unit CU ₄ is mounted on the camera unit CU _4. It is mounted on the camera mounter CM ₄ . Note that the number of cameras installed in one camera unit is not limited.

In the experimental example described later, as shown in FIG. 7, ten camera units each including one camera are arranged so as to surround the volleyball court. The volleyball court has a rectangular size of 18 m × 9 m, and a free zone having a width of 3 m is provided around the rectangle. Each camera is arranged so that it can photograph a space having a height of 3 m in an area including at least the free zone. Each camera is arranged by mounting it on a tripod leveled at a predetermined position or by fixing it to a handrail. Each camera is calibrated in advance with the position and orientation fixed, and the position and orientation are maintained during shooting.

The camera network of the motion capture camera system shown in FIG. 7 is shown in FIG. It should be noted that although 12 cameras can be connected to the camera network, 10 cameras are used in the embodiment. The camera network according to this embodiment includes a plurality of cameras, a transmission box, a camera hub, one or a plurality of computers, a power supply, and a waveform generator that generates a synchronization signal. Each camera is electrically connected to the transmission box, the transmission box and the camera hub are electrically connected by hybrid copper / fiber cable, and the camera hub and the computer are electrically connected. The cameras are synchronized.

Each camera acquires an image with a resolution of 1920x1200 at 120Hz. Each camera has a USB3.0 interface, and USB3 to Fiber Optic Extender is used for image signal transmission. The transmitter box is equipped with a USB3toFiberOpticExtender Transmitter, the camera hub is equipped with a USB3toFiberOpticExtender Receiver, and image signals are sent to the computer via the USB3toFiberOpticExtender. .. The computer includes a data receiving unit, a processing unit, and a storage unit. The image data received by the data receiving unit is appropriately compressed by the processing unit and stored in the storage unit.

Each camera is mounted on a camera mounter, and the position and orientation of each camera is fixed with respect to the camera mounter. The internal parameters, external parameters, and distortion coefficient of the camera are obtained by the calibration, and these camera parameters are stored in the storage unit of the computer. The camera parameters are used for three-dimensional reconstruction of the motion of the object using each camera image.

The motion capture camera system according to the present embodiment further includes a calibration means and one or a plurality of surveying instruments having an angle measuring function. The calibration means is composed of a computer (including an input unit, a calculation unit, a storage unit, an output unit, etc.). The calibration means acquires camera parameters for three-dimensionally reconstructing each camera image. The calibration according to the present embodiment is characterized in that the survey data of the survey instrument is used in the calibration calculation. More specifically, the calibration means acquires external parameters by using the position and orientation of each camera mounter in the global coordinate system, and the position and orientation of the camera mounter in the global coordinate system is one. Alternatively, it is acquired using a plurality of surveying instruments. In one aspect, the position / orientation of the camera is fixed and the surveying instrument is not used during the motion capture. Typically, the calibration means acquires the camera parameters by optimization calculation, and the calibration means includes an optimization calculation unit that executes optimization calculation.

Theodolite and total station are known as high-precision surveying technology in a large outdoor space. Theodolite is an angle measuring instrument with a typical angle measuring function. It has a movable part (with a telescope) that can rotate horizontally and vertically, and can measure the horizontal angle and altitude angle with high accuracy. .. A total station is a device that has a theodolite equipped with an angle measuring function and a distance measuring function (laser measurement). The total station used in the experiment described later has a pan angle (horizontal angle) and tilt angle (vertical angle) accuracy of 20 seconds, and a laser range finder accuracy of ± 1 mm at 28.8 m.

INDUSTRIAL APPLICABILITY The present invention relates to a motion capture camera system used for measurement in a large space, such as a surveying instrument, specifically, a theodolite or a total station, and, if necessary, a surveying instrument (reflection prism or reflection marker used for these measurements. It is possible to easily and accurately calibrate camera parameters even in a large space by configuring a camera system including

[A-2] When the camera mounter is composed of a surveying instrument In the first embodiment of the camera mounter, the camera mounter of each camera unit consisting of one or a plurality of cameras is comprised of a theodolite or a total station surveying instrument. It In the embodiment shown in FIG. 1, the camera mounters CM ₁ , CM ₂ , CM ₃ and CM ₄ are composed of surveying instruments. Each camera unit is mounted on the movable part of the surveying instrument, and can rotate in the horizontal and vertical directions integrally with the movable part when measuring the angle. Specifically, a camera stand equipped with one or a plurality of cameras is fixedly installed on a theodolite or a total station. The theodolite and the total station are mounted, for example, on a leveled tripod, and the camera unit is mounted thereon (see FIG. 4).

By configuring the camera mounter with a surveying instrument, the attitude / position of the camera mounter in the global coordinate system can be acquired by using the surveying technology. Each camera unit, a camera stand, and a plurality of cameras mounted on the camera stand are mounted on the surveying instrument. For example, four cameras are radially arranged at equal angles so that the optical axes of the cameras are located in the same plane and the optical axes of all the cameras intersect at one point. In another example, each camera is located on the side of a regular polygonal pyramid and the optical axes of all cameras intersect at one point on the axis of the regular polygonal pyramid. The number of cameras in each camera unit and the arrangement mode are not limited. When a plurality of camera mounters are configured by surveying instruments, in one aspect, all the camera mounters are configured by surveying instruments. In another aspect, a portion of the plurality of camera mounters comprises a surveying instrument. When two or more camera mounters are surveying instruments, at least one camera mounter may be a total station.

Here, a theodolite equipped with a camera is disclosed in, for example, Patent Documents 1-3 and Non-Patent Document 2, and is also known as a video theodolite. The camera in the conventional video theodolite is in a rotatable state during normal use, and the camera also rotates together with the rotation of the angle measuring unit, and uses the camera image information together with the angle measuring information. On the other hand, in the camera unit using the surveying instrument in the present embodiment as a camera mounter, the angle measuring function of the surveying instrument is used only in the calibration of the camera, and the position of the camera is fixed during use (during motion capture). However, the theodolite angle measurement function is not used. In the present invention, use of the angle measuring function of the surveying instrument at the time of motion capture is not excluded, and the angle measuring function may be used in the dynamic calibration described later.

[A-3] When the camera mounter is provided with a marker that can be measured by a surveying instrument In the second embodiment of the camera mounter, the camera mounter is provided with a plurality of markers (characteristic points), and the marker is used by the surveying instrument. Can be surveyed. More specifically, the camera mounter includes a plate (for example, a square of 30 cm × 30 cm to 40 cm × 40 cm) having a flat surface of a predetermined shape and size, and the flat surface portion of the plate is a reflection that serves as a target for a theodolite or a total station. Three or more (for example, four) markers are provided. In the embodiment shown in FIG. 2, the camera mounters CM ₁ , CM ₂ , CM ₃ and CM ₄ are composed of a plate having a flat surface having a plurality of reflection markers.

A camera whose relative position is fixed is mounted on the flat plate. For example, each camera unit includes one camera, and the camera is fixed at the center of the width direction in the lower portion of the square flat plate. A flat plate that has three or more reflection markers and is fixed in its relative position to the camera is attached to the camera base (head) together with the camera, and the camera base is fixed to a tripod or the like. A plurality of cameras may be mounted on the camera mounter including a plate having a plurality of markers.

By using the surveying technology, the posture / position of the camera mounter (flat plate) in the global coordinate system can be acquired. When the camera mounter has a marker that can be measured by a surveying instrument, it is advantageous to use a surveying instrument (typically a total station) having an angle measuring function and a distance measuring function for the measurement of the marker. ..

It will be understood by those skilled in the art that other instruments, surveying methods and calculation methods used in the surveying technique can be used in using the surveying instrument. Examples of surveying instruments other than theodolite and total station include a tripod, aluminum stuff, aluminum rod, pole, level, auto level, reflecting prism, and prism pole.

[B] Camera Calibration Camera calibration for three-dimensional reconstruction in large space motion capture will be described. The three methods of the first method, the second method, and the third method will be described below. The first method is a known method, but the third method is a combination of the first method and the second method. Therefore, the first method will be referred to together with the description of the camera parameters.

[B-1] General bundle adjustment using a feature point group whose position is unknown (first method)
Calibration of each camera of the camera system used in this embodiment will be described. Here, the bundle adjustment means that the bundle of rays from the three-dimensional feature point to the camera is optimally adjusted for both the position and orientation of the three-dimensional feature point and the camera. .. This optimization is generally solved as a non-linear optimization problem that minimizes reprojection error of feature points.

The perspective projection transformation matrix based on the pinhole camera model, which is the simplest camera model, is used for reprojection of the feature points. This perspective projection transformation matrix can be expressed as follows for each camera.

Where i is the camera number, matrix X = (x ₁ ... x _n ) is the position of the n feature points in the global coordinate system x _i = (x _i y _i z _j 1) ^T , matrix ⁱ Y = ( ^I y ₁ ... ⁱ y _n ) is the position of the n feature points in the image plane in pixel units ⁱ y _j = ( ⁱ u _j ⁱ v _j 1) ^T , matrix ⁱ S = diag { ⁱ s ₁ , ..., ⁱ s _n } refers to the scale ⁱ s _j = (001) ⁱ AiBx _j for projection onto the image plane.

The matrices ⁱ A and ⁱ B that represent the internal and external parameters of the camera can be expressed as follows.

Where ⁱ f _x and ⁱ f _y are the focal lengths in pixel units of the x and y axes on the image plane, and ⁱ c _x and ⁱ c _y are the principal points on the image plane (the intersection of the image plane and the optical axis). position of) the matrix ⁱ R is the attitude of the global coordinate system in the camera coordinate system, ⁱ t represents the position of the origin of the global coordinate system in the camera coordinate system. FIG. 11 shows a projection from a three-dimensional space onto an image plane along coordinate axes commonly used in computer vision. According to this, the internal parameter matrix A of the camera is defined by the following formula.

Here, the matrix ⁱ A, and ^{_{^{_{i f x = i f y =}}}} i f, i c x, the ⁱ c _y that the center of the image plane, it is possible to reduce the number of optimization variables. This is because it can be assumed that the aspect ratio of the size of the light receiving element is 1: 1 and that the center of the image plane intersects the optical axis. The matrix ⁱ R can be treated as ⁱ R ( ⁱ θ _α , ⁱ θ _β , ⁱ θ _γ ) as three variables when expressed in Euler angles. In this specification, ZYX Euler angles ⁱ R as representation of the rotation matrix ^{_{^{_{(i θ α, i θ β}}}} , i θ γ) = R Z (i θ α) R Y (i θ β) R X (i θ γ ) Is adopted.

The nonlinear least squares problem that minimizes the reprojection error of feature points in multiple cameras can be expressed as follows.

Here, the focal lengths ⁱ f _x , ⁱ f _y , which are optimization variables, and the camera position ⁱ t have a relationship such that if one is fixed, the other is not determined. Therefore, either one must be fixed, or the upper and lower limits must be set narrower for each.

The data obtained for multi-camera calibration is the pixel position in the image plane of the tie points (a group of feature points whose positions in the global coordinate system are unknown, such as the locus of the center position of the color ball) for multiple cameras. _Suppose it was only ⁱ Y _tie . At this time, the position X _tie of the feature point group in the global coordinate system is determined depending on the positions and orientations of the plurality of cameras. This bundle adjustment can be expressed as:

Here, the above system of nonlinear equations can be solved using the Levenberg-Marquardt method.

[B-2] Bundle adjustment based on direct measurement by surveying instrument (second method)
The data obtained for calibration is the position ⁱ X in the global coordinate system of the control point (the feature point group of the feature points whose position in the global coordinate system is known, such as the feature point on the court measured by the survey instrument). _{It is} assumed that there are three, _control , pixel position ⁱ Y _control in the image plane, and camera mounter position / posture ⁱ B _m . The values of the control point and the position / orientation ⁱ B _m of the mounter of the camera were obtained by direct measurement using a surveying instrument. At this time, since there is no constraint between the cameras, optimization can be performed for each camera. This bundle adjustment can be expressed as:

Here, the above nonlinear equation can be solved using the Levenberg-Marquardt method or the Trust-Region-Reflective method. In the Trust-Region-Reflective method, for optimization variables,

You can set a range constraint such as. Based on the obtained camera mounter position / orientation ⁱ B _m , by setting the initial values of the optimization variables and narrow range constraints, the above-mentioned focal lengths ⁱ f _x , ⁱ f _y and camera position ⁱ t You can add surveyingly relevant constraints to the relationship.

The distortion coefficient of the camera is an internal parameter using the OpenCV function cv :: calibrateCamera by using multiple images taken at various distances and angles for a checkerboard with known distances between grid points. Ask with. In this embodiment, the internal parameters are also optimized as variables for bundle adjustment. As an optimization solver, optimization was performed using the function lsqnonlin that solves the nonlinear least squares problem in the Optimization Toolbox of MATLAB.

[B-3] Bundle adjustment that uses a feature point group whose position is unknown and a survey value (third method)
The data obtained for calibration is the pixel position ⁱ Y _tie in the image plane of the _tie point, the position ⁱ X _control of the control point in the global coordinate system and the pixel position ⁱ Y _control in the image plane, and the camera It is assumed that there are four positions / postures ⁱ B _m of the mounter. This bundle adjustment can be expressed as:

Here, n _tie and n _control refer to the number of tie points and control points, and k _tie and k _control refer to weights for two types of feature point groups where k _tie > 0 and k _control > 0. In this embodiment, k _tie = k _control = 0.5. The above non-linear equation can be solved using the Levenberg-Marquardt method.

[B-4] Experimental Example For motion capture in a volleyball game, the following three types of multi-camera calibration were performed.
(A) General bundle adjustment using a feature point group whose position is unknown (first method),
(B) Bundle adjustment based on direct measurement with a surveying instrument (second method), and
(C) Bundle adjustment using a combination of feature points whose position is unknown and direct measurement by a surveying instrument (third method).
The arrangement of a plurality of cameras used for these measurements is shown in FIG. In the experiment, 6 cameras out of 10 cameras were calibrated.

(A) Using the image position information of the loci of color balls on multiple synchronized cameras, the bundle adjustment of multiple cameras was performed to determine the position and orientation of the cameras. In the experiment, the yellow color ball was moved as shown in FIG. 8 within the volleyball court, which is the measurement range, and its trajectory was photographed by a plurality of synchronized cameras. The position and radius of the center of the color ball on the image plane were calculated using OpenCV, and processing to exclude outliers was performed on MATLAB. The center position of the color ball was able to acquire 2223 frames excluding outliers. This data is used as a tie point. Calibration is performed by solving the equation (4).

(B) The characteristic points on the court and the position / orientation of the camera mounter were obtained using a surveying instrument, and the bundle position adjustment was performed for each camera to obtain the camera position / orientation. The camera mounter with a survey marker and the survey instrument used in the experiment are shown in FIGS. 5 and 6, respectively. The surveying instrument is a total station (angle surveyer / laser rangefinder), the camera mounter is a 40 cm × 40 cm flat plate, and surveying markers are provided at the four corners. FIG. 9 shows 18 on-court feature points (white line intersections). The position information of these feature points in the global coordinate system is acquired using a surveying instrument.

In the global coordinate system, the feature point number 0 on the court is the origin, the X axis is the direction from the origin to the feature point number 9 on the court is positive, and the Z axis is the vertical upward direction determined by the leveling of the total station. I am trying. According to this definition, the axis of the coordinate system obtained as a result of optimization is rotated, and the three-dimensional position obtained by surveying is also translated / rotated. As a result, the on-court feature point whose position in the global coordinate system is known is used as the control point. In addition, the position / orientation of the camera mounter with surveying markers in the global coordinate system is converted into the local coordinate system of the camera mounter, which is used as the initial value of the camera external parameter matrix. Calibration is performed by solving the equation (5). Further, although optimization is performed for each camera based on direct measurement by the surveying instrument, bundle adjustment of a plurality of cameras may be performed based on direct measurement by the surveying instrument. This is the evaluation function

Can be treated as a minimization problem.

(C) The camera position / orientation was obtained by solving the equation (6) by performing bundle adjustment using both the trajectory of the color ball (a group of characteristic points whose position is unknown) and direct measurement by a surveying instrument.

In this experiment, the camera calibration data and the video motion capture system (see Non-Patent Document 1) were used to reconstruct the whole-body motions of 12 players during a volleyball game. In this experiment, we photographed a volleyball game in the arena and captured the movements of volleyball players during the game. The motion capture camera system according to the present embodiment can be applied to any motion measurement without being limited to the type and location of sports. Examples of target sports include soccer, futsal, rugby, baseball, volleyball, handball, tennis, and gymnastics. Examples of the shooting location include an outdoor sports field, an indoor sports field, an outdoor court, an indoor court, an arena, a dome, and a gymnasium.

[B-5] Dynamic Calibration The calibration according to the present embodiment should be applied to video motion capture from a plurality of camera images for broadcasting / recording in which a cameraman manually manipulates the focal length, pan / tilt, and the like. You can In this case, the optical parameter (lens distortion), the internal parameter (focal length), and the external parameter all change manually. These parameters are calculated by using fixed feature points (for example, points on the line or points on the goal) on elements that form a sports field or court, and feature points of unknown positions (for example, balls in ball games). All parameters of time are dynamically obtained by optimization calculation.

First, before motion capture, position information in a global coordinate system of a plurality of immovable feature points is acquired and stored by a surveying instrument having a angle measuring function (which may have a distance measuring function). Fixed feature points are points on elements that form a sports field or court, such as intersections (including corners) of lines on the field or court, intersections, and points on goal posts (intersection between post and crossbar). The lower end of a part or a post) can be illustrated.

At the time of motion capture, using position information in each camera image of the plurality of fixed feature points and position information in each camera image of one or more feature points whose position information in the global coordinate system is unknown, The camera parameters are obtained by the optimization calculation. A ball in a ball game can be exemplified as the feature point whose position information in the global coordinate system is unknown. Moreover, you may use the mark of the specific position on the uniform which the player is wearing, or the specific point of the shoes as the feature point whose position is unknown.

In the optimization calculation, the variation width of some parameters may be set small or the change speed of some parameters may be set small by using an empirical rule. Further, the camera mounter may be configured by a surveying instrument having an angle measuring function, and the posture information of the camera mounter acquired by using the surveying instrument may be used for the optimization calculation. In the optimization calculation, the time series information of the measurement value (angle) of the surveying instrument is acquired and stored, or the optimization calculator receives the measurement value of the surveying instrument in real time and The optimization calculation may be performed with.

[C] Acquisition of position / orientation of camera mounter [C-1] Direct measurement by surveying instrument
1) Definition of relative position / orientation of measurement points on a survey pole by angle surveying:
According to FIG. 12, the relative position / orientation of the measurement points A and B on the similarly surveyed pole and the angle surveyer are obtained by the leveled i-th angle surveyer as follows. Here, points O, A, and B are measurement points (high and low) on the pole, and point O is the origin of the global coordinate system (the intersection of the pole and the ground). The point T _i is the origin of the local coordinate system of the i-th angle measuring instrument, the point Hi is the foot of the perpendicular from T _i on the pole, and the axes X _Ti , Y _Ti , and X _Ti are of the local coordinate system of the angle measuring instrument. Refers to each axis. Also, the angle θ ⁱ _A is the angle

The angle θ ⁱ _B is the angle

And Each line segment is as follows.

Therefore, the vector

Is as follows.

If there are actual measured values for the line segments T _i A and T _i B using the laser rangefinder, consider the values and optimize the vector.

Ask.

[C-2] and an angle survey meter T ₀ as the position and orientation based angle survey meter in the global coordinate system, the orientation of the local coordinate system of the angle survey meter T ₀ of the global coordinate system posture, the origin O on the survey pole Is the origin of the global coordinate system. At this time, the vector obtained in the previous section

Using, the homogeneous transformation matrix ^O ₀ H of the local coordinate system of the angle surveying instrument T _{0 with} respect to the global coordinate system can be defined as follows according to FIG.

The positions and orientations of other angle surveyers in the global coordinate system are also determined as follows.

Where angle θ ⁰ _Ti is the angle

The angle θ ⁱ _T0 is the angle

And

When the theodolite is the mounter of the camera, the external parameter matrix ⁱ B _m representing the position / orientation can be defined as follows.

[C-3] Position / Attitude of Camera Mounter with Survey Marker In FIG. 14, the origin of the local coordinate system of the camera mounter with survey marker is F _i . The position ^O p _Fi and the posture ^O _Fi R of this coordinate system with respect to the global coordinate system are obtained by optimization as follows.

Here, ^O p _iPj is the point (j = 1, ..., 4) of each survey marker.
w _frill is the design value of the distance between markers, and ⁱ L _slopej , ⁱ L _horizontalj , and ⁱ L _verticalj are the oblique distance, horizontal distance, and height difference to the marker measured by the laser range finder. Further, the angles ⁱ θ _horizontalj and ⁱ θ _verticalj indicate the direction angle and the altitude angle to the marker measured by the angle range finder. When the camera mount with surveying markers is used as the camera mount, the external parameter ⁱ B _m is defined as follows, as in the previous section.

[C-4] Definition and optimization of relative external parameters:
Only the external parameter matrix ⁱ B _{m of} the camera mounter can be directly measured by the surveying instrument with respect to the external parameter matrix ⁱ B of the camera. They are relative external parameter matrices

Can be converted as follows.

When the relative outer parameter matrix is optimized by the second method, the following problem occurs.

If it is possible to perform not only on-site measurement but also preliminary calibration for obtaining only the relative extrinsic parameter matrix, it is preferable to obtain the focal length of the camera and the relative extrinsic parameter matrix between the camera and the mounter by optimization. If the focal length is obtained in advance and fixed, a proper camera position can be obtained in the field. At this time, if an angle measuring instrument is used as the camera mounter, the focal length of the camera can be obtained more accurately. The procedure is specifically described. First, for a leveled surveying pole, a level coordinate system is used to define the global coordinate system as described above. Next, the surveying pole is photographed by moving it at regular intervals (1 ° step, ± 10 ° range). This allows control points to be defined on the survey pole (see Figure 15). You can get the position of the control point guaranteed to the accuracy of theodolite.

[D] Camera calibration in camera system using surveying instrument for camera mounter Calibration in the embodiment using surveying instrument for camera mounter will be described in detail. The camera system includes a camera unit including a plurality of (for example, 4 or 2) cameras, and each camera unit is mounted on a surveying instrument (theodolite or total station). Each camera unit is mounted on the movable part of the surveying instrument, and can rotate in the horizontal and vertical directions integrally with the movable part when measuring the angle. In one aspect, one camera unit is mounted on the total station and another camera unit is mounted on the theodolite. Of course, a plurality of camera units may be mounted in a plurality of total stations. In the description of this section, i indicates the theodolite serial number, and j indicates the j-th camera mounted on the i-th theodolite. Note that in section [C] above, i indicates the camera serial number, and the camera mounter (including the theodolite) is assigned the camera serial number.

[D-1] Theodolite coordinate system in the world coordinate system The acquisition of the homogeneous conversion matrix from the world coordinate system to the theodolite coordinate system will be described. First, the theodolite and total station are mounted on a tripod. Set each tripod so that the field of view of each camera includes the measurement range. Level the theodolite and total station using a spirit level. Set up a survey pole at the center of the measurement range. The surveying pole is a well-known pole generally used in surveying, and for example, red and white stripes are formed with a width of 10 cm. The upper measurement point on the pole is A, and the lower measurement point is B.

At each theodolite or each total station, aim at measuring point A on the pole and read Yaw _A and Pitch _A. Lock Yaw, aim at measurement point B, and read Pitch _B. Set Yaw _A as the initial Yaw angle for theodolite and total station. That is, Yaw _A is used to reset Yaw.

In each theodolite or total station, set the coordinate system as follows. With the center of each theodolite or each total station as the origin, the z-axis is Yaw = Yaw _A and Pitch = 0 (horizontal), and the x-axis is Pitch = 90 ° (vertical).

The coordinate system of the total station is the coordinate system {0}. Counterclockwise with the theodolite coordinate system as the coordinate system {1}, {2}, ..., {M}. Here, M is the theodolite number. M = 3.

At the measurement point B of the pole, global coordinates {a} are set. The z-axis and x-axis are made to coincide with the z-axis and x-axis of the total station coordinate system {0}.

The Yaw angle between the camera mounters (eg, one total station and multiple theodolites) is measured using the theodolites or each total station. For example, the Yaw angle from the coordinate system {0} to the coordinate system {1} is represented by α ₀₁ . In this measurement, the tip of the pendulum weight of the theodolite or total station that is the target is the target (characteristic point). The angle α _ij (i = 0, ..., M; J = 0, ... M; i ≠ j) is acquired by measuring the Yaw angle between the plurality of camera mounters.

Homogeneous transformation matrix from world coordinate system to theodolite coordinate system by trigonometry using the fact that the distance between measurement point A and measurement point B is known

To get

[D-2] Camera internal parameters Acquisition of camera internal parameters will be described. Each camera unit is equipped with four fixed cameras on a horizontal plane that can rotate integrally with the Yaw rotation of the theodolite or total station. The camera is fan-shaped with a sector angle of 22.5 °.

, The axis is symmetric with respect to the zx plane. Each axis

Rotate around 9.65 °.

As for the camera, each total station and theodolite is set to i = 0, ..., M, and each camera on each total station and theodolite is set to j = 0,1,2,3 in the clockwise direction of the Yaw axis. Indexed to.

The camera is squeezed open and focused on ∞. The amount of light is controlled by the shutter speed. Show the checkerboard to the camera in 15 different positions and orientations. Using cvCalibrateCamera2 of OpenCV, the internal parameters P _ij (i = 1, ..., M; J = 1,2,3,4) of the camera are acquired.

3D position in camera coordinate system

Appears in the camera image I _ij in the following format.

The distortion-free image I _ij ^* is generated from I _ij using P _ij in the following format,

The distortion-free image is the three-dimensional position of the camera coordinate system.

To

Indicate.

[D-3] Acquisition of homogeneous conversion matrix from the camera coordinate system theodolite coordinate system to the camera coordinate system in the theodolite / total station coordinate system will be described. Stand the pole vertically using a spirit level. For the theodolite / total station (i = 0, ..., M), set the theodolite or total station in the coordinate system {i} vertically using a level. The height of the camera is, for example, 1.7 m from the horizontal plane, and the distance from the pole to the camera is, for example, 10 m. Set the theodolite / total station pitch angle to 0 ° (horizontal). Then adjust Yaw to collimate the center of the pole and reset the Yaw angle (0 °).

Determine two points A and B on the pole axis, and determine the three-dimensional position of points A and B.

Is calculated by trigonometry. Set the rotation angle of theodolite or total station to β ₀ = -33.75 °, β ₁ = -11.25 °, β ₂ = 11.25 °, β ₃ = 33.75 °. For cameras (j = 0, ..., M) and angles (k = 0, ..., 20), set Yaw of coordinate system {i} to γ _jk , where

Is. Then, the angle (k = 0, ..., 20) is changed to acquire the image I _ijk of the camera (i, j).

An optimization calculation is performed so as to satisfy the above two equations, and a homogeneous transformation matrix from the theodolite coordinate system to the camera coordinate system.

To get. Similarly, the simultaneous conversion matrix is acquired for the next camera j, and the same conversion matrix is acquired for the next theodolite to acquire the simultaneous conversion matrix.

The result of the calibration according to this embodiment is represented by the following three formulas.

[E] Motion Capture System The motion capture system according to this embodiment will be described with reference to FIG. 16. The motion capture system is a so-called video motion capture system (see Non-Patent Document 1), and three-dimensional reconstruction is performed from joint positions estimated using deep learning from images of a plurality of cameras acquired by the motion capture camera system. The target does not need to attach any marker or sensor, and the measurement space is not limited. Motion capture is performed completely unconstrained from the images of multiple RGB cameras. In principle, if images can be acquired from indoor living spaces to large outdoor sports fields, motion measurement is possible. It is a technology. The motion capture system described below is an example, and the motion capture used with the motion capture camera system according to the present invention is not limited, and other methods may be adopted.

The number of objects included in one image is not limited. For example, in competitive sports, by capturing the motions of multiple people with a camera, each image includes multiple people. In each image, the joint positions of one person selected from the plurality of persons or an arbitrary number of a plurality of persons are acquired. When one image contains multiple people, for example, PAF and PCM (Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields.InInProceedings By using IEEE Conference on Computer Vision and Pattern Recognition.CVPR 2017, 2017.), joint position can be acquired for each person at the same time. Further, if the joint position is identified for each person at the initial stage of measurement, then each person can be identified by tracking the joint position with continuity between frames. In the motion acquisition by the motion capture system, the object has a link structure or an articulated structure. The object is typically a human with a skeletal structure, but the object may be a robot.

The video motion capture system uses the motion capture camera system that captures the motion of the target and the degree of certainty of the position of the feature points (Keypoints) including joint positions in color intensity based on the images captured by the camera system. The heat map acquisition unit that acquires the heat map information to be displayed, the joint position acquisition unit that acquires the target joint position using the heat map information acquired by the heat map acquisition unit, and the joint that is acquired by the joint position acquisition unit A smoothing processing unit that smoothes the position, a storage unit that stores the skeletal structure of the target body, time series data of images acquired by the video system, time series data of the joint positions acquired by the joint position acquisition unit, and the like. And a display for displaying a skeleton structure or the like corresponding to a target image and a target pose acquired by the camera system.

The motion capture / camera system includes a plurality of cameras, and a plurality of synchronized cameras capture the motion of the subject, and each camera outputs an RGB image at a predetermined frame rate. A plurality of camera images acquired at the same time are transmitted to the heat map acquisition unit. The heat map acquisition unit generates a heat map based on the RGB image. The heat map represents the spatial distribution of the likelihood of the position of the feature points on the body. The heat map acquisition unit generates a two-dimensional or three-dimensional spatial distribution of the likelihood of the likelihood of the position of the feature points (keypoints) on the body including each joint position based on the input image, and calculates the likelihood of the likelihood. Display the spatial distribution in heat map format. The heat map acquisition unit typically uses a convolutional neural network (CNN) to calculate the position (typically a joint position) of a feature point on the target body from a single input image as a heat map. Estimate as.

The generated heat map information is transmitted to the joint position acquisition unit, and the joint position acquisition unit acquires the joint position. The joint position acquisition unit estimates the joint position candidate using the heat map information acquired from the heat map acquisition unit, and executes the optimization calculation based on the inverse kinematics using the joint position candidate to calculate the skeleton model. Update the joint angle and joint position. The joint position acquisition unit estimates the joint position candidate based on the heat map data, and the joint position candidate inverse motion that performs the optimization calculation based on the inverse kinematics using the joint position candidate to calculate the joint angle. And a forward kinematics calculation unit that performs forward kinematics calculation using the calculated joint angle to calculate the joint position. The acquired joint position data is stored in the storage unit as time series data of the joint position.

The acquired joint position data is transmitted to the smoothing processing unit, and the smoothed joint position and the joint angle are acquired. The smoothed joint position acquisition unit of the smoothing processing unit performs smoothing processing on the acquired joint position using the joint position in the past frame to smooth the temporal movement of the joint position. To Perform the inverse kinematics-based optimization calculation again using the positions of the smoothed feature points to obtain the target joint angle, and execute the forward kinematics calculation using the obtained joint angle to obtain the target joint angle. Get the joint position of. The target pose is determined based on the smoothed joint position or joint angle and the skeletal structure of the target body, and the target motion made up of time series data of the pose is displayed on the display.

FIG. 17 exemplifies the processing steps of motion analysis using motion capture. The target motion is acquired by the motion capture according to this embodiment. Time series data of the acquired joint angle and joint position is acquired. Further, based on this, the joint torque is obtained by inverse dynamics calculation, and the joint torque is used to optimize the wire tension in the musculoskeletal model including the wire imitating a muscle (quadratic programming or linear programming). Method), calculate the muscle activity using the wire tension, generate a musculoskeletal image with colors assigned according to the degree of muscle activity, and visualize the musculoskeletal with muscle activity. The image is output at a predetermined frame rate and displayed on the display as a moving image. In this way, it is possible to automatically and efficiently perform the process of capturing the motion of the target, the acquisition of the three-dimensional pose of the target during the motion, and the estimation and visualization of the muscle activity required for the motion.

Claims

A plurality of camera units consisting of one or a plurality of cameras,
Multiple camera mounters equipped with each camera unit,
Calibration means for acquiring camera parameters for three-dimensionally reconstructing each camera image;
One or more surveying instruments with angle measuring function,
A motion capture camera system equipped with
The camera parameters include external parameters that represent the position and orientation of each camera in the global coordinate system,
The calibration means obtains the external parameter using the position and orientation of each camera mounter in the global coordinate system, and the position and orientation of the camera mounter in the global coordinate system is the one or more surveying points. Is acquired using a machine,
Motion capture camera system.
The motion capture camera system according to claim 1, wherein one or more surveying instruments of the one or more surveying instruments have a distance measuring function.
The motion capture camera system according to any one of claims 1 and 2, wherein some or all of the plurality of camera mounters are the surveying instrument.
The motion capture camera system according to any one of claims 1 and 2, wherein some or all of the plurality of camera mounters are provided with a plurality of markers that can be surveyed by the surveying instrument.
The motion capture camera system according to claim 4, wherein the camera mounter has a plane provided with three or more markers.
The calibration means uses position information in a global coordinate system of a plurality of feature points located in the imaging space of each camera, and position information of each feature point in each camera image,
Position information in the global coordinate system of the plurality of feature points is acquired using the surveying instrument,
The motion capture camera system according to any one of claims 1 to 5.
The calibration means uses the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system,
The motion capture camera system according to any one of claims 1 to 6.
The calibration means includes means for acquiring the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system,
The motion capture camera system according to claim 7.
The camera mounter is composed of the surveying instrument,
The position and orientation of the camera in the camera mounter coordinate system are acquired using the surveying instrument,
The motion capture camera system according to claim 8.
The motion capture camera system according to any one of claims 1 to 9, wherein the calibration means obtains the external parameter by optimization calculation.
In the optimization calculation, position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.
Motion capture camera system according to claim 10.
12. The camera parameter according to claim 10, wherein the camera parameter includes an optical parameter and an internal parameter, and the calibration unit obtains the external parameter, the optical parameter, and the internal parameter by an optimization calculation. Motion capture camera system.
A plurality of camera units consisting of one or a plurality of cameras,
Multiple camera mounters equipped with each camera unit,
A motion-capture camera system comprising: a calibration method for acquiring camera parameters for three-dimensionally reconstructing the plurality of camera images,
The camera parameters include external parameters that represent the position and orientation of each camera in the global coordinate system,
Obtaining the position and orientation of each camera mounter in the global coordinate system using one or more surveying instruments equipped with an angle measuring function,
Obtaining the extrinsic parameters using the position and orientation of each camera mounter in the global coordinate system,
Calibration method including.
The motion capture camera system according to claim 13, wherein one or more surveying instruments of the one or more surveying instruments have a distance measuring function.
15. A part or all of the plurality of camera mounters is the surveying instrument, and acquires the position and orientation of each camera mounter in the global coordinate system using surveying data of a plurality of measurement points. The calibration method according to item 1.
Some or all of the plurality of camera mounters are provided with a plurality of markers that can be surveyed by the surveying instrument, and the position and orientation of the camera mounter in the global coordinate system are acquired using surveying data of the plurality of markers. The calibration method according to any one of claims 13 and 14.
The calibration method according to claim 16, wherein the camera mounter has a flat surface provided with three or more markers.
In the acquisition of the external parameter, position information in the global coordinate system of a plurality of feature points located in the imaging space of each camera, and the position information of each feature point in each camera image, are used,
Position information in the global coordinate system of the plurality of feature points is acquired using the surveying instrument,
The calibration method according to any one of claims 13 to 17.
In the acquisition of the external parameters, the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is used.
The calibration method according to any one of claims 13 to 18.
Including the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system,
The calibration method according to claim 19.
The camera mounter is composed of the surveying instrument,
Acquiring the position and orientation of the camera in the camera mounter coordinate system using the surveying instrument,
The calibration method according to claim 20.
The calibration method according to any one of claims 13 to 21, wherein the external parameter is acquired by optimization calculation.
In the optimization calculation, position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.
The calibration method according to claim 22.
The calibration method according to any one of claims 22 and 23, wherein the camera parameter includes an optical parameter and an internal parameter, and the external parameter, the optical parameter, and the internal parameter are acquired by optimization calculation.
A plurality of camera units consisting of one or a plurality of cameras,
Multiple camera mounters equipped with each camera unit,
Calibration means for acquiring camera parameters for three-dimensionally reconstructing each camera image;
One or more surveying instruments with angle measuring function,
A motion capture camera system equipped with
The calibration means,
Position information in each camera image of a plurality of immovable feature points for which position information in the global coordinate system is obtained using the surveying instrument,
Position information in each camera image of one or more feature points whose position information in the global coordinate system is unknown,
A motion capture camera system that obtains the camera parameters by an optimization calculation using.
The motion capture camera system according to claim 25, wherein the immovable feature point is a point on an element forming a sports field or a court.
The motion capture camera system according to any one of claims 25 and 26, wherein the feature point whose position information in the global coordinate system is unknown is a ball in a ball game.
The motion capture camera system according to any one of claims 25 to 27, wherein a part or all of the plurality of camera mounters is the surveying instrument.
A plurality of camera units consisting of one or a plurality of cameras,
Multiple camera mounters equipped with each camera unit,
In a motion capture / camera system including, a calibration method for acquiring camera parameters for three-dimensionally reconstructing the plurality of camera images,
Before the motion capture, the position information in the global coordinate system of a plurality of immovable feature points is acquired by the surveying instrument with the angle measurement function.
At the time of motion capture, using position information in each camera image of the plurality of fixed feature points and position information in each camera image of one or more feature points whose position information in the global coordinate system is unknown, A calibration method for obtaining the camera parameters by optimization calculation.
The calibration method according to claim 29, wherein the immovable feature point is a point on an element forming a sports field or a court.
31. The calibration method according to claim 29, wherein the feature point whose position information in the global coordinate system is unknown is a ball in a ball game.
The calibration according to any one of claims 29 to 31, wherein a part or all of the plurality of camera mounters is a surveying instrument having an angle measuring function, and the posture information of the camera mounters is used in the optimization calculation. Method.