JP7395189B2 - Motion capture camera system and calibration method - Google Patents

Description

The present invention relates to a camera system for motion capture.

Motion capture technology is widely used for various motion measurements. However, in team sports, the movements of each player during a match are not captured and analyzed. For example, it has not been done to acquire videos of sports matches such as soccer, rugby, baseball, volleyball, and handball, and to analyze the motions of each player based on the acquired videos.

In team sports, in order to capture the movements of each player during a match and perform motion analysis, it is necessary to (a) make sure that motion capture does not degrade the performance of the players, and (b) ensure that motion capture does not affect supporting operations (TV broadcasts, team (c) large storage capacity, (d) synchronization between high-speed cameras over long distances, (e) ability to receive large amounts of data in real time, and (f) long hours. It is necessary to satisfy conditions such as a stable system with operational performance, and (g) camera calibration for accurate 3D reconstruction.

As typical conventional motion capture techniques, optical motion capture and motion capture using an inertial sensor are known. These motion capture methods have the disadvantage that it takes time to prepare the reflective markers and inertial sensors for measurement by attaching them to the subject's body, and furthermore, the markers and sensors can interfere with the smooth movement of the subject. Therefore, it cannot be used to capture the movements of each player during a match. Furthermore, the measurement space for these motion captures is limited. Optical motion capture using an infrared light source requires measurements to be taken in a space surrounded by cameras that are not affected by external light, and when receiving sensor data wirelessly, there is a limited amount of wireless signal reach. It was necessary to measure in space.

Recently, so-called markerless motion capture technology has appeared. For example, video motion capture technology (Non-Patent Document 1) captures motion from images from multiple RGB cameras in a completely unrestricted manner. In short, it is a technology that makes it possible to measure motion if images can be obtained.

In motion capture using multiple cameras, it is necessary to obtain camera parameters for three-dimensional reconstruction of multiple camera images. Camera parameters include optical parameters such as lens distortion, internal parameters such as focal length and optical center, and external parameters that represent the position and orientation of the camera in the space in which it is installed. Traditional camera calibration calibrates all of these parameters by photographing a calibration device of known shape and dimensions (such as a checkerboard or calibration wand) with one or more cameras. .

However, in a large space such as a sports field, performing 3D reconstruction for motion measurement requires highly accurate calibration of a camera that is far away from the target, which is different from traditional motion capture systems. This requires a different calibration method than the system configuration and camera calibration at relatively short distances. Among camera parameters, optical parameters including lens distortion and internal parameters such as focal length can be calibrated in advance once the camera and lens are determined, but external parameters related to the position and orientation of the camera in space can be calibrated in advance. Calibration must be performed on site where the camera is installed.

Measurement of people or robots on sports fields or outdoors is characterized by the fact that the measurement area is a large space. Although it is theoretically possible to calibrate using a conventional calibration device (checkerboard, calibration wand, etc.), the accuracy of calibration depends on the number of pixels in the image sensor. When the distance to the object is several tens of meters, the calibration accuracy decreases, making it unsuitable for three-dimensional reconstruction in motion capture. It is possible to prepare and use a larger calibration device (for example, a large checkerboard or a large calibration wand) compared to the conventional one, but if the shooting range of the camera is horizontal and vertical, When a large area is covered, it is not practical to move a large-sized calibration instrument to different positions and postures to cover such a wide area.

US Patent No. 8,625,086 WO2005/059473 Patent No. 3965781

T. Ohashi, Y. Ikegami, K. Yamamoto, W. Takano and Y. Nakamura, Video Motion Capture from the Part Confidence Maps of Multi-Camera Images by Spatiotemporal Filtering Using the Human Skeletal Model, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, 2018, pp. 4226-4231. Application of hybrid video theodolite system to human gait analysis Tetsuji Anai, Hirofumi Chikazu Photogrammetry and Remote Sensing Vol. 40 (2001) No. 2

An object of the present invention is to easily and accurately calibrate a camera even if the imaging space for motion capture is large.

The motion capture camera system according to the present invention includes:
a plurality of camera units consisting of one or more cameras;
multiple camera mounters on which each camera unit is mounted;
Calibration means for acquiring camera parameters for three-dimensionally reconstructing each camera image;
one or more surveying instruments equipped with an angle measurement function;
Equipped with
The camera parameters include external parameters representing the position and orientation of each camera in the global coordinate system,
The calibration means acquires the external parameters using the position and orientation of each camera mounter in the global coordinate system, and the position and orientation of the camera mounter in the global coordinate system are determined by the one or more surveying methods. It is obtained using a machine.
In one aspect, one or more of the one or more surveying instruments has a distance measurement function.
Examples of surveying instruments with an angle measurement function include theodolites and total stations.
A total station can be exemplified as a surveying instrument equipped with an angle measurement function and a distance measurement function.

In one aspect, some or all of the plurality of camera mounters are the surveying instruments.
In one embodiment, one camera mounter is comprised of a total station and the remaining camera mounters are comprised of theodolites.
The surveying instruments that make up the camera mounter are leveled.

In one aspect, some or all of the plurality of camera mounters include a plurality of markers (feature points) that can be measured by the surveying instrument.
In one aspect, all camera mounters are marker equipped camera mounters.
In one aspect, the camera mounter has a flat surface with three or more markers (eg, four survey markers).

In one aspect, the calibration means includes position information (three-dimensional position information) in a global coordinate system of a plurality of feature points located in the imaging space of each camera, and position information of each feature point in each camera image. (two-dimensional position information),
Position information of the plurality of feature points in a global coordinate system is acquired using the surveying instrument.
In one embodiment, the feature point is a fixed point located in the imaging space of each camera, such as an intersection (including a corner) or an intersection of lines on the court, or a point on a goal post (a point on a goal post). Intersections of crossbars and bottom ends of posts) can be used as feature points.
In one aspect, the three-dimensional position information and two-dimensional position information of the feature points are used for optimization calculation when acquiring external parameters.

In one aspect, the calibration means uses the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system (for example, when the position and orientation of the camera with respect to the camera mount are known).
In one aspect, the calibration means includes means for acquiring the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system.
In one aspect, the camera mounter is comprised of the surveying instrument,
The surveying instrument is used to obtain the position and orientation of the camera in the camera mounter coordinate system.
In one aspect, the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is known or obtained in advance.

In one aspect, the calibration means obtains the external parameter by optimization calculation.
In one aspect, the position and orientation of the camera mounter in the global coordinate system, and the three-dimensional position information and two-dimensional position information of the feature points are used in the optimization calculation.
In one aspect, in the optimization calculation, position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.

In one embodiment, the camera parameters include optical parameters and internal parameters, and the calibration means obtains some or all of the external parameters, the optical parameters, and the internal parameters through optimization calculation.
Optical and internal parameters may be obtained using conventional techniques (using calibration instruments).
For example, when the camera mounter includes the surveying instrument, the surveying instrument may be used to obtain the optical parameters and internal parameters.
In acquiring the external parameters, part or all of the acquired optical parameters and internal parameters may be used.

The present invention includes a plurality of camera units each including one or more cameras;
multiple camera mounters on which each camera unit is mounted;
A calibration method for obtaining camera parameters for three-dimensionally reconstructing the plurality of camera images in a motion capture camera system comprising:
The camera parameters include external parameters representing the position and orientation of each camera in the global coordinate system,
obtaining the position and orientation of each camera mounter in a global coordinate system using one or more surveying instruments equipped with an angle measurement function;
obtaining the external parameters using the position and orientation of each camera mounter in a global coordinate system;
including.

In one aspect, some or all of the plurality of camera mounters are the surveying instruments, and the position and orientation of each camera mounter in a global coordinate system are acquired using survey data of a plurality of measurement points (feature points). do.
In one aspect, the plurality of measurement points include two feature points A and B (with a known interval, for example, points A and B on a vertical surveying pole) that can be measured from each surveying instrument, and each camera. A feature point on the mounter (for example, the lower end of a pendulum weight).
By surveying the two feature points A and B with a surveying instrument, vectors from the origin of each camera mounter coordinate system to points A and B can be obtained. For example, the horizontal axis with the same horizontal angle as the horizontal angle of the vector is the z-axis, and the vertical axis is the x-axis.
By measuring feature points on other camera mounters with each surveying instrument, it is possible to obtain the angle between the coordinate systems of each mounter.
By aligning the z-axis and x-axis of the global coordinate system with the z-axis and x-axis of one mounter's coordinate system, and using the distance between points A and B and the above setting information, each camera can be adjusted by trigonometry. Obtain the position and orientation of the mounter in the global coordinate system.

In one aspect, some or all of the plurality of camera mounters include a plurality of markers (feature points) that can be measured by the surveying instrument, and the camera mounters in the global coordinate system are equipped with a plurality of markers (feature points) that can be measured by the surveying instrument. Obtain the position and orientation of the mounter.
In one aspect, the camera mounter has a flat surface with three or more markers (eg, four survey markers).
In one embodiment, the distances between three or more markers are known, the markers are provided on a vertical plane, and a surveying instrument (e.g., with angle measurement and ranging capabilities) By surveying markers and setting a global coordinate system with the center of the surveying instrument as the origin, the position and orientation of each camera mounter in the global coordinate system is obtained using survey information and trigonometry.

In one aspect, in acquiring the external parameters, position information in a global coordinate system of a plurality of feature points located in the imaging space of each camera and position information of each feature point in each camera image are used. ,
Position information of the plurality of feature points in a global coordinate system is acquired using the surveying instrument.
In one embodiment, the feature point is a fixed point located in the imaging space of each camera, such as an intersection (including a corner) or an intersection of lines on the court, or a point on a goal post (a point on a goal post). Intersections of crossbars and bottom ends of posts) can be used as feature points.
In one aspect, the three-dimensional position information and two-dimensional position information of the feature points are used for optimization calculation.

In one aspect, in acquiring the external parameters, the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is used.
In one aspect, the calibration means includes means for acquiring the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system.
In one aspect, the camera mounter is comprised of the surveying instrument,
The position and orientation of the camera in the camera mounter coordinate system are acquired using the surveying instrument.
In one aspect, the surveying instrument is used to set a plurality of feature points whose positions in the camera mount coordinate system are known (see FIG. 15),
Obtain the position information of each feature point in each camera image,
The position and orientation of the camera in the camera mounter coordinate system are obtained using the position information of the feature point in the camera mounter coordinate system and the position information in the camera image.
In one aspect, the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is known or obtained in advance.

In one aspect, the external parameters are obtained by optimization calculation.
In one aspect, the position and orientation of the camera mounter in the global coordinate system, and the three-dimensional position information and two-dimensional position information of the feature points are used in the optimization calculation.
In one aspect, in the optimization calculation, position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.
In one aspect, each camera is synchronized, each camera photographs the calibration instrument at a different position/orientation, and position information of a plurality of feature points on the calibration instrument in each camera image is used.

In one embodiment, the camera parameters include optical parameters and internal parameters, and the calibration means obtains some or all of the external parameters, the optical parameters, and the internal parameters through optimization calculation.
Optical and internal parameters may be obtained using conventional techniques (using calibration instruments).
For example, when the camera mounter includes the surveying instrument, the surveying instrument may be used to obtain the optical parameters and internal parameters.
In acquiring the external parameters, part or all of the acquired optical parameters and internal parameters may be used.

The present invention uses highly accurate survey data measured by a surveying instrument to calibrate the camera, thereby making it possible to easily and accurately calibrate the camera even if the imaging space for motion capture is large. can.
Therefore, by filming sports matches held on indoor and outdoor sports fields, indoor and outdoor courts, arenas, domes, gymnasiums, etc., we can perform motion capture and three-dimensional reconstruction of the movements of each player during the match, and perform motion analysis. It can be performed.

FIG. 2 is a conceptual diagram showing a mode in which the motion capture camera system according to the present embodiment is arranged to photograph a soccer court. FIG. 2 is a conceptual diagram showing a mode in which the motion capture camera system according to the present embodiment is arranged to photograph a volleyball court. FIG. 1 is a schematic diagram of a camera network according to the present embodiment. FIG. 2 is a diagram showing an example of a camera mounter according to the present embodiment, showing a theodolite as a camera mounter and three cameras mounted on the theodolite. FIG. 2 is a diagram showing an example of a camera mount according to the present embodiment, showing a flat plate as a camera mount, four markers provided at the four corners of the flat plate, and one camera mounted at the lower end of the center portion of the flat plate in the width direction. ing. 1 is a diagram showing a total station exemplified as a surveying instrument according to the present embodiment. FIG. 3 is a diagram showing a plurality of cameras arranged around a volleyball court, their FOVs, and cable wiring. It shows the trajectory of the target ball in space above the volleyball court and is used to acquire camera image data for bundle adjustment. It is a diagram showing 18 feature points on a volleyball court. FIG. 3 is a diagram showing the measured feature points on the court and the position of the camera mounter. It is a diagram showing the relationship between the camera coordinate system and the image plane on OpenCV. FIG. 3 is a diagram showing triangulation using a theodolite. It is a figure which shows the triangular side volume using the theodolite. FIG. 2 is a diagram illustrating the measurement of the position and orientation of a camera mounter with a marker using a total station. FIG. 3 is a diagram illustrating calibration of relative external parameters. 1 is an overall diagram of a motion capture system. It is a flowchart showing the process of motion analysis using motion capture.

[A] Motion capture camera system [A-1] Overall configuration The motion capture camera system according to this embodiment uses multiple cameras to film team sports played in large spaces such as sports fields and arenas. , motion capture of each player is performed using images captured by each camera. The motion capture used in this embodiment is a video motion capture technique (non-patent document 1), and its specific details will be described later.

A motion capture camera system includes a plurality of camera units including one or more cameras, and a plurality of camera mounters on which each camera unit is mounted. For example, FIG. 1 is a conceptual diagram of a motion capture camera system that photographs a soccer match, and four camera units each having four cameras are arranged to surround a soccer court. The distance from each camera to the target player is several tens of meters.

The motion capture camera system shown in FIG. 1 includes four camera units CU ₁ , CU ₂ , CU _{3 ,} and CU ₄ , and the camera unit CU ₁ has four cameras C ₁₁ , C ₁₂ , C ₁₃ , C ₁₄ , and a camera. Unit CU ₂ has four cameras C ₂₁ , C ₂₂ , C ₂₃ , C ₂₄ , camera unit CU ₃ has four cameras C ₃₁ , C ₃₂ , C ₃₃ , C ₃₄ , camera unit CU ₄ has four cameras C ₄₁ , It has C ₄₂ , C ₄₃ , and C ₄₄ . Camera unit CU ₁ is mounted on camera mounter CM ₁ , camera unit CU ₂ is mounted on camera mounter CM ₂ , camera unit CU ₃ is mounted on camera mounter CM 3, and camera unit CU ₄ is mounted on camera mounter CM ₃ . It is installed in the camera mounter CM ₄ . Although four camera units are shown in FIG. 1 for convenience, in reality, more camera units could be arranged. Note that the number of cameras mounted on one camera unit is not limited.

FIG. 2 is a conceptual diagram of a motion capture camera system that photographs a volleyball match, and four camera units each including one camera are arranged to surround a volleyball court. As shown in FIG. 2, the motion capture camera system includes four camera units CU ₁ , CU ₂ , CU ₃ , and CU ₄ , where camera unit CU ₁ has one camera C ₁₁ and camera unit CU ₂ has one camera C 11 . Camera C ₂₁ , camera unit CU ₃ includes one camera C ₃₁ , and camera unit CU4 includes one camera C ₄₁ . Camera unit CU ₁ is mounted on camera mounter CM ₁ , camera unit CU ₂ is mounted on camera mounter CM ₂ , camera unit CU ₃ is mounted on camera mounter CM 3, and camera unit CU ₄ is mounted on camera mounter CM ₃ . It is installed in the camera mounter CM ₄ . Note that the number of cameras mounted on one camera unit is not limited.

In an experimental example to be described later, ten camera units each including one camera are arranged to surround a volleyball court, as shown in FIG. The size of the volleyball court is a rectangle of 18 m x 9 m, with a free zone 3 m wide around it. Each camera is arranged so as to be able to photograph a space with a height of at least 3 m in an area including the free zone. Each camera is mounted on a tripod leveled at a predetermined position or fixed to a handrail. Each camera is calibrated in advance with a fixed position and orientation, and the position and orientation are maintained during shooting.

FIG. 3 shows the camera network of the motion capture camera system shown in FIG. 7. Note that although 12 cameras can be connected to the camera network, 10 cameras are used in the embodiment. The camera network according to this embodiment includes a plurality of cameras, a transmission box, a camera hub, one or more computers, a power source, and a waveform generator that generates a synchronization signal. Each camera is electrically connected to a transmission box, the transmission box and camera hub are electrically connected with a hybrid copper/fiber cable, and the camera hub and computer are electrically connected. Each camera is synchronized.

Each camera acquires images with a resolution of 1920x1200 at 120Hz. Each camera is equipped with a USB3.0 interface, and a USB3 to Fiber Optic Extender is used to transmit image signals. The transmission box is equipped with a USB3 to Fiber Optic Extender Transmitter, the camera hub is equipped with a USB3 to Fiber Optic Extender Receiver, and the image signal is sent to the computer via the USB3 to Fiber Optic Extender. . The computer includes a data receiving section, a processing section, and a storage section, and image data received by the data receiving section is appropriately compressed by the processing section and stored in the storage section.

Each camera is mounted on a camera mounter, and the position and orientation of each camera are fixed with respect to the camera mounter. The internal parameters, external parameters, and distortion coefficients of the camera have been obtained through calibration, and these camera parameters are stored in the storage section of the computer. The camera parameters are used for three-dimensional reconstruction of the object's motion using each camera image.

The motion capture camera system according to this embodiment further includes a calibration means and one or more surveying instruments equipped with an angle measurement function. The calibration means is composed of a computer (comprising an input section, a calculation section, a storage section, an output section, etc.). The calibration means acquires camera parameters for three-dimensionally reconstructing each camera image. The calibration according to this embodiment is characterized in that survey data from a surveying instrument is used in the calibration calculation. More specifically, the calibration means acquires external parameters using the position and orientation of each camera mounter in the global coordinate system, and the position and orientation of the camera mounter in the global coordinate system are one. Alternatively, it is obtained using multiple surveying instruments. In one aspect, during motion capture, the position and orientation of the camera are fixed, and a surveying instrument is not used. Typically, the calibration means acquires camera parameters by optimization calculation, and the calibration means includes an optimization calculation section that performs optimization calculation.

Theodolites and total stations are known as high-precision surveying techniques for large outdoor spaces. Theodolite is a typical angle measuring instrument with an angle measurement function.It has a movable part (equipped with a telescope) that can rotate horizontally and vertically, and can measure horizontal angles and altitude angles with high precision. . A total station is a device that includes a theodolite with an angle measurement function and a distance measurement function (laser measurement). The accuracy of the pan angle (horizontal angle) and tilt angle (vertical angle) of the total station used in the experiment described below is 20 seconds, and the accuracy of the laser range finder is ±1 mm at 28.8 m.

The present invention is a motion capture camera system used for measurements in large spaces, which includes a surveying instrument, specifically a theodolite and a total station, and, if necessary, surveying instruments used for these measurements (reflective prisms and reflective markers). etc.), it is possible to easily and accurately calibrate camera parameters even in a large space.

[A-2] When the camera mounter is composed of a surveying instrument In the first embodiment of the camera mounter, the camera mounter of each camera unit consisting of one or more cameras is composed of a theodolite or a total station surveying instrument. Ru. In the embodiment shown in FIG. 1, camera mounters CM ₁ , CM ₂ , CM ₃ , and CM ₄ are constructed from surveying instruments. Each camera unit is mounted on the movable part of the surveying instrument, and can rotate horizontally and vertically together with the movable part during angle measurement. Specifically, a camera stand equipped with one or more cameras is fixed and installed on a theodolite or total station. For example, the theodolite and total station are mounted on a leveled tripod, and a camera unit is mounted on top of the tripod (see FIG. 4).

By configuring the camera mounter from a surveying instrument, the attitude and position of the camera mounter in the global coordinate system can be obtained using surveying technology. It consists of each camera unit, a camera stand, and a plurality of cameras mounted on the camera stand, and the camera stand is mounted on the surveying instrument. For example, four cameras are arranged radially at equal angles so that the optical axes of each camera are located in the same plane and the optical axes of all cameras intersect at one point. In another example, each camera is placed on the side of a regular polygonal pyramid, and the optical axes of all cameras intersect at a point on the axis of the regular polygonal pyramid. The number and arrangement of cameras in each camera unit are not limited. When a plurality of camera mounters are composed of surveying instruments, in one aspect, all the camera mounters are composed of surveying instruments. In other aspects, a portion of the plurality of camera mounters is comprised of a surveying instrument. When two or more camera mounters are comprised of surveying instruments, at least one camera mounter may be comprised of a total station.

Here, a theodolite equipped with a camera is disclosed in, for example, Patent Documents 1 to 3 and Non-Patent Document 2, and is also known as a video theodolite. The camera in a conventional video theodolite is in a rotatable state during normal use, and the camera also rotates as the angle measurement section rotates, and the camera image information is used together with the angle measurement information. On the other hand, in the camera unit in this embodiment in which the surveying instrument is used as a camera mount, the angle measurement function of the surveying instrument is used only for camera calibration, and the camera position is fixed during use (motion capture). The angle measurement function of the theodolite is not used. Note that in the present invention, it is not excluded that the angle measurement function of the surveying instrument is used during motion capture, and the angle measurement function may be used in dynamic calibration, which will be described later.

[A-3] When the camera mounter is equipped with a marker that can be measured by a surveying device In the second embodiment of the camera mounter, the camera mounter is provided with a plurality of markers (feature points), and the marker can be measured by a surveying device. can be measured. More specifically, the camera mount includes a plate (for example, a 30 cm x 30 cm to 40 cm x 40 cm square) having a flat surface with a predetermined shape and dimensions, and the flat surface of the plate has a reflective surface that serves as a target for the theodolite or total station. Three or more (for example, four) markers are provided. In the embodiment shown in FIG. 2, the camera mounts CM ₁ , CM ₂ , CM ₃ , CM ₄ consist of a plate with a flat surface having a plurality of reflective markers.

The flat plate is equipped with a camera whose relative position is fixed. For example, each camera unit includes one camera, and the camera is fixed at the center in the width direction below the square flat plate. A flat plate equipped with three or more reflective markers and whose position relative to the camera is fixed is attached to a camera stand (pan head) together with the camera, and the camera stand is fixed and installed on a tripod or the like. Note that a plurality of cameras may be mounted on a camera mount made of a plate provided with a plurality of markers.

By using surveying technology, it is possible to obtain the attitude and position of the camera mounter (flat plate) in the global coordinate system. When the camera mounter is equipped with a marker that can be measured by a surveying instrument, it is advantageous to use a surveying instrument (typically a total station) equipped with an angle measurement function and a distance measurement function to measure the marker. .

It will be understood by those skilled in the art that in using the surveying instrument, other instruments, surveying methods, and calculation methods used in the surveying technology may be used. Examples of surveying instruments other than theodolites and total stations include tripods, aluminum staffs, aluminum rods, poles, spirit levels, autolevels, reflective prisms, and prism poles.

[B] Camera Calibration Camera calibration for three-dimensional reconstruction in large space motion capture will be explained. Three methods, the first method, the second method, and the third method, will be explained below. Although the first method is a known method, the third method is a combination of the first method and the second method, so the first method will also be mentioned in conjunction with the explanation of camera parameters.

[B-1] General bundle adjustment using a group of feature points with unknown positions (first method)
Calibration of each camera of the camera system used in this embodiment will be explained. Here, bundle adjustment means that the bundle of light rays from the 3D feature point to the camera is optimally adjusted for both the 3D feature point and the position/orientation of the camera. . This optimization is generally solved as a nonlinear optimization problem that minimizes the reprojection error of feature points.

ここで、iはカメラ番号、行列X＝(x₁・・・x_n)はn個の特徴点のグローバル座標系での位置x_i＝(x_i y_i z_j 1)^T、行列ⁱY＝(ⁱy₁・・・ⁱy_n)はn個の特徴点の画像平面でのピクセル単位の位置ⁱy_j＝(ⁱu_j ⁱv_j 1)^T、行列ⁱS＝diag{ⁱs₁,・・・,ⁱs_n}は画像平面に投影するためのスケールⁱs_j＝(001)ⁱAiBx_jを指す。A perspective projection transformation matrix based on a pinhole camera model, which is the simplest camera model, is used to reproject the feature points. This perspective projection transformation matrix can be expressed for each camera as follows.

Here, i is the camera number, matrix X = (x ₁ ... x _n ) is the position of n feature points in the global coordinate system x _i = (x _i y _i z _j 1) ^T , matrix ⁱ Y =( ⁱ y ₁ ... ⁱ y _n ) is the pixel unit position of n feature points on the image plane ⁱ y _j =( ⁱ u _j ⁱ v _j 1) ^T , matrix ⁱ S=diag{ ⁱ s ₁ ,..., ⁱ s _n } refers to the scale ⁱ s _j =(001) ⁱ AiBx _j for projection onto the image plane.

ここで、ⁱf_x，ⁱf_yは画像平面上のx軸、y軸それぞれのピクセル単位の焦点距離、ⁱc_x，ⁱc_yは画像平面での主点（画像平面と光軸の交点）の位置、行列ⁱRはカメラ座標系におけるグローバル座標系の姿勢、ⁱtはカメラ座標系におけるグローバル座標系の原点の位置を表す。コンピュータビジョンで一般的に使用される座標軸において、３次元空間から画像平面への投影を図１１に示す。これにしたがい、次の式でカメラの内部パラメータ行列Aを定義する。

ここで、行列ⁱAについては、ⁱf_x＝ⁱf_y＝ⁱfとし、ⁱc_x，ⁱc_yを画像平面の中心とすることで、最適化変数の数を減らすことができる。受光素子の大きさの縦横比は１：１、かつ、画像平面の中心で光軸と交わっている、と仮定できるからである。行列ⁱRについては、オイラー角で表記する場合には、ⁱＲ(ⁱθ_α,ⁱθ_β,ⁱθ_γ)として３変数として扱える。本明細書では、この回転行列の表記としてZYXオイラー角ⁱＲ(ⁱθ_α,ⁱθ_β,ⁱθ_γ)= Ｒ_Z(ⁱθ_α)Ｒ_Y (ⁱθ_β)Ｒ_X (ⁱθ_γ) を採用する。The matrices ⁱ A and ⁱ B representing the camera's internal parameters and external parameters, respectively, can be expressed as follows.

_Here ^{, IF} _{_} ^_ _{_} ^_ _{_} ), the matrix ⁱ R represents the orientation of the global coordinate system in the camera coordinate system, and ⁱ t represents the position of the origin of the global coordinate system in the camera coordinate system. FIG. 11 shows the projection from three-dimensional space onto the image plane in coordinate axes commonly used in computer vision. According to this, the camera's internal parameter matrix A is defined using the following formula.

Here, for the matrix ⁱ A, the number of optimization variables can be reduced by setting ⁱ f _x = ⁱ f _y = ⁱ f and making ⁱ c _x and ⁱ c _y the center of the image plane. This is because it can be assumed that the aspect ratio of the light receiving element is 1:1 and that it intersects the optical axis at the center of the image plane. When the matrix ⁱ R is expressed in Euler angles, it can be treated as three variables as ⁱ R( ⁱ θ _α , ⁱ θ _β , ⁱ θ _γ ). In this specification, this rotation matrix is expressed as ZYX Euler angle ⁱ R( ⁱ θ _α , ⁱ θ _β , ⁱ θ _γ )= R _Z ( ⁱ θ _α )R _Y ( ⁱ θ _β )R _X ( ⁱ θ _γ ) is adopted.

ここで、最適化変数である焦点距離ⁱf_x,ⁱf_yとカメラ位置ⁱtは、片方を固定しなければもう片方が定まらないような関係にある。そのため、片方を固定するか、それぞれに上限・下限を狭く設定するか、どちらかの必要がある。The nonlinear least squares problem for minimizing the reprojection error of feature points in multiple cameras can be expressed as follows.

Here, the optimization variables focal lengths ⁱ f _x , ⁱ f _y and camera position ⁱ t are in such a relationship that unless one is fixed, the other cannot be determined. Therefore, it is necessary to either fix one side or set narrow upper and lower limits for each.

ここで、上記非線形方程式系は、Levenberg-Marquardt法を用いて解くことができる。The data obtained for multi-camera calibration is the pixel position on the image plane of tie points (a group of feature points whose positions are unknown in the global coordinate system, such as the locus of the center position of a color ball) for multiple cameras. Suppose there was only ⁱ Y _tie . At this time, the position X _tie of the feature point group in the global coordinate system is determined depending on the positions and orientations of the plurality of cameras. This bundle adjustment can be expressed as follows.

Here, the above nonlinear equation system can be solved using the Levenberg-Marquardt method.

ここで、上記非線形方程式は、Levenberg-Marquardt法、ないし、Trust-Region-Reflective法を用いて解くことができる。Trust-Region-Reflective法では、最適化変数に対して、

のような範囲制約を設定できる。得られたカメラのマウンタの位置・姿勢ⁱB_mを基準に、最適化変数の初期値と狭い範囲制約を設定することで、前述の焦点距離ⁱf_x,ⁱf_yとカメラ位置ⁱtの関係に対して測量的に妥当な制約を加えることができる。[B-2] Bundle adjustment based on direct measurement using a surveying instrument (second method)
The data obtained for calibration is the position ⁱ _control , the pixel position ⁱ Y _control on the image plane, and the position/orientation ⁱ B _m of the camera mounter. Direct measurement was performed using a surveying instrument to obtain the values of the control points and the position and orientation of the camera mounter ⁱ B _m . At this time, since there are no restrictions between cameras, optimization can be performed for each camera. This bundle adjustment can be expressed as follows.

Here, the above nonlinear equation can be solved using the Levenberg-Marquardt method or the Trust-Region-Reflective method. In the Trust-Region-Reflective method, for optimization variables,

You can set range constraints like . Based on the obtained camera mounter position and orientation ⁱ B _m , by setting the initial values and narrow range constraints of the optimization variables, the focal lengths ⁱ f _x , ⁱ f _y and camera position ⁱ t described above can be calculated. Measurementally valid constraints can be placed on the relationship.

The distortion coefficient of the camera is calculated using internal parameters using the OpenCV function cv::calibrateCamera using multiple images taken at various distances and angles against a checkerboard with grid points with known distances. Ask with. In this embodiment, internal parameters are also optimized as variables for bundle adjustment. Optimization was performed using the function lsqnonlin, which solves nonlinear least squares problems, in MATLAB's Optimization Toolbox as an optimization solver.

ここで、n_tie, n_controlはそれぞれ、タイポイント、コントロールポイントの個数、k_tie, k_controlはk_tie＞0，k_control＞0となる２種の特徴点群に対する重みを指す。本実施形態では、k_tie＝k_control＝0.5とする。上記非線形方程式は、Levenberg-Marquardt法を用いて解くことができる。[B-3] Bundle adjustment using a combination of feature points whose positions are unknown and survey values (third method)
The data obtained for calibration are the pixel position ⁱ Y _tie in the image plane of the tie point, the position ⁱ X _control in the global coordinate system of the control point and the pixel position ⁱ Y _control in the image plane, and Assume that there are four positions and orientations of the mounter: ^i, B _{, and m} . This bundle adjustment can be expressed as follows.

Here, n _tie and n _control indicate the number of tie points and control points, respectively, and k _tie and k _control indicate weights for two types of feature point groups such that k _tie >0 and k _control >0. In this embodiment, k _tie = k _control = 0.5. The above nonlinear equation can be solved using the Levenberg-Marquardt method.

[B-4] Experimental Example The following three types of multi-camera calibration were performed for motion capture in a volleyball match.
(a) General bundle adjustment using a group of feature points whose positions are unknown (first method),
(b) Bundle adjustment based on direct measurement by a surveying instrument (second method), and
(c) Bundle adjustment using a group of feature points whose positions are unknown and direct measurement using a surveying instrument (third method).
Figure 7 shows the arrangement of multiple cameras used for these measurements. In the experiment, calibration was performed on 6 cameras out of 10 cameras.

(a) Using the image position information of the trajectory of the color ball from multiple synchronized cameras, we performed bundle adjustment for multiple cameras and determined the position and orientation of the cameras. In the experiment, a yellow colored ball was moved as shown in Figure 8 within the measurement range of a volleyball court, and its trajectory was photographed using multiple synchronized cameras. The position and radius of the center of the color ball on the image plane were calculated using OpenCV, and processing to exclude outliers was performed on MATLAB. We were able to obtain 2223 frames for the center position of the color ball, excluding outliers. This data will be used as a tie point. Calibration is performed by solving equation (4).

(b) The position and orientation of the feature points on the court and the camera mounter were determined using a surveying instrument, and bundle adjustments were made for each camera to determine the camera position and orientation. The camera mounter with a surveying marker and the surveying instrument used in the experiment are shown in Figures 5 and 6, respectively. The surveying instrument is a total station (angle surveyor/laser distance meter), and the camera mount is a 40cm x 40cm flat plate with surveying markers on the four corners. FIG. 9 shows 18 on-court feature points (white line intersections). Position information of these feature points in the global coordinate system is acquired using a surveying instrument.

を最小化する問題として扱うことができる。In the global coordinate system, the origin is the on-court feature point number 0, the X-axis is the positive direction from the origin to the on-court feature point number 9, and the Z-axis is the vertical upward direction determined by the leveling of the total station. It is said that In line with this definition, the axes of the coordinate system obtained through optimization are rotated, and the three-dimensional position obtained through surveying is also translated and rotated. As a result, a feature point on the court whose position in the global coordinate system is known is set as a control point. Furthermore, the position and orientation of the camera mounter with a survey marker in the global coordinate system are converted to the local coordinate system of the camera mounter, and these are used as initial values of the external parameter matrix of the camera. Calibration is performed by solving equation (5). Further, although optimization is performed for each camera based on direct measurement by a surveying instrument, bundle adjustment of a plurality of cameras may be performed based on direct measurement by a surveying instrument. This is the evaluation function

can be treated as a problem to minimize.

(c) The camera position and orientation were determined by performing bundle adjustment using a combination of the trajectory of the color ball (a group of feature points whose positions are unknown) and direct measurement using a surveying instrument, and solving Equation (6).

In this experiment, the whole body movements of 12 players during a volleyball match were reconstructed using camera calibration data and a video motion capture system (see Non-Patent Document 1). In this experiment, we filmed a volleyball match held in an arena and captured the movements of volleyball players during the match. The motion capture camera system according to this embodiment can be applied to any motion measurement without being limited to the type of sport or location. Examples of target sports include soccer, futsal, rugby, baseball, volleyball, handball, tennis, and gymnastics. Examples of shooting locations include outdoor sports fields, indoor sports fields, outdoor courts, indoor courts, arenas, domes, and gymnasiums.

[B-5] Dynamic Calibration Calibration according to this embodiment is applied to video motion capture from multiple camera images for broadcasting and recording in which a cameraman manually controls focal length, pan/tilt, etc. I can do it. In this case, the optical parameters (lens distortion), internal parameters (focal length), and external parameters are all manually varied. These parameters are calculated using fixed feature points on elements forming a sports field or court (for example, points on a line or points on a goal) and feature points with unknown positions (for example, a ball in a ball game). All time parameters are dynamically obtained through optimization calculations.

First, before motion capture, position information of a plurality of immovable feature points in a global coordinate system is acquired and stored using a surveying instrument equipped with an angle measurement function (which may also be equipped with a distance measurement function). Fixed feature points are points on the elements forming the sports field or court, such as intersections (including corners) and intersections of lines on the field or court, and points on goal posts (intersections of posts and crossbars). (lower end of a post).

During motion capture, using the position information of the plurality of immobile feature points in each camera image and the position information of one or more feature points whose position information in the global coordinate system is unknown in each camera image, The camera parameters are obtained by optimization calculation. An example of a feature point whose position information in the global coordinate system is unknown is a ball in a ball game. Further, a mark at a specific position on a uniform worn by a player or a specific point on shoes may be used as a feature point whose position is unknown.

In the optimization calculation, empirical rules may be used to set the fluctuation range of some parameters to be small or to set the change rate of some parameters to be small. Alternatively, the camera mounter may be configured from a surveying instrument with an angle measurement function, and posture information of the camera mounter acquired using the surveying instrument may be used for optimization calculations. In the optimization calculation, the time series information of the measured values (angles) of the surveying instrument is acquired and stored, or the optimization calculation section receives the measured values of the surveying instrument in real time and calculates it in real time. Optimization calculations may be performed using

角度θⁱ _Bは角度

とする。各線分は次のようになる。

したがって、ベクトル

は、以下のようになる。

レーザ距離計により線分T_iA、T_iB の実測値がある場合には、その値も考慮して最適化によりベクトル

求める。[C] Obtaining the position and orientation of the camera mounter [C-1] Direct measurement using a surveying instrument
1) Definition of the relative position and orientation of measurement points on the survey pole by angle measurement:
Using the i-th leveled angle measurement meter, the relative position and orientation of the angle measurement meter and measurement points A and B on the similarly leveled surveying pole can be determined as follows according to FIG. Here, points O, A, and B are measurement points (high and low) on the pole, and point O is the origin of the global coordinate system (the intersection of the pole and the ground). Point T _i is the origin of the local coordinate system of the i-th goniometer, point Hi is the foot of the perpendicular line from T _i on the pole, and axes X _Ti , Y _Ti , X _Ti are the origin of the local coordinate system of the goniometer Point to each axis. Also, the angle θ ⁱ _A is the angle

Angle θ ⁱ _B is the angle

shall be. Each line segment is as follows.

Therefore, the vector

becomes as follows.

If there are actual measured values of the line segments T _i A and T _i B using a laser distance meter, the vectors are calculated by optimization taking those values into consideration.

demand.

を用いて、グローバル座標系に対する角度測量計T₀のローカル座標系の同次変換行列^O ₀Hを図１３にしたがい次のように定義できる。

また、グローバル座標系における他の角度測量計の位置・姿勢についても、次のように定める。

ここで、角度θ⁰ _Tiは角度

角度θⁱ _T0は角度

とする。[C-2] Let the angle surveyor T ₀ be the position/orientation base of the angle surveyor in the global coordinate system, and let the orientation of the local coordinate system of the angle surveyor T ₀ be the orientation of the global coordinate system, and the origin O on the surveying pole. Let be the origin of the global coordinate system. At this time, the vector obtained in the previous section

According to FIG. 13, the homogeneous transformation matrix ^O ₀ H of the local coordinate system of the goniometer T ₀ with respect to the global coordinate system can be defined as follows.

In addition, the positions and orientations of other angle measurement meters in the global coordinate system are also determined as follows.

Here, the angle θ ⁰ _Ti is the angle

Angle θ ⁱ _T0 is the angle

shall be.

When a theodolite is used as a camera mounter, the external parameter matrix ⁱ B _m representing its position and orientation can be defined as follows.

ここで、^Op_iPj は各測量マーカの点(j＝1,…,4)である。
w_frillはマーカ間距離の設計値、ⁱL_slopej、ⁱL_horizontalj、ⁱL_verticaljは、レーザ距離計で測ったマーカへの斜距離・水平距離・高低差である。また、角度ⁱθ_horizontalj、ⁱθ_verticaljは、角度距離計で測ったマーカへの方向角・高度角を指す。測量マーカ付きカメラマウンタトをカメラのマウンタとするとき、前節と同様に外部バラメータⁱB_mは次のように定義される。

[C-3] In the position/orientation diagram of the camera mounter with survey marker 14, let F _i be the origin of the local coordinate system of the camera mounter with survey marker. The position ^O p _Fi and orientation ^O _Fi R of this coordinate system with respect to the global coordinate system are determined by optimization as follows.

Here, ^O p _iPj is the point of each survey marker (j=1,...,4).
w _frill is the design value of the distance between markers, ^{and i} L _slopej , ⁱ L _horizontalj , ^{and i} L _verticalj are the oblique distance, horizontal distance, and height difference to the marker measured with a laser range finder. Further, the angles ⁱ θ _horizontalj and ⁱ θ _verticalj refer to the direction angle and altitude angle to the marker measured by the angular distance meter. When a camera mounter with a survey marker is used as a camera mounter, the external parameter ⁱ B _m is defined as follows, as in the previous section.

を用いて次のように変換できる。

相対外部パラメ―タ行列を第２手法で最適化すると、次の問題になる。

[C-4] Definition and optimization of relative external parameters:
In contrast to the camera's external parameter matrix ⁱ B, only the camera mounter's external parameter matrix ⁱ B _m can be directly measured by a surveying instrument. They are relative extrinsic parameter matrices

It can be converted as follows using .

When the relative external parameter matrix is optimized using the second method, the following problem arises.

If it is possible to perform not only on-site measurement but also prior calibration to determine only the relative extrinsic parameter matrix, it is preferable to determine the camera focal length and the relative extrinsic parameter matrix between the camera and the mounter by optimization. Determining and fixing the focal length in advance allows you to determine a more appropriate camera position on-site. At this time, if an angle measurement meter is used as a camera mounter, the focal length of the camera can be determined more precisely. The procedure will be described in detail. First, a global coordinate system is defined as described above using a leveled angle surveyor with respect to a leveled surveying pole. Next, move it at regular intervals (in 1° increments, within a range of ±10°) and photograph the surveying pole. This allows control points to be defined on the survey pole (see Figure 15). You can obtain control point positions guaranteed to the accuracy of the theodolite.

[D] Camera Calibration in a Camera System Using a Surveying Instrument as a Camera Mounter Calibration in an embodiment in which a surveying instrument is used as a camera mounter will be described in detail. The camera system includes a camera unit consisting of a plurality of cameras (for example, four to two), and each camera unit is mounted on a surveying instrument (theodolite or total station). Each camera unit is mounted on the movable part of the surveying instrument, and can rotate horizontally and vertically together with the movable part during angle measurement. In one embodiment, one camera unit is mounted on the total station and the other camera unit is mounted on the theodolite. Of course, multiple camera units may be mounted on multiple total stations. In the description of this section, i indicates the serial number of the theodolite, and j indicates the j-th camera mounted on the i-th theodolite. Note that in Section [C] above, i indicates the serial number of the camera, and that the mounter of the camera (even if it is a theodolite) is assigned the serial number of the camera.

[D-1] Theodolite coordinate system in the world coordinate system Obtaining a homogeneous transformation matrix from the world coordinate system to the theodolite coordinate system will be described. First, mount the theodolite and total station on a tripod. Set up each tripod so that the measurement range is included in the field of view of each camera. Level the theodolite and total station using a spirit level. Place a surveying pole in the center of the measurement range. A surveying pole is a well-known pole commonly used in surveying, and has, for example, red and white stripes 10 cm wide. Let the upper measurement point on the pole be A, and the lower measurement point B.

At each theodolite or each total station, aim at measurement point A on the pole and read Yaw _A and Pitch _A. Lock Yaw, aim at measurement point B, and read Pitch _B. Set Yaw _A as the initial Yaw angle of the theodolite or total station. In other words, Yaw _A resets Yaw.

In each theodolite or each total station, set the coordinate system as follows. The origin is the center of each theodolite or each total station, the z-axis is set at Yaw=Yaw _A and Pitch=0 (horizontal), and the x-axis is set at pitch=90° (vertical).

Let the coordinate system of the total station be the coordinate system {0}. Let the coordinate system be {1}, {2}, .., {M} counterclockwise based on the coordinate system of the theodolite. Here, M is the theodolite number. M=3.

Global coordinates {a} are set at measurement point B of the pole. The z-axis and x-axis are made to coincide with the z-axis and x-axis of the total station coordinate system {0}.

The Yaw angle between a camera mount (eg, one total station and multiple theodolites) is measured using the theodolite or each total station. For example, the Yaw angle from the coordinate system {0} to the coordinate system {1} is represented by α ₀₁ . In this measurement, the tip of the pendulum weight of the target theodolite or total station is used as the target (feature point). By measuring the Yaw angle between multiple camera mounters, the angle α _ij (i=0,…,M;J=0,…M;i≠j) is obtained.

を取得するUsing the fact that the distance between measurement point A and measurement point B is known, use trigonometry to transform the homogeneous transformation matrix from the world coordinate system to the theodolite coordinate system.

get

でセットされており、軸はz-x平面に対して対称である。各軸を軸

回りに9.65°で回転させる。[D-2] Internal parameters of camera Acquisition of internal parameters of the camera will be explained. Each camera unit has four cameras fixed on a horizontal plane that can rotate in unison with the Yaw rotation of the theodolite or total station. The camera is pivoted in a fan shape with a sector angle of 22.5°.

, and the axis is symmetrical about the zx plane. axis each axis

Rotate by 9.65°.

The camera is set as camera (i, j) with each total station and theodolite as i=0,...,M, and each camera on each total station and theodolite as j=0,1,2,3 clockwise on the Yaw axis. be indexed.

Set the camera wide open and focus on ∞. The amount of light is controlled by shutter speed. Show the checkerboard to the camera in 15 different positions and orientations. Obtain the camera's internal parameters P _ij (i=1,…,M;J=1,2,3,4) using OpenCV's cvCalibrateCamera2.

は、カメラ画像I_ijにおいて、以下の形式で現れる。

歪みフリー画像I_ij ^*は、P_ijを用いてI_ijから以下のような形式で生成され、

歪みフリー画像は、カメラ座標系の３次元位置

を

で示す。3D position in camera coordinate system

appears in the camera image I _ij in the following format.

The distortion-free image I _ij ^* is generated from I _ij using P _ij in the following format,

A distortion-free image is a 3D position in the camera coordinate system.

of

Indicated by

[D-3] Camera coordinate system in theodolite/total station coordinate system Obtaining a homogeneous transformation matrix from the theodolite coordinate system to the camera coordinate system will be described. Use a spirit level to set the pole vertically. For theodolite/total station (i=0,...,M), set the theodolite or total station in coordinate system {i} vertically using a spirit level. The height of the camera is, for example, 1.7 m from the horizontal plane, and the distance from the pole to the camera is, for example, 10 m. Set the pitch angle of the theodolite/total station to 0° (horizontal). Then, adjust the Yaw to aim at the center of the pole and reset the Yaw angle (0°).

として三角法で計算する。セオドライトないしトータルステーションの回転角度を、β₀＝-33.75°、β₁＝-11.25°、β₂＝11.25°、β₃＝33.75°、に設定する。カメラ(j=0,…,M)、及び、角度(k=0,…,20)について、座標系｛i｝のYawをγ_jkに設定し、ここで、

である。そして、角度(k=0,…,20)を変化させて、カメラ(i,j)の画像I_ijkを取得する。

上記２つの式を満たすように最適化計算を行って、セオドライト座標系からのカメラ座標系への同次変換行列

を取得する。同様に、次のカメラjについて同時変換行列を取得し、次のセオドライトについて同様に計算を行って同時変換行列を取得する。Determine two points A and B on the axis of the pole, and calculate the three-dimensional positions of points A and B.

Calculate using trigonometry as . The rotation angles of the theodolite or total station are set to β ₀ =-33.75°, β ₁ =-11.25°, β ₂ = 11.25°, and β ₃ = 33.75°. For the camera (j=0,…,M) and the angle (k=0,…,20), set Yaw of the coordinate system {i} to γ _jk , where:

It is. Then, by changing the angle (k=0,...,20), the image I _ijk of the camera (i, j) is obtained.

Optimization calculations are performed to satisfy the above two formulas, and a homogeneous transformation matrix from the theodolite coordinate system to the camera coordinate system is created.

get. Similarly, a simultaneous transformation matrix is obtained for the next camera j, and a similar calculation is performed for the next theodolite to obtain a simultaneous transformation matrix.

The results of the calibration according to this embodiment are expressed by the following three equations.

[E] Motion Capture System The motion capture system according to this embodiment will be described with reference to FIG. 16. The motion capture system is a so-called video motion capture system (see Non-Patent Document 1), which performs 3D reconstruction from joint positions estimated using deep learning from images from multiple cameras acquired by a motion capture camera system. There is no need to attach any markers or sensors to the target, and the measurement space is not limited. Motion capture is performed completely unrestricted from images from multiple RGB cameras, and in principle, if images can be captured from indoor living spaces to large outdoor sports fields, it is possible to measure motion. It's technology. The motion capture system described below is an example and does not limit the motion capture used with the motion capture camera system according to the present invention, and other techniques may be employed.

The number of objects included in one image is not limited. For example, in competitive sports, the actions of multiple people are photographed using a camera, so that each image includes multiple people. In each image, the joint positions of one person or an arbitrary number of people selected from the plurality of people are acquired. When multiple people are included in one image, for example, PAF and PCM (Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017, 2017.), it is possible to simultaneously obtain joint positions for each person. Furthermore, if the joint positions of each person are identified at the initial stage of measurement, then each person can be identified by tracking the joint positions with continuity between frames. In motion capture using a motion capture system, the object has a linked structure or an articulated structure. Typically the subject is a human with a skeletal structure, but the subject may also be a robot.

The video motion capture system uses a motion capture camera system that captures the motion of the target, and a color intensity that measures the degree of certainty of the positions of keypoints, including joint positions, based on the images captured by the camera system. A heat map acquisition section that acquires heat map information to be displayed; a joint position acquisition section that acquires target joint positions using the heat map information acquired by the heat map acquisition section; and a joint position acquisition section that acquires target joint positions using the heat map information acquired by the heat map acquisition section. A smoothing processing unit that smoothes the position, and a storage unit that stores the skeletal structure of the target body, time-series data of images acquired by the video system, time-series data of joint positions acquired by the joint position acquisition unit, etc. and a display that displays an image of the target captured by the camera system and a skeletal structure corresponding to the pose of the target.

A motion capture camera system is equipped with multiple cameras that capture the movements of a subject, and each camera outputs an RGB image at a predetermined frame rate. A plurality of camera images acquired at the same time are transmitted to the heat map acquisition unit. The heat map acquisition unit generates a heat map based on the RGB image. The heat map represents the spatial distribution of the likelihood of the location of feature points on the body. The heat map acquisition unit generates a two-dimensional or three-dimensional spatial distribution of the likelihood of the position of key points on the body including the positions of each joint based on the input image, and Display spatial distribution in heatmap format. The heat map acquisition unit typically uses a convolutional neural network (CNN) to extract the positions of feature points on the subject's body (typically joint positions) from a single input image into a heat map. Estimated as.

The generated heat map information is transmitted to the joint position acquisition section, and the joint position is acquired by the joint position acquisition section. The joint position acquisition unit estimates joint position candidates using the heat map information acquired from the heat map acquisition unit, and executes optimization calculations based on inverse kinematics using the joint position candidates to estimate the skeletal model. Update joint angles and joint positions. The joint position acquisition unit includes a joint position candidate acquisition unit that estimates joint position candidates based on heat map data, and an inverse kinematics unit that calculates joint angles by performing optimization calculations based on inverse kinematics using the joint position candidates. and a forward kinematics calculation section that calculates joint positions by performing forward kinematics calculations using the calculated joint angles. The acquired joint position data is stored in the storage unit as time series data of joint positions.

The acquired joint position data is transmitted to the smoothing processing section, and smoothed joint positions and joint angles are acquired. The smooth joint position acquisition unit of the smoothing processing unit smoothes the temporal movement of the joint positions by performing smoothing processing on the acquired joint positions using the joint positions in past frames. Make it. Optimization calculations based on inverse kinematics are performed again using the smoothed feature point positions to obtain the joint angles of the target, and forward kinematics calculations are performed using the obtained joint angles to determine the target joint angles. Get the joint positions of. The pose of the target is determined based on the smoothed joint positions or joint angles and the skeletal structure of the target's body, and the motion of the target consisting of time-series data of the pose is displayed on a display.

FIG. 17 illustrates the processing steps of motion analysis using motion capture. The motion of the target is acquired by motion capture according to this embodiment. Obtain time series data of the obtained joint angles and joint positions. Furthermore, based on this, the joint torque is obtained through inverse dynamics calculation, and the joint torque is used to optimize the wire tension in a musculoskeletal model equipped with wires that imitate muscles (quadratic programming or linear programming). method), the degree of muscle activity is calculated using the wire tension, a musculoskeletal image is assigned a color according to the degree of muscle activity, and a musculoskeletal image with the degree of muscle activity is visualized. Outputs images at a predetermined frame rate and displays them on a display as a video. In this way, it is possible to automatically and efficiently perform everything from photographing the movement of the target, obtaining the three-dimensional pose of the target during movement, and estimating and visualizing the muscle activity necessary for the movement.

Claims (24)

a plurality of camera units consisting of one or more cameras;
multiple camera mounters on which each camera unit is mounted;
Calibration means for acquiring camera parameters for three-dimensionally reconstructing each camera image;
one or more surveying instruments equipped with an angle measurement function;
A motion capture camera system equipped with
The camera parameters include external parameters representing the position and orientation of each camera in the global coordinate system,
The calibration means acquires the external parameters using the position and orientation of each camera mounter in the global coordinate system, and the position and orientation of the camera mounter in the global coordinate system are determined by the one or more surveying methods. obtained using a machine,
Motion capture camera system. The motion capture camera system according to claim 1, wherein one or more of the one or more surveying instruments has a distance measurement function. The motion capture camera system according to claim 1, wherein some or all of the plurality of camera mounters are the surveying instruments. 3. The motion capture camera system according to claim 1, wherein some or all of the plurality of camera mounters are provided with a plurality of markers that can be measured by the surveying instrument. 5. The motion capture camera system according to claim 4, wherein the camera mounter has a plane with three or more markers. The calibration means uses position information in a global coordinate system of a plurality of feature points located in the imaging space of each camera, and position information of each feature point in each camera image,
Position information of the plurality of feature points in a global coordinate system is obtained using the surveying instrument,
The motion capture camera system according to any one of claims 1 to 5. The calibration means uses the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system,
The motion capture camera system according to any one of claims 1 to 6. The calibration means includes means for acquiring the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system.
The motion capture camera system according to claim 7. The camera mounter is comprised of the surveying instrument,
the position and orientation of the camera in the camera mounter coordinate system are obtained using the surveying instrument;
The motion capture camera system according to claim 8. The calibration means obtains the external parameters by optimization calculation.
The motion capture camera system according to any one of claims 1 to 9. In the optimization calculation, position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.
Motion capture camera system according to claim 10 The camera parameters include optical parameters and internal parameters, and the calibration means obtains the external parameters, the optical parameters, and the internal parameters by optimization calculation. Motion capture camera system. a plurality of camera units consisting of one or more cameras;
multiple camera mounters on which each camera unit is mounted;
A calibration method for acquiring camera parameters for three-dimensionally reconstructing multiple camera images in a motion capture camera system equipped with
The camera parameters include external parameters representing the position and orientation of each camera in the global coordinate system,
obtaining the position and orientation of each camera mounter in a global coordinate system using one or more surveying instruments equipped with an angle measurement function;
obtaining the external parameters using the position and orientation of each camera mounter in a global coordinate system;
Calibration methods including. 14. The motion capture camera system according to claim 13, wherein one or more of the one or more surveying instruments has a ranging function. Any one of claims 13 and 14, wherein some or all of the plurality of camera mounters are the surveying instruments, and obtain the position and orientation of each camera mounter in a global coordinate system using survey data of a plurality of measurement points. The calibration method described in Section 1. Some or all of the plurality of camera mounters are equipped with a plurality of markers that can be surveyed by the surveying instrument, and the position and orientation of the camera mounter in the global coordinate system are acquired using survey data of the plurality of markers. The calibration method according to any one of claims 13 and 14. 17. The calibration method according to claim 16, wherein the camera mounter has a flat surface with three or more markers. In acquiring the external parameters, position information in a global coordinate system of a plurality of feature points located in the imaging space of each camera and position information of each feature point in each camera image are used,
Position information of the plurality of feature points in a global coordinate system is obtained using the surveying instrument,
The calibration method according to any one of claims 13 to 17. In acquiring the external parameters, the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system are used;
The calibration method according to any one of claims 13 to 18. including obtaining the position and orientation of each camera mounted on the camera mounter in each camera mounter coordinate system;
The calibration method according to claim 19. The camera mounter is comprised of the surveying instrument,
acquiring the position and orientation of the camera in the camera mounter coordinate system using the surveying instrument;
The calibration method according to claim 20. The calibration method according to any one of claims 13 to 21, wherein the external parameters are obtained by optimization calculation. In the optimization calculation, position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.
The calibration method according to claim 22. 24. The calibration method according to claim 22, wherein the camera parameters include optical parameters and internal parameters, and the external parameters, the optical parameters, and the internal parameters are obtained by optimization calculation.

Images