JP7427188B2 - 3D pose acquisition method and device - Google Patents

Description

The present invention relates to motion capture, and more particularly, to a 3D pose acquisition method and apparatus.

Motion capture is an essential technology for capturing and analyzing human motion, and is widely used in fields such as sports, medicine, robotics, computer graphics, and computer animation. Optical motion capture is well known as a motion capture method. Optical motion capture involves attaching multiple optical markers coated with retroreflective material to the subject's body and photographing the subject's movements using multiple cameras such as infrared cameras. Get the target behavior.

Another known motion capture method is to attach a so-called inertial sensor such as an acceleration sensor, a gyroscope, or a geomagnetic sensor to the subject's body to obtain motion data of the subject.

The above optical methods and methods using inertial sensors can obtain highly accurate motion data, but since it is necessary to attach multiple markers and multiple sensors to the subject's body, it takes time to prepare for motion measurement. The disadvantages are that it is labor-intensive, and that the movement of the subject is restricted, potentially interfering with natural movement. Additionally, because the systems and devices are expensive, there is a problem that the technology is not widely available for general use. Optical motion capture has a drawback in that it is difficult to obtain motion data of movements outdoors or in large spaces because the measurement locations are limited.

So-called markerless motion capture, which does not require optical markers or sensors, is also known. Examples of motion capture using a camera and a depth sensor include Non-Patent Documents 1 to 5. However, these methods have difficulty measuring objects that are outdoors, far away, or moving at high speed because the laser used to acquire depth data has low temporal and spatial resolution.

As image recognition methods and accuracy have dramatically improved through deep learning, video motion capture has also been proposed, which analyzes RGB images from one viewpoint to obtain motion data (Non-Patent Documents 6 to 11). This method can be used outdoors and at long distances, and by selecting camera performance, it is possible to increase temporal and spatial resolution at a relatively low cost. However, when measuring from a single viewpoint, it is often difficult to estimate the pose of the target due to occlusion, and the accuracy is not as high as optical motion capture using multiple cameras.

Research is also being conducted to use deep learning to find a human figure from a single video image and generate a heat map that represents the spatial distribution of the likelihood of joint positions. One representative study is OpenPose (Non-Patent Document 12). OpenPose can estimate keypoints of multiple people, such as wrists and shoulders, in real time from a single RGB image. This is a study by Wei et al. that estimates the position of each joint by generating Part Confidence Maps (PCM) of each joint from a single RGB image using CNN (Non-Patent Document 13), and Part Affinity Fields by Cao et al. In a study (Non-Patent Document 14), the above method was extended to estimate joint positions in real time for multiple people by calculating a vector field called (PAF) that represents the direction of adjacent joints. It was developed based on In addition, various methods have been proposed to obtain a heat map (PCM in OpenPose), which is a spatial distribution of likelihood that represents the certainty of each joint position, and a method for estimating human joint positions from input images. A contest is also held to compete for accuracy (Non-Patent Document 15).

The present inventors have proposed video motion capture (Non-Patent Document 16), which is a method for three-dimensional reconstruction of joint positions using heat map information, and which measures motion with high accuracy like optical motion capture. is suggesting. Video motion capture captures motion from images from multiple RGB cameras in a completely unrestricted manner.In principle, if images can be captured, motion can be measured from indoor spaces to large outdoor sports fields. This is a technology that makes it possible.

Here, there are often situations where multiple people exercise in the same space (a typical example is competitive sports), and it is important to measure the movements of multiple people in motion capture. There are two known methods for estimating the pose of each person based on an image that includes multiple people. One is the bottom-up type, which estimates the joint positions of multiple people in an image using heat map information or PCM, and calculates the pose of each person using a vector field representing the orientation of the limbs (for example, PAF). Detect. The other is a top-down type, which searches the area of each person in the image, sets a bounding box, and estimates the joint position of each person within each bounding box using heat map information. Detect the pose of each person. There are several software programs that acquire heat map information from image information within a bounding box (Non-Patent Documents 18 to 20). Even if multiple people are included in the bounding box, a heat map of the single most appropriate person (for example, a person closer to the center of the image, a person whose body parts are all within the bounding box, etc.) It is trained to obtain information or PCM.

However, if the detection of the human region in the input image is not performed properly, it will not be possible to determine an appropriate bounding box (for example, the wrist or ankle will be cut in the area surrounded by the bounding box), and the joints of the person will be This will lead to erroneous position estimation. Furthermore, as the resolution of the input image becomes higher, the calculation time for detecting a human region in the input image increases. Therefore, it is a challenge to appropriately determine a bounding box with a smaller amount of calculation. Furthermore, this problem is not limited to pose estimation in a multi-person environment. Even if there is only one object in an image, calculation time can be reduced by calculating heat map information based on pixel information in a limited area surrounded by a bounding box.

Z. Zhang. Microsoft kinect sensor and its effect. IEEE MultiMedia, 19(2):4-10, Feb 2012. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. Real-time human pose recognition in parts from single depth images. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2011, CVPR '11, pages 1297-1304, Washington, DC, USA, 2011. IEEE Computer Society. J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan. Scanning 3d full human bodies using kinects. IEEE Transactions on Visualization and Computer Graphics, 18(4):643-650, April 2012. Luciano Spinello, Kai O. Arras, Rudolph Triebel, and Roland Siegwart. A layered approach to people detection in 3d range data.In Proceedings of the Twenty-Fourth AAAI Conference on ArtificialIntelligence, AAAI'10, pages 1625-1630. AAAI Press, 2010. A. Dewan, T. Caselitz, G. D. Tipaldi, and W. Burgard. Motion-based detection and tracking in 3d lidar scans. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 4508-4513,May 2016. C. J. Taylor. Reconstruction of articulated objects from point correspondences in a single uncalibrated image. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000, volume 1, pages 677-684 vol.1, 2000. I. Akhter and M. J. Black. Pose-conditioned joint angle limits for 3d human pose reconstruction. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2015, pages 1446-1455, June 2015. Dushyant Mehta, Helge Rhodin, Dan Casas, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation using transfer learning and improved CNN supervision. The Computing Research Repository, abs/1611.09813, 2016. Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shaei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with a single RGB camera. Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. arXiv:1712.06584, 2017. Xiao Sun, Jiaxiang Shang, Shuang Liang, and Yichen Wei. Compositional human pose regression. The Computing Research Repository, abs/1704.00159, 2017. Openpose. https://github.com/CMU-Perceptual-Computing-Lab/openpose. Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2016, 2016. Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017, 2017. http://cocodataset.org/#keypoints-leaderboard T. Ohashi, Y. Ikegami, K. Yamamoto, W. Takano and Y. Nakamura, Video Motion Capture from the Part Confidence Maps of Multi-Camera Images by Spatiotemporal Filtering Using the Human Skeletal Model, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, 2018, pp. 4226-4231. K. Ayusawa and Y. Nakamura. Fast inverse kinematics algorithm for large dof system with decomposed gradient computation based on recursive formulation of equilibrium. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3447-3452, Oct 2012. Y. Chen, Z.Wang, Y. Peng, Z. Zhang, G. Yu, and J. Sun. Cascaded Pyramid Network for Multi-Person Pose Estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. K. Sun, B. Xiao, D. Liu, and J. Wang. Deep High-Resolution Representation Learning for Human Pose Estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. B. Xiao, H. Wu, and Y. Wei. Simple Baselines for HumanPose Estimation and Tracking. In European Conference on Computer Vision (ECCV), 2018.

An object of the present invention is to obtain a pose with high accuracy even when an image includes a plurality of objects.

A 3D pose acquisition method according to the present invention is a 3D pose acquisition method of a target in motion capture using multiple cameras, comprising:
The object has a plurality of feature points on its body including a plurality of joints, and the 3D pose of the object is specified by the positions of the plurality of feature points,
Determine a bounding box surrounding the object on each camera image at frame t+1 using one or more 3D poses of the object at frame t and/or one or more frames before frame t. ,
In each camera image of frame t+1, a spatial distribution of the likelihood of the certainty of the position of each feature point is obtained using the image information in the bounding box, and based on the spatial distribution of the likelihood, Obtain the position of each feature point.

In one aspect, the image includes a plurality of objects,
Determine the bounding box for the selected object,
The position of each feature point of the one target in frame t+1 is obtained using the image information within the bounding box.
In one aspect, the 3D pose of each object is acquired in parallel in each frame.
Including multiple objects in an image is one aspect and does not limit the invention. The present invention also applies to pose acquisition using images containing only one object.

In one aspect, determining the bounding box comprises:
predicting a 3D pose at frame t+1 using the one or more 3D poses;
projecting the predicted 3D pose onto each camera image to predict a 2D pose in each camera image of frame t+1;
The position and size of the bounding box in each camera image are determined using position information of a plurality of feature points that specify the predicted 2D pose.
In one aspect, the location of the bounding box is determined by the center of the bounding box.
One aspect includes determining the slope of the bounding box for each camera image.
The slope of the bounding box can be obtained, for example, by obtaining the slope of the target pose with respect to the upright pose in a frame before the input image, or by obtaining the slope of the predicted target pose with respect to the upright pose in the input image. This can be determined by:

In one aspect, in the subject, the distance between adjacent joints is obtained as a constant,
Using the spatial distribution of the likelihood, obtain position candidates for one or more feature points corresponding to each feature point,
Obtaining each joint angle of the target by performing optimization calculation based on inverse kinematics using the position candidates of the feature points and the multi-joint structure of the target,
By performing forward kinematic calculation using the joint angles, the positions of feature points including the target joints are obtained.

In one aspect, the acquisition of the feature point position candidates involves setting a search range for the feature point position candidates using each feature point acquired in one or more frames;
One or more feature point position candidates for each joint position in frame t+1 are acquired using the spatial distribution of likelihoods acquired in frame t+1.
In one aspect, the search range is a space in the vicinity of the predicted position of each feature point in frame t+1.
In one aspect, the search range is a space in the vicinity of the position of each feature point acquired in frame t.
In one aspect, the search range for the feature point position candidates in acquiring the feature point position candidates is, in addition to or instead of the position of each feature point acquired in frame t, This is the neighborhood space of the position of the feature point acquired in one or more frames and/or one or more frames after frame t+2.
In one aspect, the search range is a predetermined number of points distributed three-dimensionally at predetermined intervals around the position of the feature point.
In one aspect, the spatial distribution of the likelihood is used in the optimization calculation based on the inverse kinematics.

In one aspect, the position of the feature point is smoothed in the time direction using the positions of a plurality of feature points acquired in a plurality of other frames,
Each joint angle of the target is obtained by performing an optimization calculation based on inverse kinematics using the positions of the smoothed feature points and the multi-joint structure of the target.
In one aspect, the position of a feature point including the target joint is obtained by performing forward kinematics calculation using the joint angle.

A 3D pose acquisition device according to the present invention is a 3D pose acquisition device for a target in motion capture using multiple cameras, and includes:
The object has a plurality of feature points on its body including a plurality of joints, and the 3D pose of each object is specified by the position of the plurality of feature points,
The device includes a storage unit and a processing unit,
The storage unit stores time-series data of calculated 3D poses of each object,
The processing unit includes:
Using one or more 3D poses of an object in frame t and/or one or more frames before frame t, create a bounding box surrounding the object in each camera image at frame t+1. decided,
In each camera image of frame t+1, a spatial distribution of the likelihood of the certainty of the position of each feature point is obtained using the image information in the bounding box, and based on the spatial distribution of the likelihood, The 3D position of each feature point is acquired.
Some of the above aspects of the 3D pose acquisition method are applied to a 3D pose acquisition device.

The present invention also provides a computer program that causes a computer to function as a storage unit and a processing unit of the above device, or a computer readable medium storing the computer program, or a computer program or the computer program that causes a computer to execute the above method. may be provided as a computer-readable medium having the information stored thereon.

The present invention also includes:
predicting a 3D pose of the object in the next frame using one or more acquired 3D poses;
projecting the predicted 3D pose onto the image plane of the camera to predict the 2D pose of the object on the image plane of the camera in the next frame;
Determining the position and size of a bounding box in the camera image of the next frame using position information of a plurality of feature points that specify the predicted 2D pose;
A method for determining a bounding box in top-down pose estimation is disclosed.

The present invention also includes:
multiple cameras and
a bounding box determination section;
3D pose acquisition unit,
A motion capture system equipped with
The 3D pose acquisition unit acquires a spatial distribution of the likelihood of the position of each feature point in each camera image using the image information in the bounding box, and based on the spatial distribution of the likelihood, Obtain a 3D pose,
The bounding box determining unit includes:
predicting a 3D pose of the object in the next frame using one or more acquired 3D poses;
projecting the predicted 3D pose onto the image plane of the camera to predict the 2D pose of the object on the image plane of the camera in the next frame;
Determining the position and size of a bounding box in the camera image of the next frame using position information of a plurality of feature points that specify the predicted 2D pose;
A motion capture system is disclosed.
In one aspect, the motion capture system further includes a 3D position acquisition unit different from the 3D pose acquisition unit, and the 3D position acquisition unit acquires the 3D position of the target without using a bounding box. be.
In one aspect, the 3D position acquisition unit is a 3D pose estimator.

In the present invention, the 3D pose of the target in the next frame is predicted, and the bounding box in the next frame is determined to include the predicted 2D pose on the image plane of the camera corresponding to the predicted 3D pose. can be determined.

1 is an overall diagram of a motion capture system. 3 is a flowchart showing a process of processing an input image. 12 is a flowchart showing the processing steps of camera calibration and acquisition of the initial posture and inter-joint distance of the skeletal model. The left figure shows the skeletal model according to this embodiment, and the right figure shows the feature points of OpenPose. 7 is a flowchart showing the processing steps of a joint position candidate acquisition unit. FIG. 3 is a diagram illustrating a point group that is a search range. 12 is a flowchart illustrating processing steps of a joint position acquisition unit (when using different grid intervals). 3 is a flowchart showing a process of rotating an input image to obtain a PCM. 7 is a flowchart showing video motion capture of multiple people according to the present embodiment. 3 is a flowchart illustrating an input image processing step according to the present embodiment. FIG. 3 is a diagram showing determination of a bounding box according to the present embodiment. 7 is a flowchart showing processing steps of a joint position candidate acquisition unit according to the present embodiment. FIG. 3 is a diagram illustrating how feature points are detected when occlusion occurs.

The description of embodiments of the invention consists of the following two chapters.
[I] Motion capture system [II] Motion capture for multiple targets In Chapter I, a video motion capture system will be described in detail, and in Chapter II, an embodiment in which the method according to Chapter I is applied to motion capture for multiple targets will be described. . Those skilled in the art will understand that the disclosures in Chapter I and Chapter II are closely related to each other, and that matters described in either chapter can be appropriately incorporated into the other chapter. Note that the formula numbers are assigned independently in Chapter I and Chapter II.

[I] Motion capture system [A] Overall configuration of motion capture system The motion capture system is a so-called video motion capture system (see Non-Patent Document 16), and is estimated using deep learning from images of multiple target cameras. Three-dimensional reconstruction is performed from joint positions, and there is no need to attach any markers or sensors to the target, and the measurement space is not limited. This technology performs motion capture completely unrestricted from images from multiple RGB cameras, and in principle, if images can be captured, it is possible to measure motion from indoor spaces to large outdoor sports fields. be.

As shown in FIG. 1, the motion capture system according to the present embodiment includes a video acquisition unit that acquires the motion of a target, and keypoints including joint positions based on the images acquired by the video acquisition unit. a heat map acquisition unit that acquires heat map information that displays the degree of certainty of a position in color intensity; and a joint position acquisition unit that acquires target joint positions using the heat map information acquired by the heat map acquisition unit. , a smoothing processing unit that smoothes the joint positions acquired by the joint position acquisition unit, the skeletal structure of the target body, time series data of images acquired by the video acquisition unit, and joints acquired by the joint position acquisition unit. It includes a storage unit that stores time-series position data, etc., and a display that displays an image of the target acquired by the video acquisition unit, a skeletal structure corresponding to the pose of the target, etc. Since the characteristic points on the subject's body are mainly joints, the word "joint" is used to represent the characteristic points in this specification and drawings, but "joints" and key points will be explained later. Please note that this is not a complete correspondence.

The hardware of the motion capture system according to this embodiment includes a plurality of cameras constituting the video acquisition unit, one or more local computers that acquire camera images, and one or more local computers connected to them via a network. Consists of a computer and one or more displays. Each computer includes an input section, a processing section, a storage section (RAM, ROM), and an output section. In one embodiment, one camera is associated with one local computer to acquire camera images, and a heat map acquisition section is configured at the same time, and then the joint position acquisition section, the smoothing processing section, and the storage section are configured. consists of one or more computers connected via a network. In another embodiment, a local computer connected to the camera compresses the image as necessary and transmits it over a network, and the connected computer performs the heat map acquisition section, the joint position acquisition section, and the smoothing process. section, which constitutes the storage section.

Each camera is synchronized, and each camera image acquired at the same time is transmitted to a corresponding heat map acquisition section, and a heat map is generated by the heat map acquisition section.
The heat map represents the spatial distribution of the likelihood of the location of feature points on the body. The generated heat map information is transmitted to the joint position acquisition section, and the joint position is acquired by the joint position acquisition section. The acquired joint position data is stored in the storage unit as time series data of joint positions. The acquired joint position data is transmitted to the smoothing processing section, and smoothed joint positions and joint angles are acquired. The pose of the target is determined based on the smoothed joint positions or joint angles and the skeletal structure of the target's body, and the motion of the target consisting of time-series data of the pose is displayed on a display.

The video acquisition unit includes a plurality of cameras, which are synchronized using, for example, an external synchronization signal generator. Note that the method of synchronizing multiple camera images is not limited. A plurality of cameras are arranged to surround the object, and a multi-view video of the object is obtained by simultaneously photographing the object with all or some of the cameras. For example, a 60fps, 1024×768 RGB image is acquired from each camera, and the RGB image is transmitted to the heat map acquisition unit in real time or non-real time.

In this embodiment, the video acquisition unit is composed of a plurality of cameras, and a plurality of camera images acquired at the same time are transmitted to the heat map acquisition unit. The heat map acquisition unit generates a heat map based on the image input from the video acquisition unit. In this motion capture system, the number of objects included in one image is not limited. Motion capture for multiple objects will be explained in the next chapter.

In motion capture using a motion capture system, the object has a linked structure or an articulated structure. Typically, the subject is a human being, and the multi-jointed structure is a skeletal structure of the body. Note that if the learning data used by the heat map acquisition unit can be prepared according to the target, it can also be applied to targets other than humans (for example, non-human animals and robots).

The storage unit stores measurement data and processing data. For example, time series data of images acquired by the moving image acquisition section, joint position data and joint angle data acquired by the joint position acquisition section are stored. The storage unit further stores smoothed joint position data, smoothed joint angle data acquired by the smoothing processing unit, heat map data generated by the heat map acquisition unit, and other data generated in the processing process. It's okay.

The storage unit further stores data for determining the skeletal structure of the subject's body. This data includes a file that defines a skeletal model of the body and distance data between adjacent joints of the object. The joint angles and pose of the object are determined from the position of each joint of the skeletal model, which is a multi-jointed body. The skeletal model used in this embodiment is shown in the left diagram of FIG. The skeletal model shown in the left diagram of FIG. 4 has 40 degrees of freedom, but this skeletal model is just an example. As described below, a constant representing the distance between adjacent joints of a target can be obtained during initial setup of motion capture. The distance between each joint of the target may be obtained in advance using another method, or the distance between the joints that has already been obtained may be used. In the present embodiment, by using the skeletal structure data of the target body, it is possible to provide a constraint unique to the skeletal structure in which the distance between adjacent joints remains unchanged over time in calculating joint positions.

The display displays a video of the target acquired by the video acquisition unit, a time-series skeletal image representing the pose of the target acquired by motion capture, and the like. For example, in the processing unit of a computer, skeletal image data (pose of the object) is generated for each frame using time series data of the object-specific skeletal structure, calculated joint angles, and joint positions, and the skeletal image data is output at a frame rate of , and display it on the display as a video.

[B] Heat map acquisition unit The heat map acquisition unit obtains a two-dimensional or three-dimensional spatial distribution of the likelihood of the position of key points on the body, including the positions of each joint, based on the input image. and display the spatial distribution of the likelihood in a heat map format. A heat map is a spatial display of values that spread and change in color intensity, like a temperature distribution, and allows visualization of likelihood. The likelihood value is, for example, 0 to 1, but the scale of the likelihood value is arbitrary. In this embodiment, the heat map acquisition unit stores the spatial distribution of the likelihood of the position of feature points on the body including each joint, that is, heat map information (each pixel of the image holds a value representing the likelihood). It is only necessary that the heat map has been obtained, and it is not necessarily necessary to display the heat map.

The heat map acquisition unit typically uses a convolutional neural network (CNN) to extract the positions of feature points on the subject's body (typically joint positions) from a single input image into a heat map. Estimated as. A convolutional neural network (CNN) has an input layer, a hidden layer, and an output layer.The middle layer is constructed by deep learning using training data of the locations of two-dimensional mappings of feature points onto images. ing.

In this embodiment, the likelihood acquired by the heat map acquisition unit is given to each pixel on a two-dimensional image, and the three-dimensional existence of the feature point is determined by integrating heat map information from multiple viewpoints. Information on the certainty of location can be obtained.

As the heat map acquisition unit, OpenPose (Non-Patent Document 12), which is open software, can be exemplified. In OpenPose, 18 keypoints are set on the body (see the right diagram in Figure 3). Specifically, the 18 feature points consist of 13 joints, a nose, left and right eyes, and left and right ears. OpenPose uses a trained convolutional neural network (CNN) to generate Part Confidence Maps (PCM) of 18 key points on the body from each RGB image captured by multiple synchronized cameras. Or generate it in real time and display it in heatmap format. In this specification, the term PCM may be used to refer to the spatial distribution or heat map of the likelihood of the location of feature points on the body, but the Please note that this is not intended to limit the index representing the spatial distribution of likelihood to PCM.

Other methods than OpenPose can be used in the heat map acquisition section. Various methods have been proposed to obtain a heat map that represents the certainty of the location of feature points on a target's body. For example, it is also possible to adopt a method that won a high rank in the COCO Keypoints challenge (Non-Patent Document 15). Alternatively, a convolutional neural network (CNN) may be constructed by creating a unique learning device for the heat map acquisition section.

[C] Initial settings of the motion capture system according to the present embodiment With reference to FIG. Explain acquisition.

ここで、Ｋ_iは焦点距離や光学的中心等の内部パラメータであり、Ｒ_i、ｔ_iは、それぞれ、カメラの姿勢・位置を表す外部パラメータである。カメラキャリブレーションは、既知の形状や寸法のキャリブレーション器具（チェッカーボードやキャリブレーションワンド等）を複数台のカメラで撮影することで行うことが可能である。歪みパラメータは、内部パラメータと同時に取得され得る。カメラの撮影空間が広域空間の場合には、上記キャリブレーション器具に代えて、例えば、計測領域全体に亘って球体を移動させながら複数台のカメラで撮影し、各カメラ画像中の球体の中心座標を検出するようにしてもよい。各カメラ画像中の球体の中心座標を用いて、バンドル調整によって、カメラの姿勢及び位置を最適化することで外部パラメータを取得する。なお、内部パラメータは、キャリブレーション器具等を用いることで事前に取得できるが、内部パラメータの一部あるいは全部を、最適化計算によって、外部パラメータと同時に取得してもよい。 [C-1] Camera Calibration In motion capture using multiple cameras, it is necessary to obtain camera parameters for three-dimensional reconstruction of multiple camera images. A matrix M _i for projecting an arbitrary point on the three-dimensional space onto the image plane of camera i is expressed as follows.

Here, K _i is an internal parameter such as focal length and optical center, and R _i and t _i are external parameters representing the attitude and position of the camera, respectively. Camera calibration can be performed by photographing a calibration instrument (checkerboard, calibration wand, etc.) with a known shape and dimensions using a plurality of cameras. The distortion parameters may be obtained simultaneously with the intrinsic parameters. If the shooting space of the camera is a wide area, instead of using the above-mentioned calibration equipment, for example, the center coordinate of the sphere in each camera image is taken by multiple cameras while moving the sphere over the entire measurement area. may be detected. Using the center coordinates of the sphere in each camera image, the external parameters are obtained by optimizing the posture and position of the camera through bundle adjustment. Note that the internal parameters can be obtained in advance by using a calibration instrument or the like, but some or all of the internal parameters may be obtained simultaneously with the external parameters by optimization calculation.

３次元状の任意の点をカメラiの撮影面のピクセル位置に変換する関数（行列）μ_i は記憶部に格納される。 When the projection matrix M _i is obtained for each camera, the pixel position when a point X in the three-dimensional space is projected onto each camera image plane is expressed as follows.

A function (matrix) μ _i that converts a three-dimensional arbitrary point into a pixel position on the imaging plane of camera i is stored in the storage unit.

本実施形態に係る骨格モデルの関節と、OpenPoseにおける１８個の特徴点は完全に一致してはいない。例えば、骨格モデルのpelvis(base body)、 waist、 chest、 right clavicle、left clavicle、Headに対応する特徴点は、OpenPoseには存在しない。なお、本実施形態に係る骨格モデルにおける関節、OpenPoseにおける１８個の特徴点は、共に、身体上の特徴点の代表的な特徴点であって、可能性のある全ての特徴点を網羅しているものではない。例えば、さらに詳細な特徴点を設定してもよい。あるいは、全ての身体上の特徴点が関節であってもよい。OpenPoseの１８個の特徴点だけでは決まらない関節角度については、可動範囲などの制約を考慮した最適化の結果として決定される。なお、骨格モデルの関節と、尤度の空間分布が取得される特徴点と、が初めから対応している場合には、この対応づけは不要である。 [C-2] Correspondence between the skeletal model and the feature points on the body for which a heat map is generated (excluding a, o, p, q, r). The correspondence relationship is shown in Table 1.

The joints of the skeletal model according to this embodiment and the 18 feature points in OpenPose do not completely match. For example, feature points corresponding to the pelvis (base body), waist, chest, right clavicle, left clavicle, and head of the skeleton model do not exist in OpenPose. Note that the joints in the skeletal model according to this embodiment and the 18 feature points in OpenPose are representative feature points on the body, and cover all possible feature points. It's not something that exists. For example, more detailed feature points may be set. Alternatively, all the feature points on the body may be joints. Joint angles that cannot be determined using only OpenPose's 18 feature points are determined as a result of optimization that takes into account constraints such as range of motion. Note that if the joints of the skeletal model and the feature points from which the spatial distribution of likelihood is obtained correspond from the beginning, this correspondence is not necessary.

[C-3] Obtain the initial posture and distance between joints of the skeletal model. Obtain the initial posture that is the starting point for motion measurement of the target. In this embodiment, the inter-joint distance and initial posture are estimated from the pixel positions of feature points calculated by applying OpenPose to the distortion aberration corrected image. First, an initial heat map is obtained based on the initial images obtained by each camera. In this embodiment, we consider a ray from each camera that connects the optical center of the camera and the pixel position of the center of gravity of the initial heat map of each feature point calculated from OpenPose, and calculate the length of the common perpendicular line of the rays of the two cameras. Determine the minimum two units, and when the length of the common perpendicular line is less than a predetermined threshold (for example, 20 mm), determine the midpoint of the two legs of the common perpendicular line to be the position of the three-dimensional feature point. This is used to obtain the inter-joint distance and initial posture of the skeletal model.

Those skilled in the art can use various methods for estimating the initial positions of feature points. For example, by using the position of a corresponding point on each camera image and camera parameters, the initial position of a feature point in a three-dimensional space can be estimated by a DLT (Direct Linear Transformation) method. Since three-dimensional reconstruction using the DLT method is known to those skilled in the art, detailed explanation will be omitted.

Optimization calculations based on inverse kinematics require constant distances between adjacent feature points (interjoint distances), that is, link lengths, but since link lengths differ for each target, the link length of the skeletal model is determined for each target. Calculate the length. In order to improve the precision of motion capture according to this embodiment, it is desirable to perform scaling for each object. The skeletal model is a model of the standard human skeletal structure, and is scaled for the whole body or for each part to generate a skeletal model that fits the body shape of the subject.

なお、リンク長は他の手法で取得してもよいし、予め取得したものを用いてもよい。あるいは、対象者に固有の骨格モデルが得られている場合には、それを用いてもよい。 In this embodiment, each link length of the skeletal model is updated based on the obtained initial posture. With respect to the initial link length of the skeletal model, a scaling parameter used for updating the link length is calculated from the position of each feature point based on the correspondence shown in FIG. 4 and Table 1. Among the link lengths in the left diagram of FIG. Since there are no points, the scale parameters cannot be determined in the same way. Therefore, the length is determined using other link length scale parameters. In this embodiment, since the human skeleton basically has a symmetrical length, each scaling parameter is calculated from the average of the left and right sides so that they are equal, and the initial link lengths of the skeletal model are equal on the left and right sides. . Regarding the calculation of the scaling parameter between Neck and Head, the scaling parameter is calculated using the midpoint of the feature point positions of both ears as the location where the head joint exists. Each link length of the skeletal model is updated using the obtained scaling parameters. Regarding calculation of the positions of the nose, eyes, and ears, the positions of each virtual joint (nose, eyes, ears) are calculated based on the correspondence shown in Table 2.

Note that the link length may be obtained using another method, or may be obtained in advance. Alternatively, if a skeletal model unique to the subject is available, that may be used.

[D] Joint position acquisition unit The joint position acquisition unit estimates joint position candidates using the heat map information (the spatial distribution of the likelihood of the certainty of the position of each feature point) acquired from the heat map acquisition unit, The feature is that the joint angles and joint positions of the skeletal model are updated by performing optimization calculations based on inverse kinematics using the joint position candidates. The joint position acquisition unit includes a joint position candidate acquisition unit that estimates joint position candidates based on heat map data, and an inverse kinematics unit that calculates joint angles by performing optimization calculations based on inverse kinematics using the joint position candidates. and a forward kinematics calculation section that calculates joint positions by performing forward kinematics calculations using the calculated joint angles.

The joint position acquisition unit according to the present embodiment sets the search range of each joint position candidate using the joint positions acquired in one or more frames, and thereby calculates the likelihood acquired in frame t+1. A joint position candidate in frame t+1 is obtained using the spatial distribution of . Examples of the search range include a space in the vicinity of the joint position acquired in frame t, or a space in the vicinity of the predicted position of the joint position predicted in frame t+1. The latter is advantageous when the target is moving quickly. This chapter explains the former search range, and the latter search range will be explained in the next chapter.

と表す。例えば、^t P ⁿを中心とした図６のような間隔sの11×11×11（k=5）の格子状の点を考える。格子点の距離ｓは画像ピクセルの大きさとは無関係である。 The joint position candidate acquisition unit according to the present embodiment calculates joint positions and joint angles of the current frame (frame t+1) using joint position data of the previous frame (frame t). Video motion capture of all T frames is performed by repeating the process of obtaining the joint angles and positions of frame t+1 from the joint positions of frame t until t=T. Since the change in joint position in one frame is minute, if the three-dimensional coordinates of the joint position of joint n in frame t are ^t P ⁿ and the joint position in frame t+1 is ^t+1 P ⁿ , then ^t+1 P ⁿ is considered to exist near ^t P ⁿ . Therefore, consider ^three (2k+1) grid points (k is a positive integer) with an interval s spread around ^t P ⁿ , and define the set (grid space) as

Expressed as For example, consider an 11×11×11 (k=5) grid of points with spacing s as shown in FIG. 6 centered at ^t P ⁿ . The grid point distance s is independent of the size of the image pixel.

The search range based on the joint position ^t P ⁿ in frame t is, for example, a group of points in the neighborhood space of the joint position ^t P ⁿ , and is determined by the total number of points (2k+1) ³ in the neighborhood space and the interval s between the points. be done. In determining the search range, although a cube is shown in FIG. 6, the shape of the search range is not limited, and the search may be performed in a spherical range, for example. Alternatively, the search may be performed by narrowing the search range to a rectangular parallelepiped shape or a long sphere, or by setting the center point of the search range to a different point from ^t P ⁿ based on changes in joint positions in past frames.

The search range (for example, center point, parameter k, and search width s) can be appropriately set by those skilled in the art. The search range may be changed depending on the type of movement of the target. Furthermore, the search range may be changed depending on the speed and acceleration of the object's movement (for example, the speed at which the object's pose changes). Furthermore, the search range may be changed depending on the frame rate of the camera that takes the image. Furthermore, the search range may be changed for each joint part.

によって求められる。この計算をn_j個（OpenPoseの場合は１８個）の関節全てにおいて実行する。 Note that all points in the grid space ^t L ⁿ can be transformed to pixel coordinates in the projection plane of any camera using the function μ _i . ^Let μ ⁱ be the function that converts one point ^t L ⁿ _{a, b, c} at t L n into a pixel position on the image plane of camera _i , and let ^{t + be the function that obtains the PCM value at frame t+1} from that pixel position. ¹ S ⁿ _i , the maximum point of the sum of PCM values calculated from n _c cameras can be considered to be the most probable position of joint n in frame t+1, and ^t+1 P ⁿ _{The key} is

It is determined by This calculation is executed for all n _j (18 in the case of OpenPose) joints.

なお、逆運動学に基づく最適化計算における各関節の重み^t+1Ｗⁿには

と規定されるように、各関節の予測位置におけるPCM値の和を用いる。
関節位置取得部でリアルタイムあるいは非リアルタイムで取得された各関節位置は、関節位置の時系列データとして記憶部に格納される。本実施形態では、関節位置取得部でリアルタイムあるいは非リアルタイムで取得された各関節位置は平滑化処理部によって平滑化処理されて、平滑化関節位置が生成される。 Then, focusing on the fact that each joint position t+1 P ⁿ of frame ^t+1 is a function of joint angle ^t+1 Q, as shown in equation (3), an optimization calculation based on inverse kinematics is performed. The joint angle ^t+1 Q is calculated, and the joint position ^t+1 P ⁿ is calculated by forward kinematics calculation.

In addition, the weight ^t+1 W ⁿ of each joint in the optimization calculation based on inverse kinematics is

The sum of the PCM values at the predicted position of each joint is used as defined below.
Each joint position acquired by the joint position acquisition unit in real time or non-real time is stored in the storage unit as time series data of joint positions. In this embodiment, each joint position acquired by the joint position acquisition unit in real time or non-real time is smoothed by the smoothing processing unit to generate a smoothed joint position.

For the optimization calculation based on inverse kinematics, for example, the algorithm described in Non-Patent Document 17 can be used. Several methods of optimization calculation based on inverse kinematics are known to those skilled in the art, and the specific optimization calculation method is not limited. One preferable example is a numerical solution method using a gradient method. Further, equation (4) that defines the weight of each joint in the optimization calculation based on inverse kinematics is one preferable aspect and is an example. For example, in the present embodiment, the smoothing processing unit makes the weights of all joints uniform after smoothing, and solves the optimization calculation based on inverse kinematics. Furthermore, it is understood that those skilled in the art may appropriately impose constraints when performing optimization calculations based on inverse kinematics.

In the above search method, since the search range depends on the grid interval s, if a point with the highest PCM score exists between each grid point, that point cannot be found. In this embodiment, instead of finding only the lattice point that has the maximum value in equation (2), multiple points among all lattice points are used as a joint position point group to search for a location where the PCM score is high. , an optimization calculation based on inverse kinematics may be performed. The joint position point group includes, for example, seven points. The numerical value of 7 was determined based on the expectation that there are similarly high-likelihood points in the front, back, top, bottom, left and right of the maximum point determined by equation (2).

Joint positions may be obtained using different grid spacings s. In one of the embodiments, the search for joint position candidates and the optimization calculation based on inverse kinematics are performed twice by changing the values of s to s1 and s2 (s1>s2), and from the joint position of frame t. Video motion capture of all T frames is performed by repeating the process of acquiring the joint angle and position of frame t+1 until t=T. This makes it possible to achieve both search speed and search accuracy. FIG. 7 shows the joint position acquisition process using different grid spacings.

First, a search is performed for joint position candidates in frame ^t +1 in a first search range (interval s1 of points in the neighborhood space of joint position t P ⁿ ) based on joint position ^t P ⁿ in frame t. In this step, a first joint position candidate in frame t+1 is obtained. Optimization calculations based on inverse kinematics and forward kinematics calculations are performed using the first joint position candidates to obtain the second joint position candidates at frame t+1.

Next, search for a joint position candidate in frame t+1 in a second search range (interval s2 of points in the neighborhood space of the second joint position candidate, s2 < s1) based on the second joint position candidate in frame t+1. conduct. In this step, a third joint position candidate in frame t+1 is obtained. Using the third joint position candidate, optimization calculations based on inverse kinematics and forward kinematics calculations are performed to obtain joint angles and joint positions at frame t+1.

In this embodiment, the search range is determined based on the joint positions acquired in the immediately preceding frame t, but in addition to or in place of the joint positions acquired in the immediately preceding frame t, Joint positions obtained in one or more previous frames or in one or more frames after frame t+2 may be used. For example, when acquiring joint positions in real time, the search range may be set based on joint positions in frames two or more frames before, such as frame t-1 and frame t-2. Alternatively, feature point position searches may be performed separately in parallel calculations for even frames and odd frames, and smoothing processing may be performed on feature point position candidates that are alternately generated.

In the calculation of equation (2), the likelihood to be evaluated is an index of the high precision of motion capture. A threshold value may be set for this likelihood, and if the likelihood is lower than the threshold value, it may be assumed that tracking of the target pose has failed, and the search range for joint position candidates may be expanded and searched. This may be done for some parts of the whole body pose, or it may be done for the whole body. In addition, if a feature point is temporarily lost due to occlusion, etc., offline analysis advances in time to determine a heat map of the same object, and then uses the continuity of motion to determine the time. By going back, the trajectory of the joint position of the portion where tracking failed due to occlusion may be recovered. This can minimize loss of sight due to occlusion.

In this way, in this embodiment, as a method of searching for a point in three-dimensional space where the PCM score is maximum, a group of points in three-dimensional space is projected onto a two-dimensional plane, and the PCM value of the pixel coordinates is obtained. Then, the sum (PCM score) is calculated, and the point with the highest PCM score among the point group is selected as the joint position candidate as the point in the three-dimensional space where the PCM score is the maximum. The calculation of projecting three-dimensional points onto each camera plane and calculating the PCM score is light. The search for joint position candidates according to this embodiment requires a large amount of calculation by limiting the search range using information of the previous frame and reprojecting the three-dimensional positions of grid points within the search range onto a two-dimensional image (PCM). , and eliminates outliers.

[E] Smoothed joint position acquisition unit The PCM acquisition and optimization calculations based on inverse kinematics used in the joint position acquisition unit do not take time-series relationships into account, so the output joint positions may vary in time. There is no guarantee that it will be smooth. The smoothed joint position acquisition section of the smoothing processing section uses the time series information of the joints to perform smoothing processing that takes temporal continuity into consideration. For example, when smoothing the joint positions obtained at frame t+1, typically the joint positions obtained at frame t+1, the joint positions obtained at frame t, and the joint positions obtained at frame t-1 are smoothed. The acquired joint positions are used. For the joint positions acquired in frame t and the joint positions acquired in frame t-1, the joint positions before smoothing are used, but the joint positions after smoothing can also be used. When performing the smoothing process in non-real time, joint positions acquired at a later time, for example, joint positions of frames after frame t+2, may be used. Furthermore, the smoothed joint position acquisition unit does not necessarily need to use consecutive frames. To simplify the calculation, first smoothing is performed without using body structure information. Therefore, the link length, which is the distance between adjacent joints, is not preserved. Next, using the smoothed joint positions, optimization calculations based on inverse kinematics using the target skeletal structure were performed again to obtain each joint angle of the target, thereby preserving the link length. Perform smoothing.

The smoothing joint position acquisition unit performs temporal smoothing of joint positions using a low-pass filter. Smoothing processing using a low-pass filter is applied to the joint positions acquired by the joint position acquisition unit, the smoothed joint positions are used as target positions of each joint, and optimization calculations based on inverse kinematics are performed. This makes it possible to take advantage of the smooth temporal changes in joint positions under the skeletal condition that the distance between each joint remains unchanged.

ローパスフィルタの特性上、ローパスフィルタに通した関節位置の取得はフィルタ次数の半分にあたる３フレームの遅延が生じ、かつ、関節角更新開始から３フレームはローパスフィルタを適応することができないという問題がある。本実施形態では、フィルタに適応前に、第１フレームの関節位置を第－２フレーム、第－１フレーム、第０フレームとおくことで、２フレーム分計算時間の遅れは生じるものの、空間的な誤差の少ない全フレームの関節位置の平滑処理を行う。 The smoothing processing unit will be explained in more detail. In this embodiment, an IIR low-pass filter shown in Table 3 is designed, and smoothing processing by the low-pass filter is performed on joint positions. Note that the value of the cutoff frequency is a value that can be appropriately set by a person skilled in the art, and for example, the values in Table 3 can be used empirically. The parameters of the smoothing filter may be adjusted depending on the type of movement to be measured and the frame rate of the camera to be photographed.

Due to the characteristics of the low-pass filter, there is a problem that there is a delay of 3 frames, which is half the order of the filter, when acquiring the joint position through the low-pass filter, and the low-pass filter cannot be applied for 3 frames after the joint angle update starts. . In this embodiment, by setting the joint positions of the first frame as -2nd frame, -1st frame, and 0th frame before applying the filter, although the calculation time is delayed by 2 frames, the spatial Smoothing the joint positions of all frames with few errors is performed.

If each joint position determined by the above filter is output from video motion capture, temporal smoothness of each joint can be obtained, but the condition that the distance between adjacent joints is a constant may be broken. In this embodiment, the joint positions after applying this low-pass filter are set as the target joint positions of each joint, and optimization calculations based on inverse kinematics are performed again. Equation (3) can be used for optimization calculations based on inverse kinematics, but optimization calculations based on inverse kinematics are performed with uniform weights for all joints (though not limited to this). This makes it possible to take advantage of the smoothness of temporal changes in joint positions adapted to the low-pass filter under the skeletal condition that the distance between each joint remains unchanged. Note that the range of motion of the joint angle may be added as a constraint condition for the inverse kinematics calculation.

The output of the smoothing processing unit includes, for example, joint angle information, skeletal structure, and joint position information that can be uniquely calculated from the two. For example, when drawing CG, the motion of the body is drawn using forward kinematics calculations using joint angle information and the body's skeletal structure file. Information included in the output of the smoothing processing section may be stored in the storage section.

[F] Preprocessing unit [F-1] In the calculation of the rotation heat map of the input image, for images in which a person is standing upright, for images in which a person is lying down or in an inverted position, accuracy may decrease. This is because the learning data used by the heat map acquisition unit is biased in that there are many images that are close to upright, which increases the estimation error of the lower body when the subject performs an inverted movement such as a handstand or cartwheel. In this case, the image is rotated according to the inclination of the subject's body in the previous frame, so that the subject appears in the image in a posture as close to erect as possible. In this embodiment, the PCM of the lower body was acquired from a rotated image.

Generalizing, the accuracy of heat map information is greatly degraded when the subject is in a predetermined first pose set (e.g., lying down or inverted), and when the subject is in a predetermined second pose set (e.g., standing upright). When it is known that the accuracy of the information is high, it is determined whether the target pose is included in the first pose set from the inclination of the target's body in the input image, and the target pose is included in the second pose set. The input image is rotated so that the pose (erect) included in is obtained, and heat map information is obtained. Particularly when acquiring heat map information in real time, the determination of the tilt of the object is performed based on the input image in the previous frame. The idea of acquiring heat map information by rotating an input image is a technique that can be generally applied to heat map acquisition units, independent of the motion capture according to this embodiment. Note that, with the accumulation of learning data and improvements in convolutional neural networks (CNN), it may not be necessary to rotate the input image. Furthermore, when a movable camera is used, rotation of the input image may not be necessary by physically rotating the camera itself in accordance with the movement of the object and obtaining the function μ _i for each frame.

The process of rotating an input image to obtain a PCM will be described with reference to FIG. 8. In the input image of frame t, the inclination of the subject's body (in one embodiment, the inclination of the trunk) is detected. For example, a vector connecting the target's waist and neck is calculated. Specifically, the three-dimensional coordinate positions of the Pelvis joint and Neck joint of the skeletal model shown in the left diagram of FIG. 4 are calculated. Using the function μ _i that converts a three-dimensional point to a pixel position on the image plane of camera i, we calculate the inclination of the subject's body at camera i at frame t (the vector connecting the waist and neck is orthogonally projected in each camera direction). Find the actual angle).

The necessity of image rotation processing is determined based on the inclination of the subject's body. In this embodiment, the image of frame t+1 is rotated according to the obtained body inclination (angle of the orthogonal projection vector) so that the orthogonal projection vector faces vertically upward. For example, a set of multiple rotation angles (for example, 0 degrees, 30 degrees, 60 degrees, 90 degrees, ... 330 degrees in 30 degree increments) and an angular range corresponding to each rotation angle (for example, 15 degree to 45 degrees corresponds to 30 degrees) and is stored in the storage unit as a table for determining the rotation of the input image. Referring to this table, determine which angular range the subject's body inclination (orthogonal projection vector angle) in the previous frame corresponds to, and rotate the input image by the angle corresponding to the determined angular range. Get PCM. When acquiring a heat map off-line, a PCM may be acquired for each rotation angle and stored in the storage unit, and the PCM may be selected according to the angle of the orthogonal projection vector. In rotated images, the background (four corners) is filled with black to facilitate input to the OpenPose network. Apply OpenPose to the rotated image and calculate the PCM of the target's lower body. Return the rotated image to the original image posture along with PCM. Then, a search for joint position candidates is performed. The previous frame used to determine the rotation of the input image is not only frame t, but may also be a frame before frame t-1.

[F-2] Other preprocessing The preprocessing is not limited to the process of rotating the input image according to the inclination of the subject's body. Examples of preprocessing performed using three-dimensional position information of one or more subjects in the previous frame include trimming and/or reduction, mask processing, camera selection, and stitching.

Trimming refers to trimming an image with reference to the position of the target on the image in the previous frame, and performing PCM calculation only on the trimmed portion. Reducing the computation time of PCM by trimming is advantageous in real-time acquisition of target motion. The bounding box, detailed in the next chapter, functions as a trim. Furthermore, if the input image is sufficiently large, the PCM creation accuracy of OpenPose may remain almost unchanged even if the image is reduced, so reducing the image may reduce the PCM calculation time.

Mask processing as preprocessing is a process in which, when an input image includes a person other than the target, mask processing is applied to the person other than the target to perform PCM calculation for the target. By performing mask processing, mixing of PCMs of multiple targets can be prevented. Note that the masking process may be executed in the joint position acquisition unit after PCM calculation.

Camera selection is to select an input image to be used for object motion acquisition and motion analysis by selecting a camera when the video acquisition unit includes a plurality of cameras. For example, when performing motion capture using a large number of cameras in a wide field, rather than performing motion capture and motion analysis using information from all cameras used, it is assumed that the target will be captured during preprocessing. The process is to select a camera that will be used in a video game and perform motion capture using the input image from the selected camera. Further, as pre-processing, stitching of input images may be performed. Stitching refers to stitching together camera images using acquired camera parameters and compositing them into one seamless image when each camera has an overlapping area at each angle of view. Thereby, even if the object partially appears at the edge of the input image, the PCM can be estimated favorably.

[G] Flow of obtaining the position of the target feature point from the input image The process of obtaining the joint angle and the position of the feature point from the input image according to this embodiment will be described with reference to FIG. The subject's movements are captured by multiple synchronized cameras, and each camera outputs an RGB image at a predetermined frame rate. Upon receiving the input image, the processing unit determines whether pre-processing is necessary. The preprocessing is, for example, whether or not image rotation is necessary. If it is determined that the image needs to be rotated according to a predetermined criterion, a heat map is obtained with the input image rotated. If it is determined that image rotation is not necessary, a heat map is obtained based on the input image.

For all the feature points on the body, a spatial distribution (heat map) of the likelihood of the position of the feature points on the body is generated and sent to the processing unit. The processing unit searches for position candidates for feature points. In one embodiment, when a heat map generated from the input image of frame t+1 is received, a search range is set based on the position of the feature point of frame t, and a search for a candidate position of the feature point is performed. The same process is then executed for all joints to obtain joint position candidates for all joints.

Optimization calculations based on inverse kinematics are performed for all feature point position candidates. Position candidates for feature points are associated with joints (feature points) of the skeletal model, and the skeletal model is adapted to the skeletal model unique to the subject. Based on the feature point position candidates and weights, optimization calculations based on inverse kinematics and forward kinematics calculations are performed to obtain joint angles and feature point positions.

By performing smoothing processing on the positions of the acquired feature points using joint positions in past frames, the temporal movement of the joint positions is smoothed. Optimization calculations based on inverse kinematics are performed again using the smoothed feature point positions to obtain the joint angles of the target, and forward kinematics calculations are performed using the obtained joint angles to determine the target joint angles. Get the joint positions of.

In this embodiment, the point where the PCM score is maximum is considered to be the most suitable pose in the current frame, and joint position candidates are acquired, while allowing the PCM score to decrease and performing subsequent processing. , optimization calculations based on inverse kinematics, and smoothing using a low-pass filter. By performing optimization calculations based on inverse kinematics, taking into account the skeletal structure of the target and the temporal continuity of the positions of the feature points, it was possible to reduce the estimation error of the positions of the feature points.

The motion capture according to this embodiment performs three-dimensional reconstruction from joint positions estimated using deep learning from images from multiple cameras, taking into account the structure of the human skeleton and the continuity of motion. Obtain smooth motion measurements comparable to optical motion capture. Adopting the algorithms shown in equations (1) to (4) in the joint position candidate acquisition unit in this embodiment has the following advantages. The vague joint positions, which have a spatial extent as a heat map, are optimized by referring to the human skeletal shape (optimization calculation based on inverse kinematics). In searching for joint position candidates, we use multiple spatially spread heat map information obtained from input images from each camera as is, and then consider the skeletal structure of the target to search for joint position candidates. The joint positions are determined using optimization calculations based on inverse kinematics.

In a skeletal structure, if it is necessary to determine the displacement of the degree of freedom of the skeletal structure that cannot be determined only by the positions of feature points obtained using heat map information, it may be determined by optimization that includes prior knowledge as a condition. . For example, for the hands and feet, the initial angle is determined using prior knowledge that the wrists and ankles exist, respectively. When hand and toe information is acquired, the initial angle is changed for each frame, and when hand and toe information is not acquired, the hand and toe angle is not changed from the angle of the previous frame. Fixed. In addition, prior knowledge is used to give weights and restrictions to each degree of freedom of the wrists and ankles according to the range of motion of the elbows and knees, to prevent the wrists and ankles from flipping, and to restrict the body from penetrating the ground. In this way, optimization calculations based on inverse kinematics may be performed.

[II] Motion capture system for multiple people [A] Overview of motion capture system The video motion capture for multiple people according to this embodiment uses top-down pose estimation. FIG. 9 shows a flowchart of multi-person video motion capture according to this embodiment. Multi-viewpoint images captured by multiple cameras are used as input images. The input image includes multiple people, but by surrounding each person with a bounding box, video motion capture is performed independently for each person. If a plurality of cameras are arranged at each viewpoint, one camera is selected at each viewpoint, and a bounding box surrounding the target person is determined in the selected camera image. Hit map information of feature points (keypoints) is obtained based on image information within the bounding box. In this embodiment, the heat map of each feature point is estimated using HRNet (https://github.com/HRNet), which is one of the top-down pose estimators. In the initial settings, a skeletal model unique to each person is set, and three-dimensional reconstruction of the feature points is performed using heat map information and skeletal parameters of each feature point obtained from each camera image. Regarding skeletal parameters specific to each person, reference may be made to the description in Chapter I. By acquiring the 3D position and joint angle of the feature point at each time, motion capture of the target is performed from time series information of the 3D position and joint angle of the feature point (time series information of 3D pose). Acquisition of the 3D positions and joint angles of feature points at each time is performed in parallel for multiple people, and motion capture for multiple people is performed. In one aspect, the skeletal structure corresponding to the 3D positions of feature points of each person and the time series information of joint angles (time series information of 3D poses) is simultaneously displayed on the display.

FIG. 10 shows a flowchart showing the input image processing steps according to this embodiment. A major difference from the motion capture system disclosed in the previous chapter is that the method includes a step of determining a bounding box before acquiring a heat map of each feature point. That is, a bounding box surrounding one object is determined on the input image (RGB image), and a heat map of one object is acquired in the heat map acquisition unit based on the image information within the bounding box. Pose estimation is performed using heat map information (spatial distribution of the likelihood of the location of feature points) and a skeletal model. Basically, the method disclosed in the previous chapter can be used to obtain the position of each feature point.

となる。各視点vにおいて、n_Cv台のカメラから１つのカメラを選択して、選択したカメラの画像を用いて特徴点の２Ｄ位置の推定を行う。より具体的には、選択したカメラの画像上に所定のバウンディングボックスを設定し、バウンディングボックス内のピクセル情報を用いて特徴点のヒートマップを取得する。上記カメラシステムは例示であって、複数視点のそれぞれに１台のカメラを配置してカメラシステムを構成してもよい。 The three-dimensional motion reconstruction according to this embodiment is performed using n _c cameras arranged around n _p objects. Each camera is synchronized and calibrated. When the space photographed by the camera is a wide space, a plurality of cameras with different fields of view may be placed adjacent to each other at one viewpoint in order to avoid the problem that the object is not visible to the camera. In this case, if the number of viewpoints is _nv , the camera set placed at viewpoint v is _Cv , and the number of cameras at v is _nCv , then the number of cameras is

becomes. At each viewpoint v, one camera is selected from n _Cv cameras, and the 2D position of the feature point is estimated using the image of the selected camera. More specifically, a predetermined bounding box is set on the image of the selected camera, and a heat map of feature points is obtained using pixel information within the bounding box. The camera system described above is an example, and the camera system may be configured by arranging one camera at each of a plurality of viewpoints.

[B] Initial setting Photograph the target using multiple cameras. Search the area of the target person in each image and create a bounding box. To search for human areas during initial setup, we use human detectors such as Yolov3, pose estimators that support multiple people such as OpenPose, epipolar constraints using camera parameters, face recognition, clothing recognition, etc. It can be determined using a personal identification device, etc. Alternatively, each person's area may be assigned manually. A plurality of people may be included in the bounding box.

Using a top-down pose estimator (for example, HRNet), a heat map of each feature point of one object within the bounding box is calculated, and the position of the feature point is detected. For example, the center coordinates of the heat map are estimated as the 2D positions of the feature points. By detecting the 2D position of the feature point from multiple viewpoints, we perform 3D reconstruction of the 3D position of the feature point using the multiple 2D positions of one feature point, and calculate the initial 3D pose and skeletal parameters ( (distance between joints in three-dimensional space). Skeletal parameters may be calculated from images taken by multiple cameras at a single time, or may be calculated from camera images taken at multiple times to reduce the influence of errors. Further, the skeletal parameters may be measured in advance. Further, the skeletal parameters may include the range of motion of each joint angle.

If it is determined that the initialization (estimation of the initial 3D posture and skeletal parameters) has failed, the above steps are performed at a different imaging time. Judgment indicators include heat map values when the 3D reconstructed feature point positions are projected onto the image plane, coordinate values when the 3D positions of the feature points are reprojected onto the image plane, and the initial feature An error with the coordinate value estimated as the 2D position of the point, a coefficient of each skeleton parameter, etc. can be used.

At the time of initial setting, 3D reconstruction of feature points is performed using the 2D positions of the feature points directly obtained from the heat map information of each feature point, but after that, similar to the method in the previous chapter, heat map information is used to reconstruct the feature points. 3D pose estimation is performed using (the spatial distribution of the likelihood of the certainty of the position of the feature point).

[C] Determination of bounding box in input image The motion capture system according to this embodiment uses top-down pose estimation, and determines the bounding box before calculating the heat map of each feature point. In this embodiment, appropriate dimensions and positions of bounding boxes are predicted using 3D position information of feature points acquired in other frames. The video motion capture according to this embodiment can perform motion capture with high precision equivalent to optical motion capture. If the frame rate is high enough, the current 3D pose of the object can be predicted from the previous calculated 3D pose or previous 3D poses (past 3D motion). The dimensions and location of the appropriate bounding box can be calculated using a perspective transformation (using the matrix μ _i ) based on the predicted location of the 3D pose of the object.

Determination of the bounding box will be explained with reference to FIG. 11. Assume that the 3D poses (3D positions of each feature point) of the object in frame t-2, frame t-1, and frame t have been obtained, and that the 3D pose of the object in frame t+1 is to be obtained. Bounding box determination involves predicting the 3D position of each feature point at frame t+1, and using the predicted 3D position of each feature point to determine the 2D position of each feature point on the image plane of each camera at frame t+1. and using the predicted 2D position (coordinates) of each feature point to determine the size and position of the bounding box on the image plane of each camera at frame t+1. That is, the bounding box determination unit includes a 3D position prediction unit for each feature point in the next frame, a 2D position prediction unit for each feature point in the next frame, and a bounding box size/position determination unit for the next frame. We are prepared.

The feature point 3D position prediction unit calculates the predicted 3D position of each feature point in frame t+1, for example, using the 3D positions of each feature point in frame t-2, frame t-1, and frame t. Frame t-2, frame t-1, and frame t shown in FIG. but not limited to. For example, the 3D positions of feature points in frames t-1 and t may be used, or the 3D positions of feature points in frames before frame t-2 may be used. Note that the 3D predicted positions of feature points in the initial settings of next frame 1, frame 2, and frame 3 are, for example, initial 3D pose, initial 3D pose, and 3D pose of frame 1, initial 3D pose, and 3D of frame 1, respectively. Pose, obtained using the 3D pose of frame 2.

The feature point 2D position prediction unit projects the 3D predicted position of each feature point in frame t+1 onto each camera image using perspective projection transformation, thereby predicting each feature point on the image plane of each camera in frame t+1. Obtain the 2D predicted position (coordinates). This perspective projection transformation is performed using a function (matrix) μ _i that transforms a three-dimensional arbitrary point into a pixel position on the imaging plane of camera i. The function (matrix) μ _i is obtained during calibration of each camera.

The bounding box size and position determination unit determines the size and position of the bounding box in each camera image so as to include the 2D predicted positions of all feature points. The position of the bounding box is determined, for example, by the center coordinates of the rectangular box. Note that the above calculations performed by the bounding box determiner are lightweight. Then, in the image of frame t+1, the 3D pose (3D position of each feature point) of the target at frame t+1 is obtained based on the image information of the area surrounded by the bounding box. The 3D position of each feature point in frame t+1 is stored in the storage unit and used to determine the bounding box in the image of frame t+2.

ここで、^t+1 _lB_iは、時刻t+1における人lのカメラiの画像上で予測された中心位置及び寸法を表している。^t _lP、^t-1 _lP、^t-2 _lPは、時刻t、t-1、t-2における人lの全ての関節の３Ｄ位置である。mは、対象の全身がちょうど含まれると想定されるバウンディングボックスの寸法を決定するための正の定数値である。 In one aspect, the bounding box can be determined by calculations such as the following.

Here, ^t+1 _l B _i represents the predicted center position and size on the image of camera i of person l at time t+1. ^t _l P, ^t-1 _l P, ^t-2 _l P are the 3D positions of all joints of person l at time t, t-1, t-2. m is a positive constant value to determine the dimensions of the bounding box that is assumed to exactly contain the whole body of the subject.

Equation (3) is an equation for calculating the predicted 3D position of each joint at frame t+1. Equation (3) is an example, and the coefficients and frames used are not limited to those shown in Equation (3). An equation for position prediction may be set on the assumption that the motion of the object is uniform acceleration motion, or an equation may be designed on the assumption of uniform velocity motion. The 3D positions of frames before frame t-3 may be used, or it is not excluded that the 3D positions of frames after frame t+2 are used. When using 3D positions of joints in a plurality of frames, appropriate weights may be set for the values of each frame. Furthermore, the formula may be changed depending on the object or type of movement.

Equation (2) is an equation for determining the dimensions and position of the bounding box, and is an example. In equation (2), the maximum x-coordinate value, minimum x-coordinate value, maximum y-coordinate value, and minimum y-coordinate value are obtained from the coordinates of all feature points in the image plane of each camera, and these coordinate values are The dimensions of the bounding box and the reference vertical and horizontal dimensions are determined from the above, and the coordinates of the center of these coordinate values are determined as the position of the bounding box. Determine the dimensions of the bounding box from the above vertical and horizontal dimensions and the constant m. Although the range of m is not limited, it is understood that m>0 and its value can be appropriately set by those skilled in the art. If the value of m is small, a part of the body may not be included in the bounding box, especially if the 3D predicted position of the feature point is incorrectly estimated. Furthermore, if the keypoints are only up to the eyes or wrists, if m is set to a small value, there is a risk that the head and fingertips will not be included in the bounding box. On the other hand, if the value of m is large, the advantage of using a bounding box may be diminished. For example, there is a higher possibility that other objects will be included in the bounding box.

[D] 3D Pose Acquisition Unit [D-1] Acquisition of Heat Map of Feature Points In this embodiment, a heat map of feature points is acquired using the HRNet model learned with the COCO dataset. In this embodiment, the input image can be resized to predetermined dimensions W'×H'×3 (RGB) depending on the top-down pose estimator used. For example, in the HRNet pose estimator, W′×H′=288×384. The number of feature points is n _k =17, consisting of 12 joints (shoulders, elbows, wrists, hips, knees, ankles) and 5 feature points (eyes, ears, nose). A top-down pose estimator that generates a heat map corresponding to feature points is well known, and the pose estimator that can be used in this embodiment is not limited to the HRNet model.

この式において、nは人体骨格モデルの関節位置を表している。数字は、図４左図で数字で示す位置に対応している。１つの態様では、１１個の特徴点（肩、肘、手首、目、耳、鼻）のヒートマップが、回転されたバウンディングボックスに囲まれた領域の画像情報を用いて計算される。バウンディングボックスの回転は、画像に対する相対的な回転でもよく、回転させた入力画像上にバウンディングボックスを設定してもよい。入力画像の回転については、前チャプタの入力画像の回転を参照することができる。 [D-2] Determination of rotation or tilt of bounding box The HRNet used in this embodiment is trained on the assumption that the body is not excessively tilted. Therefore, if the body is significantly tilted in the vertical direction (for example, in a headstand or cartwheel), pose estimation may fail. In this embodiment, the heat map of feature points is estimated more accurately by rotating the bounding box. The rotation angle of the bounding box is derived from the slope of the predicted vector connecting the torso and neck.

In this equation, n represents the joint position of the human skeleton model. The numbers correspond to the positions indicated by the numbers in the left diagram of FIG. In one aspect, a heat map of 11 feature points (shoulder, elbow, wrist, eye, ear, nose) is computed using image information of the region surrounded by the rotated bounding box. The bounding box may be rotated relative to the image, or the bounding box may be set on the rotated input image. Regarding the rotation of the input image, you can refer to the rotation of the input image in the previous chapter.

Ｉは、カメラ画像の解像度を表している。 [D-3] Camera Selection In this embodiment, one viewpoint is equipped with a plurality of cameras with different fields of view, so the camera that most appropriately photographs the target person is selected at the 2D position of the feature point. Select using estimates. Camera selection is performed using, for example, the following equation using each predicted joint position.

I represents the resolution of the camera image.

[D-4] Obtaining Joint Positions In 3D pose estimation, the 3D positions of keypoints are generally obtained by three-dimensionally reconstructing the 2D positions of the feature points detected from each camera. More specifically, for example, in each camera image, the central coordinates of the heat map of the feature point are estimated as the 2D position of the feature point, and these 2D positions are used to obtain the 3D position of the feature point. However, with such a simple method, for example, in a severe occlusion environment, 3D pose estimation will fail due to erroneous detection of feature points (see FIG. 13). The important point to note here is that even if the 2D position of the feature point detected from the heat map (center coordinates of the heat map) is a false detection, the heat map can still be used to determine the likelihood of the location of the feature point. This means that it is a spatial distribution of degrees and may indicate the likelihood of the correct location of the feature point.

透視投影変換を用いることで、任意の３Ｄ座標の点のカメラiの画像上の座標に投影することができ、該座標に対応する尤度（PCMスコア）を取得することができる。^t+1 _lPⁿ _predが正確に予測されていると仮定すると、最も可能性の高い特徴点の３Ｄ位置は、尤度（PCMスコア）の合計が最大となる格子上の点となる。関節位置候補取得部の処理工程を図１２に示す。 Therefore, similarly to the method in the previous chapter, a search area is set to obtain position candidates (one or more) of feature points. In the previous chapter, the search range is the space near the feature point acquired in frame t, whereas in this embodiment, the search range is the space near the predicted position of the feature point predicted in frame t+1. shall be. Specifically, a grid space centered at the 3D predicted position ^t+1 _l P ⁿ _pred of each feature point in frame t+1 is set (in Figure 6, ^t P ⁿ is replaced with ^t+1 _l P ⁿ _pred ). ), ^t+1 _l L ⁿ Let _a,b,c be one point in the grid space.

By using perspective projection transformation, it is possible to project an arbitrary 3D coordinate point onto the coordinates on the image of camera i, and obtain the likelihood (PCM score) corresponding to the coordinate. Assuming ^{that t+1} _l P ⁿ _pred is accurately predicted, the most likely 3D location of the minutiae will be the point on the grid where the sum of the likelihoods (PCM scores) is maximum. FIG. 12 shows the processing steps of the joint position candidate acquisition unit.

ここで、^t+1 _lSⁿ _i(X)は、時刻t+1でカメラiにおいて、人物lの関節nの尤度(PCMスコア)を取得するための関数である。gは、0～1の間の定数であり、当業者により適宜設定される。なお、gの最適な値は、例えば、オクルージョンの状況や、関節の部位、視点の数等によって変わり得る。 In motion capture of multiple people, occlusion may occur (see Figure 13). In this embodiment, it is assumed that the reliability of the likelihood (PCM score) decreases in an occlusion environment. A weight consisting of a constant is assigned to the likelihood (PCM score) obtained from the heat map information. The most likely minutiae locations are obtained as follows.

Here, ^t+1 _l S ⁿ _i (X) is a function for obtaining the likelihood (PCM score) of joint n of person l at camera i at time t+1. g is a constant between 0 and 1, and is appropriately set by a person skilled in the art. Note that the optimal value of g may change depending on, for example, the occlusion situation, the location of the joint, the number of viewpoints, etc.

ここで、^t+1 _lＱは、時刻t+1における人物lの関節角を表し、_lＪはヤコビアン行列を表す。 The joint positions of the skeletal model are calculated by referring to the positions of the calculated feature points. In this embodiment, the joint angles of the skeletal model can be optimized by referring to the correspondence shown in FIG. 4 and using inverse kinematic calculations using the positions of feature points as target positions.

Here, ^t+1 _l Q represents the joint angle of person l at time t+1, and _l J represents the Jacobian matrix.

しかしながら、平滑化処理が実行されると、骨格モデルが壊れて、空間連続性が失われる。さらに、上記逆運動学計算では、リング長のみが考慮されているので、各関節角は可動域を考慮していない。そこで、目標位置として滑らかにされた関節位置を用いて、再度、逆運動学計算によって骨格モデルを最適化する。

ここで、Ｑ^－、Ｑ^＋は、RoM(Range of Motion)の最小値及び最大値を表す。この計算により、より適切な関節位置及び角度（３Ｄポーズ）が取得される。 Joint positions are calculated through optimization calculations, but these positions do not take into account temporal continuity of motion. In order to obtain smooth motion, joint positions are smoothed using a low-pass filter F consisting of time-series data of joint positions.

However, when the smoothing process is performed, the skeletal model is broken and spatial continuity is lost. Furthermore, in the above inverse kinematics calculation, only the ring length is taken into consideration, so the range of motion of each joint angle is not taken into account. Therefore, the skeletal model is optimized again by inverse kinematic calculation using the smoothed joint positions as target positions.

Here, Q ⁻ and Q ⁺ represent the minimum and maximum values of RoM (Range of Motion). Through this calculation, more appropriate joint positions and angles (3D poses) are obtained.

Motion capture of a single object is performed by repeating the above process (obtaining a 3D pose in each frame). Video motion capture of multiple people can be achieved by performing the same processing on multiple targets in parallel. Motion capture of multiple players can be applied, for example, to acquiring a video of a match of a team sport (soccer, futsal, rugby, baseball, volleyball, handball, etc.) and performing motion capture of each player during the match. Note that the bounding box determination described in this embodiment can be applied not only to motion capture of multiple people but also to pose estimation using an image including a single object as an input image.

[E] Complementation of Motion Capture Flow The 3D pose estimator according to this embodiment relies on object discrimination using bounding boxes for performance. Therefore, for example,
(i) If excessive misestimation of the 3D pose of the object occurs (typically when multiple objects are extremely close together);
(ii) if the subject moves outside the capture volume;
(iii) if a new subject appears within the capture volume;
How to deal with this is also important. This section describes a mechanism that complements the motion capture flow by detecting the occurrence of the above events.

The interpolation mechanism includes a second 3D pose estimator or estimation program that estimates the 3D pose of the object without using a bounding box. In one aspect, the second 3D pose estimator is of the bottom-up type. The second 3D pose estimator operates in parallel with the 3D pose estimator (first 3D pose estimator) using a bounding box according to this embodiment. The second 3D pose estimator performs a 3D pose estimation of the object in each frame or periodically. That is, the motion capture system includes a first 3D pose estimator that uses a bounding box and a second 3D pose estimator that does not use a bounding box.

The second 3D pose estimator will be explained. Using a person detector (such as Yolo v3) or a pose estimator that supports multiple people (such as OpenPose) for each camera, the person's position on each image is estimated without using bounding box information. Next, the estimated person is matched between multiple cameras using epipolar restraint, facial recognition, clothing recognition, etc., and the 3D position of the estimated person is calculated to estimate the 3D pose. do. Note that instead of acquiring the 3D pose of the target, the approximate position of the three-dimensional space occupied by the target (for example, a rectangular parallelepiped shape or a cylindrical shape) may be acquired. In that sense, the second 3D pose estimator can be generalized as a 3D position estimator.

The complementation mechanism complements the pose estimation by the first 3D pose estimator, for example, when the event (i) above occurs. The occurrence of the event (ii) means, for example, the PCM score of the first 3D pose estimated by the first 3D pose estimator (the PCM score of the 2D pose projected onto the image plane, for example, (4) is calculated and compared with a threshold value. If the PCM score of the first 3D pose is less than the threshold value, it is determined that the PCM score of the first 3D pose is incorrectly estimated. At this time, the second 3D pose estimated by the second 3D pose estimator is adopted, for example, on the condition that the PCM score of the second 3D pose exceeds the threshold.

A first 3D pose of the object estimated by a first 3D pose estimator using a bounding box and a second 3D pose of the object estimated by a second 3D pose estimator are compared, The image forming apparatus may include a comparison/determination device that determines whether or not the 3D pose of No. 1 is incorrectly estimated. The comparison/determination by the comparison/determination device may be performed in each frame or periodically (for example, once per second or twice per second).

As a comparison method, the PCM score of the first 3D pose and the second 3D pose (the PCM score of the 2D pose projected on the image plane, for example, the sum of the PCM scores calculated by equation (4) in Chapter I) Comparison, norm error in three dimensions between the first 3D pose and the second 3D pose, the position where the first 3D pose is projected onto the two-dimensional image plane, and the second 3D pose (3D position). The degree of coincidence with the position projected onto the two-dimensional image plane can be exemplified. Based on the comparison result and using the set discriminant value, it is determined whether the 3D pose estimation of the target by the first 3D pose estimator is incorrect estimation.

If it is determined that the 3D pose estimation of the target by the first 3D pose estimator is incorrect, the 2D pose of the target is determined based on the 3D pose (3D position) estimated by the second 3D pose estimator. Correct the bounding box for estimating. When performing correction based on the second 3D pose, predetermined conditions may be imposed. For example, the condition may be that the difference between the reprojection of the second 3D pose onto the image plane and the 2D pose used to estimate the second 3D pose is within a discriminant value. A first 3D pose estimator uses the corrected bounding box to obtain a 3D pose of the object. For example, when an incorrect estimation is determined for frame t, the bounding box of frame t is corrected, and the first 3D pose estimator recalculates the 3D pose of frame t using the corrected bounding box. Alternatively, when an incorrect estimation is determined for frame t, the first 3D pose estimator calculates the 3D pose of frame t+1 using the corrected bounding box from frame t+1. In this way, event (i) is dealt with.

The complementation mechanism determines that the event (ii) has occurred. If the second 3D position estimator cannot recognize the object, it is determined that the object has moved outside the capture volume (i.e., event (ii) has occurred), and the object is detected using the first 3D pose estimator. Stop motion capture calculation. Note that the size of the capture volume is determined in advance, and if the 3D pose deviates from it, it is determined that event (ii) has occurred, and the motion capture calculation of the target using the first 3D pose estimator is performed. may be stopped.

The complementation mechanism determines that the event (iii) has occurred. If the second 3D position estimator recognizes a new object, it determines that the new object has entered the capture volume (i.e., event (iii) has occurred) and performs the first 3D pose estimation. In the device, the system is initialized for the new target and added to the motion capture process. Regarding initialization, the above description can be referred to.

If an object that has once moved out of the capture volume reenters the capture volume, it can be treated as an occurrence of event (iii). In this case, if it is recognized that the newly recognized object is the same as the object for which 3D pose estimation was stopped in accordance with the occurrence of event (ii), the pose of the newly recognized object by the first 3D pose estimator is In estimation, already acquired skeletal information can be used.

This motion capture system is suitable for sports (motion analysis, coaching, tactical suggestions, automatic scoring of competitions, detailed training logs), smart life (general healthy living, monitoring of the elderly, detection of suspicious behavior), and entertainment (live performance). It can be used in a wide variety of fields such as , CG creation, virtual reality games and augmented reality games), nursing care, and medical care.

Claims (20)

A method for acquiring a 3D pose of a target in motion capture using multiple cameras, the method comprising:
The object has a plurality of feature points on its body including a plurality of joints, and the 3D pose of the object is specified by the positions of the plurality of feature points,
Determine a bounding box surrounding the object on each camera image at frame t+1 using one or more 3D poses of the object at frame t and/or one or more frames before frame t. ,
In each camera image of frame t+1, a spatial distribution of the likelihood of the certainty of the position of each feature point is obtained using the image information in the bounding box, and based on the spatial distribution of the likelihood, Obtain the 3D position of each feature point,
How to obtain 3D pose. The image includes multiple objects,
Determine the bounding box for the selected object,
obtaining a 3D position of each feature point of the one object in frame t+1 using image information within the bounding box;
The 3D pose acquisition method according to claim 1. Acquire the 3D pose of each object in parallel in each frame,
The 3D pose acquisition method according to claim 2. The determination of the bounding box is
predicting a 3D pose at frame t+1 using the one or more 3D poses;
projecting the predicted 3D pose onto each camera image to predict a 2D pose in each camera image of frame t+1;
determining the position and size of a bounding box in each camera image using position information of a plurality of feature points that specify the predicted 2D pose;
The 3D pose acquisition method according to any one of claims 1 to 3. the position of the bounding box is determined by the center of the bounding box;
The 3D pose acquisition method according to claim 4. determining a slope of the bounding box for each camera image;
The 3D pose acquisition method according to claim 4 or 5. In the object, the distance between adjacent joints is obtained as a constant,
Using the spatial distribution of the likelihood, obtain position candidates for one or more feature points corresponding to each feature point,
Obtaining each joint angle of the target by performing optimization calculation based on inverse kinematics using the position candidates of the feature points and the multi-joint structure of the target,
obtaining the position of a feature point including the target joint by performing forward kinematics calculation using the joint angle;
The 3D pose acquisition method according to any one of claims 1 to 4. The acquisition of the feature point position candidates involves setting a search range for the feature point position candidates using the 3D position of each feature point acquired in one or more frames;
converting all or some of the points within the search range into pixel positions on each camera image plane;
8. The 3D pose according to claim 7, wherein one or more feature point position candidates for each joint position in frame t+1 are obtained using the spatial distribution of likelihoods obtained in frame t+1. Acquisition method. 9. The 3D pose acquisition method according to claim 8, wherein the search range is a space in the vicinity of the predicted 3D position of each feature point at frame t+1. 9. The 3D pose acquisition method according to claim 8, wherein the search range is a space in the vicinity of the 3D position of each feature point acquired in frame t. 11. The 3D pose acquisition method according to claim 7, wherein the spatial distribution of the likelihood is used in the optimization calculation based on inverse kinematics. smoothing the position of the feature point in the time direction using the positions of a plurality of feature points acquired in a plurality of other frames;
obtaining each joint angle of the target by performing an optimization calculation based on inverse kinematics using the smoothed feature point positions and the multi-joint structure of the target;
The 3D pose acquisition method according to any one of claims 7 to 11. A 3D pose acquisition device for a target in motion capture using multiple cameras,
The object has a plurality of feature points on its body including a plurality of joints, and the 3D pose of each object is specified by the position of the plurality of feature points,
The device includes a storage unit and a processing unit,
The storage unit stores time-series data of calculated 3D poses of each object,
The processing unit includes:
Using one or more 3D poses of an object in frame t and/or one or more frames before frame t, create a bounding box surrounding the object in each camera image at frame t+1. decided,
In each camera image of frame t+1, a spatial distribution of the likelihood of the certainty of the position of each feature point is obtained using the image information in the bounding box, and based on the spatial distribution of the likelihood, Obtain the 3D position of each feature point,
A 3D pose acquisition device configured as follows. The determination of the bounding box is
predicting a 3D pose at frame t+1 using the one or more 3D poses;
projecting the predicted 3D pose onto each camera image to predict a 2D pose in each camera image of frame t+1;
determining the position and size of a bounding box in each camera image using position information of a plurality of feature points that specify the predicted 2D pose;
The 3D pose acquisition device according to claim 13. The storage unit stores a distance between adjacent joints as a constant for the one object,
The processing unit includes:
Using the spatial distribution of the likelihood, obtain position candidates for one or more feature points corresponding to each feature point,
Obtaining each joint angle of the target by performing optimization calculation based on inverse kinematics using the position candidates of the feature points and the multi-joint structure of the target,
obtaining the position of a feature point including the target joint by performing forward kinematics calculation using the joint angle;
The 3D pose acquisition device according to claim 13 or 14. The processing unit sets a search range for feature point position candidates using each feature point acquired in one or more frames, and uses the spatial distribution of the likelihood acquired in frame t+1, The 3D pose acquisition device according to claim 15, which acquires one or more feature point position candidates for each joint position in frame t+1. The processing unit includes:
smoothing the position of the feature point in the time direction using the positions of a plurality of feature points acquired in a plurality of other frames;
obtaining each joint angle of the target by performing an optimization calculation based on inverse kinematics using the smoothed feature point positions and the multi-joint structure of the target;
It is configured as follows.
The 3D pose acquisition device according to claim 15 or 16. predicting a 3D pose of the object in the next frame using one or more acquired 3D poses;
projecting the predicted 3D pose onto the image plane of the camera to predict the 2D pose of the object on the image plane of the camera in the next frame;
Determining the position and size of a bounding box in the camera image of the next frame using position information of a plurality of feature points that specify the predicted 2D pose;
A method for determining bounding boxes in top-down pose estimation. multiple cameras and
a bounding box determination section;
3D pose acquisition unit,
A motion capture system equipped with
The 3D pose acquisition unit acquires a spatial distribution of the likelihood of the position of each feature point in each camera image using the image information in the bounding box, and based on the spatial distribution of the likelihood, Obtain a 3D pose,
The bounding box determining unit includes:
predicting a 3D pose of the object in the next frame using one or more acquired 3D poses;
projecting the predicted 3D pose onto the image plane of the camera to predict the 2D pose of the object on the image plane of the camera in the next frame;
Determining the position and size of a bounding box in the camera image of the next frame using position information of a plurality of feature points that specify the predicted 2D pose;
motion capture system. Furthermore, a 3D position acquisition section different from the 3D pose acquisition section is provided, and the 3D position acquisition section acquires the 3D position of the target without using a bounding box.
The motion capture system according to claim 19.

Images