KR20230081395A - human motion analysis system - Google Patents

Abstract

A computer vision technique for recognizing human motion in multi-view images is disclosed. Positions of joint points are determined in the multi-view images, and the momentum is calculated for each joint by sequentially applying them to the modeling body on the time axis. The total amount of exercise of the entire body can be calculated by summing up the amount of exercise for each joint. The exercise risk can be calculated and provided by analyzing the distribution of each joint of the angular momentum applied to the modeling body.

Description

Human motion analysis system {human motion analysis system}

An image processing technology, in particular, a computer vision technology for recognizing a human motion in multi-view images is disclosed.

Patent No. 1,962,045 titled 'choreography evaluation apparatus and method' detects and tracks the positions of the choreographer and learner's joints from depth image data, and compares and evaluates the choreography movements of the choreographer and learner based on the joint positions. technology is disclosed. As such, although technology for recognizing human postures or motions is developing, most applications are limited to recording or analyzing postures or motions.

The proposed invention aims to provide momentum information when performing a corresponding motion by recognizing an actor's motion.

Furthermore, the proposed invention aims to provide momentum information for each unit motion by recognizing an actor's motion.

Furthermore, the proposed invention aims to provide.

According to one aspect of the proposed invention, joint point positions are determined from multi-view images obtained from a plurality of cameras, and the motion amount for each joint is calculated by sequentially applying them to the modeling body on the time axis. The total amount of exercise of the entire body can be calculated by summing up the amount of exercise for each joint.

According to an additional aspect, unit motions may be distinguished by analyzing joint point positions on a time axis.

According to an additional aspect, the movement of the estimated joint points on the time axis may be analyzed to calculate the amount of momentum for each joint, and the amount of momentum may be calculated by summing them.

According to an additional aspect, the exercise risk may be calculated and provided by analyzing the distribution of each joint of the angular momentum applied to the modeling body.

According to an additional aspect, the degree of agreement may be evaluated by comparing the actions of the actor and the comparison actor.

According to the proposed invention, by analyzing human motions such as sports or dance by an automated system and analyzing the range of motion or momentum for each joint, it is possible to prevent injury due to excessive exercise or load concentration of a specific joint due to wrong motion. can

1 shows the overall configuration of a motion analysis system according to an embodiment.
2 is a block diagram showing the configuration of a motion analysis system according to an embodiment.
Figure 3 illustrates the arm part in such a modeling body.
4 is a block diagram illustrating an embodiment of a posture information capture unit.
5 is a diagram illustrating epipolar geometry by way of example.
6 is a block diagram showing the configuration of an embodiment of the momentum calculation unit of FIG. 2 .
7 is a block diagram showing the configuration of an embodiment of the motion information analysis unit of FIG. 2 .
8 is a flowchart illustrating the configuration of a motion analysis method according to an embodiment.

The foregoing and additional aspects are embodied through embodiments described with reference to the accompanying drawings. It is understood that the elements of each embodiment can be combined in various ways within one embodiment or with elements of another embodiment without contradiction with each other or other references. Based on the principle that the inventor can properly define the concept of terms in order to explain his/her invention in the best way, the terms used in this specification and claims have meanings consistent with the description or proposed technical idea. and should be interpreted as a concept. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Description of Figure 1>

1 shows the overall configuration of a motion analysis system according to an embodiment. As shown, the motion analysis system according to an embodiment includes a plurality of cameras 30-1 to 30-5 that photograph the actor 20 at different points of view and a server connected to the cameras through a network. It includes a computer 50. A plurality of cameras are often arranged along a spherical surface to be suitable for generating a multi-viewpoint image, and may consist of five cameras, front, left, right, top, and rear, or a larger number. The video signals output from the plurality of cameras 30-1 to 30-5 control the synchronization of shooting and capture the output images to be suitable for transmission to the remote server computer 50 and configure the transmission format. A control unit 10 may be included. The camera controller 100 may be in the form of a computer and may be installed in a studio in which a plurality of cameras 30-1 to 30-5 are installed. In addition, the camera controller 100 may be configured to obtain 2D images from each camera. , process the joint points, estimate the joint points from each of the 2-dimensional images, fuse the joint points estimated from each of the 2-dimensional images, and estimate the 3-dimensional joint points.

Managers who manage this process in the studio can connect to the server computer 50 through a network with a personal computer 71 to input necessary information, monitor output, or check result values. In addition, the analysis result for the operation can be confirmed by, for example, a smart phone 73 through a wireless Internet network.

<Description of Claim 1 Invention>

2 is a block diagram showing the configuration of a motion analysis system according to an embodiment. As shown, the motion analysis system according to an embodiment includes a plurality of cameras 30-1 to 30-5, a posture information capture unit 210, and a motion information analysis unit 230. In one embodiment, the exercise information analysis unit 230 includes a momentum calculator 600.

The posture information capture unit 210 outputs the joint position of the actor in 3D coordinates from the multiview images obtained from the plurality of cameras 30-1 to 30-5. Estimating a person's pose from multiview images is to find joint points in each image and estimate the positions of 3-dimensional joint points from the locations of the joint points, which is a technology that has been recently developed. am. Z. Zhang, C. Wang, W. Qiu, W. Qin, and W. Zeng, “Adafuse: Adaptive multiview fusion for accurate human pose estimation in the wild,” IJCV, 2020 A technique for estimating a 3-dimensional joint point by estimating the positions of joint points and fusing them by a probability heatmap is disclosed. The posture information capture unit 210 estimates joint points of an actor in each multi-view image frame using one of these known techniques and outputs the three-dimensional coordinates.

The momentum calculation unit 600 applies the joint point positions output from the posture information capture unit 210 sequentially to the modeling body on the time axis to calculate the momentum for each joint and sums them up to calculate the momentum. Modeling body information may be stored and managed in the database 640 . Figure 3 illustrates the arm part in such a modeling body. The modeling body roughly divides the human body into several parts and models the mass of each part with weights 320, 340, and 360 having only weight and zero volume. The body parts between the joints are simplified to wires 313, 335, 359 where the mass is concentrated in the weights. Positions of the weights 320, 340, and 360 in the wire model may be determined according to the shape of the body part. Defunct Patent No. 915,606 discloses a technique for constructing a modeling body that is kinematically close to a specific person. The proposed invention can be applied by simplifying the disclosed modeling body to a model including only joint points estimated by the proposed invention, for example. It is not simple to calculate the momentum for each joint centered on the joint. Since all the joints constituting the body are connected, for example, the momentum of the arm is the sum of the momentum of the wrist joint 350, the momentum of the elbow joint 330, and the momentum of the shoulder joint 310, but the momentum of the elbow joint 330 In addition to the lower beat, the hand should be considered, and the momentum varies according to the movement of the wrist joint 350. In the proposed invention, in order to simplify the calculation of the momentum of these connected joints, the momentum for a specific time interval is calculated by calculating the momentum in units of minute displacements (ΔФ) in which the shape of the joint does not change significantly and accumulating them. From the point of view of calculating the momentum, it can be assumed that the shape of the body does not change during the minute displacement. According to this, it is assumed that the wrist joint 350 does not move when calculating the amount of motion during the minute displacement of the elbow joint 330 .

The user interface unit 250 provides analyzed results and receives necessary instructions or information. In the illustrated embodiment, the user interface unit 250 includes a keyboard 251 and a display 253. The user interface unit 250 may include such a device as well as a program for controlling the device, and computer program instructions for generating a graphical user interface (GUI) that displays results graphically or accepts a user's input.

<Description of Claim 2 Invention>

4 is a block diagram illustrating an embodiment of a posture information capture unit. According to an embodiment, the posture information capture unit includes a plurality of two-dimensional posture estimation units 340-1, 340-2, ..., 340-N, and a plurality of match probability distribution generators 350-1, 350-2, ... , 350-N), and a multi-view heat map fusion unit 390. Each of the plurality of 2D posture estimation units 340-1, 340-2, ..., 340-N estimates and outputs the position of each joint point of the actor in the 2D image output from the corresponding camera. Since a method of estimating a predetermined number of body joint points in a 2D image is a known technique, a detailed description thereof will be omitted. Since the joint points estimated by the 2D posture estimator (340-1, 340-2, ..., 340-N) are not the positions of the joint points in the real space but the positions in the 2D camera image, is the coordinate value.

The match probability distribution generating unit (350-1, 350-2, ..., 350-N) arranges the joint points on the 2-dimensional image output from the corresponding 2-dimensional posture estimation unit by reflecting the camera parameters, and each isopolar line ( heat map information that expresses the probability of the true joint point for each pixel in the vicinity of the epipolar line is generated. 5 is a diagram illustrating epipolar geometry by way of example. According to the epipolar geometry, 2D image planes photographed at two camera positions are arranged by reflecting the alignment angles of the cameras, and the optical centers (C ₁ , C ₂ ) of the camera lenses are connected to each other. The true joint (X) can be obtained from the intersection of two epipolar lines, which are straight lines extending from the respective optical center points of the corresponding joint points (x,x') on two different 2D image planes. there is. In reality, since the location of the joint points on the 2D image plane estimated by the 2D posture estimators (340-1, 340-2, ..., 340-N) is not accurate, it can be expressed as the existence probability of the true joint points around the epipolar line. there is. A more detailed description can be found in the aforementioned Adafuse paper. According to this, the existence probability of a true joint point around the epipolar line is expressed as a two-dimensional Gaussian distribution.

The multi-view heat map convergence unit 390 fuses the heat map information of each 2D image, determines a match point having a high probability as a joint point, and outputs the result. A true joint point may be determined by fusing heat maps generated from epipolar lines of 2D joint points of each 2D image plane and selecting a pixel with the highest probability.

<Description of Claim 3 Invention>

According to one aspect, the posture information capture unit may further include a plurality of person detection units 330-1, 330-2, ..., 330-N. The person detectors 330-1, 330-2, ..., 330-N extract the human region from the 2D images output from the respective cameras, and the corresponding 2D posture estimators 340-1, 340-2, ... , 340-N). Since extracting a human region from an image is a well-known technique, a description thereof will be omitted. By extracting the human region in advance and estimating the 2D posture, the reliability of joint point estimation in the 2D image can be increased.

<Description of Claim 4 Invention>

According to an additional aspect, the momentum calculation unit may calculate the momentum by analyzing the estimated joint points on the time axis, calculating the momentum for each joint, and summing them up. 6 is a block diagram showing the configuration of an embodiment of the momentum calculation unit of FIG. 2 . The momentum calculation unit 600 according to an embodiment may include a unit motion classification processing unit 610 and a momentum analysis unit 631 for each unit motion. The unit motion classification processing unit 610 identifies and classifies unit motions from motion information generated by analyzing joint points on the time axis. These unit movements are defined as standard unit movements in ballet, and people can classify them according to their own criteria in sports such as soccer and volleyball. By tracking the three-dimensional joint point frames supplied consecutively on the time axis, detecting the section where the movement stops for a while, presenting candidate points for division, and determining the final division point among them with the help of a person. can be dealt with When these data are accumulated, for example, using a Recurrent Neural Network (RNN) learned for each unit motion, all sections of 3D frames in which the movement speed is normalized are processed, and the section outputting the highest probability value is a unit. action can be determined. In movements such as ballet in which the number of unit movements is in the hundreds, this processing may result in a very high processing load, but it is applicable in sports with a relatively small number of unit movements. By identifying the unit motion in the action of the actor, it is possible to visualize and output the unit motion of each section as a graphic user interface that divides the sections of the demonstration time based on the unit motion on the time axis indicated by a straight line and displays the unit motion of each section as text. In addition, there is a selection button for selecting playback for each section, and when selected, the 3D joint point frame of the corresponding section can be played through the modeling body or the corresponding section of the video can be played.

The momentum analysis unit 631 for each unit motion calculates the amount of momentum for each joint for the divided unit motion and sums it up over the modeling body to calculate the amount of momentum for each unit motion. The method for calculating the amount of exercise based on the modeling body has been described above. The process of calculating and accumulating the momentum for each minute displacement of all joints included in the modeling body may be repeated for the unit motion section to calculate the momentum for each unit motion section. In this way, it is possible to visualize how many times the performer repeated the same unit motion during the demonstration and to visualize the difference in momentum in the demonstration of the number of unit motions. In addition, in the evaluation of choreography, it is possible to evaluate whether the movement of a specific joint created is excessive or insufficient. For example, the degree of movement of each joint for each unit motion may be displayed as a color map that assigns a color to each joint of the modeling body.

<Description of Claim 5 Invention>

Additionally, the momentum calculation unit 600 may further include a total momentum analysis unit 633 . The total momentum analysis unit 633 calculates and outputs the total momentum over the entire demonstration by adding up the momentum for each unit motion. For example, in ballet, it is possible to quantify the cumulative value of the amount of motion for each joint in the entire demonstration of a specific choreography. Furthermore, it is possible to determine which joints are strained by a specific choreography, and injuries can be prevented by identifying joints that cause strain in sports.

<Description of Claims 6 and 7 Invention>

The exercise information analysis unit 230 may additionally include a calorie consumption calculation unit 650 . In one embodiment, the calorie consumption calculation unit 650 may include a calorie consumption calculation unit 651 for each unit operation. The calorie consumption calculation unit 651 for each unit motion calculates and outputs the calorie consumption for each unit motion based on the modeling body from the amount of exercise for each unit motion. The exercise amount information can be directly converted into calories. Additionally, the calorie consumption calculation unit 650 may include a total calorie consumption calculation unit 653 . The total calorie consumption calculation unit 653 may calculate and output the total calorie consumption by summing the calorie consumption for each unit operation.

<Description of the invention of claim 8>

7 is a block diagram showing the configuration of an embodiment of the motion information analysis unit of FIG. 2 . According to an additional aspect, the exercise information analysis unit may calculate and provide a motion risk for each joint. The exercise information analysis unit 230 according to an embodiment may further include an exercise risk analysis unit 231 . The motion risk analysis unit 231 analyzes the motion angular range and angular velocity for each joint among the momentum information for each joint output from the momentum calculation unit 600 on the time axis to calculate the risk and outputs the motion risk information for each joint. Risk can be calculated with one or more risk functions. For example, the risk function may be an angular motion range of a joint, an angular velocity, and an angular momentum size reflecting weight. As another example, the risk function may be a value obtained by quantifying body balance, that is, uniformity of weight distribution, calculated from locations of joint points. The risk level reflects the concentration of exercise for each joint. In other words, if the amount of exercise is not evenly distributed to all joints and is concentrated on a few specific joints, the risk may increase especially if the exercise is concentrated on weak joints in the body.

<Description of the invention of claim 9>

According to an additional aspect, the degree of agreement may be evaluated by comparing the actions of the actor and the comparison actor. The motion information analysis unit 230 may further include a motion comparison evaluation unit 233 . The motion comparison evaluation unit 233 evaluates the degree of agreement by comparing the coordinates output from the posture information capture unit with the captured motion information of the stored comparing agent. The motion information of the comparing actor may also be in the form of storing the positional coordinates of joint points together with corresponding timestamps. According to one aspect, the evaluation of motion consistency may be processed by applying dynamic time warping. The motion frame of the comparison actor corresponding to the 3D joint point frame of the actor can be determined, and the degree of coincidence of the coordinates of the joint points of the corresponding motion frame can be quantified using the concept of accumulated difference distance. Regarding the quantification of motion similarity using dynamic time warping, Jun-seok Lim and Jin-heon Kim. 2020, "Method for quantifying human body motion similarity using dynamic time warping (DTW) of keypoint information extracted from video", Journal of the Institute of Electrical and Electronics Engineers, vol.24, no.4 pp.1109-1116. For example, the technique disclosed in

<Description of Claim 11 Invention>

8 is a flowchart illustrating the configuration of a motion analysis method according to an embodiment. A method for analyzing an actor's motion according to an embodiment may be implemented as computer program instructions executed on a computer. As shown, the motion analysis method of an actor according to an embodiment includes a pose information capture step 810 and a motion information analysis step 830 . In the pose information capture step 810, the computer calculates the location of the actor's joint point from multiview images obtained from a plurality of cameras photographing the actor from different viewpoints. Output in dimensional coordinates. The motion information analysis step 830 includes a momentum calculation step 850 . In the momentum calculation step 850, the computer sequentially applies the joint point positions output in the posture information capture step 810 to the modeling body on the time axis to calculate the momentum for each joint and sums them up to calculate the momentum. These have been described with reference to FIG. 2 .

According to one aspect, the posture information capture step 810 may include a 2D posture estimation step 810 for each 2D image, a match probability distribution generating step 815, and a multi-view heat map fusion step 817. there is. In the 2D posture estimation step 810 for each 2D image, the computer estimates and outputs the position of each joint point of the actor in the 2D image output from each camera. In the match probability distribution generation step 815, the computer arranges the joint points on the 2D image output in the corresponding 2D posture estimation step by reflecting the camera parameters, and the true joint for each pixel around each epipolar line. Create heat map information that expresses the probability of points. In the multi-view heat map fusion step 817, the computer fuses the heat map information of each 2D image, determines a match point having a high probability as a joint point, and outputs the result. According to a further aspect, the posture information capture step 810 may further include a person detection step 811 before the 2D posture estimation step. In the human detection step 811, the computer extracts and outputs the human region from the 2D images output from each camera. Accordingly, the 2D posture estimation step 813 may be performed not on the 2D image but on human regions extracted from the 2D image. This has been described in detail with reference to FIGS. 3, 4 and 5.

According to an additional aspect, the momentum calculation step 850 may include a unit motion classification processing step 851 and a momentum analysis step 853 for each unit motion section. In the unit motion classification processing step 851, the computer identifies and classifies the unit motion from motion information generated by analyzing the joint points on the time axis. In the momentum analysis step for each unit motion section (853), the computer calculates the amount of momentum for each joint for the divided unit motion and sums it up over the modeling body to calculate the amount of momentum for each unit motion. Additionally, the momentum calculation step 850 may further include a total momentum analysis step 855 . In the total momentum analysis step 855, the computer calculates and outputs the total momentum by adding up the momentum for each unit motion.

According to an additional aspect, the exercise information analysis step 830 may further include an exercise risk analysis step 833 . In the motion risk analysis step 833, the computer analyzes the motion angular range and angular velocity for each joint among the momentum information for each joint output in the momentum calculation step 850 on the time axis to calculate the risk and outputs the motion risk information for each joint. This has been described with reference to FIGS. 6 and 7 .

In the above, the present invention has been described through embodiments with reference to the accompanying drawings, but is not limited thereto, and should be interpreted to cover various modifications that can be obviously derived by those skilled in the art. The claims are intended to cover these variations.

10: camera control unit 20: actor
30-1 to 30-5: multi-view cameras
50: server computer
71: personal computer 73: smartphone
210: posture information capture unit 230: exercise information analysis unit
231: momentum calculation unit 233: motion comparison evaluation unit
250: user interface unit
251: key code 253: display
310, 330, 350: joint 320, 340, 360: weight
313, 335, 359: wire
600: momentum calculation unit
610: unit motion classification processing unit 631; Momentum analysis unit for each unit motion
640: modeling body
633: total momentum analysis unit
650: Calorie consumption calculation unit
651: Calorie consumption calculation unit for each unit operation
653: Total Calorie Calculation Unit

Claims (16)

a plurality of cameras that take pictures of an actor from different viewpoints;
a pose information capture unit that outputs the position of an actor's joint point in 3D coordinates from multiview images obtained from a plurality of cameras;
A motion information analysis unit including a momentum calculation unit that calculates a momentum for each joint by sequentially applying the joint point positions output from the posture information capture unit to the modeling body on the time axis and sums them up to calculate the momentum;
a user interface unit that provides analyzed results and receives necessary instructions or information;
Motion analysis system comprising a. The method according to claim 1, the attitude information capture unit:
a plurality of 2D posture estimators for estimating and outputting the position of each joint point of the actor in the 2D image output from each camera;
The joint points on the 2D image output from the corresponding 2D posture estimation unit are arranged by reflecting the camera parameters, and heat map information expressing the probability of the true joint point for each pixel around each epipolar line is generated. a plurality of match probability distribution generating units for generating;
a multi-view heat map convergence unit that fuses heat map information of each 2D image, determines a match point with a high probability as a joint point, and outputs the result;
Motion analysis system comprising a. The method according to claim 2, the posture information capture unit
A plurality of human detecting parts for extracting a human area from a 2D image output from each camera and outputting the extracted human region to a corresponding 2D posture estimator (human detecting part);
Motion analysis system further comprising a. The method according to claim 1, the momentum calculation unit:
a unit motion classification processing unit that identifies and classifies unit motions from motion information generated by analyzing joint points on a time axis;
a momentum analysis unit for each unit motion section that calculates the amount of momentum for each joint for the divided unit motion and sums it up over the modeling body to calculate the amount of momentum for each unit motion;
Motion analysis system comprising a. The method according to claim 4, the momentum calculation unit:
A total momentum analysis unit that calculates and outputs the total momentum by summing the momentum for each unit motion;
Motion analysis system further comprising a. The method according to claim 4, exercise information analysis unit:
a calorie consumption calculation unit for each unit movement that calculates and outputs the calorie consumption for each unit movement based on the modeling body from the amount of exercise for each unit movement;
Motion analysis system further comprising a. The method according to claim 4, the momentum calculation unit:
a total calorie consumption calculation unit that calculates and outputs a total calorie consumption by summing calorie consumption for each unit operation;
Motion analysis system further comprising a. The method according to claim 1, exercise information analysis unit:
A motion risk analysis unit that analyzes the angular motion range and angular velocity for each joint among the momentum information for each joint output from the momentum calculation unit on the time axis to calculate a risk and outputs it as motion risk information for each joint;
Motion analysis system further comprising a.
The method according to claim 1, exercise information analysis unit:
An operation of comparing the coordinates output from the posture information capture unit with the captured motion information of the stored comparing agent to evaluate the degree of agreement and output the evaluation result, but to compare motions by applying dynamic time warping. comparative evaluation department;
Motion analysis system further comprising a. The system of claim 1 , wherein the system:
a modeling register for receiving new modeling body information through the user interface and storing it in a modeling body information database;
Motion analysis system further comprising a. A method for analyzing the motion of an actor implemented as computer program instructions executed on a computer,
A pose information capture step of outputting the joint position of the actor in 3D coordinates from multiview images obtained from a plurality of cameras photographing the actor from different viewpoints; and ;
A motion information analysis step including a momentum calculation step of sequentially applying the joint point positions output in the posture information capture step to the modeling body on the time axis to calculate a momentum for each joint and summing them to calculate the momentum;
Motion analysis method comprising a. The method according to claim 11, wherein the posture information capture step is:
a 2D posture estimation step of estimating and outputting the location of each joint point of the actor in the 2D image output from each camera;
The joint points on the 2D image output in the corresponding 2D posture estimation step are arranged by reflecting the camera parameters, and heat map information expressing the probability of the true joint point for each pixel around each epipolar line is generated. a match probability distribution generating step;
a multi-view heat map fusion step of fusing heat map information of each two-dimensional image, determining a match point having a high probability as a joint point, and outputting the result;
Motion analysis method comprising a. The method according to claim 12, wherein the posture information capture step is prior to the two-dimensional posture estimation step
a person detection step of extracting and outputting a human region from a 2D image output from each camera;
Motion analysis method further comprising a. The method according to claim 11, wherein the momentum calculation step is:
a unit motion classification processing step of identifying and classifying unit motions from motion information generated by analyzing joint points on a time axis;
A momentum analysis step for each unit motion section of calculating the amount of momentum for each joint for the divided unit motion and summing it over the modeling body to calculate the amount of momentum for each unit motion;
Motion analysis method comprising a. The method according to claim 14, wherein the momentum calculation step is:
A total momentum analysis step of calculating and outputting a total momentum by summing up the momentum for each unit motion;
Motion analysis method further comprising a. The method according to claim 11, wherein the motion information analysis step is:
A motion risk analysis step of calculating a risk by analyzing the motion angular range and angular velocity for each joint among the motion information for each joint output in the momentum calculation step on the time axis, and outputting the motion risk information for each joint;
Motion analysis method further comprising a.

Images