CN110711374B - Multi-modal dance action evaluation method - Google Patents

Multi-modal dance action evaluation method Download PDF

Info

Publication number
CN110711374B
CN110711374B CN201910979004.1A CN201910979004A CN110711374B CN 110711374 B CN110711374 B CN 110711374B CN 201910979004 A CN201910979004 A CN 201910979004A CN 110711374 B CN110711374 B CN 110711374B
Authority
CN
China
Prior art keywords
dance
weight
scoring
joint
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910979004.1A
Other languages
Chinese (zh)
Other versions
CN110711374A (en
Inventor
王正友
张志涛
王长明
乔丽方
马丽琴
张萍
毛立军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shijiazhuang Boom Electronics Co ltd
Shijiazhuang Tiedao University
Original Assignee
Shijiazhuang Boom Electronics Co ltd
Shijiazhuang Tiedao University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shijiazhuang Boom Electronics Co ltd, Shijiazhuang Tiedao University filed Critical Shijiazhuang Boom Electronics Co ltd
Priority to CN201910979004.1A priority Critical patent/CN110711374B/en
Publication of CN110711374A publication Critical patent/CN110711374A/en
Application granted granted Critical
Publication of CN110711374B publication Critical patent/CN110711374B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B71/00—Games or sports accessories not covered in groups A63B1/00 - A63B69/00
    • A63B71/06—Indicating or scoring devices for games or players, or for other sports activities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/05—Image processing for measuring physical parameters
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/80—Special sensors, transducers or devices therefor
    • A63B2220/806—Video cameras
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2220/00—Measuring of physical parameters relating to sporting activity
    • A63B2220/80—Special sensors, transducers or devices therefor
    • A63B2220/808—Microphones
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
    • A63B2244/00—Sports without balls
    • A63B2244/22—Dancing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Psychiatry (AREA)
  • Biophysics (AREA)
  • Databases & Information Systems (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Medical Informatics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Physical Education & Sports Medicine (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a multi-modal dance action evaluation method, and relates to the technical field of data identification methods. The method comprises the following steps: acquiring dance actions of dancers according to sound, a color camera and a depth camera, and firstly, identifying the acquired dancers; marking the joint points of the dancer according to the color image and the depth image mapping, and storing the coordinates of the joint points through the depth image; dividing dance movements according to preset music key points, comparing the divided movements with standard template movements in a preset moving window, and matching the best corresponding movements; and according to the standard template, the speed, the acceleration, the curvature proficiency, the motion trail and the integral data deviation, carrying out background scoring and combining with real-time scoring to finally obtain the total score of each module, and finally scoring the existing action according to the weight parameters trained by the expert data set to obtain the final result. The method has the advantage of high scoring accuracy.

Description

Multi-modal dance action evaluation method
Technical Field
The invention relates to the technical field of data identification methods, in particular to a multi-modal dance action evaluation method.
Background
The existing dance evaluation system is comparison software with a given difficulty coefficient, namely, the information about the dance is recorded into the system through on-line registration and registration. The APP can automatically group, combine and group, and formulate a course table. Meanwhile, the number of performers can be increased or decreased at any time and synchronized to the system, and the scores are input by the commentator in the competition process and displayed on a large screen in real time. However, in the evaluation system, evaluation by a client is still performed first, then scores are manually input, and final scores and ranking are obtained after system statistics. For example, the human is limited in energy, and as time goes on, the phenomenon of misjudgment and missed judgment may occur due to inattention; further, since different judges may not agree in the evaluation criterion and the evaluation scale, a situation may be caused in which some persons are at a disadvantage in terms of difference in the evaluation criterion.
Although the technology for recognizing and capturing the body movements of the performer is available, most of the technologies are complex and inefficient in technology and prevent the performer from playing dance by installing a capturing and collecting device on the body of the performer. Meanwhile, the problem of insufficient data amount exists when selective information acquisition is carried out through equipment worn on the body of a performer, such as LED lamp decorations, and comprehensiveness of judgment results can be affected to a certain extent.
Disclosure of Invention
The invention aims to solve the technical problem of how to provide a multi-modal dance action evaluation method with high scoring accuracy.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows: a multi-modal dance action evaluation method is characterized by comprising the following steps:
acquiring dance actions of dancers according to sound, a color camera and a depth camera, and firstly, identifying the acquired dancers; marking the joint points of the dancer according to the color image and the depth image mapping, and storing the coordinates of the joint points through the depth image;
dividing dance movements according to preset music key points, comparing the divided movements with standard template movements in a preset moving window, and matching the best corresponding movements;
the standard template comprises dance movement, speed, acceleration, proficiency and movement track information, and the test dance is processed by a grading system to obtain an evaluation score of each module of the dancer; and obtaining the evaluation weight of the expert through particle swarm iterative training, and finally obtaining the total evaluation of the dancer according to the trained modules and the weight vector of each action and the evaluation score of each current module.
The further technical scheme is as follows: the method comprises the steps of preprocessing original data, and performing real-time Holt dual-parameter linear smoothing; and the background scoring is stage scoring and Kalman filtering is adopted.
The further technical scheme is that the Holt double-parameter linear smoothing comprises the following steps:
holt double parametric linear exponential smoothing:
Figure BDA0002234574170000021
wherein alpha and gamma are smoothing parameters, xtIs the actual observed value; t' is the number of predicted times, StCorrecting the smoothed value for time t, btIs a trend value at time t, Ft+T′Indicating the prediction gain after time T'.
The further technical scheme is as follows: matching by adopting a Mahalanobis distance similarity method:
Figure BDA0002234574170000022
wherein the content of the first and second substances,
Figure BDA0002234574170000023
representing two matching motion vectors that are,
Figure BDA0002234574170000024
representing the distance of two vectors, sigma-1 being the covariance momentArraying;
the FastDTW method is adopted, and the constraint conditions are as follows:
Figure BDA0002234574170000025
wherein g (i, j) represents the dynamic programming which is the accumulated distance of two actions at the ith and jth time point, and d (i, j) represents xiAnd yjThe distance between them.
The further technical scheme is that the method based on the template action score comprises the following steps:
extracting a correlation coefficient between the test dance action and the template action through the correlation modulus of the quaternion,
Figure BDA0002234574170000031
wherein p and q represent a test sequence and a template sequence, respectively,
Figure BDA0002234574170000032
for the cross-covariance coefficients, C denotes the covariance matrix,
Figure BDA0002234574170000033
and
Figure BDA0002234574170000034
standard deviation for both actions;
Figure BDA0002234574170000035
Figure BDA0002234574170000036
q (t) represents the state of the template sequence at time t, p (t) represents the state of the test sequence at time t, pv(t) represents the mean deviation of the test data, qv(t) represents the deviation from the mean of the template data.
T represents a single dance action time, pcIndicates that the test data is expected at T interval, pvRepresenting the expected difference between each point of the sequence; τ represents in the minimum neighborhood;
the scores after adding the weight coefficients were:
Figure BDA0002234574170000037
S(wpos) The total score after the weighting is represented,
Figure BDA0002234574170000038
representing body joint point weight, S1,jRepresenting unweighted singleton scores for individual actions, and J represents the total number of dance key actions.
The further technical scheme is that the method based on the speed scoring comprises the following steps: and constructing a matching sequence according to a template matching result, grading the speed by comparing the sequence, constructing the correlation degree by the grading rule and the correlation coefficient, and simultaneously performing corresponding reduction grading on the time sequence delayed by the same action.
The further technical scheme is that the method based on the acceleration scoring comprises the following steps: and (4) estimating the continuity of the dance movements and the coordination of the force condition and the strength of the dancer in each movement according to the acceleration.
The further technical scheme is that the proficiency scoring method based on the curvature comprises the following steps:
windowing the covariance matrix of the sampled discrete points of the curve, and solving the characteristic root product of the covariance matrix, namely an estimated curvature formula, as shown in a formula (6); calculating motion smoothness according to the curvature, wherein the motion smoothness is shown as a formula (7);
Figure BDA0002234574170000041
Figure BDA0002234574170000042
wherein C represents a covariance matrix, λiDenotes the curvature, kmeanDenotes the mean curvature, knThe curvature at time n is shown and Smoothness shows Smoothness.
The further technical scheme is as follows: optimizing the scoring weight by adopting a particle swarm algorithm, wherein the optimization weight is divided into two parts: a first part: an action weight; a second part: the scoring module weight, the normalized score is shown as formula (11);
according to the difference of the structure of the human body joint point, the joint point is divided into 5 parts, namely a first part: the trunk including shoulder joints, spine joints, hip joints and hip joints; a second part, the elbow joint, comprising a left elbow joint and a right elbow joint; a third portion, a knee joint, comprising a left knee joint and a right knee joint; the fourth part, the head joint and the neck joint; the fifth part, the joints of the hand and the foot, including the wrist joint, the ankle joint and the finger joint; the weights are respectively defined as wt,we,wk,wh,wf；
The scoring module is divided into: an action module, a speed module, an acceleration module, a proficiency module and a motion trail module, wherein the weight is defined as wpos,wvel,wacc,wcur,wshpThe module normalization score is shown as formula (12);
Figure BDA0002234574170000043
wherein
Figure BDA0002234574170000051
Represents the body joint weight, S (w)pos) Is weight normalization, S1,jRepresent jth module weighted score:
Figure BDA0002234574170000052
wherein
Figure BDA0002234574170000053
Representative Module Scoring weight, TiThe base module score representing the jth joint point.
The further technical scheme is as follows: optimizing the weight value by utilizing a particle swarm optimization algorithm through data set training and iteration to finally obtain the optimal weight, wherein the fitness function is shown as a formula (13);
Figure BDA0002234574170000054
wherein C isj，C'jRespectively representing the unweighted actual score and the template expert score, wjRepresenting the current weight vector of the particle j; then, the particles are iterated, when the iteration reaches the maximum iteration times or the error is small enough, the iteration is finished, and the vector corresponding to the current position is the optimal weight vector; and finally, scoring the tested data through the weight vector to obtain the final score of the dancer.
Adopt the produced beneficial effect of above-mentioned technical scheme to lie in: according to the method, original movement data are optimized through two preprocessing methods, dance characteristics are expressed as data information of movement, speed, acceleration, proficiency and movement track, and dance characteristic weight vectors are trained and calculated according to a preset standard data set. The objective characteristic score and the characteristic weight are combined to obtain the score which is most consistent with the real condition, so that the method is high in calculation accuracy.
Drawings
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
FIG. 1 is a comparison of human joint points in a method according to an embodiment of the invention;
FIG. 2 is a block flow diagram of a method according to an embodiment of the invention;
FIG. 3 is a flow chart of Kalman filtering in a method according to an embodiment of the invention;
FIG. 4 is a graph of velocity versus acceleration in a method according to an embodiment of the present invention;
FIG. 5 is a flow chart of a particle group optimization algorithm in the method according to an embodiment of the present invention;
wherein: 1. head 2, neck 3, shoulder center 4, left thumb 5, right thumb 6, left fingertip 7, right fingertip 8, left hand 9, right hand 10, left wrist 11, right wrist 12, left elbow 13, right elbow 14, left shoulder 15, right shoulder 16, spine 17, hip center 18, left hip 19, left knee 20, left ankle 21, left foot 22, right hip 23, right knee 24, right ankle 25, right foot.
Detailed Description
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.
The Kinectv2 somatosensory device consists of a color camera, two infrared cameras and an audio acquisition array, and is used for acquiring color image information, depth image information and audio information respectively. The video acquisition frame rate is 30fps, the three-dimensional coordinate position of a human body can be calculated according to the depth image, and the test error is less than 5mm within the effective detection distance (1-4 m). The coordinates of 25 Joint points of a human body can be detected through Microsoft Kinect SDK and stored in a Joint structure, quaternions are stored in an organization structure, and Joint point comparison is shown in figure 1.
The identity of the dance testing personnel is verified by the color camera (1920X 1080); acquiring a depth image by an infrared camera, and calculating joint point information; the microphone array collects music beats corresponding to dance actions. Most dance movements can be completed within an effective detection range.
The Kinect acquires the coordinates of the joint points and contains certain noise, such as indoor illumination, wearing of people, shading and the like. The method is divided into two types in a general view, namely a system error and an accidental error. The systematic error is determined by the characteristics of the instrument, the error of 5mm is acceptable in dance motion, the accidental error is subject to positive distribution according to probability, the calculation can be carried out by using an average method in calibration, but due to the unstable characteristic of noise, filtering smoothing processing is required. The method adopts two methods, namely Holter double-parameter linear smoothing and Kalman filtering, which are real-time filtering and module filtering respectively.
Dance movements have spatiotemporal characteristics, and the movements need to be matched spatio-temporally. According to different dance and scoring rules, action weights are specifically distributed, so the method provides a multi-mode action-based scoring rule. Each dance motion must be completed within a specified time, and scores can be completed within a specified range of the standard dance completion time, otherwise, scores are not scored. The method adopts Dynamic Time Warping (DTW) to match actions in a specified range, and scores speed, acceleration, curvature proficiency, motion trail, real-time data and action standard, wherein the process is shown in figure 2.
As shown in FIG. 2, the embodiment of the invention discloses a multi-modal dance motion evaluation method, which comprises the following steps:
acquiring dance actions of dancers according to sound, a color camera and a depth camera, and firstly, identifying the acquired dancers; marking the joint points of the dancer according to the color image and the depth image mapping, and storing the coordinates of the joint points through the depth image;
dividing dance movements according to preset music key points, comparing the divided movements with standard template movements in a preset moving window, and matching the best corresponding movements;
the standard template comprises dance movement, speed, acceleration, proficiency and movement track information, and the test dance is processed by a grading system to obtain an evaluation score of each module of the dancer; and obtaining the evaluation weight of the expert through particle swarm iterative training, and finally obtaining the total evaluation of the dancer according to the trained modules and the weight vector of each action and the evaluation score of each current module.
The dance data preprocessing acquisition and optimization method comprises the following steps: the method selects two different data processing methods which are respectively based on real-time optimization and prediction optimization. Because the dance place is indoor, the ambient light is better, and image quality is better, need not carry out the preprocessing of making an uproar that falls to the image. The main preprocessing part is divided into two parts, namely a joint point accuracy and an anti-noise performance. The joint point data smooth filtering effect based on the time sequence is better when processed by the Kalman filtering and Holt double-parameter index smoothing method. However, Holt bi-parametric linear exponential smoothing is faster in real-time processing compared with kalman filtering, but the effect is inferior to kalman filtering. Therefore, the method adopts a Holt double-parameter exponential smoothing method in real-time scoring, and adopts a Kalman filtering method in background processing.
The Holt two-parameter linear smoothing comprises the following steps:
holt double parametric linear exponential smoothing:
Figure BDA0002234574170000071
wherein alpha and gamma are smoothing parameters, xtIs the actual observed value; t' is the number of predicted times, StCorrecting the smoothed value for time t, btIs a trend value at time t, Ft+T′Indicating the prediction gain after time T'.
The kalman filtering flow is shown in fig. 3, wherein,
Figure BDA0002234574170000081
and pkIs to collect the best estimated coordinate information and the covariance matrix of the prior state estimation of the original signal, FkIn order to convert the matrix, the first and second matrices,
Figure BDA0002234574170000082
Bkin order to control the matrix of the control,
Figure BDA0002234574170000083
for the control vector the project has no external control quantity, so the two terms are 0; prediction denotes the process of prediction by velocity and acceleration, QkRepresents the covariance of the random noise; hkRepresenting measured data, RkWhich represents the covariance of the sensor,
Figure BDA0002234574170000084
means for representing measured data, K ═ Σ0(∑0+∑1)-1Representing a Kalman gain;
Figure BDA0002234574170000085
and p'kRepresenting the new optimal estimated coordinates and covariance.
The dance data matching method comprises the following steps:
the weight distribution usually adopts a fuzzy analytic hierarchy process, and gives a preliminary weight ratio according to expert review opinions. Weight vector (w)per) As follows:
wper＝[wper1,wper2,...,wpern] (2)
the current methods for motion estimation mainly include hidden markov (HMM) and Dynamic Time Warping (DTW). Because dance movements have continuity and more sampling points, the DTW method is adopted for real-time matching. And matching the initial dance movements, matching the initial dance movements with the front half parts of the template movements, sequentially matching the initial dance movements backwards if the initial dance movements are wrong, and setting the front half parts to be zero. The traditional matching adopts the Euclidean distance to calculate the similarity of two actions, and because the difference of the motion amplitudes of different joint points is large, a large error is generated.
Figure BDA0002234574170000086
Figure BDA0002234574170000087
Representing two matching motion vectors that are,
Figure BDA0002234574170000088
representing the distance of two vectors, and sigma-1 is the covariance matrix.
In order to improve the running speed of the system and reduce the operation amount, the method adopts a FastDTW method, and the constraint conditions are as follows:
Figure BDA0002234574170000091
wherein: g (i, j) represents the cumulative distance of two actions at the ith, j time point, i.e. Dynamic Programming (DP)
The dance scoring method comprises the following steps:
(1) template-based action scoring
Extracting a correlation coefficient between the test dance action and the template action through the correlation modulus of the quaternion,
Figure BDA0002234574170000097
wherein p and q represent a test sequence and a template sequence, respectively,
Figure BDA0002234574170000092
for the cross-covariance coefficients, C denotes the covariance matrix,
Figure BDA0002234574170000093
and
Figure BDA0002234574170000094
standard deviation for both actions;
Figure BDA0002234574170000095
Figure BDA0002234574170000096
q (t) represents the state of the template sequence at time t, p (t) represents the state of the test sequence at time t, pv(t) represents the mean deviation of the test data, qv(t) represents the deviation from the mean of the template data.
T represents a single dance action time, pcIndicates that the test data is expected at T interval, pvRepresenting the expected difference between each point of the sequence; τ represents in the minimum neighborhood;
the scores after adding the weight coefficients were:
Figure BDA0002234574170000101
S(wpos) The total score after the weighting is represented,
Figure BDA0002234574170000102
representing body joint point weight, S1,jRepresenting unweighted singleton scores for individual actions, and J represents the total number of dance key actions.
(2) Scoring based on velocity
And constructing a matching sequence according to a template matching result, grading the speed by comparing the sequence, constructing the correlation degree by the grading rule and the correlation coefficient, and simultaneously performing corresponding reduction grading on the time sequence delayed by the same action.
(3) Scoring based on acceleration
And (4) estimating the continuity of the dance movements and the coordination of the force condition and the strength of the dancer in each movement according to the acceleration. Because dance motion continuity and key motion require strength and coordination, but the Kinect somatosensory equipment does not have a sensor for detecting strength, acceleration can powerfully reflect the strength of a dancer. FIG. 4 is a graph of speed and acceleration surf, herein for example, of a left and right Tai Chi horsehair.
(4) Proficiency scoring based on curvature
The method mainly detects the action standard degree of dancers, so that the skill degree of dancers is also listed as an important standard in the examination range. According to dance speed analysis, the dancer (except mechanical dancing) with higher proficiency moves more smoothly, the effect is more consistent, and the coordination is stronger. The method measures the smoothness of the motion using a curvature method.
From differential geometry knowledge, a curve in the neighborhood of a point on the curve in arc length approximates straight lines and parabolas. The curvature is the degree of curvature of the reaction curve. And (3) adding the window covariance matrix to the sampled discrete points of the curve, and solving the characteristic root product of the covariance matrix to obtain the estimated curvature (formula 9). The motion smoothness is obtained from the curvature (equation 10).
Figure BDA0002234574170000111
Figure BDA0002234574170000112
Wherein C represents a covariance matrix, λiDenotes the curvature, kmeanDenotes the mean curvature, knThe curvature at time n is shown and Smoothness shows Smoothness.
(5) Scoring based on motion trajectory
In some dances, the movement track is an important index for measuring the dancing quality of dancers, such as taijiquan. The method selects Spine _ base points as reference points, and the points are the most stable of human body joint points. And comparing the point track to the standard track according to the test, and grading.
The comprehensive scoring method comprises the following steps: the score of each feature method is obtained according to the dance features, but direct definition is difficult to be given to the weight ratio, and the weight ratio is difficult to be directly obtained through expert opinions. The expert does not give the score of each part in the actual scoring process, so that a function cannot be fitted to calculate the weight coefficient, the scoring of the expert in the game is used as training data in the method, iterative learning and training are carried out by utilizing a Particle Swarm Optimization (PSO), and finally the optimal model structure is found.
And taking the expert score as a training set, wherein each particle represents a data weighting, and an optimal weight is trained through iteration.
The method adopts a Particle Swarm Optimization (PSO) to optimize the scoring weight. The main optimization weight is divided into two parts: 1. an action weight; 2. and grading module weight. The normalized score is shown in equation (11).
According to the different structures of the human body joint points, the framework divides the joint points into 5 parts: the body (shoulder, spine, hip joint), elbow joint (left and right elbows), knee joint (left and right knees), head neck joint, hand and foot joint (wrist, ankle, fingers), the weight is defined as wt,we,wk,wh,wf。
The scoring module is divided into: motion, velocity, acceleration, proficiency and trajectory of motion, weight being defined as wpos,wvel,wacc,wcur,wshp. The module normalized score is shown as equation (12).
Figure BDA0002234574170000121
Wherein
Figure BDA0002234574170000122
Represents the body joint weight, S (w)pos) Is weight normalization, S1,jRepresenting the jth module weighted score.
Figure BDA0002234574170000123
Wherein
Figure BDA0002234574170000124
Representative Module Scoring weight, TiBase module score representing jth joint point
As shown in fig. 5, the particle swarm optimization algorithm is a global optimization algorithm, which is a random search method based on the swarm to find the optimal solution. The framework optimizes the weight value through data set training and iteration by using a particle swarm optimization algorithm, and finally obtains the optimal weight. The fitness function is shown as equation (13).
Figure BDA0002234574170000125
Wherein C isj，C'jRespectively representing the unweighted actual score and the template expert score, wjRepresenting the current weight vector of the particle j; then, the particles are iterated, Ebest represents an individual optimal value, Gbest represents a group optimal value, when the iteration reaches the maximum iteration number or the error is small enough, the iteration is finished, and the vector corresponding to the current position is the optimal weight vector; and finally, scoring the tested data through the weight vector to obtain the final score of the dancer.

Claims (8)

1. A multi-modal dance action evaluation method is characterized by comprising the following steps:
acquiring dance motions of a dancer according to sound, a color camera and a depth camera, firstly identifying the acquired dancer, then marking joint points of the dancer according to color images and depth image mapping, and storing the coordinates of the joint points through the depth image;
dividing dance movements according to preset music key points, comparing the divided movements with standard template movements in a preset moving window, and matching the best corresponding movements;
the standard template comprises dance movement, speed, acceleration, proficiency and movement track information, and the test dance is processed by a grading system to obtain an evaluation score of each module of the dancer; obtaining the evaluation weight of an expert through particle swarm iterative training, and finally obtaining the overall evaluation of the dancer according to the trained modules and the weight vector of each action and the evaluation score of each current module;
optimizing the scoring weight by adopting a particle swarm algorithm, wherein the optimization weight is divided into two parts: a first part: an action weight; a second part: the scoring module weight, the normalized score is shown as formula (11);
according to the difference of the structure of the human body joint point, the joint point is divided into 5 parts, namely a first part: the trunk including shoulder joints, spine joints, hip joints and hip joints; a second part, the elbow joint, comprising a left elbow joint and a right elbow joint; a third portion, a knee joint, comprising a left knee joint and a right knee joint; the fourth part, the head joint and the neck joint; the fifth part, the joints of the hand and the foot, including the wrist joint, the ankle joint and the finger joint; the weights are respectively defined as wt,we,wk,wh,wf；
The scoring module is divided into: an action module, a speed module, an acceleration module, a proficiency module and a motion trail module, wherein the weight is defined as wpos,wvel,wacc,wcur,wshpThe module normalization score is shown as formula (12);
Figure FDA0002965854040000011
wherein
Figure FDA0002965854040000012
Represents the body joint weight, S (w)pos) Is weight normalization, S1,jRepresent jth module weighted score:
Figure FDA0002965854040000021
wherein
Figure FDA0002965854040000022
Representative Module Scoring weight, TiA base module score representing a jth joint point;
optimizing the weight value by utilizing a particle swarm optimization algorithm through data set training and iteration to finally obtain the optimal weight, wherein the fitness function is shown as a formula (13);
Figure FDA0002965854040000023
wherein C isj，C'jRespectively representing the unweighted actual score and the template expert score, wjRepresenting the current weight vector of the particle j; then, the particles are iterated, when the iteration reaches the maximum iteration times or the error is small enough, the iteration is finished, and the vector corresponding to the current position is the optimal weight vector; and finally, scoring the tested data through the weight vector to obtain the final score of the dancer.
2. The multi-modal dance motion assessment method of claim 1, wherein: the method comprises the steps of preprocessing original data, and performing real-time Holt dual-parameter linear smoothing; and the background scoring is stage scoring and Kalman filtering is adopted.
3. The multi-modal dance motion assessment method of claim 2, wherein: the Holt two-parameter linear smoothing comprises the following steps:
holt double parametric linear exponential smoothing:
Figure FDA0002965854040000024
wherein alpha and gamma are smoothing parameters, xtIs the actual observed value; t' is the number of predicted times, StCorrecting the smoothed value for time t, btIs a trend value at time t, Ft+T′Indicating the prediction gain after time T'.
4. The multi-modal dance motion assessment method of claim 1, wherein:
matching by adopting a Mahalanobis distance similarity method:
Figure FDA0002965854040000025
wherein the content of the first and second substances,
Figure FDA0002965854040000026
representing two matching motion vectors that are,
Figure FDA0002965854040000027
represents the distance of two vectors, sigma-1 is a covariance matrix;
the FastDTW method is adopted, and the constraint conditions are as follows:
Figure FDA0002965854040000031
wherein g (i, j) represents the dynamic programming which is the accumulated distance of two actions at the ith and jth time point, and d (i, j) represents xiAnd yjThe distance between them.
5. The multi-modal dance motion assessment method of claim 1, wherein: the template-based action scoring method is as follows:
extracting a correlation coefficient between the test dance action and the template action through the correlation modulus of the quaternion,
Figure FDA0002965854040000032
wherein p and q represent a test sequence and a template sequence, respectively,
Figure FDA0002965854040000033
for the cross-covariance coefficients, C denotes the covariance matrix,
Figure FDA0002965854040000034
and
Figure FDA0002965854040000035
standard deviation for both actions;
Figure FDA0002965854040000036
Figure FDA0002965854040000037
q (t) represents the state of the template sequence at time t, p (t) represents the state of the test sequence at time t, pv(t) represents the mean deviation of the test data, qv(t) represents the mean deviation of the template data;
t represents a single dance action time, pcIndicates that the test data is expected at T interval, pvRepresenting the expected difference between each point of the sequence; τ represents in the minimum neighborhood;
the scores after adding the weight coefficients were:
Figure FDA0002965854040000038
S(wpos) The total score after the weighting is represented,
Figure FDA0002965854040000039
representing body joint point weight, S1,jRepresenting unweighted singleton scores for individual actions, and J represents the total number of dance key actions.
6. The multi-modal dance motion assessment method of claim 1, wherein: the method based on speed scoring is as follows: and constructing a matching sequence according to a template matching result, grading the speed by comparing the sequence, constructing the correlation degree by the grading rule and the correlation coefficient, and simultaneously performing corresponding reduction grading on the time sequence delayed by the same action.
7. The multi-modal dance motion assessment method of claim 1, wherein: the method based on acceleration scoring is as follows: and (4) estimating the continuity of the dance movements and the coordination of the force condition and the strength of the dancer in each movement according to the acceleration.
8. The multi-modal dance motion assessment method of claim 1, wherein: the method for proficiency scoring based on curvature is as follows:
windowing the covariance matrix of the sampled discrete points of the curve, and solving the characteristic root product of the covariance matrix, namely an estimated curvature formula, as shown in a formula (6); calculating motion smoothness according to the curvature, wherein the motion smoothness is shown as a formula (7);
Figure FDA0002965854040000041
Figure FDA0002965854040000042
wherein C represents a covariance matrix, λiDenotes the curvature, kmeanDenotes the mean curvature, knThe curvature at time n is shown and Smoothness shows Smoothness.
CN201910979004.1A 2019-10-15 2019-10-15 Multi-modal dance action evaluation method Active CN110711374B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910979004.1A CN110711374B (en) 2019-10-15 2019-10-15 Multi-modal dance action evaluation method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910979004.1A CN110711374B (en) 2019-10-15 2019-10-15 Multi-modal dance action evaluation method

Publications (2)

Publication Number Publication Date
CN110711374A CN110711374A (en) 2020-01-21
CN110711374B true CN110711374B (en) 2021-05-04

Family

ID=69212622

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910979004.1A Active CN110711374B (en) 2019-10-15 2019-10-15 Multi-modal dance action evaluation method

Country Status (1)

Country Link
CN (1) CN110711374B (en)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782858B (en) * 2020-03-31 2024-04-05 北京沃东天骏信息技术有限公司 Music matching method and device
CN111523517B (en) * 2020-05-26 2023-08-04 北京奇艺世纪科技有限公司 Action scoring method and device, electronic equipment and readable storage medium
CN111672087B (en) * 2020-06-13 2021-05-07 曲灏辰 Muscle vibration detection system and detection method suitable for street dance
CN111803885A (en) * 2020-07-11 2020-10-23 郭坤 Intelligent exercise method based on international standard dance
CN112221107A (en) * 2020-08-31 2021-01-15 广州市美图现电子有限公司 Intelligent fitness mirror control method and intelligent fitness mirror
CN112381035A (en) * 2020-11-25 2021-02-19 山东云缦智能科技有限公司 Motion similarity evaluation method based on motion trail of skeleton key points
CN112418153A (en) * 2020-12-04 2021-02-26 上海商汤科技开发有限公司 Image processing method, image processing device, electronic equipment and computer storage medium
CN112675520A (en) * 2020-12-30 2021-04-20 四川工商学院 Group exercise tester based on human body motion capture
CN113239848B (en) * 2021-05-27 2024-02-02 数智引力(厦门)运动科技有限公司 Motion perception method, system, terminal equipment and storage medium
CN113611387B (en) * 2021-07-30 2023-07-14 清华大学深圳国际研究生院 Motion quality assessment method based on human body pose estimation and terminal equipment
CN113617004A (en) * 2021-08-03 2021-11-09 燕山大学 Training method for improving shooting hit rate of basketball players
CN113822182B (en) * 2021-09-08 2022-10-11 河南理工大学 Motion action detection method and system
CN115100745B (en) * 2022-07-05 2023-06-20 北京甲板智慧科技有限公司 Swin transducer model-based motion real-time counting method and system
CN117077084B (en) * 2023-10-16 2024-01-26 南京栢拓视觉科技有限公司 Dance scoring method based on space-time heterogeneous double-flow convolutional network
CN117496409B (en) * 2023-11-15 2024-05-07 广东新裕信息科技有限公司 Fine granularity dance action scoring method based on multi-view three-dimensional human body reconstruction

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5022089A (en) * 1990-01-19 1991-06-04 Wilson Monti R Method and apparatus for fast registration using crosshair register marks
CN104021538A (en) * 2013-02-28 2014-09-03 株式会社理光 Object positioning method and device
CN107463898A (en) * 2017-08-01 2017-12-12 闽江学院 The stage performance abnormal behavior monitoring method of view-based access control model sensing network
CN110245623A (en) * 2019-06-18 2019-09-17 重庆大学 A kind of real time human movement posture correcting method and system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5022089A (en) * 1990-01-19 1991-06-04 Wilson Monti R Method and apparatus for fast registration using crosshair register marks
CN104021538A (en) * 2013-02-28 2014-09-03 株式会社理光 Object positioning method and device
CN107463898A (en) * 2017-08-01 2017-12-12 闽江学院 The stage performance abnormal behavior monitoring method of view-based access control model sensing network
CN110245623A (en) * 2019-06-18 2019-09-17 重庆大学 A kind of real time human movement posture correcting method and system

Also Published As

Publication number Publication date
CN110711374A (en) 2020-01-21

Similar Documents

Publication Publication Date Title
CN110711374B (en) Multi-modal dance action evaluation method
WO2018120964A1 (en) Posture correction method based on depth information and skeleton information
US8824802B2 (en) Method and system for gesture recognition
CN108764120B (en) Human body standard action evaluation method
CN111437583B (en) Badminton basic action auxiliary training system based on Kinect
JP5902204B2 (en) Motion recognition
US7565295B1 (en) Method and apparatus for translating hand gestures
US8175326B2 (en) Automated scoring system for athletics
EP3571627A2 (en) Systems, methods, apparatuses and devices for detecting facial expression and for tracking movement and location including for at least one of a virtual and augmented reality system
CN110991268B (en) Depth image-based Parkinson hand motion quantization analysis method and system
CN109308437B (en) Motion recognition error correction method, electronic device, and storage medium
Elaoud et al. Skeleton-based comparison of throwing motion for handball players
Monir et al. Rotation and scale invariant posture recognition using Microsoft Kinect skeletal tracking feature
CN113663312A (en) Micro-inertia-based non-apparatus body-building action quality evaluation method
CN115188074A (en) Interactive physical training evaluation method, device and system and computer equipment
CN113975775B (en) Wearable inertial body feeling ping-pong exercise training system and working method thereof
CN113974612A (en) Automatic assessment method and system for upper limb movement function of stroke patient
Morel et al. Automatic evaluation of sports motion: A generic computation of spatial and temporal errors
CN111353345B (en) Method, apparatus, system, electronic device, and storage medium for providing training feedback
Shahjalal et al. An approach to automate the scorecard in cricket with computer vision and machine learning
JP2021099666A (en) Method for generating learning model
Malawski et al. Automatic analysis of techniques and body motion patterns in sport
CN116665895A (en) Parkinsonism movement disorder quantitative analysis realization system
Yang et al. Wushu movement evaluation method based on Kinect
Qiu et al. Machine Learning based Movement Analysis and Correction for Table Tennis

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant