CN113283373A - Method for enhancing detection of limb motion parameters by depth camera - Google Patents
Method for enhancing detection of limb motion parameters by depth camera Download PDFInfo
- Publication number
- CN113283373A CN113283373A CN202110642003.5A CN202110642003A CN113283373A CN 113283373 A CN113283373 A CN 113283373A CN 202110642003 A CN202110642003 A CN 202110642003A CN 113283373 A CN113283373 A CN 113283373A
- Authority
- CN
- China
- Prior art keywords
- data
- neural network
- missing
- depth camera
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000001514 detection method Methods 0.000 title claims abstract description 28
- 230000002708 enhancing effect Effects 0.000 title claims abstract description 17
- 238000013528 artificial neural network Methods 0.000 claims abstract description 58
- 230000007246 mechanism Effects 0.000 claims abstract description 17
- 230000008569 process Effects 0.000 claims description 26
- 238000012549 training Methods 0.000 claims description 23
- 230000009466 transformation Effects 0.000 claims description 20
- 210000000988 bone and bone Anatomy 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 15
- 238000012360 testing method Methods 0.000 claims description 10
- 239000003550 marker Substances 0.000 claims description 8
- 238000010606 normalization Methods 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 210000003414 extremity Anatomy 0.000 description 32
- 230000036544 posture Effects 0.000 description 18
- 210000001364 upper extremity Anatomy 0.000 description 11
- 238000004458 analytical method Methods 0.000 description 7
- 230000005021 gait Effects 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 5
- 239000013589 supplement Substances 0.000 description 5
- 230000009469 supplementation Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 230000036449 good health Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Molecular Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a method for enhancing detection of limb motion parameters by a depth camera, and belongs to the field of human body posture information detection. The method comprises the following steps: collecting limb movement data of a human body by using a depth camera; s2: when the phenomenon that all mark points of collected human body limb movement data are missing at a certain moment is detected, namely the data volume is insufficient, predicting and reconstructing the missing data by utilizing a trained neural network with an attention mechanism to obtain complete skeleton three-dimensional coordinate data; when the phenomenon that part of mark points of collected body limb movement data at a certain moment are missing is detected, namely three-dimensional coordinate data of individual skeletal joint points at a certain moment are missing, the missing data points are complemented by using a trained neural network with an attention mechanism, and finally complete skeletal three-dimensional coordinate data are obtained. The invention realizes non-contact human body free motion detection and effectively eliminates the problem of data loss which may occur when a depth camera is used for collecting data.
Description
Technical Field
The invention belongs to the field of human body posture information detection, and relates to a method for enhancing the detection of limb motion parameters by a depth camera.
Background
The human body posture detection refers to measuring posture information of a human body or a local part of the human body during movement by a certain detection method, for example: the human body posture information is analyzed to obtain the instantaneous posture of the human body and the change rule of the posture of the human body.
Human motion posture detection is an essential detection technology for obtaining human body information, and is widely applied to the fields of physical training, gait analysis, action recognition, medical rehabilitation and military.
The patent application with the publication number of CN105046281A discloses a human behavior detection method based on Kinect, and the method adopts Kinect somatosensory equipment (a depth image sensor) to capture human gestures, so that compared with the traditional method of analyzing video images captured by a camera, the method reduces the influence of external environment on analysis results, and the detection results are more accurate. However, this method requires the motion of the subject to be restricted, and the subject needs to make a specified action according to the experimenter's request.
Patent application publication No. CN110477922A discloses a limb movement detection method and system, in which a movement capture device worn on a limb (including an arm and a leg, hereinafter referred to as a limb) is obtained to obtain limb movement posture data, a limb movement model is generated, and is compared and analyzed with a limb movement model of a person in good health condition, so as to determine the deviation degree between the generated limb movement model and a normal limb movement model of the person in good health condition. But wearable equipment may have certain hindrance to the motion of being tried, influences the motion condition of being tried, and receives outside environment to disturb greatly, and the operation is also relatively loaded down with trivial details.
The patent application with the publication number of CN112215172 discloses a human body lying posture three-dimensional posture estimation method fusing a human color image and depth information, because a depth camera is difficult to distinguish the human body lying posture, the method obtains the information of the human body lying posture by using a depth image and color image fusion mode, and has the advantages of low cost, no contact, high precision and the like, and the frame loss problem of the depth camera in the human body posture detection process is solved by using a spline interpolation method. However, the scheme only aims at the human body in the lying posture, and the detection and identification types of the human body movement posture are too single.
The solutions given in the market at present basically aim at some specific postures, namely, constraints are made on the movement to be tested, and the detected human posture information is not rich and comprehensive enough. Various problems may occur in the data acquisition process, for example, the hardware equipment is not operated properly, the body condition to be tested is not allowed, and the like, which results in serious shortage of the acquired data amount, a lot of frame data is lost, the data loss also has a certain influence on the subsequent data analysis result, and a phenomenon that partial bone joint point data is lost may also occur in the data acquisition process, which causes the data acquisition personnel to need to re-acquire the data to be tested, and wastes time and labor for the data acquisition personnel to be tested and the data acquisition personnel.
Therefore, a new method for detecting limb movement is needed to overcome the above drawbacks.
Disclosure of Invention
In view of the above, the present invention provides a method for enhancing the depth camera to detect the limb movement parameters, which utilizes a non-contact device, i.e. a depth camera, to capture the body posture under the condition of unconstrained free movement, perform predictive reconstruction on the limb movement data of the subject in case of insufficient data amount, or effectively and accurately complement the three-dimensional coordinate information of the individual bone joint point lost when the depth camera collects the limb movement data of the subject at a certain frame time.
In order to achieve the purpose, the invention provides the following technical scheme:
a method for enhancing detection of limb motion parameters by a depth camera specifically comprises the following steps:
s1: collecting limb movement data of a human body by using a depth camera;
s2: when the phenomenon that all mark points of collected human body limb movement data are missing at a certain moment is detected, namely the data volume is insufficient, predicting and reconstructing the missing data by utilizing a trained neural network with an attention mechanism to obtain complete skeleton three-dimensional coordinate data;
when the phenomenon that part of mark points of collected body limb movement data at a certain moment are missing is detected, namely three-dimensional coordinate data of individual skeletal joint points at a certain moment are missing, the missing data points are complemented by using a trained neural network with an attention mechanism, and finally complete skeletal three-dimensional coordinate data are obtained.
Further, the step S1 specifically includes: after the depth camera is connected with a computer, a testee stands in front of the depth camera, the three-dimensional coordinate data of the skeletal joint points in the free movement process of the testee are collected in real time by using a visual capture sensor in the depth camera, and the three-dimensional coordinate data of the skeletal joint points collected by the depth camera are stored as files.
Further, in step S2, when all the marker points after a certain time are missing, prediction reconstruction is performed on the future movement situation based on the existing data. The invention utilizes a neural network method in machine learning to predict and reconstruct the missing conditions of all the mark points after a certain time.
The neural network is actually referred to as "neural network learning" and the purpose of learning is to assign a correct value to the weight of the connection of each node of the hidden layer. These weights may determine the vector of outputs given an input vector. In the initial stage, all weights are randomly assigned. For each input value of the training set, after the forward calculation of the neural network, the obtained output value will be compared with the expected output value, and then the obtained error will be transmitted back to the previous network layer (back propagation algorithm). This error is noted and the weights are adjusted accordingly. The neural network needs to iterate repeatedly until the error of the output is lower than a set threshold, i.e. the difference between the output value and the expected output is small enough, and this process is also called a model training process.
In the invention, along with the continuous input of three-dimensional coordinate data of different tested skeletal joint points into the neural network with attention mechanism, the neural network continuously adjusts the mutual connection relationship among a large number of nodes in the neural network, the output is continuously close to the expected output, the reconstructed data is predicted to be gradually close to the real motion situation of the human body, namely the reconstructed motion track of the human body is predicted to be gradually close to the real motion track of the human body, and the predicted reconstructed data is used for calculating the limb motion parameters.
The specific steps of model training are as follows:
s201: calibrating collected three-dimensional coordinate data of skeletal joint points under the condition of free motion of a group of complete testees by using a three-dimensional motion capture system, then removing data after k frames, namely data after a certain moment, inputting missing data into a neural network, and automatically selecting a k value by an experimenter;
s202: taking the complete three-dimensional coordinate data of the skeletal joint points as expected output of the model, comparing the output of the neural network with the expected output, and forming a mapping relation between the input and the output in the neural network through multiple iterations;
s203: when the input and the output achieve a good mapping relation, the training of the neural network is completed, and then the test data is input into the neural network for prediction reconstruction. The test data is data of all the marker point missing conditions after a certain time appears in the data acquisition process.
Further, in step S2, when a partial marker point is missing at a certain time, a point is complemented based on the context information of the missing data.
The method comprises the steps of training a neural network with an attention mechanism along with continuously inputting three-dimensional coordinate data of different tested limb movement skeletal joint points into the neural network, continuously adjusting the interconnection relationship among a large number of nodes in the neural network, outputting the data which is continuously close to expected output, namely, continuously outputting a result which is close to a real result, continuously improving the accuracy of point supplement, enabling the data after the point supplement to be gradually close to the real skeleton information of a human body, namely, gradually approaching the real motion track of the human body after the point supplement, and then calculating limb motion parameters by using the data after the point supplement.
The specific steps of model training are as follows:
s211: utilizing a three-dimensional motion capture system to calibrate the collected three-dimensional coordinate data of the skeletal joint points under the condition of free motion of a group of complete testees, deducting data of a plurality of mark points from the complete data, and inputting the data of missing part of the mark points at a certain moment into a neural network;
s212: taking data containing complete information as expected output of the neural network, comparing the output of the neural network with the expected output, and forming a mapping relation between the input and the output in the neural network through multiple iterations;
s213: when the input and the output form a good mapping relation, the training of the neural network is completed, and then the test data is input into the neural network to complete the missing of part of the mark points at a certain moment. The test data is data of a part of mark point missing phenomenon at a certain moment in the data acquisition process.
Further, a neural network with attention mechanism includes: the coding component and the decoding component and the connection between the coding component and the decoding component; the coding component consists of a stack of encoders, and the decoding component consists of the same number of decoders; firstly, converting each frame of coordinate sequence input into a vector in an encoder at the bottommost layer, then enabling the vector to pass through two layers (a self-attention neural network and a feedforward neural network) of each encoder, enabling the self-attention to pay attention to the three-dimensional coordinates of each frame of the whole input sequence, helping the network to better encode the sequence, wherein the feedforward neural network aims to fit a function, and finally, the training time of the feedforward neural network is reduced by utilizing summation normalization; an attention vector set is output to a decoder after passing through all encoders, a real number vector is generated after passing through all decoders, and the real number vector is converted into a three-dimensional bone joint point sequence to be output after passing through a linear transformation layer and a flexible maximum value transfer function.
The invention has the beneficial effects that:
1) the depth camera adopted by the invention is a non-contact device, and cannot influence the movement of a tested object;
2) the invention does not restrict the movement of the testee, and the detected body and limb movement information of the human body is richer and more comprehensive;
3) the invention effectively eliminates the condition of data loss which can occur when the depth camera is used for collecting data.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention may be realized and attained by the means of the instrumentalities and combinations particularly pointed out hereinafter.
Drawings
For the purposes of promoting a better understanding of the objects, aspects and advantages of the invention, reference will now be made to the following detailed description taken in conjunction with the accompanying drawings in which:
FIG. 1 is a schematic structural diagram of a method for detecting a motion parameter of a limb according to the present invention;
FIG. 2 is a flow chart of a method of detecting a limb movement parameter of the present invention;
FIG. 3 is a schematic diagram of a neural network model training with attention mechanism according to the present invention;
FIG. 4 is a human skeletal joint points extracted by the depth camera;
FIG. 5 is a schematic diagram of the internal structure of a neural network model with attention mechanism in the present invention.
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention in a schematic way, and the features in the following embodiments and examples may be combined with each other without conflict.
Referring to fig. 1 to 5, the present invention provides a technical method for enhancing the detection of limb movement parameters by a depth camera, comprising the following steps:
the method comprises the following steps: a depth camera is selected as data acquisition equipment, and the depth camera can acquire three-dimensional coordinate information of human body bone joint points (as shown in figure 4). The testee stands in front of the equipment and freely moves at will (as shown in figure 1), a visual capture sensor in the depth camera is used for acquiring the three-dimensional data of the bone joints in the process of the free movement of the testee in real time, and the motion data acquired by the depth camera, namely the three-dimensional coordinate sequence of the corresponding bone joint points, is stored as a file.
Step two: for the case that all the mark points are missing after a certain time, the neural network with attention mechanism is used to perform the predictive reconstruction of data, as shown in fig. 5, the neural network model with attention mechanism is divided into an encoding component and a decoding component and the connection between them, the encoding component is composed of a stack of encoders, and the decoding component is composed of the same number of decoders. The coordinate sequence of each frame of input is firstly converted into a vector in the encoder at the lowest layer, and then the vector passes through two layers of each encoder (self-attention and feedforward neural network), wherein the self-attention focuses on the three-dimensional coordinates of each frame of the whole input sequence to help the model to better encode the sequence, and the aim of the feedforward neural network is to fit a function, and the aim of summation normalization is to reduce the training time of the feedforward neural network. An attention vector set is output to a decoder after passing through all encoders, a real number vector is generated after passing through all decoders, and the real number vector is converted into a three-dimensional bone joint point sequence to be output after passing through a linear transformation layer and a flexible maximum value transfer function.
The method comprises the steps that a group of complete three-dimensional coordinate sequences of skeletal joint points of a human body with sufficient data volume during limb movement needs to be collected, after a start password is audited, the motion is freely made without any restriction on the audited motion until the end password is audited, after data collected by a depth camera is calibrated with three-dimensional motion capture data, the data at a certain frame time is removed and used as input of an untrained depth self-attention transformation model, the collected complete three-dimensional coordinate sequences of the skeletal joint points are used as expected output, the weight of the depth self-attention transformation model is continuously adjusted through a back propagation algorithm, and a model which is closer to expected output is trained. And (4) carrying out prediction reconstruction on the test data by using the trained model, and then calculating the limb movement parameters by using the data after prediction reconstruction.
Step three: a neural network with an attention mechanism is used for completing the missing of part of mark points at a certain moment, a complete group of data is firstly subjected to random removal of three-dimensional coordinates of certain joint points in certain frames, then the group of data is input into an untrained deep self-attention transformation model, parameters (weights) of the model are generated randomly at the moment, the complete data is used for being compared with an output result of the model, the weights of the deep self-attention transformation model are slightly adjusted by using a back propagation algorithm, and the model with the output result closer to real data is trained. And (3) performing partial mark point missing completion on the test data at a certain moment by using the trained model, and then calculating the limb motion parameters by using the data after point completion.
Example 1: technical method and implementation for enhancing the detection of upper limb movement parameters by a depth camera, comprising the steps of:
as shown in fig. 1 to 2, the subject stands in front of the depth camera and is tried to make various movements with the upper limbs without any constraint for a prescribed time, for example: reach motion, elbow flexion and extension motion, etc. The process needs to be repeated n times, and the value of n is automatically judged by an operator according to the current situation.
Selecting a group with the most complete data and calibrated data in the n times of experimental data as training set data;
when three-dimensional coordinates of skeletal joint points of upper limb movement of a subject are acquired, situations with insufficient data amount may occur, such as: the upper limb movement of the tested person is inconvenient, the coordinate sequence acquired within the specified time is short, the subsequent data analysis is not facilitated, the data of the tested person needs to be predicted and reconstructed at the moment to obtain sufficient data quantity, the accuracy of the data after prediction and reconstruction needs to be guaranteed to a certain extent, and the closer the data is to the real data, the better the analysis result of the upper limb movement data is. Aiming at the condition that all mark points are missing after a certain moment, model training is firstly carried out, sequence information (k value is selected by an operator) of a tested complete upper limb movement skeletal joint point three-dimensional coordinate sequence after a kth frame is removed, a front k frame sequence is input into an original depth self-attention transformation model, a prediction sequence is obtained according to original weight in the model, a result sequence is compared with expected output of the user, namely the tested complete upper limb movement skeletal joint point three-dimensional coordinate sequence, the difference between the two is returned to the model through a back propagation algorithm, the depth self-attention transformation model adjusts the weight of the depth self-attention transformation model according to the difference between the two, the mapping relation between the input and the output is continuously optimized through multiple iterations, the input result of the model is gradually close to the expected output of the invention, namely, the prediction reconstruction result obtained through the depth self-attention transformation model is gradually approximate to real motion data, when this mapping is applied to all the marker-point missing instances after a certain time, the missing part of the instance can be predicted according to the attribute characteristics of the instance. Then, the predicted and reconstructed data can be used for calculating the motion parameters of the tested limb.
In the data acquisition process, the information of part of the bone joint points at a certain frame time is not acquired and is blocked, so that the condition that the three-dimensional coordinates of the part of the bone joint points are lost at certain time is caused, namely, the three-dimensional coordinate lost coordinate sequence of the individual bone joint points is incomplete, and certain influence is caused on the analysis result of the subsequent upper limb movement data of the tested upper limb, at the moment, the point supplementation is required to be carried out on the three-dimensional coordinate data of the tested complete upper limb movement bone joint points to obtain a complete coordinate sequence, and the data after the point supplementation is as close as possible to the real movement data. Aiming at the condition that part of mark points at a certain moment are missing, model training is firstly carried out, three-dimensional coordinate data of a certain skeleton joint point at certain moment are randomly removed from a complete three-dimensional coordinate sequence of the skeleton joint point of the upper limb movement, the sequence is input into an untrained depth self-attention conversion model, a result sequence is obtained according to the original weight of the untrained model, the result sequence is compared with expected output of people, namely the three-dimensional coordinate sequence of the skeleton joint point in the complete walking process of a human body, the difference between the two is returned to the model through a back propagation algorithm, the depth self-attention conversion model adjusts the weight of the depth self-attention conversion model according to the difference between the two, the mapping relation of the input and the output is continuously optimized through multiple iterations, the input result of the model is gradually close to the expected output of the people, namely, the point supplement result obtained after the depth self-attention conversion model is closer to the real movement data, when this mapping is applied to a partially marked point missing instance at a certain time, the missing part of the instance can be predicted according to the attribute characteristics of the instance. And then, calculating the limb movement parameters of the testee by using the data after point supplementation.
Example two: the technical method and implementation for enhancing detection of human gait parameters by a depth camera comprise the following steps:
as shown in fig. 1, a tested station completes a gait data acquisition process before a distance depth camera, and the process needs to be repeated for n times;
selecting a group with the most complete data in the n times of experimental data and subjected to three-dimensional dynamic capture data calibration as training set data;
in the data acquisition process, various emergencies may occur, if a phenomenon that a coordinate sequence is too short occurs when acquiring the three-dimensional coordinate data of the skeletal joint point in the walking process of the tested human body, namely the data volume is insufficient, and subsequent gait analysis cannot be performed, the three-dimensional coordinate data of the skeletal joint point in the walking process of the tested human body needs to be predicted and reconstructed to obtain sufficient data volume, but the accuracy requirement on the data after prediction and reconstruction is higher, and the gait analysis result is more accurate as the data is closer to the real data. Aiming at the condition that all mark points are missing after a certain moment, model training is firstly carried out (as shown in figure 3), sequence information after the k frame is removed from a three-dimensional coordinate sequence of skeletal joint points acquired in the process of trial walking is input into an original depth self-attention transformation model, a result sequence is obtained according to the original weight in the model, the result sequence is compared with expected output of people, namely the three-dimensional coordinate sequence of skeletal joint points in the complete human body walking process, the difference between the two is returned to the model through a back propagation algorithm, the depth self-attention transformation model adjusts the weight of the depth self-attention transformation model according to the difference between the two, the input result of the model gradually approaches the expected output of the people through a plurality of times of iterative cycle, namely, the predicted reconstruction result obtained through the depth self-attention transformation model is close to real motion data, when the mapping relation is applied to all the marker point missing examples after a certain time, the missing part of the example can be predicted according to the attribute characteristics of the example, and then the motion parameters of the tested limb can be calculated by using the data after prediction and reconstruction;
inputting the three-dimensional coordinate sequences of all the marker point deletions after a certain moment in the acquisition process into a trained depth self-attention conversion model, and calculating the model to obtain complete coordinate information after prediction and reconstruction.
In the data acquisition process, the information of part of the bone joint points at a certain frame time is not acquired and is blocked, so that the condition that the three-dimensional coordinates of the part of the bone joint points are missing at certain time is caused, namely, the three-dimensional coordinate missing coordinate sequence of the individual bone joint point is incomplete, and certain influence is caused on the subsequent gait analysis result of the tested person, at the moment, the three-dimensional coordinate data of the bone joint points in the walking process of the tested person needs to be supplemented to obtain a complete coordinate sequence, and the data after point supplementation is close to the real motion data as much as possible. Aiming at the condition that part of mark points at a certain moment are missing, model training is firstly carried out (as shown in figure 3), three-dimensional coordinate data of a certain skeleton joint point at certain moment are randomly removed from a complete three-dimensional coordinate sequence of the skeleton joint point, the sequence is input into an untrained depth self-attention transformation model, a result sequence is obtained according to the original weight of the untrained model, the result sequence is compared with expected output of people, namely the three-dimensional coordinate sequence of the skeleton joint point in the whole human body walking process, the difference between the three-dimensional coordinate sequence and the untrained depth self-attention transformation model is returned to the model through a back propagation algorithm, the depth self-attention transformation model adjusts the weight of the depth self-attention transformation model according to the difference between the two, and through multiple iterations, the mapping relation between the input and the output is continuously optimized, the input result of the model is gradually close to the expected output of people, namely, the point complementing result obtained through the depth self-attention transformation model is closer to the real motion data, when the mapping relation is applied to a part of the mark point missing example at a certain moment, the missing part of the example can be predicted according to the attribute characteristics of the example, and then the data after point supplementation can be used for calculating the limb movement parameters of the testee.
Finally, the above embodiments are only intended to illustrate the technical solutions of the present invention and not to limit the present invention, and although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions, and all of them should be covered by the claims of the present invention.
Claims (8)
1. A method for enhancing detection of limb motion parameters by a depth camera, the method comprising the steps of:
s1: collecting limb movement data of a human body by using a depth camera;
s2: when the phenomenon that all mark points of collected human body limb movement data are missing at a certain moment is detected, namely the data volume is insufficient, predicting and reconstructing the missing data by utilizing a trained neural network with an attention mechanism to obtain complete skeleton three-dimensional coordinate data;
when the phenomenon that part of mark points of collected body limb movement data at a certain moment are missing is detected, namely three-dimensional coordinate data of individual skeletal joint points at a certain moment are missing, the missing data points are complemented by using a trained neural network with an attention mechanism, and finally complete skeletal three-dimensional coordinate data are obtained.
2. The method for enhancing detection of limb movement parameters by a depth camera according to claim 1, wherein the step S1 specifically comprises: and acquiring the three-dimensional coordinate data of the skeletal joint points in the free movement process of the testee in real time by using the depth camera, and storing the three-dimensional coordinate data of the skeletal joint points acquired by the depth camera as a file.
3. The method for enhancing the detection of the limb movement parameters by the depth camera according to claim 1, wherein in step S2, when it is detected that all the marker points of the collected human limb movement data are missing at a certain time, the specific steps of training the neural network with attention mechanism are as follows:
s201: calibrating collected three-dimensional coordinate data of skeletal joint points under the condition of free motion of a group of complete testees by using a three-dimensional motion capture system, then removing data after k frames, namely data after a certain moment, inputting missing data into a neural network, and automatically selecting a k value by an experimenter;
s202: taking the complete three-dimensional coordinate data of the skeletal joint points as expected output of the model, comparing the output of the neural network with the expected output, and forming a mapping relation between the input and the output in the neural network through multiple iterations;
s203: when the input and the output reach a proper mapping relation, the training of the neural network is completed, and then the test data is input into the neural network for prediction reconstruction.
4. The method for enhancing the detection of the limb movement parameters by the depth camera according to claim 3, wherein in step S203, the test data is data of all the marker point missing situations after a certain time in the data acquisition process.
5. The method for enhancing the detection of the limb movement parameters by the depth camera according to claim 1, wherein in step S2, when the phenomenon that part of the markers of the collected human limb movement data are missing at a certain moment is detected, the training of the neural network with attention mechanism includes the following steps:
s211: utilizing a three-dimensional motion capture system to calibrate the collected three-dimensional coordinate data of the skeletal joint points under the condition of free motion of a group of complete testees, deducting data of a plurality of mark points from the complete data, and inputting the data of missing part of the mark points at a certain moment into a neural network;
s212: taking data containing complete information as expected output of the neural network, comparing the output of the neural network with the expected output, and forming a mapping relation between the input and the output in the neural network through multiple iterations;
s213: when the input and the output form a proper mapping relation, the training of the neural network is completed, and then the test data is input into the neural network to complete the missing of part of the mark points at a certain moment.
6. The method for enhancing detection of limb movement parameters by a depth camera according to claim 5, wherein in step S213, the test data is data of partial marker point missing phenomenon at a certain moment in the data acquisition process.
7. The method for enhancing depth camera detection of limb motion parameters according to claim 3 or 5, wherein the neural network comprises: assigning a correct value to the weight of the connection of each node of the hidden layer; when an input vector is given, determining an output vector according to the weight; in the initial stage, all weights are randomly assigned; for each input value of the training set, after forward calculation of the neural network, the obtained output value is compared with an expected output value, and then the obtained error is transmitted back to the previous network layer, namely a back propagation algorithm; then the weight is correspondingly adjusted according to the error; the neural network needs to iterate repeatedly until the error of the output is lower than a set threshold.
8. The method for enhancing depth camera detection of limb motion parameters according to claim 3 or 5, wherein the neural network with attention mechanism comprises: the coding component and the decoding component and the connection between the coding component and the decoding component; the coding component consists of a stack of encoders, and the decoding component consists of the same number of decoders; firstly, converting each frame of coordinate sequence input into a vector in an encoder at the bottom layer, then enabling the vector to pass through a self-attention and feedforward neural network of each encoder, enabling the self-attention to pay attention to the three-dimensional coordinates of each frame of the whole input sequence, helping the network to better encode the sequence, wherein the feedforward neural network aims to fit a function, and finally reducing the training time of the feedforward neural network by using summation normalization; an attention vector set is output to a decoder after passing through all encoders, a real number vector is generated after passing through all decoders, and the real number vector is converted into a three-dimensional bone joint point sequence to be output after passing through a linear transformation layer and a flexible maximum value transfer function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110642003.5A CN113283373B (en) | 2021-06-09 | 2021-06-09 | Method for enhancing limb movement parameters detected by depth camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110642003.5A CN113283373B (en) | 2021-06-09 | 2021-06-09 | Method for enhancing limb movement parameters detected by depth camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113283373A true CN113283373A (en) | 2021-08-20 |
| CN113283373B CN113283373B (en) | 2023-05-05 |
Family
ID=77283763
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110642003.5A Active CN113283373B (en) | 2021-06-09 | 2021-06-09 | Method for enhancing limb movement parameters detected by depth camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113283373B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115223023A (en) * | 2022-09-16 | 2022-10-21 | 杭州得闻天下数字文化科技有限公司 | Human body contour estimation method and device based on stereoscopic vision and deep neural network |
| CN117635897A (en) * | 2024-01-26 | 2024-03-01 | 腾讯科技（深圳）有限公司 | Three-dimensional object posture complement method, device, equipment, storage medium and product |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103996220A (en) * | 2014-05-26 | 2014-08-20 | 江苏大学 | Three-dimensional reconstruction method and system in intelligent transportation |
| CN105912985A (en) * | 2016-04-01 | 2016-08-31 | 上海理工大学 | Human skeleton joint point behavior motion expression method based on energy function |
| US20170020113A1 (en) * | 2014-04-01 | 2017-01-26 | Lely Patent N.V. | Arrangement and method to determine a body condition score of an animal |
| CN109086869A (en) * | 2018-07-16 | 2018-12-25 | 北京理工大学 | A kind of human action prediction technique based on attention mechanism |
| CN109816696A (en) * | 2019-02-01 | 2019-05-28 | 西安全志科技有限公司 | A kind of robot localization and build drawing method, computer installation and computer readable storage medium |
| CN111723667A (en) * | 2020-05-20 | 2020-09-29 | 同济大学 | Human body joint point coordinate-based intelligent lamp pole crowd behavior identification method and device |
| CN111815679A (en) * | 2020-07-27 | 2020-10-23 | 西北工业大学 | Binocular camera-based trajectory prediction method during loss of spatial target feature points |
| CN112149531A (en) * | 2020-09-09 | 2020-12-29 | 武汉科技大学 | Human skeleton data modeling method in behavior recognition |
| CN112487913A (en) * | 2020-11-24 | 2021-03-12 | 北京市地铁运营有限公司运营四分公司 | Labeling method and device based on neural network and electronic equipment |
-
2021
- 2021-06-09 CN CN202110642003.5A patent/CN113283373B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170020113A1 (en) * | 2014-04-01 | 2017-01-26 | Lely Patent N.V. | Arrangement and method to determine a body condition score of an animal |
| CN103996220A (en) * | 2014-05-26 | 2014-08-20 | 江苏大学 | Three-dimensional reconstruction method and system in intelligent transportation |
| CN105912985A (en) * | 2016-04-01 | 2016-08-31 | 上海理工大学 | Human skeleton joint point behavior motion expression method based on energy function |
| CN109086869A (en) * | 2018-07-16 | 2018-12-25 | 北京理工大学 | A kind of human action prediction technique based on attention mechanism |
| CN109816696A (en) * | 2019-02-01 | 2019-05-28 | 西安全志科技有限公司 | A kind of robot localization and build drawing method, computer installation and computer readable storage medium |
| CN111723667A (en) * | 2020-05-20 | 2020-09-29 | 同济大学 | Human body joint point coordinate-based intelligent lamp pole crowd behavior identification method and device |
| CN111815679A (en) * | 2020-07-27 | 2020-10-23 | 西北工业大学 | Binocular camera-based trajectory prediction method during loss of spatial target feature points |
| CN112149531A (en) * | 2020-09-09 | 2020-12-29 | 武汉科技大学 | Human skeleton data modeling method in behavior recognition |
| CN112487913A (en) * | 2020-11-24 | 2021-03-12 | 北京市地铁运营有限公司运营四分公司 | Labeling method and device based on neural network and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 魏小鹏;刘瑞;张强;肖伯祥;: "基于模板匹配的人体运动捕捉数据处理方法" * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115223023A (en) * | 2022-09-16 | 2022-10-21 | 杭州得闻天下数字文化科技有限公司 | Human body contour estimation method and device based on stereoscopic vision and deep neural network |
| CN115223023B (en) * | 2022-09-16 | 2022-12-20 | 杭州得闻天下数字文化科技有限公司 | Human body contour estimation method and device based on stereoscopic vision and deep neural network |
| CN117635897A (en) * | 2024-01-26 | 2024-03-01 | 腾讯科技（深圳）有限公司 | Three-dimensional object posture complement method, device, equipment, storage medium and product |
| CN117635897B (en) * | 2024-01-26 | 2024-05-07 | 腾讯科技（深圳）有限公司 | Three-dimensional object posture complement method, device, equipment, storage medium and product |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113283373B (en) | 2023-05-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111881887A (en) | Multi-camera-based motion attitude monitoring and guiding method and device | |
| CN112069933A (en) | Skeletal muscle stress estimation method based on posture recognition and human body biomechanics | |
| US10445930B1 (en) | Markerless motion capture using machine learning and training with biomechanical data | |
| CN113283373B (en) | Method for enhancing limb movement parameters detected by depth camera | |
| CN109635820A (en) | The construction method of Parkinson's disease bradykinesia video detection model based on deep neural network | |
| Pasinetti et al. | Assisted gait phase estimation through an embedded depth camera using modified random forest algorithm classification | |
| JP5604249B2 (en) | Human body posture estimation device, human body posture estimation method, and computer program | |
| Guo et al. | Obtaining lower-body Euler angle time series in an accurate way using depth camera relying on Optimized Kinect CNN | |
| Cai et al. | Single-camera-based method for step length symmetry measurement in unconstrained elderly home monitoring | |
| Zhu et al. | Dual-channel cascade pose estimation network trained on infrared thermal image and groundtruth annotation for real-time gait measurement | |
| CN111820902B (en) | Ankle joint ligament injury intelligent decision-making system based on activity degree characteristics | |
| Vitali et al. | A new approach for medical assessment of patient’s injured shoulder | |
| CN116543455A (en) | Method, equipment and medium for establishing parkinsonism gait damage assessment model and using same | |
| O'Malley et al. | Kinematic analysis of human walking gait using digital image processing | |
| KR20230120341A (en) | 3D human body joint angle prediction method and system using 2D image | |
| Vitali et al. | Digital motion acquisition to assess spinal cord injured (SCI) patients | |
| CN110837751B (en) | Human motion capturing and gait analysis method based on RGBD depth camera | |
| Zhang et al. | Recent development in human motion and gait prediction | |
| Kumar et al. | Prediction of lower limb kinematics from vision-based system using deep learning approaches | |
| Lau et al. | Cost-benefit analysis reference framework for human motion capture and analysis systems | |
| Talaa et al. | Computer Vision-Based Approach for Automated Monitoring and Assessment of Gait Rehabilitation at Home. | |
| CN116630551B (en) | Motion capturing and evaluating device and method thereof | |
| Patil et al. | Early Detection of Hemiplegia by Analyzing the Gait Characteristics and Walking Patterns Using | |
| CN117671791A (en) | Human body sitting trajectory prediction method | |
| Mishra et al. | XAI-based gait analysis of patients walking with Knee-Ankle-Foot orthosis using video cameras |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240202 Address after: 100102 China Electronics Taiji Information Industrial Park, No.7 Rongda Road, Chaoyang District, Beijing Patentee after: TAIJI COMPUTER Co.,Ltd. Country or region after: China Address before: 400044 No. 174 Shapingba street, Shapingba District, Chongqing Patentee before: Chongqing University Country or region before: China |
|
| TR01 | Transfer of patent right |