CN114092854A - Intelligent rehabilitation auxiliary training system for spinal degenerative disease based on deep learning - Google Patents

Abstract

The invention discloses a deep learning-based intelligent rehabilitation auxiliary training system for spinal degeneration. The system of the present invention includes a real-time classification module of traditional Chinese medicine guidance video based on deep learning and a video sequence division and evaluation module based on human skeleton representation; the former performs deep learning training by acquiring two-dimensional human skeleton data as the training data of the learning model, The generalized deep learning model is obtained; finally, the real-time frame classification result is obtained; the latter performs segmentation and real-time error correction on the skeleton sequence of the same category according to the frame classification result, and divides the segmented sequence segment with the corresponding category expert group video skeleton sequence segment Perform sequence alignment scoring. The system of the present invention does not require the guidance and intervention of medical staff, and can enable patients to carry out TCM guidance training at any time by themselves, is suitable for families and primary medical and health institutions, can reduce the pressure of medical staff, and improve the flexibility and accuracy of patient rehabilitation training.

Description

Intelligent rehabilitation auxiliary training system for spinal degenerative diseases based on deep learning

technical field

The invention belongs to the technical field of computer vision video understanding, in particular to a video sequence recognition and evaluation system based on deep learning.

Background technique

TCM Daoyin is a method of self-cultivation and treatment with Chinese characteristics. It promotes the recovery of limb movement by guiding the patient to adjust the shape based on regulating the mind and breath, that is, the main activities of the limbs. With the continuous development of computer hardware and artificial intelligence technology, the research and development of intelligent assisted rehabilitation training system has become a hot spot in related fields at home and abroad. In computer vision tasks, video action recognition is a very challenging field. Methods based on RGB frame input often cannot meet the performance requirements of real-time processing. Methods based on skeleton sequences have lower time complexity, but in the inference process It will rely on the extraction of skeleton information. The invention adopts the cyclic neural network based on skeleton sequence input as the basic network architecture of video behavior recognition, and obtains the skeleton sequence in real time through OpenPose ^[1] two-dimensional pose estimation combined with sparse optical flow tracking, and finally performs skeleton sequence segmentation according to the classification results, patient Exercise score and action correction reminder. The invention combines the field of computer vision and the rehabilitation training of traditional Chinese medicine Daoyin to develop an automatic evaluation and auxiliary training system for rehabilitation training actions for spinal degenerative diseases, so as to realize automatic, precise and intelligent rehabilitation training.

SUMMARY OF THE INVENTION

In order to increase the burden of TCM rehabilitation medical staff when patients with spinal degenerative diseases undergo rehabilitation training through TCM Daoyin surgery and improve the patient's practice effect, the present invention provides an intelligent rehabilitation auxiliary training system for spinal degenerative diseases based on deep learning , combined with artificial intelligence technology, to automatically evaluate and correct the patient's TCM Daoyin practice.

The intelligent rehabilitation auxiliary training system for spinal degenerative diseases based on deep learning provided by the present invention is composed of a deep learning-based TCM guidance video real-time classification module and a human skeleton representation-based video sequence division and evaluation module.

The deep learning-based TCM guidance video real-time classification module obtains two-dimensional human skeleton data through Openpose ^[1] , which is used as the training data of the deep learning model to perform supervised deep learning training to obtain a generalized deep learning model; Then, through OpenPose frame detection combined with sparse optical flow tracking, real-time preprocessing is performed on the input video, and the result is fed into the pre-trained classification model as input to obtain real-time frame classification results.

The segmentation and evaluation module based on human skeleton representation and video sequence, according to the frame classification result of the real-time classification module, performs segmentation and real-time error correction on the skeleton sequence of the same category, and separates the segmented sequence segment with the corresponding category expert group. Video skeleton sequence segments for sequence alignment scoring.

In the intelligent rehabilitation auxiliary training system for spinal degenerative diseases based on deep learning proposed by the present invention:

Corresponding to the described deep learning-based TCM guidance video real-time classification module, its work content is as follows;

(1) Obtain training data for deep learning;

(2) Training a deep learning model;

(3) OpenPose frame detection combined with sparse optical flow tracking to preprocess and classify video in real time;

Corresponding to the described segmentation and evaluation module based on human skeleton representation and video sequence, its work content is as follows:

(4) Segmentation and real-time error correction of the skeleton sequence based on the classification result;

(5) compare and score the segmented sequence segments;

To obtain deep learning training data as described in content (1), the specific operation process is as follows:

(11) Process video data, edit all video data according to the design of rehabilitation training exercises, and obtain short video data with a length of no more than 1000 frames;

(12) all short video data are mirrored and flipped as data enhancement;

(13) Using the Openpose model pre-trained on the BODY_25 data set, extract the two-dimensional pose of the video data samples processed in the process (11), and obtain the skeleton data sequence represented by the two-dimensional spatial coordinates of 25 key points, wherein The 25 key points are Nose, Neck, RShoulder, RElbow, RWrist, LShoulder, LElbow, LWrist, MidHip, RHip, RKnee, Rankle, LHip, LKnee, LAnkle, REye, LEye, Rear, LEar, LBigToe, LSmallToe, LHeel, RBigToe, RSmallToe, Rheel, Background;

最后进行零填充操作，在骨架序列后方填充全零，将骨架序列长度固定为1000帧。(14) Perform data preprocessing on all skeleton data sequences, first perform a translation operation, subtract the mid-hip coordinates of the first valid frame from all frame coordinates of the skeleton sequence; then perform a normalization operation to calculate the average shoulder of the skeleton sequence distance d _s , and then scale all frame coordinates of the skeleton sequence with a scaling factor of

Finally, a zero-padding operation is performed, and all zeros are filled at the back of the skeleton sequence, and the length of the skeleton sequence is fixed to 1000 frames.

The specific process of training the deep learning model in content (2) is as follows:

(21) The deep learning model adopts a classification network model, and specifically adopts a hierarchical limb attention LSTM network, which includes a perspective conversion module, a limb attention module, a limb level classification module and a body level classification module; The skeleton data is the input, first, the input sequence coordinates are subjected to two-dimensional translation transformation through the perspective conversion module, and then the coordinates of each frame of the skeleton sequence are divided into 8 limb parts, which are respectively passed through the LSTM layer and the Dropout layer of the limb level classification module; Then, the eigenvectors of the 8 limbs are cascaded and passed through the limb attention module to calculate the spatiotemporal attention weight of each limb; finally, the eigenvectors corresponding to the 8 limbs are cascaded and weighted according to the spatiotemporal attention weight, and are classified by the body level. The LSTM layer, Dropout layer and softmax layer of the module finally get the classification score of the skeleton sequence on each frame.

(22) Setting model hyperparameters;

The main hyperparameters in the model are: training conditions, batch size, learning rate, dropout rate, LSTM orthogonal initialization multiplication factor, and the maximum number of iterations;

(23) Start training, take the validation loss value of the model during training as the benchmark, when the validation loss value of the model no longer decreases and continues for 30 iterations, it means that the network has been accepted, and the training ends;

(24) Adjust the hyperparameters multiple times to obtain the model with the best generalization performance;

Among them, the operation process of the perspective conversion module in the hierarchical limb attention LSTM network in step (21) is:

In order to reduce the impact of the change of shooting angle on the model classification performance, the input skeleton sequence is subjected to adaptive two-dimensional translation and rotation operations through the perspective conversion module, and the coordinates of each frame of the input skeleton sequence are adjusted. Among them, the specific calculation process defined as:

S' _{t, j} = [x' _{t, j} , y' _{t, j} ]' = R _t (S _{t, j -} d _t ), (1)

代表第t帧对应的平移向量，R_t代表第t帧对应的二维旋转矩阵，具体表示为：Wherein, S _{t, j} =[x _{t, j} , y _{t, j} ]′ represents the two-dimensional coordinates of the j-th key point in the t-th frame of the input skeleton sequence;

Represents the translation vector corresponding to the t-th frame, and R _t represents the two-dimensional rotation matrix corresponding to the t-th frame, specifically expressed as:

分别表示对骨架序列第t帧所有坐标沿着横轴和纵轴的平移量，α_t表示对骨架序列第t帧所有坐标逆时针旋转的弧度。in,

Represents the amount of translation along the horizontal axis and the vertical axis for all coordinates of the t-th frame of the skeleton sequence, respectively, and α _t represents the radian of counterclockwise rotation of all the coordinates of the t-th frame of the skeleton sequence.

In the process (21), the skeleton information of the perspective conversion module is divided into 8 limbs with a certain overlap according to the distribution of the human body, namely the head, the left arm, the right arm, the torso, the left leg, the right leg, and the left foot. , right foot, and then the information of different limbs is calculated by separate LSTM layer and Dropout layer and then concatenated into the overall skeleton information.

In the process (21), the operation process of the limb attention module is:

_Ht =LSTM(concat(Ht _,1 ,..., _Ht,L )), (3)

a _t =W ₁ tanh(W ₂ H _t +b ₂ )+b ₁

Among them, H _{t, i} represents the i-th limb information of the t-th frame skeleton sequence, 1≤i≤L, (L=8), H _t represents the 8 limb feature information after being cascaded and passed through the LSTM layer and the Dropout layer. The obtained feature information, W ₁ , W ₂ are learnable parameter matrices, b ₁ , b ₂ are bias vectors, and then calculate the weight vector α _{t, l} of each limb through soffmax activation, and finally get the weighted weight of each limb. feature vectors, and concatenate all weighted limb feature vectors as input to subsequent modules:

H' _{t, l} = α _{t, l} ·H _{t, l} , (5)

H' _t = concat(H' _{t, 1} , ..., H' _{t, L} ), (6)

The cascaded weighted limb feature vectors are passed through two layers of LSTM and Dropout and one fully connected layer, and then the classification scores are obtained through softmax activation.

Content (3) Use the trained model to classify the video with classification. The specific operation process is as follows:

(31) Obtain and process the skeleton sequence and classification from the input video signal source, use Openpose frame detection combined with sparse optical flow tracking to obtain skeleton sequence information in real time, perform OpenPose pose estimation every 5 frames, and obtain 25 human body key points Coordinates, where the 25 key points are Nose, Neck, RShoulder, RElbow, RWrist, LShoulder, LElbow, LWrist, MidHip, RHip, RKnee, Rankle, LHip, LKnee, LAnkle, REye, LEye, Rear in the index order of 0-24 , LEar, LBigToe, LSmallToe, LHeel, RBigToe, RSmallToe, Rheel, Background. The Lucas-Kanade method is used to track the two-dimensional coordinate information of 25 human body key points in each of the following 4 frames. The specific process is as follows:

S _t+1 =calOpticalFlowPyrLK(I _t , It ₊₁ , S _t ), (7)

Among them, S _t and It respectively represent the grayscale image and skeleton information of the _t -th frame, and calOpticalFlowPyrLK is an implementation of the Lucas-Kanade method by the OpenCV open source library.

(32) The classification of skeleton sequence frames is slightly different in the inference process and the training process. In the inference process, skeleton information preprocessing and classification are performed after every 10 frames of skeleton information are obtained. The preprocessing process includes translation, normalization The translation process is no longer based on the coordinates of the mid-hip point (MidHip) of the first frame, but maintains a mid-hip coordinate as the origin during system operation, and the coordinates are updated every 10 seconds; normalization The process of ization and zero padding is the same as step (14).

(33) In the inference process, since the classification is performed every 10 frames instead of the entire video, it is necessary to make all LSTMs in the model maintain their own parameter information after each classification during the operation of the system, i.e. Turn on the LSTM stateful mode, in which the hierarchical limb attention LSTM network retains the state of all LSTM layers after each inference and serves as the initial state for the next inference.

In content (4), the skeleton sequence is segmented and real-time error correction is performed based on the classification result. The process is as follows:

(41) Skeleton sequence segmentation based on classification results:

During the operation of the system, the exercises performed by patients generally include multiple different actions. In order to enable the subsequent scoring process to accurately compare the skeleton sequence information of the corresponding category of expert groups, it is necessary to perform real-time sequence segmentation after the sequence frame classification. It is: when the first frame of the sequence or the end frame of the previous sequence segment is determined, the category of the current sequence segment is determined according to the classification result of the current frame, and the basis for determining the end of a sequence segment is either the frame category and the current sequence segment category are continuous Inconsistent and either the maximum error tolerance length (100 frames) is reached or the input sequence ends.

(42) Real-time error correction based on classification results, the process is as follows:

During the operation of the system, different parameters are calculated according to the skeleton information and classification results of each frame, and error correction text information is generated. The relationship between action categories and calculation parameters is described in Table 1.

Table 1. Reference parameter names for generating error correction texts for different action categories

Action category parameter name 0 The relative lift of the hands, the angle between the upper arm and the lower arm 1 The relative extent of the left and right spread of the hands and the left and right rotation of the head 2 The relative longitudinal height of the elbows, the relative lateral width of the two axes 3 The relative longitudinal height of the elbows, the relative lateral width of the two axes 4 Relative reach of hands 5 The ratio of the relative extension of the hands to the average length of the calf 6 Longitudinal distance between hip and neck 7 The ratio of the difference in the longitudinal distance of the feet to the longitudinal distance from the hip to the neck

In content (5), compare and score the sequence segments at the end of the segmentation

During the operation of the system, whenever a new sequence segment is determined, the sequence alignment score is performed according to the category of the sequence segment. First, the system maintains a segment of expert group skeleton sequence information Qn for each action analogy. For a sequence segment C _m , the similarity between the sequences at both ends is calculated by the dynamic time rule (DTW) algorithm. The DTW algorithm uses the idea of dynamic programming to find two The optimally aligned path for a sequence of segments, and compute the sum of Euclidean distances for the sequence frames along the path:

Cost(i,j)=D(i,j)+min[Cost(i-1,j),Cost(i,j-1),Cost(i-1,j-1)], (9)

Among them, x, y represent the two-dimensional coordinates in the sequence segment, D(i, j) represents the Euclidean distance between the i-th frame of the sequence Q _n and the j-th frame of the sequence C _m , Cost(i, j) in the two sequences The cumulative Euclidean distance sum of the (i, j) positions of the best alignment path, Cost (n, m) the similarity of the two sequences. According to the similarity of the sequence, calculate the alignment score of the current sequence segment and the expert group skeleton sequence (percentage system). Similarity 0-2), 75-90 (corresponding to similarity 2-4), 60-75 (corresponding to similarity 4-6), 40-60 (corresponding to similarity 6-∞), for the similarity of each file, The score calculation process is:

score=low+(len-10*2 ^{cost-highCost} ), (10)

Among them, low and highCost represent the lowest score and highest similarity of the current file, len represents the length of the score range of the current file, and cost represents the similarity of the sequence.

To sum up, the present invention realizes an intelligent rehabilitation auxiliary training system for patients with spinal degenerative diseases by innovatively combining artificial intelligence technology with traditional Chinese medicine rehabilitation medicine. Including: (1) Deep learning-based real-time classification model of traditional Chinese medicine Daoyin surgery video: The two-dimensional human skeleton data obtained through Openpose is used as the training data of the deep learning model, and supervised deep learning training is carried out to obtain a generalized deep learning model. Then, the video is preprocessed in real time through Openpose frame detection combined with sparse optical flow tracking, and the result is fed into the pre-trained deep learning model as input to obtain real-time classification results; (2) Video sequence division and evaluation based on human skeleton representation: Based on the above real-time classification results, the skeleton sequences of the same category are segmented and error-corrected, and the segmented subsequences are scored. Among them, the scoring process uses the dynamic time warping algorithm to compare the sub-sequences with the expert group video sequences, calculates the video similarity and converts it into a percentile score; The characteristics of the skeleton data pre-define reminder information, and when the pre-defined characteristic conditions are met, the patient is reminded in the form of voice broadcast. Experiments show that the real-time classification model proposed in the present invention achieves a good balance in accuracy and speed performance, and the intelligent rehabilitation auxiliary training system for spinal degenerative diseases based on this classification model has high application value.

The invention applies artificial intelligence technology to the field of rehabilitation training of traditional Chinese medicine daoyin, and promotes the recovery of limb motor function by automatically guiding patients' limb movements on the basis of regulating mind and breath. The present invention, through technologies such as computer vision and deep learning, can identify and evaluate the accuracy of the patient's action posture in real time during the patient's training of traditional Chinese medicine guidance, and timely correct it in the form of a voice reminder, and at the end of the patient's training Afterwards, the patient's training was scored according to the training process and the results of the comparison of the movements of the expert group. The intelligent rehabilitation auxiliary training system designed by the present invention for patients in the remission period of spinal degenerative diseases does not require the guidance and intervention of medical staff, and enables patients to carry out the training of traditional Chinese medicine guidance at any time by themselves. It is suitable for families and primary medical and health institutions, and can greatly Reduce the pressure of medical staff and improve the flexibility and accuracy of patient rehabilitation training.

Description of drawings

FIG. 1 is a general flow block diagram of the present invention.

The left side of FIG. 2 is a schematic diagram of the joint points of the BODY_25 dataset in OpenPose, and the right side is a schematic diagram of dividing 25 key points into 8 limbs in the classification method of the present invention.

3 is an architecture diagram of a hierarchical limb attention LSTM network in the present invention, which is divided into four modules from left to right: a perspective conversion module, a body level module, a limb attention module, and a limb level module.

Detailed ways

其中T＝1000，J＝25，分别代表骨架序列的时间长度和骨架的关键点数据。In the present invention, the structure of the real-time classification model (hierarchical limb attention LSTM network) of TCM daoyin movements is shown in Figure 3, wherein the input skeleton only contains the two-dimensional coordinate information of 25 key points after preprocessing, as shown in Figure 2 shown, where each backbone sequence can be expressed as

where T=1000 and J=25, representing the time length of the skeleton sequence and the keypoint data of the skeleton, respectively.

The raw skeleton sequence data is input into the hierarchical limb attention LSTM for classification after preprocessing including translation, normalization and zero padding, and finally outputs the classification score of each frame of the skeleton sequence.

The implementation of the present invention includes two parts: the training of the classification model (hierarchical limb attention LSTM network) and the realization of the intelligent rehabilitation auxiliary training system, wherein the specific implementation process is as follows:

(1) Preparation of the dataset

Due to the unique applicability of the present invention, we use our own collection of spinal rehabilitation exercise datasets (RDSD: rehabilitation movements for spinal degenerative diseases) to conduct experiments. RDSD consists of 1012 videos in 9 action categories (8 kungfu actions and standing actions), in which the average length of each video is 18.4 seconds (30 frames per second). All videos are mirror-flipped as data enhancement, and finally divided into training set and test set according to the ratio of 75:25.

(2) Data preprocessing

For the RDSD data set proposed by the present invention, the skeleton sequence is extracted by two-dimensional pose estimation through the open source multi-person pose skeleton code tool library OpenPose ^[1] , wherein each frame of the skeleton sequence consists of 25 key points of two-dimensional space coordinates. .

最后进行零填充操作，在不足1000帧的骨架序列后填充全零，对超过1000帧的骨架序列进行截取，保证所有骨架序列的时间长度为1000 帧。Based on each skeleton sequence, the present invention preprocesses the data set, which includes three steps of translation, normalization and zero padding: first, a translation operation is performed, and the first valid frame is subtracted from all coordinates of all frames of the skeleton sequence. The mid- _hip coordinates of

Finally, the zero-padding operation is performed, and all zeros are filled after the skeleton sequence of less than 1000 frames, and the skeleton sequence of more than 1000 frames is intercepted to ensure that the time length of all skeleton sequences is 1000 frames.

(3) Model training

The main hyperparameters in the model are: training conditions, batch size, learning rate, dropout rate, LSTM orthogonal initialization multiplication factor, maximum number of iterations;

In the present invention, the settings of the hyperparameters of the model are as follows: training conditions: a single GTX1070 GPU; batch size: set to 64; learning rate: the initial learning rate is set to 0.005, and ReduceLROnPlateau is used as the learning rate adjustment strategy, that is, if every 10 The validation loss value of the next iteration model does not decrease, and the learning rate is reduced by 10 times; the dropout rate: set to 0.1; the LSTM orthogonal initialization multiplication factor: set to 0.001; the maximum number of iterations: 300, when the model's validation loss value no longer decreases And after 30 iterations, the training is terminated early, and the total number of training is generally more than 100 times;

(4) Experimental results

In order to study the effectiveness of each module in the classification network (hierarchical limb attention LSTM network), the present invention compares the comparison experiments between each model with/without addition and the baseline network. Among them, the baseline network is RNNs, which consists of three layers of LSTM+Dropout; HRNNs indicates that the network consists of one layer of Limb-level LSTM+Dropout and two layers of body-level (Body-level) LSTM+Dropout; VT-RNNs compared to RNNs Added View-transformation Module as shown in Figure 3; VT-HRNNs vs HRNNs Added View-transformation Module as shown in Figure 3; VT-HRNNs-ATT vs VT-HRNNs The limb-attention module (Limb-Attention) shown in Figure 3 is added, which is the hierarchical limb-attention LSTM network of the present invention. As can be seen from Table 2, the use of the hierarchical structure (adding Limb-level) compared to all using the body-level (Body-level) LSTM improves the test accuracy by 1.22%, and the addition of the perspective conversion module improves the test accuracy. 1.04%, the addition of the limb attention module improves the test accuracy by 2.41%, which illustrates the effectiveness of each module in the hierarchical limb attention LSTM network in the present invention.

Table 2, the ablation experiment of each module of the hierarchical limb attention LSTM network of the present invention on the RDSD data set

network structure training accuracy test accuracy RNNs (Baseline) 95.30% 92.01% HRNNs 96.30% 93.23% VT-RNNs 95.43% 93.05% VT-HRNNs 96.06% 92.93% VT-HRNNs-ATT 96.64% 95.34%

(5) Realization of intelligent rehabilitation auxiliary training system

The intelligent rehabilitation auxiliary training system in the present invention includes the following four functions: real-time acquisition and classification of skeleton sequences based on OpenPose[1] frame-by-frame detection combined with sparse optical flow tracking, segmentation and real-time error correction of skeleton sequences, and skeleton sequence segmentation score, as shown in Figure 1.

The system receives the image frame signal source from the ordinary two-dimensional RGB camera, and obtains the skeleton sequence in real time. Specifically, by running OpenPose ^[1] 2D pose estimation every 5 frames, and according to the obtained 25 human body keypoint 2D coordinates, the human body keypoint 2 of the next 4 frames is obtained by the Lucas-Kanade sparse optical flow tracking method. dimensional coordinates.

Each time the latest 10 frames of human skeleton sequences are obtained, the classification network (hierarchical limb attention LSTM network) in the present invention is run once to obtain the action category labels of the 10 frames of sequences. Specifically, 10 frames of human skeleton sequences are preprocessed including translation, normalization and zero padding, and then fed into the pre-trained hierarchical limb attention LSTM network to obtain real-time classification results. Note that the Hierarchical Limb Attention LSTM network maintains the memory state of all LSTM units from the start of system operation and clears the memory state every 30s.

For the skeleton sequence of each frame, according to the two-dimensional coordinate information of its key points and the action category of the frame, the corresponding text error correction information is generated to remind the patient of the action points of the current exercise. Specifically, in the present invention, different reference parameters are respectively defined for 8 kinds of exercises, as shown in Table 1, which are used to dynamically calculate and generate text information, and remind the user through voice broadcast.

The system dynamically manages all classified frames that have not yet been divided into action segments, that is, action segment segmentation. Specifically, the class of the action segment is determined according to the action class of the unclassified frames of the latest several frames, and the inconsistency of the action class of the longest 100 frames is tolerated.

For the segmented action segment, the system selects the corresponding pre-loaded expert group video skeleton sequence information according to its action category, and calculates the similarity distance between the current action segment and the expert group action segment through the dynamic time warping algorithm, and then according to Predefined distance-score conversion rules to calculate the percentile score of the current action segment.

The system is implemented based on the Python3 language, which mainly uses public code libraries such as Tensorflow, OpenCV and multi-process processing, and runs in real time on the portable min-PC in the form of BS, which has a high intelligent auxiliary application for the rehabilitation training of patients with spinal degenerative diseases value.

references

[1] Cao Z, Hidalgo G, Simon T, et al. OpenPose: Realtime Multi-Person 2DPose Estimation using Part Affinity Fields [J]. IEEE Transactions on PatternAnalysis and Machine Intelligence, 2018.

[2] Donald J Berndt and James Clifford. 1994. Using Dynamic Time Wraping to Find Patterns in Time Series. Proceedings of the AAAI Conference onArtificial Intelligence (AAAI). 359-370pages.

Claims (5)

1. A spine degenerative disease intelligent rehabilitation auxiliary training system based on deep learning is characterized by comprising a traditional Chinese medicine guidance video real-time classification module based on deep learning and a video sequence dividing and evaluating module based on human body skeleton representation;

the traditional Chinese medicine guidance video real-time classification module based on deep learning obtains two-dimensional human skeleton data through Openpos, the two-dimensional human skeleton data serve as training data of a deep learning model, supervised deep learning training is carried out, and a generalized deep learning model is obtained; then, performing real-time preprocessing on the input video by combining OpenPose frame separation detection with sparse optical flow tracking, and taking the result as an input feed-in pre-trained classification model to obtain a real-time frame classification result;

and the human body skeleton representation and video sequence division and evaluation module is used for carrying out segmentation and real-time error correction on skeleton sequences of the same category according to the frame classification result of the real-time classification module, and carrying out sequence comparison grading on the segmented sequence segments and the video skeleton sequence segments of the corresponding category expert group.

2. The intelligent rehabilitation auxiliary training system for the deep learning-based degenerative spine disease as claimed in claim 1, wherein:

the working content of the real-time classification module of the traditional Chinese medicine guidance video based on deep learning is as follows;

(1) acquiring training data of deep learning;

(2) training a deep learning model;

(3) the method comprises the steps of preprocessing videos in real time by combining OpenPose frame separation detection with sparse optical flow tracking and classifying the videos;

corresponding to the human body skeleton representation-based and video sequence partitioning and evaluating module, the working contents are as follows:

(4) carrying out segmentation and real-time error correction on the skeleton sequence based on the classification result;

(5) carrying out comparison scoring on the sequence segments after the segmentation is finished;

in the content (1), the specific operation flow of obtaining deep learning training data is as follows:

(11) processing video data, and clipping all the video data according to the design of the rehabilitation training skill and action to obtain short video data with the length not more than 1000 frames;

(12) performing left and right mirror image turnover on all short video data to serve as data enhancement;

(13) performing two-dimensional posture extraction on a video data sample processed in the process (11) by using an Openpos model pre-trained on a BODY _25 dataset to obtain a skeleton data sequence represented by two-dimensional space coordinates of 25 key points, wherein the 25 key points are respectively Nose, Neck, RShoulder, Relbow, RChrist, LShoulder, LElbow, LWrist, MidHip, Rip, RKne, Rankle, LHip, Lknee, Lankle, REye, LEye, reader, LEAr, LBigToe, LSmalToe, LHeel, RBigToe, RSmalToe, Rheel, Back;

(14) performing data preprocessing on all the framework data sequences, firstly performing translation operation, and subtracting the hip middle coordinate of the first effective frame from all the frame coordinates of the framework sequences; then, normalization operation is carried out, and the average distance d between shoulders of the skeleton sequence is calculated_sThen, all frame coordinates of the skeleton sequence are scaled by a scaling factor of

Finally, zero filling operation is carried out, all zeros are filled behind the framework sequence, and the length of the framework sequence is fixed to 1000 frames;

in the content (2), the training of the deep learning model specifically includes:

(21) the deep learning model adopts a classification network model, particularly adopts a hierarchical limb attention LSTM network, and the network comprises a visual angle conversion module, a limb attention module, a limb level classification module and a body level classification module; the hierarchical limb attention LSTM takes skeleton data as input, firstly, two-dimensional translation transformation is carried out on input sequence coordinates through a visual angle conversion module, then, each frame coordinate of a skeleton sequence is divided into 8 limb parts, and the limb parts pass through an LSTM layer and a Dropout layer of a limb level classification module respectively; then, the feature vector magnitude of 8 limb parts is connected and the space-time attention weight of each limb is calculated through a limb attention module; finally, according to the space-time attention weight, corresponding to the feature vector magnitude of 8 limbs and weighting, and finally obtaining the classification score of the skeleton sequence on each frame through an LSTM layer, a Dropout layer and a softmax layer of a body level classification module;

(22) setting a model hyper-parameter;

the main hyper-parameters in the model are: training conditions, batch size, learning rate, discarding rate, LSTM orthogonal initialization multiplication factor and maximum iteration number;

(23) starting training, namely taking the verification loss value of the model during training as a reference, and when the verification loss value of the model does not decrease and continues for 30 iterations, indicating that the network is already stored, and ending the training;

(24) adjusting the hyper-parameters for multiple times to obtain a model with the best generalization performance;

in the content (3), the behavior classification is carried out on the video with classification by using the trained model, and the specific operation flow is as follows:

(31) acquiring and processing a skeleton sequence and classification from an input video signal source, acquiring skeleton sequence information in real time by adopting an OpenPose frame separation detection and sparse optical flow tracking mode, and performing OpenPose attitude estimation once every 5 frames to obtain the coordinates of the key points of the 25 human bodies; tracking two-dimensional coordinate information of key points of 25 human bodies of each frame in subsequent 4 frames by adopting a Lucas-Kanade method, wherein the specific process comprises the following steps:

S_t+1＝calOpticalFlowPyrLK(I_t，I_t+1，S_t)，

wherein S is_t，I_tRespectively representing a gray image and skeleton information of a t-th frame, wherein the calOpticalFlowPyrLK is one implementation of an OpenCV open source library to a Lucas-Kanade method;

(32) the classification of the skeleton sequence frame is slightly different from the classification in the reasoning process and the training process, in the reasoning process, skeleton information preprocessing and classification are carried out once after 10 frames of skeleton information are obtained, and the preprocessing process comprises translation, normalization and zero filling;

(33) in the reasoning process, all LSTMs in the model keep the parameter information of the LSTM after each classification, namely, an LSTM stateful mode is started, and in the mode, the LSTM network with hierarchical limb attention keeps the states of all LSTM layers after reasoning each time and is used as the initial state in the next reasoning;

in the content (4), the skeleton sequence is segmented and corrected in real time based on the classification result, and the process comprises the following steps:

(41) and (3) skeleton sequence segmentation based on classification results:

after the sequence frames are classified, real-time sequence segmentation is performed, specifically: when the first frame of the sequence or the last frame of the last sequence section is determined, judging the type of the current sequence section according to the classification result of the current frame, and determining the basis for ending the sequence section, namely whether the frame type is continuously inconsistent with the type of the current sequence section and the maximum error tolerance length is reached or the input sequence is ended;

(42) the real-time error correction based on the classification result comprises the following processes:

calculating different parameters according to the skeleton information and the classification result of each frame, and generating error correction text information;

the comparative scoring of the segmentation-ended sequence segments in the content (5) comprises:

after a new sequence segment is determined, carrying out sequence comparison scoring according to the category of the sequence segment; first, the system maintains an expert group skeleton sequence information Q for each action analogy_nFor a sequence segment C_mCalculating the similarity of the sequences at the two ends by a dynamic time rule (DTW) algorithm, finding the optimal alignment path of the two sequences by the DTW algorithm through the idea of dynamic programming, and calculating the Euclidean distance sum of sequence frames on the path:

Cost(i，j)＝D(i，j)+min[Cost(i-1，j)，Cost(i，j-1)，Cost(i-1，j-1)]，

wherein x, y represent two-dimensional coordinates in the sequence segment, and D (i, j) represents the sequence Q_nFrame i and sequence C_mThe cumulative euclidean distance sum of Cost (i, j) at the (i, j) position of the best alignment path of the two sequences, and the similarity of the two sequences of Cost (n, m); according to the similarity of the sequences, calculating the comparison score of the current sequence segment and the expert group framework sequence, and adopting a percent system; in order to make the score distribution more uniform, the scores are divided into 4 grades according to the similarity of the sequences: respectively 90-100, corresponding to similarity 0-2; 75-90, corresponding to similarity 2-4; 60-75 parts of; the corresponding similarity is 4-6; 40-60, corresponding to the similarity of 6-infinity, and for the similarity of each grade, the score calculation process is as follows:

score＝low+(len-10*2^{cost-highCost})，

wherein low and highpost represent the lowest score and highest similarity of the current gear, len represents the length of the score range of the current gear, and cost represents the similarity of the sequence.

3. The intelligent rehabilitation training system for spinal degenerative disease based on deep learning of claim 2, wherein the operation process of the view transformation module in the LSTM network for level limb attention in the step (21) is as follows:

in order to reduce the influence of shooting angle change on the classification performance of the model, the visual angle conversion module is used for carrying out self-adaptive two-dimensional translation and rotation operation on the input skeleton sequence, and coordinates of each frame of the input skeleton sequence are adjusted, wherein the specific calculation process is defined as:

S′_t，j＝[x′_t，j，y′_t，j]′＝R_t(S_t，j-d_t)，

wherein S is_t，j＝[x_t，j，y_t，j]' two-dimensional coordinates representing the jth key point of the tth frame of the input skeleton sequence;

representing a translation vector corresponding to the t-th frame; r_tThe two-dimensional rotation matrix corresponding to the t-th frame is represented as:

wherein,

respectively represents the translation quantity alpha of all coordinates of the t-th frame of the skeleton sequence along the horizontal axis and the vertical axis_tRepresenting the radian of anticlockwise rotation of all coordinates of the t frame of the skeleton sequence;

the skeleton information passing through the visual angle conversion module is divided into 8 limbs with certain overlap according to human body distribution, namely a head, a left arm, a right arm, a trunk, a left leg, a right leg, a left foot and a right foot, and then the information of different limbs is calculated by an independent LSTM layer and a Dropout layer and then is cascaded into the integral skeleton information again.

4. The intelligent rehabilitation training system for spinal degenerative disease based on deep learning of claim 3, wherein the operation process of the limb attention module in the hierarchical limb attention LSTM network in the step (21) is as follows:

wherein H_t，iI-th limb information indicating the t-th frame skeleton sequence, i ≦ L (L ≦ 8), H_tRepresenting the characteristic information obtained after 8 limb characteristic information passes through an LSTM layer and a Dropout layer after being cascaded, W₁，W₂Is a learnable parameter matrix, b₁，b₂Is a deviation vector, then a weight vector a of each limb is calculated by softmax activation_t，lFinally, obtaining the weighted feature vector of each limb, and cascading all weighted limb feature vectors as the input of a subsequent module:

H′_t，l＝a_t，l·H_t，l，

H′_t＝concat(H′_t，1，...，H′_t，L)，

after the cascaded weighted limb feature vectors pass through two layers of LSTM and Dropout and a full connection layer, classification scores are obtained through softmax activation.

5. The intelligent rehabilitation training system for spinal degenerative disease based on deep learning as claimed in claim 3, wherein the process (42) calculates different parameters according to the skeleton information and classification result of each frame, and the relationship between the motion category and the calculated parameters is as follows:

the action categories are: 0, 1, 2, 3, 4, 5, 6, 7; the corresponding parameter names are as follows in sequence: the two hands form an included angle with the lifting amplitude and the upper arm and the lower arm; the relative amplitude of left-right unfolding of the hands and the left-right rotation amplitude of the head; the relative longitudinal height of the double elbows and the relative transverse width of the double axes; the relative longitudinal height of the double elbows and the relative transverse width of the double axes; the relative downward extension range of the two hands; the ratio of the relative downward extension amplitude of the hands to the average length of the legs; longitudinal distance between the hip and the neck; the ratio of the difference in longitudinal distance between the feet to the longitudinal distance between the hip and the neck.

Images