CN115985462A - Rehabilitation and intelligence-developing training system for children cerebral palsy - Google Patents
Rehabilitation and intelligence-developing training system for children cerebral palsy Download PDFInfo
- Publication number
- CN115985462A CN115985462A CN202310093200.5A CN202310093200A CN115985462A CN 115985462 A CN115985462 A CN 115985462A CN 202310093200 A CN202310093200 A CN 202310093200A CN 115985462 A CN115985462 A CN 115985462A
- Authority
- CN
- China
- Prior art keywords
- training
- rehabilitation
- infant
- game
- children
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a rehabilitation and intelligence-developing training system for children cerebral palsy, which relates to the technical field of cerebral palsy rehabilitation and has the technical scheme that: the game training system comprises a camera, a mobile terminal and a game training library module. The system integrates the intelligence-developing game into the rehabilitation training of the children patients, so that the system is more suitable for the game nature of the children patients, is beneficial to improving the enthusiasm of the children patients in the rehabilitation training and realizes efficient rehabilitation training; meanwhile, emotion information of the infant is fed back to the family members in time in the whole process of rehabilitation training incorporating emotion detection, so that the family members are assisted to perform psychological intervention on the infant, the compliance of rehabilitation training of the infant is improved, and the generation of adverse emotion is prevented; in addition, this system is based on considering the infant, directly gathers infant motion information through the camera, has avoided the connection and the wearing of a large amount of peripheral hardware, comfort level when having improved infant rehabilitation training makes the preparation operation more convenient simultaneously, more does benefit to and carries out rehabilitation training at home.
Description
Technical Field
The invention relates to the technical field of cerebral palsy rehabilitation, in particular to a cerebral palsy rehabilitation intelligence development training system for children.
Background
Cerebral palsy is a group of persistent central motor and postural developmental disorders, the restricted activity syndrome, which is caused by non-progressive brain injury in the developing fetus or infant. Dyskinesias of cerebral palsy are often accompanied by sensory, perceptual, cognitive, communication and behavioral disorders, as well as epilepsy and secondary musculoskeletal problems.
The main mode of treatment of cerebral palsy is rehabilitation. Because cerebral palsy children involve dysfunction in various aspects such as nervous system, motor system (including muscles, tendons, bone joints and the like), speech, psychology and the like, rehabilitation therapy comprehensively utilizes various methods such as motor training therapy, operation therapy, speech therapy, psychological intervention and the like to inhibit abnormal nerve reflex and posture, and promotes normal motor function, speech function, psychological health and development of the nervous system.
Deep Learning (DL) is a new research direction in the field of Machine Learning (ML), and is an intrinsic rule and expression level for Learning sample data. The final purpose is to enable the machine to have the same learning and analyzing capability as human beings, to recognize complex data such as characters, images and sounds, and to solve more and more problems in the deep neural network along with the continuous development of deep learning.
The existing rehabilitation training therapy for children cerebral palsy needs to complete training under the field guidance of a professional doctor, so that the learning cost is high, and a large amount of medical resources are occupied. In response to this problem, the applicant invented a system for assisting the rehabilitation and intelligence-developing training of children's cerebral palsy based on deep learning.
Disclosure of Invention
The invention aims to provide a rehabilitation and intelligence-developing training system for children cerebral palsy, which solves the technical problems mentioned in the background technology.
The technical purpose of the invention is realized by the following technical scheme: the children cerebral palsy rehabilitation intelligence development training system comprises a camera, a mobile terminal and a game training library module;
the camera is used for collecting motion data and facial expression information of the sick child in real time and transmitting the information to the game training library module;
the game training library module is stored with a plurality of training games for guiding rehabilitation training of different parts of the infant, detecting a moving target of a picture acquired by the camera, evaluating and analyzing according to the motion situation of the infant to generate a training report, and generating real-time data to be sent to the mobile terminal when the infant trains;
the mobile terminal comprises a family end and a doctor end;
the family end is used for receiving the training data output by the analysis transmission module in real time, and can receive the training report generated by the data analysis transmission module after training is finished; meanwhile, part of training reports and evaluation results can be sent to the doctor end, so that doctor-patient data can be exchanged and shared;
the doctor end is used for receiving the training report and the evaluation result of the children patients transmitted by the family members end.
Further, the game training library module comprises a neck training unit, an upper limb training unit, a lower limb training unit, a standing and sitting training unit and a comprehensive training unit; the game training library module comprises a neck training unit, an upper limb training unit, a lower limb training unit, a standing and sitting training unit and a comprehensive training unit, wherein a plurality of corresponding training games are stored in the training unit, and the training games are analyzed according to the motion condition of the sick child and generate a training report.
Further, the method for processing the images by the game training library module comprises the following steps:
s1: carrying out moving target detection on the motion data and facial expression information of the infant patient in the image acquired by the camera;
s2: detecting bone key points of the children by using Python API provided by a Python module in OpenPose;
s3: and matching the skeleton point space-time change of the child patient by using the ST-GCN neural network model to complete action recognition and action evaluation, and realizing expression recognition by using the CNN neural network.
Further, the specific method for detecting the moving object of the game training library module in the S1 is as follows:
s1-1: processing the image input through a Gaussian mixture model;
s1-2: extracting the processed image data by a Gaussian filtering method;
s1-3: adopting expansion and erosion morphological treatment aiming at the possible crushing and breaking conditions;
s1-4: and the block with the maximum outline area is used as a complete moving target outline map, so that the moving target can be accurately detected.
Further, the game training library module evaluates the detection result by using an intersection ratio (IoU), a transmission frame number per second (FPS), an average precision (mAP), a precision (P), a recall rate (R) and a cost (loss value) when the moving object is detected.
Further, the bone key points of the children patient are detected and evaluated by adopting a percentage comparison model of correct key points of indexes.
In conclusion, the invention has the following beneficial effects:
1. compared with the existing repeated and boring rehabilitation training mode, the system integrates the intelligence-developing game into the rehabilitation training of the infant, better accords with the game nature of the infant, is beneficial to improving the rehabilitation training enthusiasm of the infant and realizes efficient rehabilitation training;
2. compared with the prior art that the psychological change of the infant is neglected in the rehabilitation training, the system brings emotion detection into the whole rehabilitation training process, timely feeds the emotion information of the infant back to the family members, assists the family members to perform psychological intervention on the infant, improves the compliance of the rehabilitation training of the infant, and prevents the generation of adverse emotion;
3. compared with the existing complex wearable equipment and peripheral equipment of rehabilitation products, the system directly collects motion information of the sick children through the cameras based on consideration of the sick children, avoids connection and wearing of a large number of peripheral equipment, improves the comfort level of the sick children during rehabilitation training, simultaneously enables preparation operation to be more convenient and faster, and is more beneficial to home rehabilitation training;
4. compared with the prior rehabilitation products which neglect the popularization of the family professional knowledge, the system brings the professional medical advice and the rehabilitation guidance into the whole training process, makes up the defect of the family specialty and improves the specialty of the family rehabilitation training;
5. compared with the high demand of the existing rehabilitation products for offline doctor knowledge, the system develops an on-cloud service platform, realizes on-cloud doctor-seeking inquiry, constructs an online offline doctor-patient communication closed loop, improves the utilization of medical resources and relieves social medical pressure.
Drawings
FIG. 1 is a block diagram of a system of a rehabilitation and intelligence-improving training system for cerebral palsy of children according to an embodiment of the present invention;
FIG. 2 is a flow chart of the overall system in an embodiment of the invention;
FIG. 3 is a schematic structural diagram of a game training library module in the embodiment of the present invention;
FIG. 4 is a flow chart of a Gaussian filtering method according to an embodiment of the invention;
FIG. 5 is a diagram of a moving object detection neural network architecture incorporating Gaussian filtering in an embodiment of the present invention;
FIG. 6 is a network structure diagram of the OpenPose algorithm based on CNN in the embodiment of the present invention;
FIG. 7 is a schematic diagram of a convolutional neural network of ST-GCN space-time diagrams in the embodiment of the present invention
Fig. 8 is a structural diagram of a CNN convolutional neural network in the embodiment of the present invention.
Detailed Description
The present invention is described in further detail below with reference to figures 1-8.
Example (b): a rehabilitation and intelligence-developing training system for children cerebral palsy, as shown in fig. 1 to 8, comprising a camera, a mobile terminal and a game training library module;
the camera is used for collecting motion data and facial expression information of the infant patient in real time and transmitting the information to the game training library module;
the game training library module is stored with a plurality of training games for guiding rehabilitation training of different parts of the infant, the picture collected by the camera is subjected to moving target detection, meanwhile, evaluation and analysis are carried out according to the motion situation of the infant to generate a training report, and real-time data is generated and sent to the mobile terminal when the infant trains;
the mobile terminal comprises a family end and a doctor end;
the family members store training evaluation questionnaires, the family members can fill in the training evaluation questionnaires in the APP of the family members, and the system generates evaluation results according to the filling contents of the questionnaires to assist the family members in mastering the rehabilitation training progress of the children patients in time. In the training process, the family members receive the computer emotion data output in real time through the mobile phone, master the emotion of the infant patient, intervene in the training process in time and perform psychological intervention. After training is finished, family members can receive a training report of the child patient from the mobile phone, and the training condition of the child patient can be checked; meanwhile, partial training reports and evaluation results can be sent to an APP (application) of a doctor end, doctor suggestions are received, and doctor-patient data exchange and sharing are achieved;
the doctor end is used for receiving the infant training report and the evaluation result transmitted by the family member end, comprehensively mastering the training process and the training progress of the infant, timely carrying out rehabilitation training diagnosis and treatment guidance and constructing an effective doctor-patient communication closed loop.
The game training library module comprises a neck training unit, an upper limb training unit, a lower limb training unit, a standing and sitting training unit and a comprehensive training unit; the game training library module comprises a neck training unit, an upper limb training unit, a lower limb training unit, a standing and sitting training unit and a comprehensive training unit, wherein a plurality of corresponding training games are stored in the training unit, and the training games are analyzed according to the motion condition of the sick child and generate a training report.
In this embodiment, a game of 'small regular-movement balls' is stored in the neck training unit, and the infant patient controls the small balls to impact the drumheads in different directions through head movement according to voice and picture prompts. The game aims to realize the rehabilitation training of neck muscles of children patients and prevent or reduce the occurrence of secondary neck muscle deformity as much as possible. Meanwhile, the unit carries out training analysis according to the head movement swing of the sick child to generate a training report.
The upper limb training unit stores games of 'small painters' and 'catching butterflies together', and the training mode of the small painters is as follows: and (3) simulating the movement of a painting brush by the upper limb and hand movement of the infant according to the game demonstration, and completing the missing lines or graphic areas in the picture. The game aims to enable the sick children to complete specific tasks under the coordination of various psychological activities such as perception, attention and the like by means of the movement of muscles or small muscle groups at the parts such as upper limbs, fingers and the like, so as to realize upper limb training and improve fine motor ability. Meanwhile, the unit module conducts training analysis according to the upper limb movement actual condition of the child patient to generate a training report. The training mode of catching butterflies together is as follows: the infant controls the virtual character to touch the butterfly randomly generated by the system by waving the arm according to the picture prompt. The game aims to enable the infant patient to realize the training of the shoulder, elbow, wrist and metacarpophalangeal joints through the movement of the two upper limbs and the hands, improve the movement capability and flexibility of the upper limb joints and improve the coordination and reaction capability of the upper limbs. Meanwhile, the game analyzes the flexibility of the shoulder, elbow, wrist and metacarpophalangeal joints of the children patients, provides a targeted training suggestion and generates a training report.
The lower limb training unit stores games of 'football boys', 'rhythm master' and 'I are small athletes', and the training mode of the football boys is as follows: the infant kicks the virtual football on the screen by completing standard kicking motions, and only when the system recognizes that the actions of the infant are standard and rapid enough, the infant can kick the ball into the goal. The game aims at carrying out leg muscle training on the children patients, so that the children patients have basic leg movement ability; preventing or minimizing the occurrence of secondary leg muscle deformities. Meanwhile, the unit analyzes the swing of the legs of the sick children to generate a training report. The training mode of the rhythm master is as follows: the infant patient moves to the corresponding mark before the arrow rolls to the bottom line according to the rolling arrow prompt in the game. The game aims at children with partial walking ability, and the children can carry out walking training such as forward walking, side walking, backward walking and the like by giving image instructions, so that the lower limb strength and the walking ability of the children are enhanced. Meanwhile, the unit analyzes the stride and the stride frequency of the sick child to generate a training report. The training mode of my small athletes is as follows: the sick children hide according to the shape and the position of the barrier on the virtual track in a jumping, squatting and left-right moving mode. The game aims at enabling the sick children to continuously carry out squatting training in the process of avoiding obstacles through the interaction between the sick children and the game picture, thereby achieving the purpose of exercising the muscle strength of the legs. Meanwhile, the unit analyzes the motion degree of the infant patient to generate a training report.
The stand and seat training unit stores a game of 'best circus', and the training mode of the game is as follows: the infant completes the balance stunt by keeping the acrobatics in the sitting or standing position control screen. The game aims to correct the abnormal sitting posture and the abnormal standing posture of the patient through the sitting position and the standing position training. Meanwhile, the system analyzes the standard degree and duration of sitting posture and standing posture, provides a targeted training suggestion and generates a training report.
The comprehensive training unit stores games of 'not needing caterpillar, needing small safflower' and 'crossing foam wall', and the training mode of the game of the small safflower is that: the sick children control the virtual character to avoid the caterpillars and touch the small safflowers through body movement. Aims to improve the control capability of the infant to each key point of the body and simultaneously exercise the muscle strength of the neck, the upper limbs and the lower limbs of the infant by finishing the actions of swinging the head, moving the shoulder joints, swinging the upper limbs, moving the hip joints, swinging the lower limbs and the like. Meanwhile, the unit analyzes the action of the infant patient to generate a training report. The training mode of the crossing foam wall is as follows: the infant needs to put out a corresponding posture according to the cavity model on the virtual wall to control the virtual character to pass through the wall. The system aims to encourage the sick children to finish the self-training of the joints of the whole body and improve the control capability of the joints of the upper limbs and the lower limbs; various actions are organically combined, and the coordinated development of all joints is promoted; visual and auditory stimulation is realized through action simulation, and the hypoevolutism of nerve movement is avoided. The unit generates a training report by analyzing the action standard degree of the sick child.
In this embodiment, the method for processing the image by the game training library module is as follows:
s1: carrying out moving target detection on the motion data and facial expression information of the infant patient in the image acquired by the camera; the specific steps of moving object detection are shown in fig. 4:
s1-1: firstly, introducing a Gaussian mixture model to process image input; the treatment is a primary treatment, and the accuracy of the result of the treatment is lowThe device is easily influenced by rapid background change and illumination shadow change; in this embodiment, a Gaussian Mixture Model (GMM) is modified from a single Gaussian distribution Model with a time complexity O (n) 2 The input image is subjected to learning training through the Gaussian mixture model, a complete background initialization model can be extracted and used as the basis for tracking and identifying the foreground moving target, and the formula is as follows:
s1-2: extracting the processed image data by a Gaussian filtering method, and realizing data noise reduction without damaging a precursor of a moving target; gaussian filtering (gaussian), also known as gaussian smoothing, is often used to blur or soften images or to remove unwanted noise from images. The gaussian filtering process of the image is to perform convolution operation on the image and the normal distribution, so that the used gaussian filtering pixel weight distribution matrix is a basic 3 x 3 type matrix. Therefore, after the foreground target is obtained, the noise can be effectively eliminated, excessive elimination of the foreground image of the broken and fractured motion and interference and influence of the noise on the target contour track are avoided, and the integrity of target contour extraction is effectively improved.
S1-3: adopting expansion and erosion morphological treatment aiming at the possible crushing and breaking conditions; firstly, carrying out expansion repairing on the image, and then obtaining a rough outline by utilizing erosion; dilation and erosion are commonly used methods in morphological processing, and are mainly used for solving the problem of object contour fragmentation and fracture in image processing. In order to solve the problem of object contour breakage and fracture after Gaussian filtering, a moving object contour is more finely processed by adopting an expansion and erosion method. In order to avoid excessive expansion while realizing effective patching, one percent of the height of an input image is rounded to be used as an expansion coefficient, and the expansion coefficient is reduced by one to be used as an erosion coefficient, so that an ideal effect is achieved.
S1-4: in order to enhance the self-adaptive setting capability of the background, a block with the largest outline area is used as a complete moving target outline graph to realize accurate detection of the moving target; the maximum area judgment aims to solve the problem that misjudgment is caused by rapid change of background objects in results after Gaussian filtering and expansion erosion processing are completed by an algorithm. In the embodiment, the contour searching method is adopted to extract all object contours in the image and record the object contours in the contour matrix, the area of the block is calculated according to the contour matrix, and finally the contour of the moving object is judged according to the maximum area, so that the moving object is accurately extracted and identified.
In this embodiment, the motion detection evaluation index is evaluated by an IoU (intersection over intersection). The strength of the correlation is reflected by calculating the size of the overlapping rate of the 'predicted object frame' (DT) and the 'real object frame' (GT). In the evaluation index, we set a threshold to 0.8, and if IoU is greater than the threshold, the detection is considered valid.
In addition, the commonly used evaluation indexes FPS (Frames Per Second — frame Per Second) and mAP (Average accuracy — Mean Average Precision) in the target detection algorithm are also important references. Also, the present embodiment also evaluates the moving object detection learning effect based on P (Precision) and R (Recall) and cost (loss value).
S2: detecting bone key points of the children by utilizing Python API provided by a Python module in OpenPose;
OpenPose is an open source library which is built based on a convolutional neural network and supervised learning and takes caffe as a framework, can realize the tracking of facial expressions, trunks, limbs and even fingers of people, is suitable for a single person and a plurality of people, and has better robustness. The skeletal key point markers of OpenPose encompass the following three parts:
a. extracting image features by a convolutional network to obtain a group of feature Maps, then dividing the feature Maps into two branches, and extracting Part Confidence Maps and Part Affinity Fields by using a CNN network respectively;
b. then, part Association is solved according to Bipartite Matching, the joint points of the same object are connected, and the generated even Matching accuracy is higher due to the vector of the PAF, so that the even Matching accuracy can be finally combined into the whole framework of the infant patient;
c. and solving the Multi-Person matching based on the PAFs, thereby converting the Multi-Person matching problem into a graphs problem, and finally completing the bone key point detection with the help of Hungarian Algorithm (Hungary Algorithm).
The Open Pose algorithm network structure based on the CNN is shown in FIG. 6.
The Convolutional Neural Network (CNN) is a feedforward neural network, generally comprises an input layer, a convolutional layer, a pooling layer, a full-link layer and an output layer, can effectively extract deep image features, and in addition, the local sensing, weight sharing and pooling characteristics of the CNN can greatly reduce the calculation complexity, improve the training speed and realize the invariance of translation, scaling and rotation. Openpos is an improvement on the basis of the convolutional neural network CNN.
At present, the multi-person skeleton key point detection generally adopts a top-down idea, firstly carries out pedestrian detection, then segments each object, and finally extracts respective human body key points based on each independent individual. However, in the embodiment, only single bone point identification of the infant is needed, and multi-person identification is often designed in the traditional top-down bone point detection model, so that the single calculation amount is large, the consumed time is long, and the problem of repeated identification exists.
In the embodiment, on the basis of the top-down method, a solution is formulated by taking the recognition and detection thought of bottom-up in OpenPose as a reference, and an OpenPose library is utilized to construct a proper CNN neural network, so that a better detection effect and a better calculation speed are obtained.
After localization to the infant, we used PAF (partial area Affinity-Part Affinity Fields) to encode 2D vectors of limb position and orientation in the image domain; and marking the Confidence coefficient (namely a commonly-called 'heat map') of each key point by means of CMP (Part Detection Confidence Maps), so that the association between the key point positions and the key points is jointly learned with the help of two branches, thereby improving the clustering accuracy and avoiding the occurrence of wrong connection problems. Meanwhile, a Greedy matching Algorithm (Greedy matching Algorithm) is utilized, so that the global situation can be sufficiently coded under the condition of low operation cost, and a high-quality result is obtained.
The evaluation index for detecting the bone key points of the children patients is to evaluate the model by using the detection performance of the bone key points of the human body to evaluate the Percentage (PCK) of the correct key points of the common indexes. The PCK determines the accuracy of the detected key points by measuring the distance between the positions of the predicted key points and the positions of the real key points. If the distance is within the threshold value, the detected key point is correctly positioned. The higher the PCK value, the better the model performance.
In addition, common evaluation indexes such as Object Keypoint Similarity (OKS), average Accuracy (AP), mean average precision (mean average) and the like are used as reference bases, and comparison analysis is performed on the MPII and MSCOCO data sets.
S3: and matching the skeleton point space-time change of the child patient by using the ST-GCN neural network model to complete action recognition and action evaluation, and realizing expression recognition by using the CNN neural network.
The ST-GCN (Spatial Temporal Graph Convolutional Network-space-time diagram) is proposed based on the GCN as shown in fig. 7. The core view is to combine the TCN and the GCN to process the graph structure data with timing relationship. The network is divided into two parts:
GCN-Net: the input data is subjected to spatial convolution, namely, the convolution is applied to data at different points of the same time sequence without considering time factors.
TCN-Net: and performing time sequence convolution on the data, and performing convolution on the data of the same point in different time sequences by considering the relation of the same characteristic point in different time sequences.
Compared with the convolutional neural network, the ST-GCN neural network directly learns the input skeletal key point data, so that the picture input and the background noise generated when all videos are split into one frame and one frame are avoided, the data input amount of the ST-GCN is one eighths of ten thousand of that of the convolutional neural network, the calculation burden is reduced, the operation speed is greatly improved, and the real-time requirement of a project is met.
In the embodiment, the human body key point information output by openpos recognition can be regarded as a topological structure, so that the efficiency of the GCN in extracting the spatial characteristics is obviously superior to that of a convolutional neural network. Meanwhile, the sick children need a certain time when completing a certain rehabilitation training action, namely, in the process of completing the rehabilitation training action, human key points of the sick children have certain characteristics in the time dimension, so that the TCN has obvious advantages in processing the space-time change relation compared with a convolutional neural network. Therefore, the ST-GCN can be convolved with the real-time current time sequence and space dimensions and extract features to obtain a model.
For the NTU-RGB + D data set, 10% of all data are used as a test set, then the data are input into an ST-GCN time-space diagram convolution neural network model, and then a confusion matrix is generated according to the matching rate of a prediction result and an actual label. And according to the accuracy (accuracycacy) of the prediction result and the precision (precision), recall rate (recall), accuracy and recall rate harmonic value (F-score), macro-average (macro-average) and weighted-average (weighted-average) of each action classification, taking the accuracy and the recall rate harmonic value (F-score) of each action as the generalization performance index of the model evaluation. And evaluating the fitting effect of the model according to respective loss values (cost) of the training data set and the testing data set.
Meanwhile, the embodiment adopts a plurality of data sets to evaluate the generalization capability of the trained network model, wherein the confusion matrix is generated according to the matching rate of the prediction result and the actual label after the first 10% of the data sets such as the classical Activity-Net, the Kinetics-Skeleton and the like are transmitted into the neural network model. And evaluating the fitting effect of the model according to the accuracy (accuracycacy) of the prediction result, the precision (precision) of each action classification, the recall rate (call), the accuracy and recall rate harmonic value (F-score), the macro-average (macro-average) and the weighted-average (weighted-average) as the generalization performance indexes of the model evaluation and the respective loss values (cost) of the training data set and the testing data set.
The action evaluation is to evaluate the completion quality of the specific action. The multifunctional training aid is generally applied to the professional fields of sports, dancing, taijiquan and the like, can assist referees and coaches in scoring, and more importantly helps people to perform action analysis and training. Meanwhile, the action recognition and the action evaluation are closely related, the two have many common points on technical processes and methods, and the action evaluation is often required to be completed on the basis of the action recognition.
Therefore, each intelligence developmental game training in the game library of the embodiment is provided with an action evaluation algorithm, namely, a set threshold value is set for the action completion degree, and only when the rehabilitation action of the sick child reaches the standard degree and exceeds the threshold value, the game animation can be activated to enter the next game rehabilitation training. Meanwhile, the system scores the actions according to the action completion standard degree, and accordingly generates a final game score and a training report.
The realization scheme of the game training library module in the expression recognition is as follows:
1) Data set
The present embodiment employs the database CK +. The CK + database is an expansion on the basis of Cohn-Kanade data set, and comprises 123 subjects and 593 imagesequence including the table of expressions and the table of Action Units. When the method is used, the picture obtained after the data set is cut needs to be saved as an h5 file, and the data set is converted to generate data of the type of torch. In the CK + database, the image is divided into a front view and a 30 degree view, and may also be divided into an 8-bit grayscale, digitized as a 640x490 pixel array, and a 24-bit color map, digitized as a 640x480 pixel array. The expression data of the database are obtained under the laboratory condition, and information contribution participants cover the huge data categories of different sexes, different ages and different ethnicities, so that the model is more rigorous and reliable in application training.
2) Convolutional Neural Network (CNN)
In the training process, the ImageDataGenerator is used for realizing data enhancement, label is divided according to file names through flow _ direction, an SGD optimization algorithm and a Softmax classification method are selected for fully learning the sample, and the problem that gradient vanishing training is difficult because normalization is not performed is solved by taking the hard saturated ReLU as an activation function.
Meanwhile, in the training process, in order to solve the problem that the number of CK + data set samples is small, the pictures are cut and mirrored in the upper left corner, the lower left corner, the upper right corner, the lower right corner and the center, so that the pictures in the database are enlarged ten times, then the pictures are sent into a model, and the probability value is taken to obtain the maximum output classification as the corresponding expression recognition result. Therefore, the data volume of the database is enlarged, data overfitting is slowed down, the robustness of the training network is enhanced, and the accuracy of the prediction classification result is improved.
3) Evaluation index
For the CK + data set, 20% of all data are used as a test set, and then the data are input into a convolutional neural network model to generate a confusion matrix according to the matching rate of a prediction result and an actual label. And according to the accuracy (accuracycacy) of the prediction result and the precision (precision), recall rate (recall), accuracy and recall rate harmonic value (F-score), macro-average (macro-average) and weighted-average (weighted-average) of each expression classification, taking the accuracy and the recall rate harmonic value (F-score) as the generalization performance indexes of the model evaluation. And meanwhile, evaluating the fitting effect of the model according to respective loss values (cost) of the training data set and the test data set.
Meanwhile, a plurality of data sets are adopted to evaluate the generalization capability of the trained network model, and in the embodiment, the first 15% of the data sets such as classical FER2013 (Goodfellow et al, 2013) and TFD (therotofacedatabase) are transmitted into the neural network model, and then a confusion matrix is generated according to the matching rate of the prediction result and the actual label. And evaluating the fitting effect of the model according to the accuracy (accuracycacy) of the prediction result, the precision (precision), recall rate (call), accuracy and recall rate harmonic value (F-score), macro-average (macro-average) and weighted-average (weighted-average) of each expression classification as the generalization performance indexes of the model evaluation and the respective loss values (cost) of the training data set and the test data set.
System design
1. Data structure
The family members firstly register account numbers and verify identity information before logging in the system, and the account numbers of the family members of the patient can realize simultaneous logging in of a computer terminal and a mobile phone terminal. Doctors also need to register account numbers and verify identity information, and the account numbers are bound with family members of patients to establish a one-to-many inquiry mode.
When the infant is subjected to game training at the computer end, the system can record the motion data and the emotion information of the infant, and the emotion information is fed back to the family mobile phone end in real time. And generating a game score and a training report after the training is finished, wherein the training report is automatically stored in a computer self-built folder and is simultaneously sent to the family mobile phone terminal. All the motion data and the expression data are transmitted to the cloud server for calculation and analysis, and the results are transmitted back to the computer terminal.
The family mobile phone end can receive the emotion information of the infant and the corresponding training report transmitted by the computer end, fill in the training effect evaluation questionnaire, and finally transmit the training report and the evaluation questionnaire to the doctor end through the cloud server.
The doctor end can check and collect the training report and the evaluation questionnaire and make diagnosis and treatment judgment, and effectively interact with the family end.
In conclusion, the deep learning model relates to One-Hot coding, matrix and vector representation, interaction of a three-terminal App application program at a computer terminal, family members and a doctor mobile phone terminal and front and back ends of a cloud server is represented by Json data, and the attribute structure of children patients training and account data is shown in the following table.
Infant training and account data attribute structure
| Introduction of data | Properties | Note that |
| Account number | Character string | Registered account information of patient family and doctor |
| Duration of training | Floating point type data | Training time for children patient |
| Training Game id | Integer number | Intelligence development game id number selected by children patients |
| Assessment questionnaire | Integer number | Storing each selection result of the evaluation questionnaire according to labels |
| Training report | Character string | Storing each index and corresponding result of the training report in sequence |
| Expression information | Character string | Real-time emotional information derived by means of detection of a child patient by a camera |
| Training movements | Character string | Behavior category judged by detection information |
| Identity information | Character string | Including authentication information of doctor, patient and family members |
2. Database with a plurality of databases
The system not only completes rehabilitation training and expression recognition, but also relates to information interaction between a doctor end and a family end and guiding error correction of rehabilitation training of a sick child by a computer end. After the infant starts training, the system records the training duration, the training content, the training score, the action completion degree and other related information of the infant, and realizes long-term analysis decision guidance for the rehabilitation training of the infant; and the doctor-patient account binding and file data transmission are realized in the doctor-patient communication plate, and a timely and effective doctor-patient communication platform is constructed. The database of the system mainly comprises the relation among children patients, family members and doctors and is used for the communication between rehabilitation training and doctors and patients.
In the infant training stage, the system collects various data during infant training and stores the data into a database, and data analysis and training report generation are realized; and for the doctor-patient information, the account numbers and the patient information are stored in a database, so that the identity binding of the doctor-patient information and the patient information is realized, and a doctor-patient communication platform is built.
3. Server
Compared with a common server, the ECS cloud server to be used by the embodiment is simpler and more efficient, the processing capacity can be elastically expanded and contracted, and hardware does not need to be purchased in advance. The applications built on the cloud are more stable and safer, and meanwhile, the difficulty in developing operation and maintenance and the overall IT cost can be reduced, so that the development of core functions is concentrated. The server is mainly responsible for processing various data, is connected with interfaces such as an App application program and hardware equipment, runs a deep learning core function code, and is responsible for managing and integrating various function modules.
The present embodiment is only for explaining the present invention, and it is not limited to the present invention, and those skilled in the art can make modifications of the present embodiment without inventive contribution as needed after reading the present specification, but all of them are protected by patent law within the scope of the claims of the present invention.
Claims (6)
1. The children cerebral palsy rehabilitation intelligence-developing training system is characterized in that: the game training system comprises a camera, a mobile terminal and a game training library module;
the camera is used for collecting motion data and facial expression information of the sick child in real time and transmitting the information to the game training library module;
the game training library module is stored with a plurality of training games for guiding rehabilitation training of different parts of the infant, detecting a moving target of a picture acquired by the camera, evaluating and analyzing according to the motion situation of the infant to generate a training report, and generating real-time data to be sent to the mobile terminal when the infant trains;
the mobile terminal comprises a family end and a doctor end;
the family end is used for receiving the training data output by the analysis transmission module in real time, and can receive the training report generated by the data analysis transmission module after training is finished; meanwhile, part of training reports and evaluation results can be sent to the doctor end, so that doctor-patient data can be exchanged and shared;
the doctor end is used for receiving the infant training report and the evaluation result transmitted by the family member end.
2. The rehabilitation and intelligence-development training system for children's cerebral palsy according to claim 1, wherein: the game training library module comprises a neck training unit, an upper limb training unit, a lower limb training unit, a standing and sitting training unit and a comprehensive training unit; the game training library module comprises a neck training unit, an upper limb training unit, a lower limb training unit, a standing and sitting training unit and a comprehensive training unit, wherein a plurality of corresponding training games are stored in the training unit, and the training games are analyzed according to the motion condition of the sick child and generate a training report.
3. The rehabilitation and intelligence-development training system for children's cerebral palsy, according to claim 1, wherein: the method for processing the images by the game training library module comprises the following steps:
s1: carrying out moving target detection on the motion data and facial expression information of the infant patient in the image acquired by the camera;
s2: detecting bone key points of the children by using Python API provided by a Python module in OpenPose;
s3: and matching the skeleton point space-time change of the child patient by using the ST-GCN neural network model to complete action recognition and action evaluation, and realizing expression recognition by using the CNN neural network.
4. The rehabilitation and intelligence-development training system for children's cerebral palsy according to claim 3, wherein: the specific method for detecting the moving target of the game training library module in the S1 comprises the following steps:
s1-1: processing the image input through a Gaussian mixture model;
s1-2: extracting the processed image data by a Gaussian filtering method;
s1-3: adopting expansion and erosion morphological treatment aiming at the possible crushing and breaking conditions;
s1-4: and the block with the maximum outline area is used as a complete moving target outline map, so that the moving target can be accurately detected.
5. The rehabilitation and intelligence-development training system for children's cerebral palsy, according to claim 3, wherein: at the time of detecting the sports object, the game training library module evaluates the detection result by using an intersection ratio (IoU), a transmission frame number per second (FPS), an average precision (mAP), a precision (P), a recall rate (R) and a cost (loss value).
6. The rehabilitation and intelligence-development training system for children's cerebral palsy according to claim 3, wherein: and evaluating the bone key points of the children patient by adopting a percentage comparison model of correct key points of indexes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310093200.5A CN115985462A (en) | 2023-02-10 | 2023-02-10 | Rehabilitation and intelligence-developing training system for children cerebral palsy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310093200.5A CN115985462A (en) | 2023-02-10 | 2023-02-10 | Rehabilitation and intelligence-developing training system for children cerebral palsy |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115985462A true CN115985462A (en) | 2023-04-18 |
Family
ID=85968059
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310093200.5A Pending CN115985462A (en) | 2023-02-10 | 2023-02-10 | Rehabilitation and intelligence-developing training system for children cerebral palsy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115985462A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117275676A (en) * | 2023-11-21 | 2023-12-22 | 深圳市健怡康医疗器械科技有限公司 | Department of neurology guardianship potential safety hazard analysis method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107998601A (en) * | 2018-02-06 | 2018-05-08 | 湖州健凯康复产品有限公司 | A kind of interactive mode children with cerebral palsy device for healing and training |
| CN109806113A (en) * | 2019-03-14 | 2019-05-28 | 郑州大学 | A kind of ward ICU horizontal lower limb rehabilitation intelligent interaction robot group system based on ad hoc network navigation |
| CN112057082A (en) * | 2020-09-09 | 2020-12-11 | 常熟理工学院 | Robot-assisted cerebral palsy rehabilitation expression training system based on brain-computer interface |
| CN113571153A (en) * | 2021-08-19 | 2021-10-29 | 中国科学院计算技术研究所 | Passive training sensing system for limb rehabilitation of cerebral palsy patient and client side thereof |
| CN115331783A (en) * | 2022-08-11 | 2022-11-11 | 郑州世卓医药科技有限公司 | Training method for visual stimulation of autism of children based on VR (virtual reality) interaction technology |
-
2023
- 2023-02-10 CN CN202310093200.5A patent/CN115985462A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107998601A (en) * | 2018-02-06 | 2018-05-08 | 湖州健凯康复产品有限公司 | A kind of interactive mode children with cerebral palsy device for healing and training |
| CN109806113A (en) * | 2019-03-14 | 2019-05-28 | 郑州大学 | A kind of ward ICU horizontal lower limb rehabilitation intelligent interaction robot group system based on ad hoc network navigation |
| CN112057082A (en) * | 2020-09-09 | 2020-12-11 | 常熟理工学院 | Robot-assisted cerebral palsy rehabilitation expression training system based on brain-computer interface |
| CN113571153A (en) * | 2021-08-19 | 2021-10-29 | 中国科学院计算技术研究所 | Passive training sensing system for limb rehabilitation of cerebral palsy patient and client side thereof |
| CN115331783A (en) * | 2022-08-11 | 2022-11-11 | 郑州世卓医药科技有限公司 | Training method for visual stimulation of autism of children based on VR (virtual reality) interaction technology |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117275676A (en) * | 2023-11-21 | 2023-12-22 | 深圳市健怡康医疗器械科技有限公司 | Department of neurology guardianship potential safety hazard analysis method and system |
| CN117275676B (en) * | 2023-11-21 | 2024-02-09 | 深圳市健怡康医疗器械科技有限公司 | Department of neurology guardianship potential safety hazard analysis method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Thar et al. | A proposal of yoga pose assessment method using pose detection for self-learning | |
| Avola et al. | An interactive and low-cost full body rehabilitation framework based on 3D immersive serious games | |
| Tang | Hybridized hierarchical deep convolutional neural network for sports rehabilitation exercises | |
| Kamel et al. | An investigation of 3D human pose estimation for learning Tai Chi: A human factor perspective | |
| CN114998983A (en) | Limb rehabilitation method based on augmented reality technology and posture recognition technology | |
| Osgouei et al. | An objective evaluation method for rehabilitation exergames | |
| CN115985462A (en) | Rehabilitation and intelligence-developing training system for children cerebral palsy | |
| Farsi et al. | The effect of observational practice by a point-light model on learning a novel motor skill | |
| CN109126045A (en) | intelligent motion analysis and training system | |
| Patil et al. | Body posture detection and motion tracking using ai for medical exercises and recommendation system | |
| CN112966370A (en) | Design method of human body lower limb muscle training system based on Kinect | |
| Su | Artificial intelligence in the auxiliary guidance function of athletes’ movement standard training in physical education | |
| Gao | Construction and simulation of athlete’s wrong action recognition model in sports training based on embedded wireless communication and computer vision | |
| Talaat | Novel deep learning models for yoga pose estimator | |
| Wang et al. | Automatic emotion recognition based on non-contact gaits information | |
| Chang et al. | BodyTracker: A deep learning based 3D limb trajectory tracking system for rehabilitation | |
| Liu et al. | Judgment of Athlete Action Safety in Sports Competition Based on LSTM Recurrent Neural Network Algorithm | |
| Davids et al. | Artificial Intelligence for Physiotherapy and Rehabilitation | |
| Yongmao et al. | Application of wearable devices based on deep learning algorithm in rope skipping data monitoring | |
| Sun | Aerobics movement decomposition action teaching system based on intelligent vision sensor | |
| Potigutsai et al. | Hand and fingertip detection for game-based hand rehabilitation | |
| Weir et al. | OPEN ACCESS EDITED BY | |
| Lai | Optimization of Innovative Path of Physical Education Teaching in Colleges and Universities under Information Integration Technology | |
| Du | The Selecting Optimal Ball-Receiving Body Parts Using Pose Sequence Analysis and Sports Biomechanics | |
| Cavicchio et al. | Towards a theory of compositionality in displays of extreme emotion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |