CN113297919A - Rehabilitation action detection method, device, equipment and medium based on posture recognition - Google Patents
Rehabilitation action detection method, device, equipment and medium based on posture recognition Download PDFInfo
- Publication number
- CN113297919A CN113297919A CN202110474993.6A CN202110474993A CN113297919A CN 113297919 A CN113297919 A CN 113297919A CN 202110474993 A CN202110474993 A CN 202110474993A CN 113297919 A CN113297919 A CN 113297919A
- Authority
- CN
- China
- Prior art keywords
- joint point
- frame
- picture
- rehabilitation
- joint
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 230000009471 action Effects 0.000 title claims abstract description 74
- 238000001514 detection method Methods 0.000 title claims abstract description 48
- 230000008859 change Effects 0.000 claims abstract description 23
- 230000033001 locomotion Effects 0.000 claims description 51
- 238000000034 method Methods 0.000 claims description 35
- 210000000245 forearm Anatomy 0.000 claims description 29
- 210000000689 upper leg Anatomy 0.000 claims description 29
- 238000012549 training Methods 0.000 claims description 27
- 210000003414 extremity Anatomy 0.000 claims description 26
- 244000309466 calf Species 0.000 claims description 23
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 8
- 238000003860 storage Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 201000010099 disease Diseases 0.000 description 5
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 210000003127 knee Anatomy 0.000 description 3
- 210000003423 ankle Anatomy 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 210000000707 wrist Anatomy 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000010924 continuous production Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 230000035876 healing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 230000003863 physical function Effects 0.000 description 1
- 230000035479 physiological effects, processes and functions Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 210000002027 skeletal muscle Anatomy 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
- G06V20/53—Recognition of crowd images, e.g. recognition of crowd congestion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/30—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Psychiatry (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Social Psychology (AREA)
- Biophysics (AREA)
- Physical Education & Sports Medicine (AREA)
- Epidemiology (AREA)
- Medical Informatics (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention discloses a rehabilitation action detection method, a device, equipment and a medium based on gesture recognition, which comprises the following steps: inputting the acquired rehabilitation action picture into a preset HigherHRNet model to obtain the joint point position in each frame of picture; calculating the joint point distance between the continuous frames and the change angle of the limbs according to the joint point position in each frame of picture; calculating the moving speed of the joint point according to the joint point distance between the continuous frames; and obtaining the accuracy of the detected rehabilitation action according to the joint point position in each frame of picture, the joint point distance between the continuous frames, the moving speed of the joint point, the change angle of the limbs and the preset characteristic importance parameter. The detection method provided by the embodiment of the disclosure focuses more on the continuity of the rehabilitation action in the video, adopts the multi-frame image for detection, improves the detection accuracy, can adapt to various actions, and can change the judgment standard only by presetting some action characteristic importance parameters.
Description
Technical Field
The invention relates to the technical field of gesture recognition, in particular to a rehabilitation action detection method, a rehabilitation action detection device, rehabilitation action detection equipment and a rehabilitation action detection medium based on gesture recognition.
Background
The criteria for medically judging whether a patient is successfully cured are: successful healing is 50% of surgical success + 50% of recovery success. With the comprehensive understanding of people on disease treatment, rehabilitation training plays an increasingly important role in the whole disease treatment process. Firstly, physical strength of a patient can be obviously recovered through rehabilitation training, physical functions of the patient can be well recovered through simple skeletal muscle movement and aerobic training, and resistance of the patient can be improved; secondly, in the training process, the confidence of the patient on the victory disease can be improved, and the victory disease has certain treatment effect on the disease per se in physiology and psychology; thirdly, the training device has obvious promotion effect on special joints and motion functions, especially on operations of hands, joints, spines and the like, and the early functional training after the operation is very important for restoring the functions of operation organs. Therefore, the rehabilitation training after the operation is performed in an early stage and is performed gradually under the guidance of the doctor, so that a good rehabilitation training effect can be achieved.
The existing joint point recognition is mainly applied to posture recognition on conventional actions, and the posture recognition on rehabilitation actions is mainly to compare a single-frame image joint point with a single-frame standard action after recognition to obtain the standard degree of the action. The technology firstly ignores that rehabilitation training is a continuous process, and the action process is also important and can not be ignored. Secondly, the rehabilitation action has hundreds of different actions, when the actions are faced with the different actions, the prior art needs to classify the action to which kind of action belongs first, and then compares the actions with the kind of action to obtain a standard degree, on one hand, the hundreds of classifications can cause incorrect classification and wrong guidance, on the other hand, the excessive classifications can cause too long time and can not meet the requirement of real-time property.
Disclosure of Invention
The embodiment of the disclosure provides a rehabilitation action detection method, a rehabilitation action detection device, rehabilitation action detection equipment and a rehabilitation action detection medium based on posture recognition. The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
In a first aspect, an embodiment of the present disclosure provides a rehabilitation motion detection method based on gesture recognition, including:
inputting the acquired rehabilitation action picture into a preset HigherHRNet model to obtain the joint point position in each frame of picture;
calculating the joint point distance between the continuous frames and the change angle of the limbs according to the joint point position in each frame of picture;
calculating the moving speed of the joint point according to the joint point distance between the continuous frames;
and obtaining the accuracy of the detected rehabilitation action according to the joint point position in each frame of picture, the joint point distance between the continuous frames, the moving speed of the joint point, the change angle of the limbs and the preset characteristic importance parameter.
In an optional embodiment, before inputting the acquired rehabilitation motion picture into the preset highherhrnet model, the method further includes:
identifying one or more rehabilitation training users by adopting a face identification method;
when the rehabilitation training user is successfully identified and positioned, the video image of the user for rehabilitation training is shot.
In an alternative embodiment, calculating the joint distance between successive frames from the joint position in each frame of the picture comprises:
taking continuous multi-frame pictures as a sample, calculating the joint point distance between frames in the sample according to the following formula,
wherein I represents the number of joint points, 13 joint points in total, N represents the number of frames in one sample,
representing the distance from frame to frame at all the joints,
indicating the position of each joint point in the sample nth frame picture,
representing the distance from frame to frame of a particular joint.
In an alternative embodiment, the calculation of the angle of change of the limbs according to the position of the joint point in each frame of picture comprises:
respectively obtaining left and right forearm slopes, left and right big arm slopes, left and right shank slopes and left and right thigh slopes according to joint points connecting four limbs;
and calculating the angles of the left and right small arms and the left and right large arms and the angles of the left and right calves and the left and right thighs according to the slopes of the left and right small arms, the slopes of the left and right large arms, the slopes of the left and right calves and the slopes of the left and right thighs.
In an alternative embodiment, calculating the angle of the left and right forearm and the angle of the left and right calf and the left and right thigh from the left and right forearm slope, the left and right upper arm slope, the left and right calf slope, and the left and right thigh slope comprises:
wherein l1Represents the slope of the left forearm, l2Represents the slope of the right forearm, l3Indicates the slope of the left forearm, l4Represents the slope of the right big arm,/5Representing the slope of the left calf, l6Representing the slope of the right calf, l7Denotes the left thigh slope, l8Represents the slope of the right thigh, θ1Denotes the angle of the left forearm and the left forearm, theta2Representing the angle, θ, of the right forearm and the right forearm3Representing the angle of the left calf to the left thigh, θ4Representing the angle of the right calf to the right thigh.
In an alternative embodiment, calculating the moving speed of the joint based on the joint distance between successive frames comprises:
calculating the moving speed of the joint point according to the time occupied by the N frames of pictures in the sample and the joint point distance between the frames, wherein the specific formula is as follows:
wherein, ViRepresents the moving speed of the i-th joint point, DiIndicating the distance from frame to frame at the ith joint, N the number of frames in a sample, and fps the number of frames transmitted per second of the picture.
In an optional embodiment, the obtaining the detected rehabilitation action accuracy according to the joint point position in each frame of picture, the joint point distance between consecutive frames, the moving speed of the joint point, the change angle of the limbs and the preset feature importance parameter includes:
wherein alpha ise(e-1, 2,3,4) represents the preset characteristic importance parameter of each rehabilitation action, S represents the accuracy of the detected rehabilitation action,
indicating the position of the joint point in the f-th frame picture,
representing the distance, V, of the joint point from frame to frameIRepresenting the speed of movement, theta, of the joint point(i,j)Indicating the varying angle of the limbs.
In a second aspect, an embodiment of the present disclosure provides a rehabilitation motion detection apparatus based on gesture recognition, including:
the input module is used for inputting the acquired rehabilitation action picture into a preset HigherHRNet model to obtain the joint point position in each frame of picture;
the first calculation module is used for calculating the joint point distance between the continuous frames and the change angle of the limbs according to the joint point position in each frame of picture;
the second calculation module is used for calculating the moving speed of the joint point according to the joint point distance between the continuous frames;
and the detection module is used for obtaining the detected rehabilitation action accuracy according to the joint point position in each frame of picture, the joint point distance between the continuous frames, the moving speed of the joint point, the change angle of the limbs and the preset characteristic importance parameter.
In a third aspect, the disclosed embodiments provide a rehabilitation motion detection device based on gesture recognition, including a processor and a memory storing program instructions, where the processor is configured to execute the rehabilitation motion detection method based on gesture recognition provided by the above embodiments when executing the program instructions.
In a fourth aspect, the disclosed embodiments provide a computer-readable medium, on which computer-readable instructions are stored, where the computer-readable instructions are executable by a processor to implement a rehabilitation motion detection method based on gesture recognition provided in the foregoing embodiments.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
the rehabilitation action detection method based on posture recognition provided by the embodiment of the disclosure not only focuses on posture recognition of a single frame, but also extracts the rehabilitation action process. In addition, the method can adapt to various actions, the judgment standard can be changed only by presetting some action characteristic importance coefficients, and the adaptability of the method is improved. The joint point locating method can also identify rehabilitation training of multiple persons, can identify target patients and non-target patients, and enables the joint points to be located more accurately by adopting a network structure of HigherHRNet.
The method automatically detects the standard degree of the rehabilitation action, so that the patient can adjust the rehabilitation action, the rehabilitation action of the patient is more standard, the patient can be better recovered, and the burden of a rehabilitation doctor is also reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a flow diagram illustrating a method of gesture recognition based rehabilitation motion detection, according to an exemplary embodiment;
FIG. 2 is a schematic diagram illustrating a HigherHRNet network architecture in accordance with an exemplary embodiment;
FIG. 3 is a schematic diagram illustrating a three-dimensional body coordinate system according to an exemplary embodiment;
fig. 4 is a schematic structural diagram illustrating a rehabilitation motion detection device based on gesture recognition according to an exemplary embodiment;
FIG. 5 is a schematic diagram illustrating a configuration of a gesture recognition based rehabilitation motion detection device according to an exemplary embodiment;
FIG. 6 is a schematic diagram illustrating a computer storage medium in accordance with an exemplary embodiment.
Detailed Description
The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems and methods consistent with certain aspects of the invention, as detailed in the appended claims.
In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The rehabilitation motion detection method based on gesture recognition according to the embodiment of the present application will be described in detail with reference to fig. 1 to 3.
Referring to fig. 1, the method specifically includes the following steps;
s101, inputting the acquired rehabilitation action picture into a preset HigherHRNet model to obtain the joint point position in each frame of picture.
In a possible implementation manner, before performing step S101, the method further includes recognizing one or more rehabilitation training users by using a face recognition method, and when the rehabilitation training users are successfully recognized and located, capturing video images of the rehabilitation training users.
In an exemplary scene, a mobile terminal such as a tablet computer and a mobile phone is used for shooting a rehabilitation training video of a user, a face recognition system is installed in the mobile terminal, one or more rehabilitation training users can be recognized from the video, if the rehabilitation training users are successfully recognized and located through the mobile terminal, the user is prompted to start to do rehabilitation training actions through voice, and the mobile terminal device starts to continuously shoot the video, wherein the FPS of the video is not lower than 20, and the resolution ratio can be 1080 x 1060.
Further, posture recognition is carried out according to the shot rehabilitation training video, and joint point positions in each frame of picture are obtained.
Specifically, the shot rehabilitation training video is input into a preset HigherHRNet model, and the joint point position in each frame of picture is calculated and obtained through an AI algorithm aiming at each frame of rehabilitation action picture transmitted in real time
Where f denotes the several pictures and i denotes the several joints.
The high herHRNet model has the advantages that the joint points of a plurality of persons can be accurately identified, and the position positioning precision of the joint points is high. In the HigherHRNet model in the embodiment of the disclosure, a heat map with better precision is generated by combining quadratic interpolation and deconvolution, so that the precision of joint point positioning can be finally higher, and a specific network structure is shown in fig. 2.
Referring to fig. 2, the highherhrnet model may include a stem module, a first-stage network module, a second-stage network module, a third-stage network module, a deconvolution module, and a feature refinement module, which are connected in sequence.
Inputting the rehabilitation action picture into a stem module for preprocessing, extracting a first feature map of the rehabilitation action picture, and reducing the resolution of the first feature map to 1/4 of the original picture, wherein the stem module can comprise two convolution layers, and the convolution kernels are the same. For example, a convolution kernel of 3 × 64 may be used, with a step size of 2 and a padding parameter (padding) of 1 for the convolution operation.
The first level network module may be configured to reduce the width of the first profile to a predetermined value C. Specifically, the first-stage network module may include two residual error units and one convolution layer, wherein each residual error unit is formed by a bottleneck layer (bottleeck) with a width (number of channels) of 64; the convolutional layer may be a 3x3 convolutional layer, reducing the width of the first feature map to C. For example, C may be set to 32.
The second level network module may include one or two resolution blocks, each of which may include two 3x3 convolutional layers, and three residual units.
The third-level network module comprises three resolution modules and three residual error units, wherein each resolution module comprises 2 3 × 3 convolutions, and the product widths of the convolutions are C, 2C and 4C respectively.
And the decoding module takes the feature map from the HigherHrnet and the prediction heat map as input and generates a new feature map with the resolution 2 times larger than that of the input feature map. In the present embodiment, a high-quality and high-resolution feature map can be efficiently generated by deconvolution.
The feature refining module can comprise 4 residual error units to refine the output feature map, and finally obtain and output joint point positions in one frame of rehabilitation action picture.
In addition, the network module connected between the third-stage network module and the deconvolution module in fig. 2 functions to realize aggregation of feature maps, and "1/4" in fig. 2 indicates the resolution of the feature map output by the corresponding module relative to the size of its original image.
According to the steps, the position information of the joint points in each frame of picture can be accurately obtained.
S102, calculating the joint point distance between the continuous frames and the change angle of the limbs according to the joint point position in each frame of picture.
In an alternative embodiment, calculating the joint distance between successive frames from the joint position in each frame of the picture comprises: taking a continuous multi-frame picture as a sample, calculating the joint point distance between frames in the sample according to the following formula, wherein N is the number of frames in the sample, and the skilled person can set the joint point distance by himself/herself, for example, the number of frames is generally not greater than 10 and not less than 5.
Wherein I represents the number of joint points, 13 joint points in total, N represents the number of frames in one sample,
representing the distance from frame to frame at all the joints,
indicating the position of each joint point in the sample nth frame picture,
representing the distance from frame to frame of a particular joint.
Further, calculating the change angle of the limbs according to the positions of the joint points in each frame of picture, comprising: respectively obtaining left and right forearm slopes, left and right big arm slopes, left and right shank slopes and left and right thigh slopes according to joint points connecting four limbs; and calculating the angles of the left and right small arms and the left and right large arms and the angles of the left and right calves and the left and right thighs according to the slopes of the left and right small arms, the slopes of the left and right large arms, the slopes of the left and right calves and the slopes of the left and right thighs.
As the movement of four limbs of the human body is three-dimensional, when the movement angle is required to be obtained, a three-dimensional coordinate system is required to be set firstly, the direction opposite to the human face is taken as the y-axis direction, the direction perpendicular to the y-axis on the ground is taken as the x-axis, and the intersection point from the head to the x-axis and the y-axis is taken as the z-axis. Fig. 3 is a schematic diagram of a three-dimensional coordinate system of a human body.
Specifically, the embodiment of the present disclosure includes 13 joint points, which are the head, the left and right shoulders, the left and right elbows, the left and right wrists, the left and right hips, the left and right knees, and the left and right ankles, respectively. The slope l of the left and right forearm can be obtained according to the joint point connecting the wrist and the elbow1,l2The slope l of the left and right upper arms can be obtained from the joint point connecting the elbow and the shoulder3,l4The left and right calf slopes l can be obtained from the joint point connecting the ankle and the knee5,l6Left and right thigh slopes l can be obtained from a joint point connecting the knee and hip7，l8。
Further, calculating the angle between the left and right forearm and the angle between the left and right calf and the left and right thigh according to the slope of the left and right forearm, the slope of the left and right calf, and the slope of the left and right thigh, comprising:
wherein l1Represents the slope of the left forearm, l2Represents the slope of the right forearm, l3Indicates the slope of the left forearm, l4Represents the slope of the right big arm,/5Representing the slope of the left calf, l6Representing the slope of the right calf, L7Denotes the left thigh slope, l8Represents the slope of the right thigh, θ1Denotes the angle of the left forearm and the left forearm, theta2Representing the angle, θ, of the right forearm and the right forearm3Representing the angle of the left calf to the left thigh, θ4Representing the angle of the right calf to the right thigh.
S103 calculates the moving speed of the joint point from the joint point distance between the successive frames.
In an alternative embodiment, calculating the moving speed of the joint based on the joint distance between successive frames comprises: calculating the moving speed of the joint point according to the time occupied by the N frames of pictures in the sample and the joint point distance between the frames, wherein the specific formula is as follows:
wherein, ViRepresents the ithSpeed of movement of joint points, DiIndicating the distance from frame to frame at the ith joint, N the number of frames in a sample, and fps the number of frames transmitted per second of the picture.
S104, according to the joint point position in each frame of picture, the joint point distance between the continuous frames, the moving speed of the joint point, the change angle of the limbs and the preset characteristic importance parameter, the detected rehabilitation action accuracy is obtained.
In the prior art, rehabilitation motions have hundreds of different motions, and when the motions are different, the motions belong to which class of motions need to be classified first, and then the classification is compared with the class of standard motions to obtain a standard degree, on one hand, the classification of hundreds of classes can cause incorrect classification and wrong guidance, on the other hand, too many classes can cause too long time and cannot meet the real-time requirement, so that the single motion standard cannot adapt to various motions.
Therefore, the embodiment of the present disclosure uses the joint point position in each frame of picture, the joint point distance between consecutive frames, the moving speed of the joint point, and the change angle of the limbs as the motion characteristics, and different motion characteristics may be set in advance to have different motion characteristic importance parameters, for example, the rehabilitation motion of knee surgery requires that the lower leg and the thigh are bent at 90 °, and the motion speed cannot be too slow. Different action characteristic importance parameters can be set according to different rehabilitation actions.
Further, the detected rehabilitation action accuracy is obtained according to the joint point position in each frame of picture, the joint point distance between the continuous frames, the moving speed of the joint point, the change angle of the limbs and the preset characteristic importance parameter, and comprises the following steps:
wherein alpha ise(e-1, 2,3,4) represents the preset characteristic importance parameter of each rehabilitation action, S represents the accuracy of the detected rehabilitation action,
indicating the position of the joint point in the f-th frame picture,
representing the distance, V, of the joint point from frame to frameIRepresenting the speed of movement, theta, of the joint point(i,j)Indicating the varying angle of the limbs.
According to the step, the detection method can adapt to various different actions, the judgment standard can be changed only by presetting some action characteristic importance coefficients, and the adaptability of the method is improved.
Optionally, after the detected rehabilitation action accuracy is obtained, a voice prompt can be sent in real time in a voice broadcasting mode, and the user can adjust the rehabilitation action according to voice prompt information to improve the rehabilitation training effect.
Optionally, after the detected rehabilitation action accuracy is obtained, the rehabilitation action accuracy information can be sent to the doctor end and the user end, so that the doctor can know the training condition of the patient conveniently.
According to the rehabilitation action detection method based on posture recognition provided by the embodiment of the disclosure, not only is the single-frame posture recognition emphasized, but also the rehabilitation action process is extracted, and the detection accuracy is improved. In addition, the embodiment can also adapt to various actions, the judgment standard can be changed only by presetting some action characteristic importance coefficients, and the adaptability of the method is improved.
The embodiment of the present disclosure further provides a rehabilitation motion detection apparatus based on gesture recognition, which is used for executing the rehabilitation motion detection method based on gesture recognition of the above embodiment, as shown in fig. 4, the apparatus includes:
the input module 401 is configured to input the acquired rehabilitation action picture into a preset highherhrnet model to obtain a joint point position in each frame of picture;
a first calculating module 402, configured to calculate joint distances between consecutive frames and change angles of limbs according to the joint positions in each frame of the image;
a second calculating module 403, configured to calculate a moving speed of the joint point according to the joint point distance between consecutive frames;
the detection module 404 is configured to obtain the detected rehabilitation action accuracy according to the joint point position in each frame of the picture, the joint point distance between consecutive frames, the moving speed of the joint point, the change angle of the limbs, and a preset feature importance parameter.
It should be noted that, when the rehabilitation motion detection device based on gesture recognition provided in the above embodiment executes the rehabilitation motion detection method based on gesture recognition, the above division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions. In addition, the rehabilitation motion detection device based on posture recognition provided by the above embodiment and the rehabilitation motion detection method based on posture recognition belong to the same concept, and details of the implementation process are found in the method embodiment and are not described herein again.
The embodiment of the present disclosure further provides an electronic device corresponding to the rehabilitation motion detection method based on gesture recognition provided in the foregoing embodiment, so as to execute the rehabilitation motion detection method based on gesture recognition.
Please refer to fig. 5, which illustrates a schematic diagram of an electronic device according to some embodiments of the present application. As shown in fig. 5, the electronic apparatus includes: the processor 500, the memory 501, the bus 502 and the communication interface 503, wherein the processor 500, the communication interface 503 and the memory 501 are connected through the bus 502; the memory 501 stores a computer program that can be executed on the processor 500, and the processor 500 executes the rehabilitation motion detection method based on gesture recognition provided by any one of the foregoing embodiments of the present application when executing the computer program.
The Memory 501 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 503 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The processor 500 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 500. The Processor 500 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 501, and the processor 500 reads the information in the memory 501, and completes the steps of the method in combination with the hardware thereof.
The electronic device provided by the embodiment of the application and the rehabilitation motion detection method based on gesture recognition provided by the embodiment of the application have the same inventive concept and have the same beneficial effects as the method adopted, operated or realized by the electronic device.
Referring to fig. 6, the computer readable storage medium is an optical disc 600, on which a computer program (i.e., a program product) is stored, and when the computer program is executed by a processor, the computer program performs the rehabilitation motion detection method based on gesture recognition provided in any of the embodiments.
It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.
The computer-readable storage medium provided by the above-mentioned embodiment of the present application and the rehabilitation motion detection method based on gesture recognition provided by the embodiment of the present application have the same inventive concept, and have the same beneficial effects as the method adopted, run or implemented by the application program stored in the computer-readable storage medium.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only show some embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A rehabilitation motion detection method based on posture recognition is characterized by comprising the following steps:
inputting the acquired rehabilitation action picture into a preset HigherHRNet model to obtain the joint point position in each frame of picture;
calculating the joint point distance between the continuous frames and the change angle of the limbs according to the joint point position in each frame of picture;
calculating the moving speed of the joint point according to the joint point distance between the continuous frames;
and obtaining the accuracy of the detected rehabilitation action according to the joint point position in each frame of picture, the joint point distance between the continuous frames, the moving speed of the joint point, the change angle of the limbs and the preset characteristic importance parameter.
2. The method according to claim 1, wherein before inputting the acquired rehabilitation motion picture into the preset highherhrnet model, the method further comprises:
identifying one or more rehabilitation training users by adopting a face identification method;
and when the rehabilitation training user is successfully identified and positioned, shooting a video image of the user for rehabilitation training.
3. The method of claim 1, wherein calculating the joint distance between successive frames according to the joint position in each frame of the picture comprises:
taking continuous multi-frame pictures as a sample, calculating the joint point distance between frames in the sample according to the following formula,
wherein I represents the number of joint points, 13 joint points in total, N represents the number of frames in one sample,
representing the distance from frame to frame at all the joints,
indicating the position of each joint point in the sample nth frame picture,
representing the distance from frame to frame of a particular joint.
4. The method of claim 1, wherein calculating the angle of change of the limbs according to the position of the joint point in each frame of the picture comprises:
respectively obtaining left and right forearm slopes, left and right big arm slopes, left and right shank slopes and left and right thigh slopes according to joint points connecting four limbs;
and calculating the angles between the left and right small arms and the left and right large arms and the angles between the left and right shanks and the left and right thighs according to the slopes of the left and right small arms, the slopes of the left and right large arms, the slopes of the left and right shanks and the slopes of the left and right thighs.
5. The method of claim 4, wherein calculating the left and right forearm to left and right forearm and calf to left and right thigh angles from the left and right forearm slope, left and right calf slope, and left and right thigh slope comprises:
wherein l1Represents the slope of the left forearm, l2Represents the slope of the right forearm, l3Indicates the slope of the left forearm, l4Represents the slope of the right big arm,/5Representing the slope of the left calf, l6Representing the slope of the right calf, l7Denotes the left thigh slope, l8Represents the slope of the right thigh, θ1Denotes the angle of the left forearm and the left forearm, theta2Representing the angle, θ, of the right forearm and the right forearm3Representing the angle of the left calf to the left thigh, θ4Representing the angle of the right calf to the right thigh.
6. The method of claim 3, wherein calculating the velocity of movement of the articulation point based on the articulation point distance between the successive frames comprises:
calculating the moving speed of the joint point according to the time occupied by the N frames of pictures in the sample and the joint point distance between the frames, wherein the specific formula is as follows:
wherein, ViRepresents the moving speed of the i-th joint point, DiIndicating the distance from frame to frame at the ith joint, N the number of frames in a sample, and fps the number of frames transmitted per second of the picture.
7. The method of claim 1, wherein obtaining the detected rehabilitation action accuracy according to the joint point position in each frame of picture, the joint point distance between the consecutive frames, the moving speed of the joint point, the change angle of the limbs and the preset feature importance parameter comprises:
wherein，αe(e-1, 2,3,4) represents the preset characteristic importance parameter of each rehabilitation action, S represents the accuracy of the detected rehabilitation action,
indicating the position of the joint point in the f-th frame picture,
representing the distance, V, of the joint point from frame to frameIRepresenting the speed of movement, theta, of the joint point(i,j)Indicating the varying angle of the limbs.
8. A rehabilitation motion detection device based on posture recognition is characterized by comprising:
the input module is used for inputting the acquired rehabilitation action picture into a preset HigherHRNet model to obtain the joint point position in each frame of picture;
the first calculation module is used for calculating the joint point distance between the continuous frames and the change angle of the limbs according to the joint point position in each frame of picture;
the second calculation module is used for calculating the moving speed of the joint point according to the joint point distance between the continuous frames;
and the detection module is used for obtaining the detected rehabilitation action accuracy according to the joint point position in each frame of picture, the joint point distance between the continuous frames, the moving speed of the joint point, the change angle of the limbs and the preset characteristic importance parameter.
9. A rehabilitation motion detection device based on gesture recognition, characterized by comprising a processor and a memory storing program instructions, the processor being configured to perform the rehabilitation motion detection method based on gesture recognition according to any one of claims 1 to 7 when executing the program instructions.
10. A computer readable medium having computer readable instructions stored thereon which are executable by a processor to implement a method of gesture recognition based rehab motion detection as claimed in any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110474993.6A CN113297919A (en) | 2021-04-29 | 2021-04-29 | Rehabilitation action detection method, device, equipment and medium based on posture recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110474993.6A CN113297919A (en) | 2021-04-29 | 2021-04-29 | Rehabilitation action detection method, device, equipment and medium based on posture recognition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113297919A true CN113297919A (en) | 2021-08-24 |
Family
ID=77320614
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110474993.6A Withdrawn CN113297919A (en) | 2021-04-29 | 2021-04-29 | Rehabilitation action detection method, device, equipment and medium based on posture recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113297919A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114052726A (en) * | 2021-11-25 | 2022-02-18 | 湖南中科助英智能科技研究院有限公司 | Thermal infrared human body gait recognition method and device in dark environment |
| CN117409930A (en) * | 2023-12-13 | 2024-01-16 | 江西为易科技有限公司 | Medical rehabilitation data processing method and system based on AI technology |
-
2021
- 2021-04-29 CN CN202110474993.6A patent/CN113297919A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114052726A (en) * | 2021-11-25 | 2022-02-18 | 湖南中科助英智能科技研究院有限公司 | Thermal infrared human body gait recognition method and device in dark environment |
| CN117409930A (en) * | 2023-12-13 | 2024-01-16 | 江西为易科技有限公司 | Medical rehabilitation data processing method and system based on AI technology |
| CN117409930B (en) * | 2023-12-13 | 2024-02-13 | 江西为易科技有限公司 | Medical rehabilitation data processing method and system based on AI technology |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110969114A (en) | Human body action function detection system, detection method and detector | |
| US20210315486A1 (en) | System and Method for Automatic Evaluation of Gait Using Single or Multi-Camera Recordings | |
| CN107909060A (en) | Gymnasium body-building action identification method and device based on deep learning | |
| JP2021064367A (en) | Motion recognition method, motion recognition device, and electronic device | |
| CN111476097A (en) | Human body posture assessment method and device, computer equipment and storage medium | |
| Chaudhari et al. | Yog-guru: Real-time yoga pose correction system using deep learning methods | |
| CN113297919A (en) | Rehabilitation action detection method, device, equipment and medium based on posture recognition | |
| CN113255522B (en) | Personalized motion attitude estimation and analysis method and system based on time consistency | |
| WO2021064963A1 (en) | Exercise recognition method, exercise recognition program, and information processing device | |
| CN112568898A (en) | Method, device and equipment for automatically evaluating injury risk and correcting motion of human body motion based on visual image | |
| CN114022645A (en) | Action driving method, device, equipment and storage medium of virtual teacher system | |
| CN111368787A (en) | Video processing method and device, equipment and computer readable storage medium | |
| US20220222975A1 (en) | Motion recognition method, non-transitory computer-readable recording medium and information processing apparatus | |
| Solongontuya et al. | Novel side pose classification model of stretching gestures using three-layer LSTM | |
| CN115035037A (en) | Limb rehabilitation training method and system based on image processing and multi-feature fusion | |
| CN114973048A (en) | Method and device for correcting rehabilitation action, electronic equipment and readable medium | |
| Kishore et al. | Smart yoga instructor for guiding and correcting yoga postures in real time | |
| CN112861606A (en) | Virtual reality hand motion recognition and training method based on skeleton animation tracking | |
| CN112102451A (en) | Common camera-based wearable virtual live broadcast method and equipment | |
| CN115346640B (en) | Intelligent monitoring method and system for closed-loop feedback of functional rehabilitation training | |
| CN116343325A (en) | Intelligent auxiliary system for household body building | |
| CN113842622B (en) | Motion teaching method, device, system, electronic equipment and storage medium | |
| KR20230009676A (en) | 3d human pose estimation apparatus and method thereof | |
| Rodrigues et al. | Supervised classification of motor-rehabilitation body movements with rgb cameras and pose tracking data | |
| Zeng et al. | Machine learning based automatic sport event detection and counting |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210824 |