CN117953591A

CN117953591A - Intelligent limb rehabilitation assisting method and device

Info

Publication number: CN117953591A
Application number: CN202410354728.8A
Authority: CN
Inventors: 汪杰; 李宏增; 张华�
Original assignee: Air Force Medical University of PLA
Current assignee: Air Force Medical University of PLA
Priority date: 2024-03-27
Filing date: 2024-03-27
Publication date: 2024-04-30
Anticipated expiration: 2044-03-27
Also published as: CN117953591B

Abstract

The invention relates to the technical field of computer vision, in particular to an intelligent limb rehabilitation assisting method and equipment. The method comprises the following steps: acquiring limb movement data through an image sensor to obtain limb movement image data; carrying out gesture estimation on the limb motion image data to obtain gesture estimation data, and carrying out key point detection on the gesture estimation data to obtain limb motion key point detection data; performing motion recognition on the limb movement key point detection data to obtain limb movement motion recognition data; and generating action parameters of the limb movement key point detection data according to the limb movement identification data to obtain limb action parameter data so as to perform intelligent limb rehabilitation auxiliary operation. The invention can provide accurate and reliable intelligent limb rehabilitation auxiliary parameters so as to reduce the dependence on manpower and reduce the manpower cost and time cost.

Description

Intelligent limb rehabilitation assisting method and device

Technical Field

The invention relates to the technical field of computer vision, in particular to an intelligent limb rehabilitation assisting method and equipment.

Background

Conventional limb rehabilitation auxiliary methods often adopt rehabilitation technology or auxiliary equipment to perform limb rehabilitation auxiliary operation, in the process, the limb rehabilitation auxiliary operation is often dependent on manual auxiliary operation or simple equipment assistance, the limb rehabilitation auxiliary method is often dependent on experience and skills of a rehabilitation engineer, but human resources are limited, and all requirements cannot be met. The computer vision is a science for researching how to make a machine "see", and more specifically, a camera and a computer are used to replace human eyes to identify, track and measure targets, and the like, and further, graphic processing is performed, so that the computer is processed into images which are more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can obtain "information" from images or multidimensional data. The information referred to herein refers to Shannon-defined information that may be used to assist in making a "decision". Because perception can be seen as the extraction of information from sensory signals, computer vision can also be seen as science of how to "perceive" an artificial system from images or multi-dimensional data. How to combine the limb rehabilitation assistance method with the computer vision technology becomes a problem.

Disclosure of Invention

The invention provides an intelligent limb rehabilitation assisting method and equipment for solving the technical problems, and aims to solve at least one of the technical problems.

The application provides an intelligent limb rehabilitation assisting method, which comprises the following steps:

step S1: acquiring limb movement data through an image sensor to obtain limb movement image data;

Step S2, including:

step S21: carrying out gesture estimation on the limb moving image data to obtain gesture estimation data;

step S22: detecting local key points of the gesture estimation data to obtain first limb movement key point detection data;

step S23: performing global key point detection on the gesture estimation data to obtain second limb movement key point detection data;

Step S3: performing motion recognition on limb movement key point detection data to obtain limb movement motion recognition data, wherein the limb movement key point detection data comprise first limb movement key point detection data and second limb movement key point detection data;

Step S4, including:

performing action classification mapping according to the limb movement identification data to obtain action classification mapping data;

Performing action time sequence division according to limb movement key point detection data and action classification mapping data to obtain action time sequence division data;

Extracting action characteristics according to the action time sequence dividing data to obtain action characteristic data, wherein the action characteristic data comprises action duration time data, action speed data, action acceleration data and action angle change data;

And generating action parameters according to the action classification mapping data, the action time sequence division data and the action characteristic data to obtain limb action parameter data so as to perform intelligent limb rehabilitation auxiliary operation.

According to the invention, the image sensor is used for acquiring limb movement data, so that invasive detection or operation on a user is avoided, and the comfort and acceptance of the user are improved. The data collected by the image sensor can be processed and analyzed in real time, so that the rehabilitation auxiliary operation can be adjusted and fed back in time according to the actual situation of the user. By carrying out gesture estimation and key point detection on the limb moving image data, the motion state and key point information of the user can be accurately acquired, so that parameter formulation generation and monitoring which are more fit with the actual situation of the user are realized. According to the method, the limb movement key point detection data are analyzed by using the action recognition technology, so that the movement action of the user can be automatically recognized, and manual intervention is not needed. And corresponding action parameters are automatically generated according to the identification result, so that the operation flow of rehabilitation auxiliary operation is further simplified. Compared with the traditional rehabilitation auxiliary method, the method does not need expensive equipment or complicated operation, has lower cost, is easy to implement and popularize, is beneficial to improving the coverage rate and popularity of rehabilitation service and reduces the dependence on manpower operation.

According to the invention, through the posture estimation in the step S21, the whole posture information of the limb can be obtained, so that the whole movement state of the limb can be better understood, and an important reference is provided for the subsequent key point detection. Step S22 performs local key point detection on the posture estimation data, so that local key points of the limb can be finely identified, and the local key points can represent important parts and motion characteristics of the limb, thereby being helpful for more accurately analyzing and understanding details of the limb motion. Step S23 carries out global key point detection on the gesture estimation data, can capture key point information of the limbs on the whole, and the global key points can reflect the whole structure and the motion state of the limbs, thereby being beneficial to more comprehensively analyzing and understanding the whole motion characteristics of the limbs. The invention can improve the accuracy and stability of the key point detection, is beneficial to more accurately identifying and positioning the key points of limbs, and provides reliable data support for subsequent action identification and parameter generation.

According to the invention, by performing action classification mapping according to the limb movement identification data, the movement data can be mapped into specific action categories, so that classification and identification of different actions are realized, the system is facilitated to better understand the movement behaviors of the user, and effective data support is provided for subsequent rehabilitation assistance. According to the limb movement key point detection data and the movement classification mapping data, the movement time sequence is divided, the whole movement process can be decomposed into different time periods or stages, finer and more accurate analysis of the movement process is facilitated, and a more reliable basis is provided for subsequent feature extraction and parameter generation. The motion characteristic extraction is carried out according to the motion time sequence division data, information about motion such as duration, speed, acceleration, angle change and the like can be obtained from multiple aspects, the characteristic data can reflect various aspects of the motion, and more reference bases are provided for subsequent parameter generation and rehabilitation assistance. According to the action classification mapping data, the action time sequence division data and the action characteristic data, action parameters are generated, and the type, time sequence and characteristic information of actions can be comprehensively considered, so that more accurate limb action parameter data are obtained.

Preferably, step S1 is specifically:

Step S11: acquiring first limb movement data by an image sensor according to preset first threshold angle data to obtain first limb movement image data;

step S12: acquiring second limb movement data by using an image sensor according to preset second threshold angle data to obtain second limb movement image data;

Step S13: and performing image stitching on the first limb moving image data and the second limb moving image data to obtain limb moving image data.

According to the invention, the image sensor is used, so that limb movement data of a user can be accurately acquired in real time. The first threshold angle data and the second threshold angle data can be adjusted according to specific situations of users so as to adapt to the movement capacities of different users, and the acquired data has higher operability and accuracy. By setting different threshold angles, limb movement data may be collected from different angles, such that the user's movement pose is observed and analyzed from multiple perspectives, more accurately assessing the state and characteristics of limb movement. And the first and second limb moving image data are spliced, so that the information richness of the image can be increased, the spliced image can present a more complete and comprehensive limb moving state, and the accuracy and stability of subsequent gesture estimation and key point detection can be improved.

Preferably, step S13 is specifically:

step S131: performing first limb angle detection on the first limb moving image data according to preset first threshold angle data to obtain first limb angle detection data;

Step S132: performing second limb angle detection on second limb moving image data according to preset second threshold angle data to obtain second limb angle detection data;

Step S133: performing image correction on the first limb moving image data according to the first limb angle detection data to obtain first limb moving image correction data, and performing image correction on the second limb moving image data according to the second limb angle detection data to obtain second limb moving image correction data;

Step S134: extracting characteristic points of the first limb moving image correction data and the second limb moving image correction data to obtain first limb moving image characteristic point data and second limb moving image characteristic point data;

Step S135: performing feature matching on the first limb moving image feature point data and the second limb moving image feature point data to obtain limb moving image feature matching data;

Step S136: performing perspective transformation on the first limb moving image data and the second limb moving image data according to the limb moving image feature matching data to obtain limb moving image perspective transformation data;

step S137: performing image fusion on the first limb moving image perspective transformation data and the second limb moving image perspective transformation data to obtain limb moving image fusion data;

step S138: and performing edge restoration on the limb moving image fusion data to obtain limb moving image data.

In the present invention, steps S131 and S132 can accurately detect the angle information of the first and second limbs by using the angle detection technique. Through image correction (step S133), the acquired image data can be corrected, errors caused by angular deviation are reduced, and accuracy of subsequent processing is improved. Steps S134 and S135 utilize feature point extraction and matching techniques, which can extract key feature points from the corrected image, and correlate through feature matching, thereby helping to accurately capture key information of limb movements and maintaining consistency. The steps S136 and S137 utilize perspective transformation and image fusion technology, and can synthesize the two corrected images to obtain more comprehensive and complete limb moving image data, and improve the information richness and quality of the images, so that limb movement can be analyzed more accurately. Step S138 is performed on the image to repair the edge problem caused by the processes of correction, transformation and the like in the image, so that the image is clearer and more accurate, and the effect of subsequent processing is improved.

Preferably, step S134 is specifically:

step S101: clustering calculation is carried out on the first limb moving image correction data and the second limb moving image correction data to obtain first limb moving image clustering data and second limb moving image clustering data;

Step S102: extracting a clustering center of the first limb moving image clustering data and the second limb moving image clustering data to obtain first image clustering center data and second image clustering center data;

Step S103: cluster radius extraction is carried out on the first limb moving image cluster data and the second limb moving image cluster data to obtain first cluster radius data and second cluster radius data;

Step S104: weighting calculation is carried out on the first cluster radius data by using the first limb moving image clustering data to obtain first cluster radius weighting data, and weighting calculation is carried out on the second cluster radius data by using the second limb moving image clustering data to obtain second cluster radius weighting data;

Step S105: performing field pixel selection on the first image clustering center data by using the first cluster radius weighting data to obtain first limb moving image field pixel data, and performing field pixel selection on the second image clustering center data by using the second cluster radius weighting data to obtain second limb moving image field pixel data;

Step S106: and carrying out dynamic pixel gray level judgment on the first limb moving image field pixel data and the second limb moving image field pixel data to obtain first limb moving image feature point data and second limb moving image feature point data.

In the step S134, the image data can be processed and analyzed in a refined way, which is helpful for capturing the characteristic information in the image more accurately and improving the precision and reliability of the subsequent processing. In the steps S104 and S105, the weighting processing and the pixel selection can be carried out on the image data according to the cluster data and the cluster radius data by utilizing the weighting calculation and the field pixel selection technology, so that the key information in the image is highlighted, and the extraction accuracy and the extraction stability of the feature points are improved. In step S106, the gray level of the dynamic pixel is determined, so that the feature points can be screened and determined according to the gray level information of the image, the sensitivity to noise and interference is reduced, and the recognition and extraction effects on the feature points are improved. The method improves the extraction accuracy and stability of the feature points of the limb moving image by analyzing the image data in a multi-layer and multi-angle manner. The traditional feature point extraction is usually carried out by adopting feature point acquisition based on a fixed threshold, and the invention carries out adaptive adjustment through the self-characteristics of the image, thereby improving the extraction effect on the feature point and the quality of the image data, and overcoming the problems of insufficient precision and unstable extraction effect in the prior art.

Preferably, step S106 is specifically:

acquiring illumination angle data and illumination intensity data through an illumination sensor;

Performing angle calculation on preset first threshold angle data and preset second threshold angle data according to the illumination angle data to obtain first shooting illumination angle data and second shooting illumination angle data;

Gray threshold generation is carried out according to the first shooting illumination angle data and the illumination intensity data to obtain first gray threshold data, gray threshold generation is carried out according to the second shooting illumination angle data and the illumination intensity data to obtain second gray threshold data;

And performing pixel gray judgment on the first limb moving image field pixel data through the first gray threshold data to obtain first limb moving image feature point data, and performing pixel gray judgment on the second limb moving image field pixel data through the second gray threshold data to obtain second limb moving image feature point data.

According to the invention, the influence of the illumination direction on the image can be considered by utilizing the illumination angle data acquired by the illumination sensor. The gray threshold value is calculated according to the illumination angle data, so that the self-adaptive adjustment of the image according to different illumination conditions is facilitated, and the stability and the robustness of image processing are improved. Step S106 dynamically generates a gray threshold according to the illumination angle data and the illumination intensity data, can adjust the gray threshold according to the real-time illumination condition, avoids the problem of unstable image processing or misjudgment caused by different illumination conditions, and improves the extraction accuracy of the image feature points. By carrying out gray judgment on the pixels of the image according to the gray threshold value generated by the illumination angle data, the self-adaptive pixel gray judgment can be realized, the characteristic points of the image can be accurately extracted under different illumination conditions, and the problem that the characteristic extraction effect is unstable when the illumination change is large in the prior art is solved. Step S106 calculates a gray threshold value and performs pixel gray judgment according to the illumination angle data, so as to improve the robustness of image processing. For images under different illumination conditions, the characteristic points can be extracted more accurately, so that the effect and reliability of subsequent processing are improved.

Preferably, step S22 is specifically:

according to the gesture estimation data, branch network detection selection is carried out to obtain branch network detection data;

Performing specific position key point detection on the gesture estimation data according to the branch network detection data to obtain specific position key point detection data;

And carrying out feature fusion on the specific position key point detection data corresponding to the different branch network detection data to obtain first limb movement key point detection data.

According to the invention, by introducing a plurality of branch networks, each branch network is responsible for detecting the key point of a specific part, the movement information of different parts in limb movement can be more accurately captured, so that the accuracy and the comprehensiveness of detection are improved. On the basis of branch network detection selection, key point detection is respectively carried out on each part, so that the detection result is finer and more specific, and the method is helpful for more accurately analyzing and evaluating the motion state of the user in the process. The feature fusion is carried out on the specific position key point data detected by different branch networks, and the movement information of different positions can be considered, so that more accurate limb movement key point detection data is obtained, and the understanding and analysis capability of the rehabilitation auxiliary system on the movement state of the user are improved.

Preferably, step S3 is specifically:

Extracting features of the limb movement key point detection data to obtain limb movement key point feature data;

feature selection is carried out on the feature data of the key points of the limb movement, so that feature selection data of the key points are obtained;

and performing action recognition on the key point feature selection data to obtain limb movement action recognition data.

According to the invention, through the feature extraction in the step S3, key point feature data representing the limb movement feature can be extracted from limb movement key point detection data, so that the complex limb movement can be converted into more specific feature data which is easier to analyze. The feature selection process in the step S3 can select the most representative and distinguishing features from the extracted key point feature data, so that feature dimensions are reduced, recognition accuracy is improved, complexity of data processing is reduced, important information on limb movement is reserved, and action recognition effect is improved. The data subjected to feature extraction and selection can better reflect the feature of limb movement, so that the motion recognition is more accurate. By selecting representative characteristics and combining an appropriate recognition algorithm, different limb movement actions can be recognized more reliably, and more accurate data support is provided for intelligent limb rehabilitation assistance. Through feature selection, the calculation amount of the recognition algorithm can be reduced, and the calculation efficiency is improved. The carefully selected features can better reflect the essential features of the limb movement, thereby reducing unnecessary calculation cost and accelerating the speed of motion recognition. As the characteristic dimension and the calculated amount are reduced, the system can process limb movement data more rapidly, and the instantaneity is improved.

Preferably, the present application also provides an intelligent limb rehabilitation assistance device for performing the intelligent limb rehabilitation assistance method as described above, the intelligent limb rehabilitation assistance device comprising:

The limb movement data acquisition module is used for acquiring limb movement data through the image sensor to obtain limb movement image data;

The limb movement key point detection module is used for carrying out gesture estimation on limb movement image data to obtain gesture estimation data, and carrying out key point detection on the gesture estimation data to obtain limb movement key point detection data;

The motion recognition module is used for performing motion recognition on the limb motion key point detection data to obtain limb motion recognition data;

And the intelligent limb rehabilitation auxiliary operation module is used for generating action parameters of limb movement key point detection data according to limb movement action identification data to obtain limb action parameter data so as to perform intelligent limb rehabilitation auxiliary operation.

The invention has the beneficial effects that: the image sensor is utilized to collect limb movement data, so that invasive detection or operation of a user is avoided, and comfort and acceptance of the user are improved. The data collected by the image sensor can be processed and analyzed in real time, so that the rehabilitation auxiliary operation can be adjusted and fed back in time according to the actual situation of the user. By carrying out gesture estimation and key point detection on the limb moving image data, the motion state and key point information of the user can be accurately acquired, so that parameter formulation generation and monitoring which are more fit with the actual situation of the user are realized. According to the method, the limb movement key point detection data are analyzed by using the action recognition technology, so that the movement action of the user can be automatically recognized, and manual intervention is not needed. And corresponding action parameters are automatically generated according to the identification result, so that the operation flow of rehabilitation auxiliary operation is further simplified. Compared with the traditional rehabilitation auxiliary method, the method does not need expensive equipment or complicated operation, has lower cost, is easy to implement and popularize, and is beneficial to improving the coverage rate and popularity of the rehabilitation auxiliary service.

Drawings

Other features, objects and advantages of the application will become more apparent upon reading of the detailed description of a non-limiting implementation, made with reference to the accompanying drawings in which:

FIG. 1 shows a flow chart of steps of a smart limb rehabilitation assistance method of an embodiment;

FIG. 2 is a flow chart illustrating the steps of a method of acquiring limb movement data in accordance with one embodiment;

FIG. 3 is a flowchart showing steps of a method for stitching a limb moving image in accordance with one embodiment;

fig. 4 is a flowchart showing steps of a feature point extraction method of a limb moving image according to an embodiment.

Detailed Description

The following is a clear and complete description of the technical method of the present patent in conjunction with the accompanying drawings, and it is evident that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, are intended to fall within the scope of the present invention.

Furthermore, the drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. The functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor methods and/or microcontroller methods.

It will be understood that, although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.

Referring to fig. 1 to 4, the present application provides an intelligent limb rehabilitation assistance method, which includes the following steps:

specifically, an RGB camera or a depth camera is used for shooting a user, and image data of limb movement is obtained. For example, a camera may be mounted on a wall or ceiling of a rehabilitation training room to capture limb movements of a user while performing rehabilitation training.

Step S2: carrying out gesture estimation on the limb motion image data to obtain gesture estimation data, and carrying out key point detection on the gesture estimation data to obtain limb motion key point detection data;

Specifically, the posture estimation is performed on the limb moving image using a deep learning model such as OpenPose or PoseNet, thereby obtaining the posture estimation data of the user. Then, key points such as joint positions of elbows, wrists, knees and the like are detected from the gesture estimation data, and key point detection data of limb movements are obtained.

Specifically, the posture estimation is performed on the limb moving image by using OpenPose to obtain posture estimation data. The pose estimation data contains coordinates and confidence of each node of interest, with a total of 18 key points. And extracting key point information from the posture estimation data, wherein the key point information comprises joint positions such as shoulders, elbows, wrists, hips, knees and ankles. Position information of 6 key points such as shoulders, elbows, wrists, hips, knees and ankles is extracted from the posture estimation data.

Step S3: performing motion recognition on the limb movement key point detection data to obtain limb movement motion recognition data;

specifically, the limb movement key point detection data is processed and analyzed by using a machine learning or deep learning algorithm, so that specific actions executed by a user are identified. For example, the sequence data may be classified using a Recurrent Neural Network (RNN) or a Convolutional Neural Network (CNN) to identify actions performed by the user, such as lifting a hand, bending a knee, and the like.

Specifically, for each frame of image, it is first necessary to convert the key point data into feature vectors including relative positions between joints, angle information, speed information, and the like. And combining successive frames of data in a time series into a sequence to capture dynamic characteristics of the motion. The extracted features are trained using a deep learning model, such as a Recurrent Neural Network (RNN) or Convolutional Neural Network (CNN). The training data set comprises marked limb movement key point detection data and corresponding action labels, such as lifting hands and bending down. And inputting the new limb movement key point detection data into a trained model for prediction to obtain corresponding action labels on each frame of image, wherein the finally obtained limb movement action identification data comprises actions executed on each frame of image and corresponding time information.

Step S4: and generating action parameters of the limb movement key point detection data according to the limb movement identification data to obtain limb action parameter data so as to perform intelligent limb rehabilitation auxiliary operation.

Specifically, according to the motion recognition data and the key point detection data, various parameters of the limb motion of the user, such as angle, speed, acceleration and the like, are calculated so as to reflect the motion state of the user and the quality of executing the motion. For example, limb motion parameter data can be generated through the change of joint angles and the analysis of motion trajectories, and personalized rehabilitation assistance guidance and feedback can be provided for a user according to the parameter data.

Specifically, the user is photographed by an RGB camera or a depth camera, and image data of the limb movement is captured. Let the resolution of the image be 1920x1080 pixels. Pose estimation and keypoint detection are performed using a deep learning model, such as OpenPose. And processing according to the image data to obtain attitude estimation data and key point detection data. It is assumed that the coordinate position and posture data detected by the image data of each limb movement have 100 key points. The coordinate position of each key point is represented by an (x, y) coordinate, and the posture data is represented by an angle value. The keypoint detection data is motion identified and the sequence of keypoints is classified using a Recurrent Neural Network (RNN). For example, the position coordinates and the gesture data of each key point may be input as a sequence, and the result of classification of the motion may be obtained through RNN model processing. Assume a total of 10 different action classifications. According to the motion recognition data and the key point detection data, various parameters of limb motion are calculated, for example, the change condition of joint angle, the motion speed, the acceleration and the like can be calculated. It is assumed that for each joint, the device calculates its angular change, velocity and acceleration values. Taking the elbow joint as an example, assume that the angle of the elbow joint varies from 0 to 180 degrees, the velocity ranges from 0 to 100 pixels/second, and the acceleration ranges from-10 to 10 pixels/second.

According to the invention, the image sensor is used for acquiring limb movement data, so that invasive detection or operation on a user is avoided, and the comfort and acceptance of the user are improved. The data collected by the image sensor can be processed and analyzed in real time, so that the rehabilitation auxiliary operation can be adjusted and fed back in time according to the actual situation of the user. By carrying out gesture estimation and key point detection on the limb moving image data, the motion state and key point information of the user can be accurately acquired, so that parameter formulation generation and monitoring which are more fit with the actual situation of the user are realized. According to the method, the limb movement key point detection data are analyzed by using the action recognition technology, so that the movement action of the user can be automatically recognized, and manual intervention is not needed. And corresponding action parameters are automatically generated according to the identification result, so that the operation flow of rehabilitation auxiliary operation is further simplified. Compared with the traditional rehabilitation auxiliary method, the method does not need expensive equipment or complicated operation, has lower cost, is easy to implement and popularize, and is beneficial to improving the coverage rate and popularity of rehabilitation services.

Preferably, step S1 is specifically:

Specifically, an RGB camera or a depth camera is used for shooting a user, a preset first threshold angle is set, for example, the maximum extension angle of an arm is set, then the arm movement is monitored in the acquisition process, and when the arm reaches the preset angle, the frame of image data is recorded as first limb moving image data.

Specifically, the user is photographed through the RGB camera, and a preset first threshold angle is set to 120 degrees, that is, the maximum extension angle of the arm. In the acquisition process, the system monitors the movement condition of the arm in real time, and when the stretching angle of the arm reaches a preset 120 degrees, the frame of image data is recorded as first limb moving image data. When a user performs rehabilitation training, the system can continuously capture images of the arms and calculate the stretching angles of the arms in real time through an image processing algorithm. When the angle of the arm reaches or exceeds 120 degrees, the system automatically saves the frame image data as first limb moving image data.

Specifically, in another case, the preset first threshold angle data is an angle threshold set at the time of acquiring limb movement data, for determining a photographing angle of the image sensor, for example, if the set first threshold angle is 60 degrees, the image sensor will start acquiring limb movement data at the time of 60 degrees.

Specifically, the RGB camera or the depth camera is also used for shooting a user, a preset second threshold angle is set, for example, the maximum bending angle of the arm is set, then the arm movement is monitored in the acquisition process, and when the arm reaches the preset angle, the frame of image data is recorded as second limb moving image data.

Specifically, the user is photographed through the RGB camera as well, and the preset second threshold angle is set to be 60 degrees, namely, the maximum bending angle of the arm. In the acquisition process, the system monitors the movement condition of the arm in real time, and when the bending angle of the arm reaches a preset 60 degrees, the frame of image data is recorded as second limb moving image data. When a user performs rehabilitation training, the system can continuously capture images of the arm and calculate the bending angle of the arm in real time. When the angle of the arm reaches or exceeds 60 degrees, the system automatically saves the frame image data as second limb moving image data.

Specifically, in another case, the preset second threshold angle data is an angle threshold set at the time of acquiring limb movement data, for determining a photographing angle of the image sensor, for example, if the set second threshold angle is 120 degrees, the image sensor will start acquiring limb movement data at the time of photographing angle is 120 degrees.

Specifically, the first limb moving image data and the second limb moving image data are spliced, and image processing software or programming can be adopted to realize image splicing. For example, by superimposing or stitching the two images in a specific manner, such as according to known shooting angles and camera parameters (preset first threshold angle data and preset second threshold angle data), a perspective transformation matrix may be calculated to align the two images in space, the perspective transformation may adjust the geometry and spatial position of the images so that the two images are visually aligned, cropping or filling the adjusted images to ensure that they have the same size and boundary, or using a feature point matching algorithm, including SIFT (scale invariant feature transform) or SURF (accelerated robust feature) to detect and match feature points in the two images, then performing a position adjustment on the images according to the matching results, fusing the first limb motion image and the second limb motion image (image fusion based on a hybrid model, pixel level blending or gradient blending), enabling smooth transition and merging of the two images together, and adjusting the position of the image edges using a gradient-based edge alignment or edge filling technique so that the two images are stitched at the same edge, so that the two images are continuously stitched at the edges, containing the final motion data of the image is the final image processing step, which is the motion data of the image processing.

Preferably, step S13 is specifically:

specifically, image deformation, such as affine transformation or perspective transformation, is performed on the first limb moving image data according to preset first threshold angle data, then the contour or key points of the limb are extracted by using an edge detection algorithm or a feature point detection algorithm, and the angle of the limb is calculated according to the information.

Specifically, image deformation, such as affine transformation or perspective transformation, is performed on the second limb moving image data according to the preset second threshold angle data, and then angle detection is performed on the second limb image by a method similar to step S131 to acquire angle detection data of the second limb.

Specifically, the first limb moving image data and the second limb moving image data are corrected based on the first limb angle detection data and the second limb angle detection data. For example, the image may be rotated, scaled, or cropped based on the detected angle information to bring the position and scale of the limb in the image to the desired level.

Specifically, for the first limb moving image correction data and the second limb moving image correction data, a feature point extraction algorithm such as SIFT (scale invariant feature transform) or SURF (speeded up robust feature) or the like is used to extract key feature points from the image. These feature points are typically significant locations of corner points, edges, etc. in the image.

Specifically, the feature point data of the first limb moving image and the feature point data of the second limb moving image are subjected to feature matching, and a matching algorithm (such as a nearest neighbor-based method) is used for matching the feature points in the first image with the feature points in the second image, so that a corresponding relation between the feature points is established.

Specifically, based on the feature matching data, the first limb motion image data and the second limb motion image data are corrected and aligned using a perspective transformation (PERSPECTIVE TRANSFORMATION) technique. Perspective transformation can correct for distortions and aberrations in the images so that the two images are aligned and kept consistent in space.

specifically, the first limb moving image data and the second limb moving image data which are subjected to perspective transformation processing are subjected to image fusion, and the image fusion is realized through technologies such as image superposition, mixing or weighted average so as to retain important information of the two images and eliminate overlapping and inconsistency.

Specifically, edge restoration is performed on the image fusion data. The purpose of edge restoration is to fill up missing parts in the image or to restore discontinuities at edges to ensure the integrity and continuity of the image, by interpolation algorithms or pixel filling based on edge information, etc.

Preferably, step S134 is specifically:

Specifically, the first limb moving image correction data and the second limb moving image correction data are subjected to clustering calculation using a clustering algorithm such as K-means clustering or hierarchical clustering. Each cluster represents a set of similar pixels in the image, thereby obtaining first limb moving image cluster data and second limb moving image cluster data.

K-means clustering calculation is respectively carried out on the first limb moving image correction data and the second limb moving image correction data. The obtained first limb moving image clustering data is assumed to contain 5 clusters, and the pixel points of each cluster are respectively: cluster 1: contains 30 pixels, cluster 2: contains 25 pixels, cluster 3: containing 20 pixels, cluster 4: contains 15 pixels, cluster 5: the clustering condition of the clustering data of the second limb moving image is similar and also comprises 5 clusters, and the number of the pixels of each cluster is different.

Specifically, representative pixel points are selected from each cluster to serve as cluster centers, and the pixel point with the smallest average distance inside the cluster can be selected as the cluster center, so that first image cluster center data and second image cluster center data are obtained and used for subsequent processing.

Representative pixel points are extracted from each cluster as cluster centers. It is assumed that cluster centers extracted from the first limb moving image cluster data are respectively: cluster 1 center: (10, 20), cluster 2 center: (30, 40), cluster 3 center: (50, 60), cluster 4 center: (70, 80), cluster 5 center: (90,100). Similarly, the cluster center extracted from the second limb moving image cluster data also has corresponding coordinates.

Specifically, the average distance from all pixel points in each cluster to the cluster center is calculated and used as the cluster radius of the cluster, so that cluster radius data of first limb moving image cluster data and cluster radius data of second limb moving image cluster data are obtained.

And carrying out cluster calculation on the first limb moving image cluster data and the second limb moving image cluster data, and calculating the average radius of each cluster. It is assumed that the cluster radius data of the obtained first limb moving image cluster data is 15, 20, 25, 30, 35, and the cluster radius data of the second limb moving image cluster data is 18, 22, 27, 32, 38.

Specifically, the first limb moving image clustering data is analyzed and processed to obtain first cluster radius data. And weighting the first cluster radius data by using a weighting calculation method according to the distribution condition and the characteristics of the first limb moving image cluster data. For example, the first cluster radius weighting data may be obtained by performing a weighting calculation according to factors such as the size, density, shape, etc. of each cluster. Similarly, the second limb moving image cluster data is analyzed and processed to obtain second cluster radius data, and the second cluster radius weighted data is obtained by a weighted calculation method.

Specifically, the cluster radius data is weighted by a linear weighting calculation method, and the weights are 0.2,0.3,0.4,0.5,0.6 respectively. For the first limb movement image, the weighted cluster radius data is 3.0,6.0,10.0,15.0,21.0, and for the second limb movement image, the weighted cluster radius data is 3.6,6.6,10.8,16.0,22.8.

Specifically, the first cluster radius weighting data is used for analyzing the first image cluster center data, and the domain range of each cluster center is determined. The domain pixel data associated with each cluster center is selected based on the first cluster radius weighting data. The size and shape of the field may be determined according to the degree of weighting to ensure that the selected pixels are representative and valid. Similarly, the second image cluster center data is analyzed and domain pixel selection is performed using the second cluster radius weighting data.

Specifically, according to the weighted cluster radius data, the domain range of each cluster center is determined, and corresponding pixels are selected as domain pixel data. Assume that in the first limb moving image, 100 pixels around each cluster center are selected as the domain pixel data. The same applies to the second limb moving image, and the field pixel data is selected.

Specifically, the dynamic pixel gradation judgment is performed on the first limb moving image domain pixel data and the second limb moving image domain pixel data. The dynamic pixel gray level judgment is to classify or filter the pixels in the image according to the gray level value and the motion characteristics of the pixels. For example, the pixels may be classified according to their brightness, contrast, color, etc. in combination with information such as a moving direction and a speed, etc., so as to screen out representative feature point data. And finally obtaining first limb moving image feature point data and second limb moving image feature point data for subsequent action parameter generation and rehabilitation auxiliary operation.

Specifically, for the selected field pixel data, judging according to the gray value and the motion characteristic of the pixel, and screening out representative characteristic point data. It is assumed that 50 feature point data are finally screened out of each set of domain pixel data for subsequent analysis and processing according to the gray value and motion characteristics of the pixels.

Specifically, the gray value of a certain pixel point isWherein/>Representing lines,/>Representing a column. The goal of the dynamic pixel gray level judgment is to determine whether the pixel point belongs to an action characteristic point. Firstly, defining a dynamic pixel gray level judging function/>Representing pixel gray value/>As to the judgment result of the action feature points, namely:

；

Then, whether the pixel belongs to the action feature point can be judged by calculating the difference between the gray value of the pixel and the gray value of surrounding pixels. Representing pixel points/>The surrounding set of neighborhood pixel points, the dynamic pixel gray scale decision function can be defined as:

；

Wherein the method comprises the steps of Is dynamic pixel gray judgment threshold data for controlling sensitivity of dynamic pixel gray judgment. If pixel dot/>Gray value of (c) and its surrounding pixel points (/ >)Representation/>Amplitude of variation range of (e.g. /)Is (1, 1) representsPixels within 1 surrounding area, e.g./>、/>、/>、/>、/>、/>、、/>、/>) The difference in average gray values of (2) is greater than the threshold/>Then the pixel point is judged to be an action feature point (i.e./>) Otherwise not (i.e./>）。

Preferably, step S106 is specifically:

Specifically, an illumination sensor, such as a photoresistor or a photodiode, is used to measure the illumination angle and the illumination intensity in the environment, and if the illumination angle is to be measured, the photoresistor or the photodiode can be turned mechanically or electronically to face the direction of interest, and then the illumination angle is determined by rotating or moving the sensor and measuring the illumination intensity at different positions. A plurality of photoresistors or photodiodes are arranged in an array in the device, and light is captured from different directions by placing a plurality of sensors (photoresistors or photodiodes) in the array. From the measurements of these sensors, the angular information of the illumination can be deduced. In particular, if illumination from a particular direction is stronger, the corresponding sensor will show a higher signal strength, while illumination from other directions will result in a lower signal strength. The incident angle of the light is determined by comparing the signal intensities of the different sensors. These sensors may be mounted on the device to acquire ambient light information in real time during the acquisition of the moving image data.

Specifically, shooting angle data of each image is acquired. The device now compares these shooting angles with a preset first and second threshold angle to calculate the actual illumination angle data. For example, let the illumination angle data be The preset first threshold angle data is/>The preset second threshold angle data is/>The first shooting illumination angle data is/>The second shooting illumination angle data is/>。

；

And generating a gray threshold according to the first shooting illumination angle data and the illumination intensity data to obtain first gray threshold data, and generating a gray threshold according to the second shooting illumination angle data and the illumination intensity data to obtain second gray threshold data.

Specifically, the image used is a grayscale image (or a non-grayscale image, then the grayscale image is first turned to determine feature points), and the feature points are determined from the grayscale values of the pixels. The first gray threshold is set to 100 and the second gray threshold is set to 200. For first limb motion image domain pixel data: each pixel is traversed to obtain its gray value. If the gray value of the pixel is larger than or equal to the first gray threshold value, the pixel is marked as a characteristic point, and the coordinates of the characteristic point are added into the first limb moving image characteristic point data. For second limb motion image domain pixel data: likewise, each pixel is traversed to obtain its gray value. If the gray value of the pixel is larger than or equal to the second gray threshold value, the pixel is marked as a feature point, and the coordinates of the feature point are added into the second limb moving image feature point data.

For first limb motion image domain pixel data: let the gray value of a pixel selected by the device be 120 and the first gray threshold be 100. Since 120 is 100 or more, the apparatus marks the pixel as a feature point and adds its coordinates to the first limb moving image feature point data. For second limb motion image domain pixel data: similarly, assume that the gray value of another pixel selected by the device is 180 and the second gray threshold is 200. Since 180 is less than 200, the device does not mark the pixel as a feature point.

Preferably, the limb movement key point detection data includes first limb movement key point detection data and second limb movement key point detection data, and step S2 specifically includes:

Specifically, the limb moving image data is processed using a deep learning model or a conventional computer vision algorithm to acquire posture information of the limb. For example, a Convolutional Neural Network (CNN) based pose estimation model, such as OpenPose, may be used to detect the location of human pose keypoints.

In particular, based on pose estimation data, local keypoints are detected using a specific keypoint detection algorithm (e.g., a single keypoint detector based on deep learning) by defining a region of interest (ROI) around a specific body part or joint. For example, for hand movement keypoint detection, a deep learning model for hand keypoint detection, such as Hand Keypoint Detection model, may be used.

Step S23: and carrying out global key point detection on the gesture estimation data to obtain second limb movement key point detection data.

Specifically, comprehensive keypoint detection is performed on the whole human gesture by using gesture estimation data to obtain global keypoint information, and the global keypoint information is obtained by applying a global keypoint detection algorithm, such as a target detection algorithm of YOLO or fast R-CNN, on the whole image, and the algorithms can detect a plurality of keypoints simultaneously and provide position information thereof, so as to obtain global keypoint detection data.

Preferably, step S22 is specifically:

specifically, a multi-branch network architecture is designed, with each branch being responsible for detecting key points for a particular body part. For example, one branch may be designed for detecting head keypoints, another branch for detecting arm keypoints, another branch for detecting leg keypoints, etc. These branches may be separate convolutional neural networks or network structures sharing parameters.

specifically, key points for a particular body part are located in the pose estimation data based on the results detected by each branch network. For example, if a branch network is dedicated to detecting arm keypoints, the position of the arm may be determined in the pose estimation data based on the detection result of the branch, and arm keypoint information may be extracted.

Specifically, feature fusion is carried out on the key point detection data of the specific body part detected by each branch network so as to comprehensively reflect the key point information of the whole limb movement, and the feature fusion is realized by means of simple feature splicing, weighted average and the like. For example, the key point data of arms, legs and the like detected by different branches can be fused to obtain complete limb movement key point detection data.

A Convolutional Neural Network (CNN) is designed to include multiple branches, each of which is specifically responsible for detecting key points of a specific part of a human body, such as an input layer: the input is gesture estimation data, which contains the position information of the key points of the human gesture. Multi-branch convolutional neural network: head branch: input layer: attitude estimation data, convolution layer: the characteristic extraction is carried out by a plurality of convolution layers, and the pooling layer: reducing the size of the feature map, retaining the main features, convolution layer: further extracting features, full connection layer: mapping the features to coordinates of head key points, and outputting the layer: coordinates of head keypoints. Shoulder branches: input layer: attitude estimation data, convolution layer: the characteristic extraction is carried out by a plurality of convolution layers, and the pooling layer: reducing the size of the feature map, retaining the main features, convolution layer: further extracting features, full connection layer: mapping the features to coordinates of the shoulder key points, outputting the layer: coordinates of the key points of the shoulder. Arm branch: input layer: attitude estimation data, convolution layer: the characteristic extraction is carried out by a plurality of convolution layers, and the pooling layer: reducing the size of the feature map, retaining the main features, convolution layer: further extracting features, full connection layer: mapping the features to coordinates of the arm key points, and outputting the coordinates: coordinates of the arm keypoints. Knee branching: input layer: attitude estimation data, convolution layer: the characteristic extraction is carried out by a plurality of convolution layers, and the pooling layer: reducing the size of the feature map, retaining the main features, convolution layer: further extracting features, full connection layer: mapping the features to coordinates of knee keypoints, outputting layer: coordinates of knee key points. Ankle branch: input layer: attitude estimation data, convolution layer: the characteristic extraction is carried out by a plurality of convolution layers, and the pooling layer: reducing the size of the feature map, retaining the main features, convolution layer: further extracting features, full connection layer: mapping the features to the coordinates of the ankle keypoints, outputting the layer: coordinates of the ankle keypoints. To train such a network, a Loss function of a keypoint detection task, such as mean square error (Mean Squared Error, MSE) or smoothed L1 Loss (smoothL 1 Loss), is employed by acquiring pose estimation training data and performing model/network training based on the pose estimation training data. For each keypoint locationThe loss function is expressed as:

；

training loss values for models,/> Estimating training data quantity data for pose,/>Training data sequence order item data for pose estimation,/>For/>Network prediction data of individual pose estimation training data,/>For/>The individual poses estimate the true position data of the training data. The optimization function may select a random gradient descent (Stochastic GRADIENT DESCENT, SGD) or a variation thereof, such as Adam or RMSProp. Input: and the posture estimation data comprises position information of key points of the human body posture. And (3) outputting: coordinates of key points of each specific part of the human body, such as (x, y) coordinate pairs. For example: the first branch is used to detect head keypoints. The second branch is used to detect arm keypoints. The third branch is used to detect leg keypoints. And each branch network obtains the key point detection data of the corresponding part through processing the gesture estimation data. For example: the head branch detects the position of the head keypoint. The arm branch detects the position of the arm keypoint. The leg branches detect the location of the leg keypoints. From the results detected by each branch network, the location of the keypoints for the specific body-part is determined in the pose estimation data. For example: the detection result of the head branch can be used to locate the head position in the pose estimation data and extract head keypoint information. The detection result of the arm branch can be used for locating the arm position in the gesture estimation data and extracting the arm key point information. The detection of leg branches may be used to locate leg positions in the pose estimation data and extract leg keypoint information. And carrying out feature fusion on the specific body part key point detection data detected by different branches to obtain complete limb movement key point detection data. For example: the key point data detected by branches such as head, arm, leg and the like can be subjected to simple characteristic stitching or weighted average so as to comprehensively reflect the key point information of the whole limb movement. /(I)

Preferably, step S3 is specifically:

Specifically, meaningful features are extracted from limb movement keypoint detection data by various feature extraction methods, for example, image features around the keypoints are extracted using a deep neural network (such as a convolutional neural network), or appearance and shape features of the keypoints are described using manually designed feature descriptors (such as SIFT, HOG, etc.).

Specifically, the most representative and discriminant features are screened out by a feature selection algorithm to reduce data dimension and redundant information. For example, feature selection may be performed by using Principal Component Analysis (PCA), linear Discriminant Analysis (LDA), information gain, or the like, and a feature subset that is most critical for motion recognition may be selected.

Specifically, the motion recognition is performed on the data after feature selection using a machine learning or deep learning algorithm, and for example, the motion recognition may be performed using a classifier such as a Support Vector Machine (SVM), random Forest (Random Forest), convolutional Neural Network (CNN), or the like. The models are trained to distinguish between different limb movement movements and are used to classify new movement data during testing to obtain limb movement identification data.

Preferably, step S4 is specifically:

Specifically, an action recognition model is built, such as an input layer: the input is limb movement identification data, which may be a time series of key point coordinates or a sequence of images. Convolutional Neural Network (CNN) section: the CNN part is responsible for extracting spatial features from the input data. Multiple convolution layers and pooling layers may be used to progressively extract features. Cyclic neural network (RNN) part: the RNN part is responsible for processing time series data and capturing time evolution information of actions. RNN units such as LSTM or GRU may be used to build the network. The output sequence of the CNN part was taken as the input sequence of RNN. Full tie layer: a full concatenation layer is added on the output of the RNN portion for converting the sequence information into a fixed length vector representation. Dropout layers were added to reduce overfitting. Output layer: the output layer is classified using a softmax function, each class representing a particular action. The number of nodes of the output layer is equal to the number of action categories. Loss function and optimization function: the loss function selects a cross entropy loss function (Cross Entropy Loss) for the multi-class classification task. The optimization function selects optimization algorithms such as Adam, SGD and the like. Input/output: input: the limb movement identification data may be a time series of key point coordinates or a series of images. And (3) outputting: the action classification result, namely the probability distribution of which action category belongs to, or the action recognition model is a set sequence mapping array, and the limb movement recognition data is input into the model for classification, wherein each category represents a specific action. And mapping each action to a corresponding category according to the output result of the model to obtain action classification mapping data.

Specifically, the start and end moments of the action are determined according to the key point detection data, and the action is time-sequence divided. Using a time window based approach, the key point data is divided in time into segments, each representing an action phase. A specific action category for each action phase is determined from the action classification mapping data.

specifically, features related to an action are extracted in each action phase. For example, the duration of each motion phase, the average speed, the maximum acceleration, and the angular change between the keypoints may be calculated.

Specifically, according to the action time sequence division data and the action characteristic data, action parameters describing each action stage are generated by combining the action classification mapping relation established previously, wherein the action parameters comprise information such as action type, start time, end time, duration time, speed, acceleration, angle change and the like and are used for executing and monitoring intelligent limb rehabilitation auxiliary operation.

Specifically, the action recognition model may recognize the following three actions: lifting the article, continuously maintaining the lifted state, and lowering the article. When the model identifies an action, the device maps it to a corresponding category, for example, based on its output: 1 denotes lifting the article, 2 denotes continuously holding the lifted state, and 3 denotes lowering the article. The action time sequence dividing algorithm of the supposing device determines the following moments according to the key point detection data: action start time: t_start=10s, action end time: t_end=30s, the device divides this period into three phases: stage 1: t_start to t_1 (15 s) represent the process of lifting the article. Stage 2: t_1 to t_2 (20 s) represent the process of continuously maintaining the lifted state. Stage 3: t_2 to t_end (30 s) represent the process of depositing the item. For each stage, the device extracts the following motion feature data: duration of time: stage 1 is 5 seconds, stage 2 is 5 seconds, and stage 3 is 10 seconds. Speed of: the average speed of each stage is calculated from the key point data. Acceleration: average acceleration of each stage is calculated from the speed data. Angle change: and calculating the angle change condition among the key points in each stage according to the key point data. According to the action classification mapping data, the action time sequence division data and the action characteristic data, the device generates the following limb action parameter data: the type for each stage is determined from the mapping, e.g. stage 1 is lifting the item, stage 2 is continuously holding the lifted state, and stage 3 is dropping the item. Start time and end time: and determining according to the time sequence division. The generated limb action parameter data comprise information such as duration, speed, acceleration, angle change and the like.

The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent limb rehabilitation assisting method is characterized by comprising the following steps of:

Step S2, including:

Step S4, including:

2. The method according to claim 1, wherein step S1 is specifically:

3. The method according to claim 2, wherein step S13 is specifically:

4. A method according to claim 3, wherein step S134 is specifically:

Step S103: cluster radius extraction is carried out on the first limb moving image cluster data and the second limb moving image cluster data to respectively obtain first cluster radius data and second cluster radius data;

5. The method according to claim 4, wherein step S106 is specifically:

6. The method according to claim 1, wherein step S22 is specifically:

7. The method according to claim 1, wherein step S3 is specifically:

8. A smart limb rehabilitation assistance device for performing the smart limb rehabilitation assistance method of claim 1, the smart limb rehabilitation assistance device comprising: