KR20230120350A - Rehabilitation exercise evaluation device and method of operating the same - Google Patents

Abstract

A method of operating a rehabilitation exercise evaluation apparatus according to an embodiment of the present disclosure includes obtaining monitoring data corresponding to a user's rehabilitation exercise, and obtaining local image data by extracting a user's musculoskeletal node based on the monitoring data. obtaining global image data by extracting information related to rehabilitation exercise based on the local image data, evaluating the level of rehabilitation exercise based on the global image data, based on the global image data, user A step of determining whether or not there is a risk of the risk, and a step of generating an evaluation report including the evaluated level and whether or not the risk is determined.

Description

Rehabilitation exercise evaluation device and its operating method {REHABILITATION EXERCISE EVALUATION DEVICE AND METHOD OF OPERATING THE SAME}

The present disclosure relates to a rehabilitation motion evaluation device, and more particularly, to a rehabilitation motion evaluation device and a method of operating the same.

Rehabilitation patients have physical functional impairments and disabilities, and perform rehabilitation exercises in daily life to improve health. Professional institutions such as medical institutions can provide safe and efficient rehabilitation exercises or treatments in consideration of physical functional impairment and disability of rehabilitation patients, but safety of rehabilitation exercises when performing rehabilitation exercises along with daily life after discharge from hospital and efficiency are limited.

Therefore, in order to ensure the safety and efficiency of rehabilitation exercises in daily life while overcoming functional impairment and disability of rehabilitation patients, the effectiveness or level of rehabilitation exercises performed by rehabilitation patients is evaluated, and rehabilitation patients performing rehabilitation exercises are at risk. A rehabilitation exercise evaluation device that determines whether or not the user is in a situation may be required.

According to an embodiment of the present disclosure, an apparatus for evaluating a rehabilitation exercise and a method for operating the same are provided.

According to an embodiment of the present disclosure, a method of operating a rehabilitation exercise evaluation apparatus includes acquiring monitoring data corresponding to a user's rehabilitation exercise, extracting a musculoskeletal node of the user based on the monitoring data, and extracting a local image. Acquiring data; extracting related information of the rehabilitation exercise based on the local image data to obtain global image data; evaluating a level of the rehabilitation exercise based on the global image data; Based on the global image data, determining whether or not the user is in danger, and generating an evaluation report including the evaluated level and the determined risk or not.

In some embodiments, acquiring the monitoring data corresponding to the rehabilitation exercise of the user may include obtaining first frame data indicating a first posture of the user in a 3D space at a first time point; and obtaining second frame data indicating a second posture of the user in the 3D space at a second time point after the first time point, wherein the monitoring data includes the first frame data and the second frame data and the rehabilitation exercise includes the first posture and the second posture.

In some embodiments, the obtaining of the local image data by extracting the musculoskeletal node of the user based on the monitoring data includes graph data corresponding to the user on the monitored space based on the monitoring data. Extracting a spatial object node, extracting a musculoskeletal node of the user based on the monitoring data, and obtaining the local image data based on a node relationship between the spatial object node and the musculoskeletal node. include

In some embodiments, the obtaining of the global image data by extracting the related information of the rehabilitation exercise based on the local image data may include removing information having little relevance to the rehabilitation exercise from the local image data. Generating running data, generating a time-scaled 4D scene graph based on the pruning data, and acquiring the global image data based on a neural network operation of the 4D scene graph. include

In some embodiments, the neural network operation is based on a graph convolutional network (GCN).

In some embodiments, generating the evaluation report including the evaluated level and the determined risk includes providing the evaluation report to the user and a manager of the user.

In some embodiments, acquiring the monitoring data corresponding to the rehabilitation exercise of the user may include requesting the user to perform the rehabilitation exercise, and while the user is performing the rehabilitation exercise, the user and acquiring the monitoring data by monitoring.

In some embodiments, the rehabilitation exercise evaluation device is implemented on the personal terminal of the user.

According to an embodiment of the present disclosure, an apparatus for evaluating a rehabilitation exercise includes a monitoring device configured to obtain monitoring data corresponding to a rehabilitation exercise of a user, a rehabilitation exercise analysis module configured to obtain global image data based on the monitoring data, the A rehabilitation exercise evaluation module configured to evaluate a level of the rehabilitation exercise based on global image data and determine whether the user is at risk, and a report configured to generate an evaluation report including the evaluated level and the determined risk or not. Include a generating module.

In some embodiments, the rehabilitation exercise analysis module may include a local image processor configured to obtain local image data by extracting a musculoskeletal node of the user based on the monitoring data, and based on the local image data, the rehabilitation exercise and a global image processor configured to obtain the global image data by extracting related information of .

In some embodiments, the local image processor may include a spatial object node extractor configured to extract a spatial object node of graph data corresponding to the user on the monitored space based on the monitoring data, based on the monitoring data, and a musculoskeletal node extraction unit configured to extract a musculoskeletal node of the user, and a node relationship generation unit configured to generate the local image data based on a node relationship between the spatial object node and the musculoskeletal node.

In some embodiments, the global image processor may include a node relational pruning unit configured to generate pruning data by removing information having little relevance to the rehabilitation exercise from the local image data, and based on the pruning data, a time scale A time scale 4D scene graph generator configured to generate a 4D scene graph, and a neural network learner configured to generate the global image data based on a neural network operation of the 4D scene graph.

In some embodiments, the rehabilitation exercise level evaluation module may include a rehabilitation exercise level evaluation module configured to evaluate a level of the rehabilitation exercise based on the global image data and generate level evaluation data, and the user based on the global image data. and a risk situation determination module configured to determine whether risk is present and to generate risk assessment data indicating whether the determined risk is present, and wherein the report generation module performs the evaluation based on the level evaluation data and the risk evaluation data. It is further configured to generate reports.

In some embodiments, the rehabilitation exercise evaluation device further includes a rehabilitation exercise device requesting the user to perform the rehabilitation exercise, wherein the monitoring device performs the rehabilitation exercise by referring to the rehabilitation exercise device. while doing so, it is further configured to monitor the user to obtain the monitoring data.

According to an embodiment of the present disclosure, an apparatus for evaluating a rehabilitation exercise and a method for operating the same are provided.

In addition, by applying a neural network operation to a 4-dimensional scene graph on a time scale, the validity and risk situation of rehabilitation exercises are accurately judged, and rehabilitation exercises with improved interface convenience are provided by providing evaluation results of rehabilitation exercises through natural language-based evaluation reports. An evaluation device and its operating method are provided.

1 is a block diagram illustrating an apparatus for evaluating a rehabilitation exercise according to an embodiment of the present disclosure.
2 is a diagram illustrating monitoring data of FIG. 1 according to some embodiments of the present disclosure.
3 is a block diagram illustrating an apparatus for evaluating a rehabilitation exercise according to some embodiments of the present disclosure.
FIG. 4 is an embodiment of the local image processor of FIG. 3 according to some embodiments of the present disclosure.
FIG. 5 is a diagram embodying the global image processor of FIG. 3 according to some embodiments of the present disclosure.
FIG. 6 is a diagram illustrating the 4D scene graph of FIG. 5 according to some embodiments of the present disclosure.
7 is a flowchart illustrating a method of operating a rehabilitation exercise evaluation apparatus according to some embodiments of the present disclosure.

In the following, embodiments of the present invention will be described clearly and in detail so that those skilled in the art can easily practice the present invention.

Terms such as “unit” and “module” used below or functional blocks shown in the drawings may be implemented in the form of a software configuration, a hardware configuration, or a combination thereof. Hereinafter, in order to clearly explain the technical spirit of the present invention, detailed descriptions of overlapping components are omitted.

1 is a block diagram illustrating an apparatus for evaluating a rehabilitation exercise according to an embodiment of the present disclosure. Referring to FIG. 1 , a rehabilitation exercise evaluation apparatus 100 is shown.

Recently, with the aging of the population, the number of rehabilitation patients such as the elderly and the disabled and the burden of medical expenses are increasing. Accordingly, in order to help rehabilitation patients return to daily life early, there is an increasing need to continuously perform rehabilitation exercises at home, training centers, and the like.

Since rehabilitation patients have functional impairments and disabilities, rehabilitation exercise treatment is performed according to rehabilitation exercise guidelines customized for rehabilitation patients in specialized institutions such as hospitals. In order to ensure the safety and effectiveness of rehabilitation exercise, an infrastructure that quantitatively evaluates the rehabilitation patient's rehabilitation exercise level during the rehabilitation exercise process and sends an alarm to the control center of a specialized institution when the rehabilitation patient is in a dangerous situation may be required. .

Meanwhile, the biggest difficulty in performing rehabilitation exercises as part of home training by rehabilitation patients in daily life such as at home may be to check whether the rehabilitation patients perform rehabilitation exercises with correct motions. If the rehabilitation patient performs a rehabilitation exercise with an incorrect motion, the rehabilitation patient may be in a dangerous situation by causing pain due to body imbalance to the rehabilitation patient. However, since rehabilitation patients themselves have limitations in determining the suitability of rehabilitation exercises, evaluating the level of rehabilitation exercises may be one of conditions that precede effective rehabilitation exercises.

According to an embodiment of the present disclosure, a rehabilitation exercise evaluation apparatus 100 may be provided that evaluates the level of rehabilitation exercise performed by a user and determines whether the user is at risk.

The user may be a subject who performs rehabilitation exercise, such as a rehabilitation patient, a patient with a musculoskeletal disorder, a disabled person, or the elderly. A user may also be referred to as an object. In other words, the user can generate actions and events of the rehabilitation exercise by temporally continuously performing the rehabilitation exercise in a real world space (ie, a three-dimensional space) such as a home or a training center.

Hereinafter, detailed components of the rehabilitation exercise evaluation apparatus 100 will be described.

The rehabilitation exercise evaluation device 100 includes a rehabilitation exercise device 110, a monitoring device 120, a rehabilitation exercise analysis module 130, a rehabilitation exercise evaluation module 140, a report generation module 150, and a connection interface 160. can include

The rehabilitation exercise device 110 may request a user to perform a rehabilitation exercise. For example, the rehabilitation exercise device 110 may be an auxiliary device necessary for performing a rehabilitation exercise, such as mounted on a part of the user's body, or a user may perform a rehabilitation exercise in a visual form such as an image or video. It may be a display device that provides The user may perform rehabilitation exercises with reference to the rehabilitation exercise device 110 .

The monitoring device 120 may obtain monitoring data MD corresponding to the user's rehabilitation exercise. The monitoring device 120 may include a 3-dimension depth sensor. The monitoring data MD may be data obtained by monitoring the user while the user performs a rehabilitation exercise with reference to the rehabilitation exercise device 110 .

In some embodiments, the monitoring data MD may include 4-dimensional monitoring information. The 4D monitoring information may refer to 3D spatial information sequentially obtained over time. For example, the 4D monitoring information may refer to information of an object defined along a time axis and a first direction-axis, a second direction-axis, and a third direction-axis defining a 3D space. A more detailed description of the monitoring data MD will be described later with reference to FIG. 2 .

The rehabilitation exercise analysis module 130 may analyze the rehabilitation exercise performed by the user based on the monitoring data MD. For example, the rehabilitation exercise analysis module 130 may acquire local image data based on the monitoring data MD. The local image data may include a node relationship between a spatial object node extracted from the monitoring data MD and a user's musculoskeletal node. In local image data, as an object (i.e., a user) is expressed as a node relationship of musculoskeletal nodes over time, the skeleton and joints of the object may be expressed as data in the form of a 4-dimensional graph having a topology relationship on a time scale. can Accordingly, the rehabilitation exercise analysis module 130 may automatically extract musculoskeletal information of the user from the monitoring data MD including the 4-dimensional monitoring information.

The rehabilitation motion analysis module 130 may acquire global image data based on the local image data. The global image data may be data obtained by extracting information associated with a rehabilitation exercise from local image data.

The rehabilitation exercise evaluation module 140 may evaluate a rehabilitation exercise based on the information analyzed by the rehabilitation exercise analysis module 130 . For example, the rehabilitation exercise evaluation module 140 may receive global image data from the rehabilitation exercise analysis module 130 . The rehabilitation exercise evaluation module 140 may evaluate the level of the rehabilitation exercise performed by the user based on the global image data, and determine whether the user is in danger based on the global image data.

The report generation module 150 may generate an evaluation report based on information evaluated by the rehabilitation exercise evaluation module 140 . The evaluation report may be a natural language-based report that is identifiable by a user (eg, a report representing evaluated content as text). The evaluation report may indicate an evaluated level of rehabilitation exercise performed by the user. The evaluation report may indicate whether or not the user performing the rehabilitation exercise is at risk. The report generation module 150 may provide an evaluation report to a user.

In some embodiments, the report generation module 150 may provide the evaluation report to the user's manager. The manager may be a doctor, a physical therapist, a rehabilitation exercise manager, or the like who manages a user who is a rehabilitation patient, or may refer to a terminal used by them. The report generating module 150 may report the evaluation report to an external specialized institution, such as a hospital where an administrator is located.

The connection interface 160 may support communication between the rehabilitation exercise device 110 , the monitoring device 120 , the rehabilitation exercise analysis module 130 , the rehabilitation exercise evaluation module 140 , and the report generation module 150 . For example, the connection interface 160 may support wired communication between different circuits, modules, or chips within the same electronic device or wireless communication between separated electronic devices.

In some embodiments, the rehabilitation exercise evaluation device 100 may be implemented on a personal terminal such as a smart phone, a smart watch, or a tablet. For example, the monitoring device 120 may be implemented as a camera module of a smart phone, the rehabilitation exercise device 110 may be implemented on a display panel of a smart phone, and the rehabilitation exercise analysis module 130, rehabilitation exercise device 110 may be implemented. The exercise evaluation module 140 and the report generation module 150 may be implemented as applications executed by a processor in a smart phone.

However, the scope of the present disclosure is not necessarily limited thereto, and the rehabilitation exercise device 110 may be implemented as a separate device mountable by a user, and includes a rehabilitation exercise analysis module 130, a rehabilitation exercise evaluation module 140, And the report generation module 150 may be implemented on a separate server device communicating with the personal terminal.

As described above, according to an embodiment of the present disclosure, the rehabilitation exercise evaluation apparatus 100 may provide an automated rehabilitation exercise evaluation service to a user. The rehabilitation exercise evaluation apparatus 100 may report the user's dangerous situation to an external specialized institution by recognizing the user's abnormal motion situation while evaluating the level of the user's rehabilitation exercise.

2 is a diagram illustrating monitoring data of FIG. 1 according to some embodiments of the present disclosure. Referring to FIGS. 1 and 2 , the monitoring device 120 may generate monitoring data MD. The monitoring data MD may include a plurality of frame data according to time. Each of the plurality of frame data may include location information (eg, 3D coordinates of skeletons and joints of the object) of an object in a 3D space at a corresponding specific point in time.

The 3D space may be a space defined by the first direction D1 , the second direction D2 , and the third direction D3 . The first direction D1 may be a direction parallel to the ground where the user is located. The second direction D2 may be perpendicular to the first direction D1. The third direction D3 may be perpendicular to the plane defined by the first and second directions D1 and D2.

For example, the monitoring data MD may include first to Nth frame data FD1 to FDN respectively corresponding to the first to Nth views. N is any natural number. The first to Nth views may refer to successive sequential views.

More specifically, the monitoring data MD may include first frame data FD1 corresponding to the first time point t1. The first frame data FD1 may indicate a first posture according to positions of the user's skeletons and joints in the 3D space at the first time point t1.

The monitoring data MD may include second frame data FD2 corresponding to a second time point t2 after the first time point t1. The second frame data FD2 may indicate a second posture according to positions of bones and joints of the user in the 3D space at the second time point t2 . The second posture may be a posture indicated by the user immediately after performing the first posture.

Similarly, the monitoring data MD may include third frame data FD3 corresponding to the third time point t3 after the second time point t2. The third frame data FD3 may indicate the user's third posture in the 3D space. The monitoring data MD may include the Nth frame data FDN corresponding to the Nth time point tN after the N−1th time point tN−1. The Nth frame data FDN may indicate the user's Nth posture in the 3D space.

The plurality of frame data included in the monitoring data MD may be used by the rehabilitation exercise evaluation apparatus 100 to evaluate the user's rehabilitation exercise.

3 is a block diagram illustrating an apparatus for evaluating a rehabilitation exercise according to some embodiments of the present disclosure. Referring to FIG. 3 , a rehabilitation exercise evaluation apparatus 200 is shown. The rehabilitation exercise evaluation apparatus 200 may communicate with a user and a manager. A user may be a subject who performs rehabilitation exercise, or may also be referred to as an object. The manager may be a doctor, a physical therapist, a rehabilitation exercise manager, etc., who help the user with rehabilitation exercise, or may refer to a terminal used by them.

The rehabilitation exercise evaluation device 200 includes a monitoring device 220, a local image processor 230, a global image processor 235, a rehabilitation exercise level evaluation module 240, a risk situation determination module 245, and a report generation module ( 250) may be included.

The monitoring device 220 may correspond to the monitoring device 120 of FIG. 1 . The local image processor 230 and the global image processor 235 may correspond to the rehabilitation exercise analysis module 130 of FIG. 1 . The rehabilitation exercise level evaluation module 240 and the risk situation determination module 245 may correspond to the rehabilitation exercise evaluation module 140 of FIG. 1 . The report generating module 250 may correspond to the report generating module 150 of FIG. 1 .

The monitoring device 220 may obtain monitoring data MD by monitoring the user's rehabilitation exercise. In some embodiments, the rehabilitation exercise evaluation apparatus 200 may further include a rehabilitation exercise apparatus (eg, the rehabilitation exercise apparatus 110 of FIG. 1 ) requesting rehabilitation exercise from a user. The monitoring device 220 may obtain monitoring data MD by monitoring a user performing a rehabilitation exercise with reference to the rehabilitation exercise device.

The local image processor 230 may receive monitoring data MD from the monitoring device 220 . The local image processor 230 may obtain local image data LID by extracting a node relationship of the user's musculoskeletal node based on the monitoring data MD.

The global image processor 235 may receive local image data LID from the local image processor 230 . The global image processor 235 may obtain global image data GID by extracting information associated with a rehabilitation exercise performed by a user based on the local image data LID.

The rehabilitation exercise level evaluation module 240 may receive global image data GID from the global image processor 235 . The rehabilitation exercise level evaluation module 240 may evaluate the level of the rehabilitation exercise performed by the user based on the global image data (GID). The level of rehabilitation exercise may quantitatively indicate whether the rehabilitation exercise is suitable, effective, or effective. Rehabilitation exercise level assessment module 240 may generate level assessment data (LED) indicating the assessed level.

In some embodiments, the rehabilitation exercise level evaluation module 240 may learn the rehabilitation exercise level in advance. For example, the rehabilitation exercise level evaluation module 240 may learn the rehabilitation exercise level in advance based on the first training data. The rehabilitation exercise level evaluation module 240 may determine the rehabilitation exercise level corresponding to the global image data (GID) by applying regression or classification according to the pre-learned rehabilitation exercise level.

The dangerous situation determination module 245 may receive the global image data GID from the global image processor 235 . The dangerous situation determination module 245 may determine whether the user is in danger based on the global image data (GID). Whether or not the user is at risk may indicate whether the user is at risk of injury or is at risk due to inappropriate rehabilitation exercise in consideration of positional changes of the user's skeletons and joints. The dangerous situation determination module 245 may generate risk assessment data DED indicating whether or not the determined risk exists.

In some embodiments, the dangerous situation determination module 245 may learn the dangerous situation in advance. For example, the dangerous situation determination module 245 may learn a dangerous situation in advance based on the second training data. The dangerous situation determination module 245 may determine whether a situation corresponding to the global image data GID corresponds to a dangerous situation by referring to a previously learned dangerous situation.

In some embodiments, the risk situation determination module 245 may be implemented as a lightweight machine learning model. More specifically, the dangerous situation determination module 245 may be implemented as a lightweight machine learning model to be executable on a personal terminal such as a smart phone. The dangerous situation determination module 245 may learn the user's dangerous situation in advance through a lightweight machine learning model. The dangerous situation determination module 245 may determine the dangerous situation of the user in real time based on the global image data (GID) by referring to the previously learned dangerous situation.

The report generation module 250 may receive level evaluation data (LED) from the rehabilitation exercise level evaluation module 240 . The report generation module 250 may receive the risk assessment data DED from the risk situation determination module 245 . The report generating module 250 may generate an evaluation report (ER) including an evaluated level and a determined risk based on the level evaluation data (LED) and the risk evaluation data (DED). The evaluation report (ER) may be a report generated based on natural language. For example, an evaluation report (ER) may be provided in text format.

The report generation module 250 may provide an evaluation report (ER) to users and administrators. Based on the evaluation report ER, the user may maintain the posture of the rehabilitation exercise, stop the dangerous rehabilitation exercise, or change the posture of the rehabilitation exercise to a more effective posture. Based on the evaluation report (ER), the administrator can diagnose the user or confirm the user's dangerous situation.

In some embodiments, the report generating module 250 may infer a causal relationship related to rehabilitation exercise by performing commonsense reasoning with reference to a database including universal knowledge. The report generation module 250 may generate an evaluation report (ER) including the inferred causal relationship.

FIG. 4 is an embodiment of the local image processor of FIG. 3 according to some embodiments of the present disclosure. Referring to FIGS. 3 and 4 , the local image processor 230 may communicate with the monitoring device 220 and the global image processor 235 . The local image processor 230 may receive monitoring data MD from the monitoring device 220 . The local image processor 230 may acquire local image data LID based on the monitoring data MD and provide the local image data LID to the global image processor 235 .

The local image processor 230 may include a spatial object node extractor 231 , a musculoskeletal node extractor 232 , and a node relationship generator 233 .

The spatial object node extractor 231 may extract a spatial object node of graph data corresponding to an object on the monitored space based on the monitoring data MD. For example, the monitoring data MD may include a plurality of frame data. The spatial object node extractor 231 may extract spatial object nodes by referring to predefined spatial object information for each of a plurality of frame data.

In some embodiments, the spatial object node extractor 231 may extract spatial object nodes using general software such as Detectron of Facebook.

The musculoskeletal node extractor 232 may extract the user's musculoskeletal node based on the monitoring data MD. For example, the musculoskeletal node extractor 232 configures a musculoskeletal joint sequence by extracting the position of the user's musculoskeletal joint based on the monitoring data (MD) or the monitoring data preprocessed by the spatial object node extractor 231. can do. The musculoskeletal node may include a musculoskeletal joint sequence configured by the musculoskeletal node extraction unit 232 . A musculoskeletal node may also be referred to as a musculoskeletal pose node.

In some embodiments, the musculoskeletal node extraction unit 232 may extract the musculoskeletal node using general software such as Facebook's Detectron.

The node relationship generator 233 may receive the spatial object node extracted by the spatial object node extractor 231 and the musculoskeletal node extracted by the musculoskeletal node extractor 232 . The node relationship creation unit may create a node relationship between the spatial object node and the musculoskeletal node, and generate local image data LID including the node relationship. A node relationship can be expressed as a line connecting different nodes.

However, the scope of the present disclosure is not limited thereto, and node relationships include rehabilitation exercises according to musculoskeletal joint sequences, motion relationships or spatial relationships for objects in space, spatial distribution of spatial object nodes, and relationships between health states of objects. It can contain multiple relationships.

FIG. 5 is a diagram embodying the global image processor of FIG. 3 according to some embodiments of the present disclosure. Referring to FIGS. 3 and 5 , the global image processor 235 may communicate with the local image processor 230 , the rehabilitation exercise level evaluation module 240 , and the risk situation determination module 245 . The global image processor 235 may receive local image data LID from the local image processor 230 . The global image processor 235 acquires the global image data (GID) based on the local image data (LID), and sends the global image data (GID) to the rehabilitation exercise level evaluation module 240 and the risk situation determination module 245. can provide

The global image processor 235 may include a node relation pruning unit 236 , a time scale 4D scene graph generating unit 237 , and a neural network learning unit 238 .

The node-relationship pruning unit 236 may generate the pruning data PD by removing information that has little relevance to the rehabilitation exercise based on the local image data LID. For example, the local image data LID may include various node relationships to spatial object nodes and musculoskeletal nodes. At this time, the node relationships include information that is not related to rehabilitation exercises, such as relationships that are invalid or lack meaning in the graph node, relationships that are not related to rehabilitation exercises, and relationships that are not related to the risk situation of rehabilitation patients. can include The node relation pruning unit 236 may remove information that has little relevance to rehabilitation exercise as a pre-processing process.

The time scale 4D scene graph generator 237 may receive the pruning data PD from the node relation pruning unit 236 . The time-scaled 4D scene graph generator 237 may generate a time-scaled 4D scenegraph FSG based on the pruning data PD. A 4-dimensional scene graph (FSG) on a time scale may represent changes over time of nodes having relationships in a 3-dimensional space. A more detailed description of the 4D scene graph (FSG) will be described later with reference to FIG. 6 .

The neural network learner 238 may receive the 4D scene graph FSG from the time scale 4D scene graph generator 237 . The neural network learner 238 may generate the global image data GID based on the neural network operation of the 4D scene graph FSG.

In some embodiments, the neural network learning unit 238 may include a graph convolutional network (GCN). The neural network learning unit 238 uses a graph convolutional network (GCN) to determine information about a series of rehabilitation exercises performed by a rehabilitation patient and the phase relationship of nodes in space, and to obtain global image data (GID). can create

FIG. 6 is a diagram illustrating the 4D scene graph of FIG. 5 according to some embodiments of the present disclosure. Referring to FIGS. 5 and 6 , the time scale 4D scene graph generator 237 may generate a 4D scene graph FSG based on the pruning data PD.

The pruning data PD may include first pruned frame data PFD1 and second pruned frame data PFD2. For example, the first pruned frame data PFD1 and the second pruned frame data PFD2 are the first frame data FD1 and the second frame data FD2 of the monitoring data MD of FIG. 2 , respectively. ) can respond.

The first pruned frame data PFD1 may correspond to the first time point t1 and may be data from which information low in relevance to rehabilitation exercise in the 3D space is removed. The first pruned frame data PFD1 may include node relationships of the musculoskeletal nodes Na1 , Nb1 , ..., Nm1 of the object.

The second pruned frame data PFD2 may correspond to the second time point t2 and may be data from which information low in relation to rehabilitation exercise in the 3D space is removed. The second time point t2 may be after the first time point t1. The second pruned frame data PFD2 may include node relationships of musculoskeletal nodes Na2, Nb2, ..., Nm2 of the object.

The time scale 4D scene graph generation unit 237 configures the musculoskeletal nodes Na1 , Nb1 , ..., Nm1 to each of the musculoskeletal nodes Na2 based on the first and second pruned frame data PFD1 and PFD2 . , Nb2, ..., Nm2), it is possible to generate a 4-dimensional scene graph (FSG) including semantic information (eg, position change of nodes over time) of the rehabilitation exercise performed by the user. there is. The 4D scene graph (FSG) may be used to determine the effectiveness of rehabilitation exercise and the user's risk situation through neural network calculation.

7 is a flowchart illustrating a method of operating a rehabilitation exercise evaluation apparatus according to some embodiments of the present disclosure. Referring to FIG. 7 , an operating method of the rehabilitation exercise evaluation apparatus will be described. The rehabilitation exercise evaluation apparatus may evaluate a rehabilitation exercise performed by a user. The rehabilitation exercise evaluation apparatus may correspond to the rehabilitation exercise evaluation apparatus 100 of FIG. 1 and the rehabilitation exercise evaluation apparatus 200 of FIG. 3 .

In step S110, the rehabilitation exercise evaluation apparatus may acquire monitoring data corresponding to the user's rehabilitation exercise.

In some embodiments, step S110 may include obtaining first frame data indicating a first posture of a user in a 3D space at a first point of view, and a first frame data of a user in a 3D space at a second point in time after the first point of view. It may include acquiring second frame data indicating 2 postures. In this case, the monitoring data may include first frame data and second frame data, and the rehabilitation exercise performed by the user may include the first posture and the second posture.

In some embodiments, operation S110 may include requesting the user to perform a rehabilitation exercise, and obtaining monitoring data by monitoring the user while the user performs the rehabilitation exercise. For example, the rehabilitation exercise evaluation device may include a rehabilitation exercise device requesting a user to perform a rehabilitation exercise. The rehabilitation exercise device may visually display a rehabilitation exercise or may be mounted on a part of a user's body when performing a rehabilitation exercise.

In step S120 , the rehabilitation exercise evaluation apparatus may obtain local image data by extracting a user's musculoskeletal node based on the monitoring data.

In some embodiments, step S120 includes extracting a space object node of graph data corresponding to the user on the monitored space based on the monitoring data, extracting a musculoskeletal node of the user based on the monitoring data, and space It may include obtaining local image data based on the node relationship of the object node and the musculoskeletal node.

In operation S130 , the rehabilitation exercise evaluation apparatus may obtain global image data by extracting information related to the rehabilitation exercise based on the local image data.

In some embodiments, step S130 includes generating pruning data by removing information that has little relevance to rehabilitation exercise from local image data, generating a time-scaled 4-dimensional scene graph based on the pruning data, and It may include obtaining global image data based on a neural network operation of a 4D scene graph. In this case, the neural network calculation may be based on a graph convolutional network (GCN).

In step S140 , the rehabilitation exercise evaluation apparatus may evaluate the level of the rehabilitation exercise based on the global image data.

In step S150, the rehabilitation exercise evaluation apparatus may determine whether the user is in danger based on the global image data.

In step S160, the rehabilitation exercise evaluation apparatus may generate an evaluation report including the level evaluated in step S140 and whether or not the risk was determined in step S150.

In some embodiments, step S160 may include providing the evaluation report to the user and the user's manager.

In some embodiments, the rehabilitation exercise evaluation device may be implemented on a user's personal terminal.

The foregoing are specific embodiments for carrying out the present invention. The present invention will include not only the above-described embodiments, but also embodiments that can be simply or easily changed in design. In addition, the present invention will also include techniques that can be easily modified and practiced using the embodiments. Therefore, the scope of the present invention should not be limited to the above-described embodiments and should not be defined by the following claims as well as those equivalent to the claims of this invention.

100: rehabilitation exercise evaluation device
110: rehabilitation exercise device
120: monitoring device
130: rehabilitation exercise analysis module
140: rehabilitation exercise evaluation module
150: report generation module
200: rehabilitation exercise evaluation device

Claims (14)

In the operation method of the rehabilitation exercise evaluation device:
acquiring monitoring data corresponding to a user's rehabilitation exercise;
obtaining local image data by extracting a musculoskeletal node of the user based on the monitoring data;
obtaining global image data by extracting related information of the rehabilitation exercise based on the local image data;
evaluating a level of the rehabilitation exercise based on the global image data;
determining whether the user is in danger based on the global image data; and
and generating an evaluation report including the evaluated level and the determined risk. According to claim 1,
Acquiring the monitoring data corresponding to the rehabilitation exercise of the user includes:
obtaining first frame data indicating a first posture of the user in a 3-dimensional space at a first point of view; and
obtaining second frame data indicating a second posture of the user in the 3D space at a second time point after the first time point;
The monitoring data includes the first frame data and the second frame data, and
The method of claim 1 , wherein the rehabilitation exercise includes the first posture and the second posture. According to claim 1,
Based on the monitoring data, extracting the musculoskeletal node of the user to obtain the local image data includes:
extracting a spatial object node of graph data corresponding to the user in the monitored space, based on the monitoring data;
Extracting a musculoskeletal node of the user based on the monitoring data; and
and acquiring the local image data based on a node relationship between the spatial object node and the musculoskeletal node. According to claim 1,
The step of obtaining the global image data by extracting the associated information of the rehabilitation exercise based on the local image data:
generating pruning data by removing information that has little to do with the rehabilitation exercise from the local image data;
generating a time-scaled 4-dimensional scene graph based on the pruning data; and
and acquiring the global image data based on a neural network operation of the 4D scene graph. According to claim 4,
The method of claim 1 , wherein the neural network operation is based on a graph convolutional network (GCN). According to claim 1,
Generating the evaluation report including the evaluated level and the determined risk:
and providing the evaluation report to the user and the user's manager. According to claim 1,
Acquiring the monitoring data corresponding to the rehabilitation exercise of the user includes:
requesting the user to perform the rehabilitation exercise; and
and obtaining the monitoring data by monitoring the user while the user performs the rehabilitation exercise. According to claim 1,
The rehabilitation exercise evaluation device is implemented on the personal terminal of the user. a monitoring device configured to acquire monitoring data corresponding to a user's rehabilitation exercise;
a rehabilitation motion analysis module configured to acquire global image data based on the monitoring data;
a rehabilitation exercise evaluation module configured to evaluate a level of the rehabilitation exercise based on the global image data and determine whether the user is at risk; and
Rehabilitation exercise evaluation device comprising a report generation module configured to generate an evaluation report including the evaluated level and the determined risk. According to claim 9,
The rehabilitation exercise analysis module:
a local image processor configured to obtain local image data by extracting a musculoskeletal node of the user based on the monitoring data; and
and a global image processor configured to obtain the global image data by extracting related information of the rehabilitation exercise based on the local image data. According to claim 10,
The local image processor:
a spatial object node extraction unit configured to extract, based on the monitoring data, a spatial object node of graph data corresponding to the user on the monitored space;
a musculoskeletal node extraction unit configured to extract a musculoskeletal node of the user based on the monitoring data; and
and a node relationship generation unit configured to generate the local image data based on the node relationship between the spatial object node and the musculoskeletal node. According to claim 10,
The global image processor:
a node relation pruning unit configured to generate pruning data by removing information having little relevance to the rehabilitation exercise from the local image data;
a time-scale 4-dimensional scene graph generator configured to generate a time-scale 4-dimensional scene graph based on the pruning data; and
and a neural network learning unit configured to generate the global image data based on a neural network operation of the 4-dimensional scene graph. According to claim 9,
The rehabilitation exercise evaluation module:
a rehabilitation exercise level evaluation module configured to evaluate a level of the rehabilitation exercise based on the global image data to generate level evaluation data; and
a dangerous situation determination module configured to determine whether the user is in danger based on the global image data, and to generate risk assessment data indicating whether the user is in danger; and
The report generating module is further configured to generate the evaluation report based on the level evaluation data and the risk evaluation data. According to claim 9,
Further comprising a rehabilitation exercise device requesting the user to perform the rehabilitation exercise,
wherein the monitoring device is further configured to acquire the monitoring data by monitoring the user while the user performs the rehabilitation exercise with reference to the rehabilitation exercise device.

Images