KR102566211B1 - Method and system for providing treatment using artificial intelligence posture estimation model and motion analysis model - Google Patents

Abstract

The present invention relates to a method and system for analyzing a patient's motor motion. The method for providing exercise therapy according to the present invention includes the steps of receiving prescription information related to exercise for a patient from a doctor terminal. Allocating an exercise plan including at least one prescription exercise to the patient's account based on the prescription information, receiving an exercise image of the exercise according to the prescription exercise from a patient terminal, extracting key points respectively corresponding to a plurality of preset joint points from the motion image using an intelligent posture estimation model; through an artificial intelligence motion analysis model learned using learning data related to the joint points; Analyzing a relative positional relationship between keypoints, and based on the analysis of the positional relationship, analyzing the exercise motion of the patient for the prescription exercise, and transmitting the analysis result of the motion motion of the patient to the patient terminal. may include steps to

Description

Motion therapy providing method and system using artificial intelligence posture estimation model and motion analysis model

The present invention relates to a method and system capable of providing exercise therapy to a patient using an artificial intelligence posture estimation model and an artificial intelligence motion analysis model specialized for musculoskeletal disorders.

Musculoskeletal disorders refer to pain or damage occurring in the musculoskeletal system, such as muscles, nerves, tendons, ligaments, bones and surrounding tissues. Musculoskeletal disorders appear in various parts of our body, such as the neck, waist, arms and legs.

According to the report of the World Health Organization (WHO-World Health Organization), economic loss due to musculoskeletal disorders is the fourth highest among all diseases, and musculoskeletal disorders are chronic pains that affect not only daily life but also economic activities. .

On the other hand, treatment of musculoskeletal disorders, in principle, is performed starting with less invasive treatment, and non-drug conservative treatment (ex: exercise treatment and education, cognitive therapy or relaxation therapy, etc.) is performed first, followed by drug treatment, Surgical treatment should be considered sequentially.

In the treatment guidelines, non-drug conservative treatment of musculoskeletal disorders is actively recommended, and research on methods for performing non-drug conservative treatment of musculoskeletal disorders is being actively conducted, mainly in the United States and Europe.

On the other hand, as technology develops, the spread of electronic devices (eg, smart phones, tablet PCs, etc.) has become popular, and accordingly, dependence on the Internet is gradually increasing in many parts of daily life.

In this way, thanks to the development of various technologies including the Internet, consumption patterns that were previously highly dependent on offline have gradually moved to online, and now, consumption centered on online is exponentially increasing. is increasing

In line with these trend changes, providing medical services online is also becoming common in industries that are based off-line due to the characteristics of services, such as the medical industry.

Accordingly, in recent years, various medical services have been provided online, and patients, that is, users, can have medical consultations about diseases with a medical staff with just a few clicks through an electronic device connected to the Internet.

As such technology, Korean Patent Registration No. 10-2195512 discloses a technology related to a server and system providing an online medical platform, and provides information about a medical service point through online to a patient.

In line with this trend, there is a need for a method of providing online non-drug conservative treatment of musculoskeletal disorders.

The present invention relates to an exercise therapy providing method and system capable of providing exercise therapy for musculoskeletal disorders on an online basis.

In particular, the present invention relates to a method and system for providing exercise therapy capable of analyzing a motion motion of a patient performing a prescribed exercise based on motion images.

In particular, the present invention relates to a method and system for providing exercise therapy capable of analyzing a patient's exercise motion from a motion image based on an artificial intelligence model specialized for musculoskeletal disorders.

Furthermore, the present invention relates to a method and system for providing exercise therapy capable of providing a user environment in which patients can easily access treatment of musculoskeletal disorders.

In order to solve the problems described above, the exercise treatment providing method according to the present invention includes the steps of receiving prescription information related to exercise for a patient from a doctor terminal, based on the prescription information, the patient Allocating an exercise plan including at least one prescription exercise to an account of the patient, receiving an exercise image of a motion according to the prescription exercise from a patient terminal, using an artificial intelligence posture estimation model, the exercise extracting keypoints corresponding to a plurality of preset joint points from the image; analyzing the relative positional relationship between the keypoints through an artificial intelligence motion analysis model; and based on the analysis of the positional relationship, the prescription motion Analyzing the exercise motion of the patient and transmitting an analysis result of the exercise motion of the patient to the patient terminal.

Furthermore, the method for estimating the provision of exercise therapy according to the present invention includes the steps of outputting the exercise image to the patient terminal in real time in association with the motion image being captured by the patient terminal, and the patient performing the exercise motion of the patient. The method may further include overlapping and providing a graphic object corresponding to the extracted keypoint in an area where a subject corresponding to the patient is located in the motion image so as to be able to recognize a joint point for which analysis is performed for .

In an example, in the step of extracting the keypoint, visible joint points of the subject that are visible in the motion image are specified among the plurality of preset joint points, and the specified visible joint points are selected as the keypoints. can be extracted as

In one example, the motion analysis model predicts an invisible joint point of the subject that is not visible in the motion image among the plurality of preset joint points based on the learning data, and the An exercise motion of the patient may be analyzed based on the visible joint points and the non-visible joint points.

In this case, the keypoints may include keypoints corresponding to the visible joint points and the invisible joint points.

In one example, the learning data is first data in which positional information about visible joint points to be learned and non-visible joint points to be studied that are estimated based on the visible joint points of a subject included in the target image to be learned are sequentially listed. and a second data group including a data value indicating whether each of the visible joint points to be learned and the non-visible joint points to be learned are visible.

In this case, the order in which data values included in the second data group are listed may have the same order as the order in which the visible joint points to be learned and the non-visible joint points to be learned are listed.

In one example, in the step of analyzing the exercise motion of the patient, the relative positional relationship between the key points is analyzed based on the rule information related to the prescribed exercise, and the relative positional relationship between the key points satisfies the rule information. It is possible to analyze the exercise motion of the patient by determining whether or not.

In one example, the visual appearance of the graphic object overlapping the motion image may be different depending on whether the relative positional relationship between the extracted keypoints satisfies the rule information.

In one example, the analysis result of the exercise motion of the patient, in a state in which the motion image is being captured by the patient terminal, the graphic object corresponding to the key point to a different visual appearance based on the rule information, A second analysis result including the patient's evaluation score for the prescribed exercise based on a first analysis result overlapping the exercise image and provided in real time and a key point extracted from each of a plurality of frames constituting the exercise image can include

In this case, the first analysis result is generated by a motion analysis model of an application installed in the patient terminal, and the second analysis result is made in a cloud server that interworks with the application, and the first analysis result and the first analysis result are generated. 2, all of the analysis results may be transmitted to the doctor terminal.

Furthermore, the exercise therapy exercise therapy providing system according to the present invention, based on the communication unit and the prescription information for receiving exercise-related prescription information for a patient from a doctor terminal, the patient's account, A control unit for allocating an exercise plan including at least one prescription exercise, wherein the control unit receives, from a patient terminal, an exercise image obtained by photographing an exercise according to the prescription exercise, and from the exercise image, sets a plurality of preset exercises. Keypoints corresponding to each joint point are extracted, and the relative positional relationship between the keypoints is analyzed through an artificial intelligence motion analysis model, and based on the analysis of the positional relationship, the motion motion of the patient for the prescribed motion is determined. analysis, and an analysis result of the exercise motion of the patient may be transmitted to the patient terminal.

Furthermore, the exercise treatment providing system according to the present invention includes a communication unit that receives prescription information related to exercise for a patient from a doctor terminal and, based on the prescription information, at least one in the patient's account. A control unit for allocating an exercise plan including a prescription exercise of the exercise, wherein the control unit receives an exercise image obtained by photographing an exercise according to the prescription exercise from a patient terminal, and uses an artificial intelligence motion analysis model to perform the exercise. It is characterized in that the exercise motion of the patient for the prescription exercise is analyzed from the video, and the analysis result of the exercise motion of the patient is transmitted to the patient terminal.

Furthermore, a program executed by one or more processes in an electronic device and stored in a computer-readable recording medium receives prescription information related to exercise for a patient from a doctor terminal; Allocating an exercise plan including at least one prescription exercise to the patient's account based on prescription information, receiving an exercise image obtained by photographing an exercise according to the prescription exercise from a patient terminal, the exercise image extracting keypoints corresponding to a plurality of preset joint points, analyzing the relative positional relationship between the keypoints through an artificial intelligence motion analysis model, and based on the analysis of the positional relationship, the prescription exercise It may include instructions for performing the step of analyzing the exercise motion of the patient for the patient and the step of transmitting the analysis result of the motion motion of the patient to the patient terminal.

As described above, the exercise therapy providing system and method according to the present invention receives prescription information related to exercise for a patient from a doctor terminal, and based on the prescription information, the patient's account For example, an exercise plan including at least one prescription exercise may be assigned. Through this, even if there is no face-to-face meeting between a doctor and a patient for exercise treatment for musculoskeletal disorders, the doctor can prescribe to the patient and the patient is provided with an exercise plan according to the doctor's prescription, thereby providing a spatial and temporal improvement in exercise treatment. , can address economic constraints and increase access to exercise therapy.

Furthermore, the system and method for providing exercise therapy according to the present invention extract key points corresponding to a plurality of preset joint points from a motion image, thereby focusing on the joints required for exercise treatment of musculoskeletal disorders, thereby helping the user's health. movement can be analyzed.

Furthermore, the exercise therapy providing system and method according to the present invention is based on a posture estimation model learned using a learning data set including location information on joint points, and specifies a user included in an exercise image. An exercise motion related to the exercise motion may be analyzed. Through this, in the present invention, it is possible to accurately analyze the patient's posture from the motion image, and in particular, it is possible to improve the quality of medical service by obtaining information on the patient's joint range of motion, alignment state, and deviation state. there is.

Furthermore, the exercise therapy providing system and method according to the present invention transmits the analysis result of the patient's exercise motion to the patient terminal, so that the patient receives feedback on the exercise image even if he or she does not directly visit a hospital located at a distance ( feedback) can be provided to improve the exercise treatment effect, and the patient's compliance with exercise can be improved.

1 is a conceptual diagram illustrating an exercise therapy providing system according to the present invention.
2 and 3 are flow charts for explaining a method for providing exercise therapy according to the present invention.
4A and 4B are conceptual diagrams for explaining a doctor's prescription.
5 and 6 are conceptual diagrams for explaining a method of analyzing a patient's exercise motion from a motion image.
7, 8a, 8b, 8c, 8d, 8e, and 8f are conceptual diagrams for explaining an artificial intelligence posture estimation model.
9A, 9B, and 9C are conceptual diagrams for explaining a user environment in which an analysis result of a patient's exercise motion is provided.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same reference numerals will be assigned to the same or similar components regardless of reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

An object of the present invention is to analyze a patient's exercise motion included in an exercise image based on an exercise image received from a patient's terminal, and to provide an analysis result. Particularly, the present invention relates to a method of analyzing a movement motion based on a joint point of a patient using an artificial intelligence model specialized for musculoskeletal disorders.

In the present invention, the description is centered on the motion analysis of rehabilitation exercises for musculoskeletal disorders, but is not necessarily limited thereto. That is, motion analysis in the present invention may include analysis of various motions, such as motions in daily life and motions during stretching, as well as motions during exercise.

On the other hand, the “exercise action” described in the present invention is a gesture (action) made in the process of performing an exercise, and includes “movement”, “action”, “movement”, and “gesture” of the body. It can be used interchangeably with terms such as “.

And, as shown in FIG. 6, the “exercise image” is an image (image or video) in which a process of the patient performing an exercise motion is recorded (including), and at least a part of the body of the patient U may be included. there is.

In the present invention, the patient object included in the motion image may be named and described as “subject U”. In the present invention, “subject (U)” may mean a patient or a body part of a patient exercising in an exercise image. In the present invention, “subject” and “patient” can be used interchangeably, and the same reference numeral “U” can be given for explanation.

Hereinafter, a motion therapy providing method and system using an artificial intelligence posture estimation model and a motion analysis model according to the present invention will be described in detail along with accompanying drawings. 1 is a conceptual diagram illustrating an exercise therapy providing system according to the present invention. 2 and 3 are flow charts for explaining the exercise treatment providing method according to the present invention, FIGS. 4a and 4b are conceptual diagrams for explaining a doctor's prescription, and FIGS. 5 and 6 analyze a patient's exercise motion from a motion image. 7, 8a, 8b, 8c, 8d, 8e, and 8f are conceptual diagrams for explaining an artificial intelligence posture estimation model, and FIGS. 9a, 9b, and 9c are conceptual diagrams for explaining a method for These are conceptual diagrams for explaining a user environment in which a patient's movement motion analysis result is provided.

As shown in FIG. 1 , the motion therapy providing system 1000 according to the present invention analyzes the motion motion of a patient in motion images received from the patient terminal 10 using an artificial intelligence posture estimation and motion analysis model. As such, it may be configured to include at least one of an application (Application, 100) and an artificial intelligence server (or cloud server, 200) installed in the patient terminal 10. In addition, the exercise treatment providing system 1000 according to the present invention may include a posture estimation model and a motion analysis model learned using learning data.

The application 100 in the present invention is installed in the patient terminal 10 to perform a function of analyzing the exercise motion of a patient U having a musculoskeletal disorder and providing feedback information based on the analysis result. can Accordingly, the application 100 according to the present invention is “digital exercise treatment solution”, “digital rehabilitation treatment solution”, “digital exercise evaluation solution”, “non-face-to-face exercise treatment solution”, “non-face-to-face rehabilitation treatment solution”, “non-face-to-face rehabilitation treatment solution” Face-to-face exercise evaluation solution”, “mobile exercise treatment program”, “mobile rehabilitation treatment program”, “mobile exercise evaluation program”, and “mobile orthopedic rehabilitation assistant (MORA) may be named.

The application 100 according to the present invention is installed on the patient terminal 10 to connect a patient with musculoskeletal disorders (U) and an orthopedic surgeon (D) to perform a role of helping the patient (U) rehabilitate. can Hereinafter, for convenience of description, the application 100 installed in the patient terminal 10 will be named and described as “exercise treatment application”.

Meanwhile, the exercise therapy application 100 according to the present invention may be installed in the patient terminal 10 . The patient terminal 10 described in the present invention refers to an electronic device logged into a user account of the patient U, and for example, the electronic device includes a smart phone, a mobile phone, a tablet PC, and a kiosk ( KIOSK), computers, notebooks, digital broadcasting terminals, personal digital assistants (PDAs), and portable multimedia players (PMPs).

Here, the user account of the patient U may mean an account of the patient U previously registered in the exercise therapy providing system 1000 according to the present invention. The user account of the patient U may be understood as a “patient account” or a “patient ID (identification, identification number)”. In the present invention, “patient”, “patient account (or patient user account)”, and “patient terminal” may be used interchangeably.

Meanwhile, the doctor may give a prescription related to exercise to the patient U through the doctor terminal 20 . In the present invention, the doctor terminal 20 may refer to an electronic device logged in with the doctor D's user account. The user account of the doctor D is the account of the doctor D previously registered in the exercise treatment providing system 1000 according to the present invention, and may be understood as a “doctor account” or “doctor ID (identification, identification number)”. can In the present invention, “doctor”, “doctor account (or doctor user account)”, and “doctor terminal” may be used interchangeably.

The doctor D may make a prescription for the patient U by referring to the user DB 30 including user information of the patient U.

User information (or patient information) of the patient U matched to each patient account may exist in the user DB 30 . The user information of the patient U may include various types of information necessary for providing exercise therapy. For example, the user information of the patient U may include at least one of disease information, age information, gender information, surgery history information, exercise plan information, exercise performance information, height information, and weight information of the patient U. can However, the above-described user information of the patient is only an example, and it is natural that the user information of the patient may include various information necessary for providing exercise therapy to the patient.

On the other hand, the motion therapy application 100 described in the present invention is installed in the patient terminal 10, and the exercise motion of the patient who performed the exercise according to the prescription of the doctor (D), the artificial intelligence posture estimation model and the artificial intelligence motion It may be analyzed through an analysis model and provided on the patient terminal 10 .

Then, the motion therapy application 100 may be configured to perform communication with the artificial intelligence server 200, and the motion motion analysis result of the patient analyzed by the artificial intelligence server 200 is displayed on the patient terminal 10. can provide The motion analysis result of the patient analyzed by the artificial intelligence server 200 is generated by at least one of the artificial intelligence motion analysis unit 212 and the rule-based motion analysis unit 213 included in the motion analysis unit 210. It can be.

The exercise therapy application 100 is made to transmit and receive mutual data through wireless communication with the artificial intelligence server 200, and there is no limitation on the wireless communication method. The exercise therapy application 100 according to the present invention may communicate with the artificial intelligence server 200 using a communication module included in the patient terminal 10 . Communication modules included in the patient terminal 10 may be varied.

For example, the communication module provided in the patient terminal 10 includes WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro ( Wireless Broadband), World Interoperability for Microwave Access (WiMAX), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), 5G (5th Generation Mobile Telecommunication), Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct , Wireless USB (Wireless Universal Serial Bus) technology, using at least one, can be made to perform communication with the artificial intelligence server (200).

Meanwhile, the artificial intelligence server 200 described in the present invention may be a cloud server that analyzes the motion motion of the patient U from motion images. The artificial intelligence server 200 may analyze the motion motion of the patient U by using the motion image received from the motion therapy application 100 . The "artificial intelligence server" described in the present invention may be named "artificial intelligence exercise treatment server", "artificial intelligence rehabilitation treatment server", "digital treatment server", and the like. Hereinafter, for convenience of explanation, it will be named and described as “artificial intelligence server”.

On the other hand, at least one of the exercise therapy application 100 and the artificial intelligence server 200 according to the present invention, as shown in FIG. 7, a learned posture estimation model 52 using learning data related to joint points It is possible to analyze the relative positional relationship between the key points P1 and P2 corresponding to the plurality of joint points of the patient U extracted from the motion image 300 through the artificial intelligence posture estimator 121a of Fig. 1). there is. Relative location analysis between keypoints may be performed by the motion analyzers 120 and 210 . In particular, exercise motion analysis may be performed by one of the artificial intelligence motion analysis units 122 and 212 and the rule-based motion analysis units 123 and 213 of the motion analysis unit. One of the artificial intelligence motion analysis units 122 and 212 or the rule-based motion analysis units 123 and 213 may be referred to as an artificial intelligence motion analysis model.

Here, “joint points” may mean a plurality of joints of the patient U (or parts of the body of the patient U including the joints).

Also, “key points” may refer to regions corresponding to each of a plurality of joint points of the subject U in the exercise image 300 .

Accordingly, in the present invention, “joint points” and “key points” may be used interchangeably, and the same reference numerals “P1 and P2” may be given to each of the joint points and key points for description.

The exercise treatment providing system 1000 extracts keypoints P1 and P2 corresponding to joint points from the motion image of the patient using the posture estimation model 52, and positions between the extracted keypoints P1 and P2. Based on the analysis of the relationship, the exercise motion of the patient U may be analyzed. In the present invention, a series of processes of analyzing a patient's movement motion from a motion image using key points extracted through the artificial intelligence posture estimation model 52 may be named "movement motion analysis process".

This exercise motion analysis process may be performed by at least one of the exercise treatment application 100 and the artificial intelligence server 200 . Specifically, the exercise motion analysis process includes i) a first data processing method made by the exercise treatment application 100, ii) a second data processing method made by the artificial intelligence server 200, iii) a exercise treatment application 100 ) and at least one of the third data processing method performed by both the artificial intelligence server 200.

Here, in the third data processing method, data processing may be performed sequentially or concurrently in the exercise therapy application 100 and the artificial intelligence server 200, respectively.

Therefore, in the present invention, it can be explained that the exercise motion analysis process is performed in the exercise therapy providing system 1000 without separately distinguishing a physical space and a subject.

Meanwhile, as shown in FIG. 7 , the exercise motion analysis process may be performed using key points extracted from the artificial intelligence posture estimation model 52 . The artificial intelligence posture estimation model 52 may specify or estimate the patient's joint points from motion images through learning on learning data specific to the joint points, and extract key points corresponding thereto.

In the present invention, learning data for which the artificial intelligence posture estimation model 52 performs learning may be stored and present in the database 40, and this database 40 may also be referred to as “learning data DB”. Details of the learning data will be described later.

As shown in FIG. 7 , the posture estimation server 50 may include at least one of a learning unit 51 and a posture estimation model 52 . The posture estimation server 50 may be provided inside the exercise therapy providing system 1000 according to the present invention or may be configured as an external server. That is, the posture estimation server 50 according to the present invention performs a function of learning about posture estimation, and it can be understood that there is no restriction on physical space. Details of the posture estimation server 50 will be described later along with learning data.

On the other hand, as shown in Figure 1, exercise therapy application 100 according to the present invention, the image receiving unit 110, motion analysis unit 120, image processing unit 130 and control unit 140, at least one configuration can include

The image receiving unit 110 of the exercise treatment application 100 may be configured to receive a motion image including a patient's motion from the patient terminal 10 in which the application 100 is installed. These motion images may be captured by a camera installed in the patient terminal 10 . In the present invention, "receiving an exercise image from the patient terminal 10" means that the image receiver 110 of the exercise treatment application 100 accesses the exercise image recorded in the memory of the patient terminal 10. (access) can be understood.

The motion analysis unit 120 of the motion therapy application 100 may analyze the motion motion (or motion posture) of the patient based on the motion image received from the patient terminal 10 . To this end, the motion analysis unit 120 of the exercise therapy application 100 includes at least one configuration of a keypoint extraction unit 121, an artificial intelligence motion analysis unit 122 and a rule-based motion analysis unit 123 It can be done. The artificial intelligence motion analysis unit 122 or the rule-based motion analysis unit 123 may be named an “artificial intelligence motion analysis model”.

The keypoint extractor 121 may extract keypoints P1 and P2 in the form of pairs of x-axis and y-axis coordinate information from the motion image. In this case, the keypoint extractor 121 may extract keypoints from the image using an artificial intelligence model.

In the present invention, keypoint extraction using an artificial intelligence model can be described as being performed by the artificial intelligence posture estimation unit 121a included in the keypoint extraction unit 121.

The artificial intelligence posture estimator 121a may be referred to as an "artificial intelligence posture estimation model", and uses an artificial intelligence model learned for object detection from images to determine the patient's joint points from motion images. Corresponding keypoints can be extracted. The artificial intelligence posture estimation model may be a model based on object detection. For example, the artificial intelligence posture estimator 121a may extract a key point from a motion image using an object detection artificial intelligence model that ensemble a plurality of bounding boxes. Meanwhile, the artificial intelligence posture estimator 121a may use various object detection artificial intelligence models, and the above-described object detection artificial intelligence model corresponds to an example.

Furthermore, in the present invention, the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 use at least one of the motion image received from the patient terminal and the keypoint extracted by the keypoint extraction unit 120, An analysis of the patient's athletic motion (or athletic posture) may be performed.

More specifically, the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 i) analyze the motion motion of the patient based on the motion image, or ii) analyze the motion motion of the patient based on key points Motion analysis may be performed, or iii) analysis of the motion motion of the patient may be performed using both the motion image and key points.

Hereinafter, for convenience of description, a method of performing motion analysis of a patient based on a key point will be mainly described. However, the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 may receive an exercise image, not a key point, as input data, and perform motion analysis of the patient directly from the motion image. It is natural that there is Meanwhile, the artificial intelligence motion analysis unit 122 or the rule-based motion analysis unit 123 may also be expressed as the aforementioned “artificial intelligence motion analysis model”.

On the other hand, the artificial intelligence motion analysis unit 122, based on the artificial intelligence model (or posture estimation model, see reference numeral 52 in FIG. 7) learned to analyze the motion motion (or motion posture) of the patient from key points , exercise type classification of exercise performed by the patient (or exercise type specification), and accuracy determination of exercise motion can be performed.

Further, the rule-based motion analysis unit 123 classifies the exercise type of the exercise performed by the patient (or identifies the exercise type) and determines the accuracy of the exercise motion based on the rule information defined for analyzing the motion motion of the patient. can do.

Here, “rule information” is information including various rules used for motion analysis, and may include, for example, reference joint motion range information for each motion motion (or motion type). Such rule information may be used interchangeably with terms such as “reference information” and “standard information”.

Furthermore, the rule information includes, in addition to the movable range of the joint, the movable distance of the joint, the movement speed (or acceleration) of the joint, the body balance, body balance, and body alignment state of the subject (corresponding to the patient) included in the motion image to be analyzed ( EX: leg alignment state, spine alignment state, etc.) may include various rule information for analyzing at least one. The rule-based motion analyzer 123 may derive various analysis results from the patient's motion image to be analyzed based on the rule information.

In the present invention, at least one of the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 may perform analysis of the motion motion of the patient from motion images.

Specifically, in the present invention, i) analysis of the patient's exercise motion is performed by the artificial intelligence motion analysis unit 122 (“first analysis method”), or ii) rule-based motion analysis unit 123 Analysis of the patient's motion is performed by ("second analysis method"), or iii) the motion motion of the patient by both the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 analysis may be made ("Third Analysis Method").

Here, in the third analysis method, data processing may be performed sequentially or concurrently in each of the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 .

Meanwhile, the image processing unit 130 of the exercise treatment application 100 overlaps a graphic object corresponding to the extracted keypoints P1 and P2 to the subject U of the patient included in the exercise image 300. Or it can be made to render (rendering). Through this, the patient can intuitively recognize the joint point at which the analysis of the exercise motion of the patient is made.

The control unit 140 of the exercise therapy application 100 may be configured to perform overall control of components included in the exercise therapy application 100 . The control unit 140 of the exercise therapy application 100 may control components of the exercise therapy application 100 by using a central processing unit (CPU) of the patient terminal 10, and furthermore, the patient terminal 10 It is possible to control the provided components (ex: communication module, camera module, sensing module, output module (ex: display, speaker), input module (ex: touch screen, microphone)).

On the other hand, as shown in FIG. 1, the artificial intelligence server 200 is a cloud server configured to perform an exercise posture of a patient using an artificial intelligence posture estimation model, and among the motion analysis unit 210 and the control unit 220 It may include at least one component.

The motion analyzer 210 of the artificial intelligence server 200 may analyze the motion motion (or motion posture) of the patient based on the motion image received from the patient terminal 10 .

The motion analysis unit 210 of the artificial intelligence server 200 may receive motion images of the patient from the motion therapy application 100, and motion image reception is performed by the communication unit (or communication module) of the artificial intelligence server 200. can be done by

The motion analyzer 210 of the artificial intelligence server 200 may include at least one of a keypoint extractor 211, an artificial intelligence motion analyzer 212, and a rule-based motion analyzer 213. . The artificial intelligence motion analysis unit 212 or the rule-based motion analysis unit 213 may be named an “artificial intelligence motion analysis model”.

Each of the keypoint extraction unit 211, the artificial intelligence motion analysis unit 212, and the rule-based motion analysis unit 213 included in the artificial intelligence server 200 is the keypoint extraction unit of the exercise therapy application 100 described above ( 121), the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 may perform the same function. Accordingly, a detailed description thereof will be omitted.

The control unit 220 of the artificial intelligence server 200 may be configured to perform overall control of components included in the artificial intelligence server 200.

Hereinafter, a motor motion analysis process of analyzing a motion motion of a patient U from a motion image and providing a motion motion analysis result will be described using the above configuration of the motion therapy providing system 1000 according to the present invention. .

As shown in FIG. 2 , an exercise prescription for the patient U may be made in the doctor terminal 20 (S210). In the exercise treatment providing system 1000, the doctor terminal 20 provides an exercise prescription for the patient. Based on this, prescription information on the exercise prescription may be received from the doctor terminal 20 .

Based on the prescription information being received from the doctor terminal 20 , the exercise treatment providing system 1000 may allocate an exercise plan including at least one prescription exercise according to the prescription information to the patient's account. The assigned exercise plan may be transmitted to the patient terminal 10 (S220).

The exercise therapy providing system 1000 may include a communication unit that communicates with at least one of the patient terminal 10 , the doctor terminal 20 , the user DB 30 , and the database 40 . For example, the communication unit includes Wireless LAN (WLAN), Wireless-Fidelity (Wi-Fi), Wireless Fidelity (Wi-Fi) Direct, Digital Living Network Alliance (DLNA), Wireless Broadband (WiBro), and World Interoperability for Microwave (WiMAX). Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth ™), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Wi-Fi Direct, Wireless Universal Serial Bus (USB) technologies Communication may be performed using at least one of the above.

Meanwhile, the patient terminal 10 may capture an exercise image of a patient performing a prescription exercise included in an exercise plan (S230). The exercise therapy application 100 may activate a camera provided in the patient terminal 10 to control a motion image to be captured.

A motion image captured by the patient terminal 10 may be used as data to be analyzed (or a motion image to be analyzed) for motion analysis of a patient by the motion therapy providing system 1000 .

As described above, the motion motion analysis process may be performed in at least a part of the motion therapy application 100 and the artificial intelligence server 200 installed in the patient terminal 10, and in the present invention, the motion motion analysis process is performed in the physical It can be described as being made in the exercise therapy providing system 1000 without separately distinguishing the space and the subject.

Meanwhile, the motion therapy providing system 1000 may extract key points P1 and P2 corresponding to a plurality of joint points from the motion image. Extraction of the keypoints P1 and P2 may be performed by at least some of the keypoint extractor 121 included in the exercise therapy application 100 and the keypoint extractor 221 included in the artificial intelligence server 200.

The exercise treatment providing system 1000 may analyze the relative positional relationship between the extracted keypoints P1 and P2 (S250). Further, the exercise treatment providing system 1000 may analyze the exercise motion of the patient U based on the analysis of the positional relationship between the key points P1 and P2 (S260). Such exercise motion analysis may be performed by at least a part of the motion analyzer 120 of the application 100 and the motion analyzer 210 of the artificial intelligence server 200 .

Furthermore, the exercise treatment providing system 1000 may provide the motion analysis result of the patient U to the patient terminal 10 as feedback information and to the doctor terminal 20 as monitoring information. (S270).

As such, the exercise treatment providing system 1000 may perform overall control over the exercise motion analysis process, which may be understood to be performed by the control unit of the exercise treatment providing system 1000 in the present invention. That is, the control unit of the exercise therapy providing system 1000 has a concept including the control unit 140 of the exercise therapy application 100 and the control unit 220 of the artificial intelligence server 200, and You can exercise overall control over

Hereinafter, an exercise motion analysis process performed by the exercise treatment providing system 1000 will be described in more detail.

In the present invention, a process of receiving prescription information related to exercise for a patient from a doctor terminal may proceed (S310, see FIG. 3).

As shown in FIGS. 4A and 4B , the exercise treatment providing system 1000 displays an exercise prescription page (or exercise assignment) including a prescription function related to a patient's exercise on the doctor terminal 20 logged in with a doctor account. page) can be provided. In the present invention, “exercise prescription” may be used interchangeably with “exercise allocation”.

The exercise treatment providing system 1000 may provide an exercise prescription page for each patient account on the doctor terminal 20 so that a prescription may be made for a specific patient U account among patient accounts matched to the doctor account. there is.

For example, in the present invention, it is assumed that a first patient account (ex: “Kim Woo-young” patient account) and a second patient account (ex: “Kim So-hee” patient account) are matched to a specific doctor (D) account. The exercise treatment providing system 1000, based on receiving an exercise prescription request for a first patient account (ex: “Kim Woo-young” patient account) from the doctor terminal 20, displays the first patient on the doctor terminal 20. An exercise prescription page corresponding to the account may be provided.

The exercise treatment providing system 1000 may receive prescription information for a specific patient from the doctor terminal 20 based on user selection (or user input) made on an exercise prescription page corresponding to the specific patient. Prescription information may include various types of information for prescribing exercise to a patient. For example, in the prescription information, i) information on at least one exercise action to be included in the exercise plan (ex: “Calf stretch with a wall”, “Sit down and roll a ball and massage the soles of the feet”), ii) Difficulty information of the exercise action , iii) information on maintenance time of exercise motion, iv) information on the number of times of exercise motion, v) information on schedule for performing motion motion, vi) body information matched to exercise motion (ex: “ankle”, “knee”), vii) precautions It may include at least one of information (ex: “Please apply ice when the exercise is over”) (see (a) of FIGS. 4A and 4B).

The exercise treatment providing system 1000 prescribes a prescription for a specific patient U from the doctor terminal 20 based on input (or selection) of prescription information for the specific patient in an exercise prescription page corresponding to the specific patient. information can be received. In this case, guidance information notifying that a prescription for a specific patient has been made may be output on the doctor terminal 20 (see (b) of FIG. 4 ).

Meanwhile, in the present invention, based on the prescription information, a process of allocating an exercise plan including at least one prescription exercise to the patient's account may proceed (S320, see FIG. 3).

As shown in (a) of FIG. 5 , the exercise treatment providing system 1000 includes an exercise plan (E) including at least one prescription exercise in a specific patient account based on prescription information for a specific patient U. ), and the allocated exercise plan (ex: “digital treatment for femoral patella arthritis”, E) can be provided on the patient terminal 10 logged in with a specific patient account.

Here, “prescription exercise” may be understood as an exercise action specified based on prescription information among a plurality of exercise actions (or exercise types) included in the exercise treatment providing system 1000 and assigned to a patient account. Accordingly, in the present invention, “prescribed exercise” may be used interchangeably with “exercise action”. And, in the present invention, “exercise plan” may be used interchangeably with “digital therapy”.

The exercise treatment providing system 1000 performs a prescription exercise included in the exercise plan for the patient based on receiving a request for providing an exercise plan assigned to a specific patient account from the patient terminal 10 logged in with a specific patient account. An exercise page linked to the exercise guide image providing function may be provided on the patient terminal 10 so as to be able to do so.

As shown in (b) of FIG. 5, the exercise page may include an exercise list (L), and the exercise list (L) includes a plurality of prescription exercises included in an exercise plan assigned to a specific account (ex “ Items (V1 to V6) corresponding to each exercise guide image may be included.

The exercise treatment providing system 1000, when the exercise plan includes a specific prescription exercise (ex: “straight lower extremity lift”) with a plurality of exercise sets, as many as the number of sets (ex: “3”) , Items (V1 to V3) corresponding to the exercise guide image of a specific prescription exercise can be controlled to be included in the exercise list (L).

On the other hand, the exercise treatment providing system 1000, based on receiving an exercise start request from the patient terminal 10, based on the order of the items V1 to V6 included in the exercise list L, a plurality of Exercise guide images may be controlled to be sequentially reproduced on the patient terminal 10 .

Meanwhile, in the present invention, a process of receiving an exercise image obtained by photographing an exercise according to a prescription exercise from a patient terminal may proceed (S330, see FIG. 3).

As shown in FIG. 6 , in the exercise therapy providing exercise therapy providing system 1000, based on the motion guide image being reproduced on the patient terminal 10, the camera provided in the patient terminal 10 controls the patient U ) can be controlled to capture a motion image.

The exercise therapy application 100 installed on the patient terminal 10 controls the activation state of the camera provided in the patient terminal 10 from a deactivated state to an active state so that the camera performs an exercise operation according to the exercise guide image. It may be controlled to capture a movement image of the patient U.

As shown in (a) of FIG. 6, the motion therapy application 100 is a movement image (or A guidance comment (ex: “Please stand within the screen”) may be output on the patient terminal 10 so that all of the contents are contained within a specific area of the display of the patient terminal.

Exercise therapy application 100, on the basis that the subject (U) corresponding to the entire patient's body is included in a specific area, to detect the subject (U) from the image 300 using an object detection (Object Detection) algorithm can

The motion therapy application 100 may use various object detection algorithms. For example, the exercise therapy application 100 may use an algorithm (Weighted Box Fusion, WBF) that ensemble a plurality of bounding boxes. However, it is natural that the exercise therapy application 100 is not limited to the above-described object detection algorithm, and can use various object detection algorithms capable of detecting an object corresponding to the subject U from the exercise image 300 to be learned. do.

Furthermore, the exercise therapy application 100 may capture an exercise image of a patient taking an exercise motion according to a prescribed exercise through a camera based on the detection of a subject U corresponding to the entire patient's body within a specific area. can

In this case, the exercise therapy application 100 may photograph the patient performing the prescribed exercise while the exercise guide image corresponding to the prescribed exercise assigned to the patient is reproduced.

In addition, the exercise treatment application 100 matches the exercise image captured by the camera of the patient terminal 10 to an exercise plan (or each of a plurality of prescription exercises included in the exercise plan), It can be controlled to be written to memory.

Meanwhile, in the present invention, a process of extracting key points respectively corresponding to a plurality of preset joint points from a motion image may be performed (S340, see FIG. 3).

In the present invention, key points P1 and P2 corresponding to preset joint points P1 and P2 may be extracted from an exercise image by at least a part of the exercise treatment application 100 and the artificial intelligence server 200. As described above, the keypoints (P1, P2) extraction is i) by the motion therapy application 100, ii) by the artificial intelligence server 200, or iii) by the motion therapy application 100 and artificial All of the intelligence server 200 can be done. Hereinafter, it will be described that the extracting of the keypoints P1 and P2 is performed by the exercise therapy providing system 1000 without separately distinguishing a subject.

The motion therapy providing system 1000 may extract, from the motion image 300, a region corresponding to a predefined (or preset) joint point among a plurality of joint points of the patient as key points P1 and P2. .

Here, “joint points” may mean a plurality of joints of the patient U (or parts of the body of the patient U including the joints).

Also, “key points” may refer to regions corresponding to each of a plurality of joint points of the subject U in the exercise image 300 .

In the present invention, "joint point" and "key point" can be used interchangeably, and the same reference numerals "P1 and P2" can be given to each of the joint point and key point for description.

On the other hand, the human body is composed of about 200 bones, joints are parts where bones are connected, and the human body may be composed of a plurality of joints.

In the present invention, among a plurality of joint points constituting the human body, joint points that are target key points may be designated in advance and exist as the joint point definition information 500 . For example, in the joint point definition information 500, a first joint point P1 corresponding to the center of the head 510 and a second joint point P2 corresponding to the center of the neck 520 may be predefined and present. Yes (see Fig. 8d).

The exercise treatment providing system 1000 determines the joint points from the motion image 300 based on the posture estimation model 52 learned using a learning data set including position information of preset joint points. Corresponding keypoints (P1, P2) can be extracted.

In this case, the exercise treatment providing system 1000 provides motion images 300 based on the fact that the positional information of each joint point previously set by the posture estimation model is extracted in the form of a pair of x-axis and y-axis coordinate information. ), the positions of the keypoints P1 and P2 can be specified.

Meanwhile, the system for providing exercise therapy 1000 selects keypoints P1 and P2 corresponding to the joint points based on whether the joint points are visible in the motion image 300 through a first keypoint extraction process and a second keypoint extraction process. It can be extracted (or specified) according to any one of the keypoint extraction processes.

In the present invention, whether a joint point is visible may be understood to mean whether the joint point is visible in the motion image 300 .

The system for providing exercise treatment 1000 may determine that the joint points of the motion image 300 are visible when a body part of the subject U corresponding to the joint points is included in the motion image 300 .

When a specific joint point is visible in the motion image 300, the exercise therapy providing system 1000 may extract a keypoint corresponding to the specific joint point according to a first keypoint extraction process.

Specifically, the exercise treatment providing system 1000 may specify a visible joint point of the subject U that is visible in the motion image 300 from among a plurality of preset joint points. For example, when a first joint point and a second joint point among a plurality of preset joint points are visible in a motion image, the system for providing exercise therapy 1000 performs the first joint point and the second joint point. The point and the second joint point may be specified as visible joint points.

In addition, the exercise treatment providing system 1000 may extract the specified visible joint point as a key point.

In this case, the exercise treatment providing system 1000 may extract the keypoint corresponding to the visible joint point by extracting location information of a region (or pixel, pixel) corresponding to the visible joint point in the exercise image. For example, the exercise therapy providing system 1000 may extract keypoints corresponding to the visible joint points by extracting location information of visible joint points using an object detection algorithm.

In the present invention, location information of visible joint points extracted according to the first keypoint extraction process may be described by naming “first type information (first type location information)” or “entity location information”.

On the other hand, the exercise treatment providing system 1000 may determine that the joint points are not visible in the motion image 300 if the body part of the subject U corresponding to the joint point is not included in the motion image 300. there is.

When a specific joint point is not visible in the motion image 300, the exercise treatment providing system 1000 predicts a keypoint corresponding to the specific joint point using the posture estimation model 52 according to the second keypoint extraction process. can be extracted by

The motion treatment providing system 1000, based on the posture estimation model 52, among a plurality of preset joint points, the invisible joint points of the subject U that are not visible in the motion image 300. The location information of can be predicted. In this case, the posture estimation model 52 may predict the positional information of the invisible joint point based on the positional information of the visible joint point.

In the present invention, location information of joint points extracted according to the second keypoint specification process can be described by naming “second type information (second type location information)” or “predicted location information”.

The exercise treatment providing system 1000 may extract (or specify) a keypoint corresponding to an invisible joint point by matching predicted position information of an invisible joint point with a keypoint corresponding to the invisible joint point.

As such, in the present invention, based on the posture estimation model that has performed learning on the location information of joint points according to whether predefined joint points are visible in the motion image 300, different processes are performed. Accordingly, keypoints P1 and P2 corresponding to the joint points may be extracted (or specified). Therefore, in the present invention, it is possible to analyze non-visible joint points that are not visible in motion images.

Meanwhile, the exercise treatment providing system 1000 may extract key points P1 and P2 from the exercise image in real time in association with the motion image being captured by the patient terminal 10 . In addition, the exercise treatment providing system 1000 provides the extracted key points P1 and P2 to the patient terminal 10 in real time so that the patient can intuitively recognize the joint points where the motion motion is analyzed. can do.

Specifically, as shown in (b) and (c) of FIG. 6 , the exercise therapy providing system 1000 interlocks with the motion image 300 being captured by the patient terminal 10, and the patient terminal 10 ), the motion image 300 can be output in real time. In addition, the exercise therapy providing system 1000 may overlap and provide graphic objects corresponding to the extracted keypoints P1 and P2 to an area of the subject U corresponding to the preset joint points. .

Data processing for overlapping and providing keypoint graphic objects to the exercise image 300 may be performed by the image processing unit 130 of the exercise treatment application 100 . The image processing unit 130 places each of the graphic objects corresponding to the extracted keypoints P1 and P2 on the area of the subject U corresponding to the joint points P1 and P2 matched to the keypoints P1 and P2. You can render.

Furthermore, when the position of a preset joint point is changed as the patient performs an exercise operation, the image processing unit 130 overlaps and provides a keypoint graphic object on the area of the subject U corresponding to the changed joint point. can do. That is, the image processing unit 130 may overlap a keypoint graphic object on an area corresponding to a joint point in a motion image so that the location of a joint point that is changed in real time is reflected.

Meanwhile, in the present invention, the relative positional relationship between the keypoints is analyzed from the keypoints extracted through the posture estimation model learned using the learning data related to the joint points, and based on the analysis of the positional relationship, the prescription exercise A process of analyzing the motion of the patient may proceed (S350, see FIG. 3). Relative location analysis between keypoints may be performed by the motion analyzers 120 and 210 . In particular, exercise motion analysis may be performed by one of the artificial intelligence motion analysis units 122 and 212 and the rule-based motion analysis units 123 and 213 of the motion analysis unit.

As shown in (d) of FIG. 6, the exercise therapy application 100 provides, on the patient terminal 10, guidance information for guiding the progress of the analysis (ex: “The result is being calculated” or “Kim Cheol-soo's exercise I will provide the results of motion analysis”) to guide the patient in motion analysis. Hereinafter, a method for analyzing a motion of a patient will be described in detail.

The system for providing exercise therapy 1000 analyzes the relative positional relationship between keypoints P1 and P2 corresponding to a plurality of preset joint points, using keypoints extracted from a posture estimation model learned using learning data. can

The system for providing exercise treatment 1000 determines the relative position between keypoints P1 and P2 corresponding to a plurality of preset joint points by using both keypoints corresponding to visible joint points and keypoints corresponding to non-visible joint points. can be analyzed.

Here, the “relative position between keypoints” refers to another keypoint (eg, second keypoint) based on a specific keypoint (eg, the first keypoint, “P1”) between at least two keypoints (P1 and P2). It can be understood as the location of the keypoint, “P2”).

Hereinafter, for convenience of description, keypoints corresponding to visible joint points will be named “first type keypoints” and keypoints corresponding to non-visible joint points will be named “second type keypoints”. .

The motion therapy providing system 1000 includes i) a relative positional relationship between a plurality of first-type keypoints, ii) a relative positional relationship between a first-type keypoint and a second-type keypoint, and iii) a plurality of second-type keypoints. At least one of the relative positional relationships between keypoints may be analyzed.

In this case, the exercise treatment providing system 1000 may analyze a relative positional relationship between some associated key points among a plurality of joint points based on the type of prescribed exercise performed by the patient.

For example, the exercise treatment providing system 1000 determines the relative position between keypoints corresponding to each of a first joint point and a second joint point among a plurality of joint points when a patient performs a prescription exercise according to a first exercise type. relationships can be analyzed.

For another example, the exercise treatment providing system 1000 may, when the patient performs a prescription exercise according to a second exercise type different from the first exercise type, a first joint point and a third joint point among a plurality of joint points. A relative positional relationship between keypoints corresponding to each point may be analyzed. As a result, this relative positional relationship can be utilized for motion analysis.

Results of motion analysis performed in the exercise therapy providing system 1000 according to the present invention may vary greatly. For example, the motion therapy providing system 1000 determines, from the extracted key points or images, the movable range of the joint, the movable distance, the speed (or acceleration) of the joint, and the subject included in the motion image to be analyzed (corresponding to the patient). At least one of body balance, body balance, and body alignment (EX: leg alignment, spine alignment, etc.) may be analyzed. Meanwhile, in the exercise therapy providing system 1000 according to the present invention, , it is possible to analyze the relative positional relationship between keypoints based on the rule information related to the prescribed exercise.

Here, rule information may be understood as information in which rules are defined in advance in order to analyze relative positional relationships between keypoints.

The exercise treatment providing system 1000 may analyze the exercise motion of the patient by determining whether the relative positional relationship between key points satisfies rule information. Hereinafter, as an example, a method of analyzing a joint movable range based on the relative positional relationship of keypoints and rule information will be described. However, the content described below is only one embodiment of analyzing the patient's motion based on the relative positional relationship of key points and rule information, and in the present invention, various motions of the patient based on the relative positional relationship of keypoints and rule information. can be analyzed.

A range of joint motion analyzed in the exercise therapy providing system 1000 according to the present invention will be described in more detail. The exercise treatment providing system 1000 may analyze a patient's range of motion of a joint according to a relative positional relationship between key points based on rule information on a range of motion of a reference joint associated with a prescribed exercise.

Furthermore, the exercise treatment providing system 1000 may analyze the relative positional relationship between the key points based on rule information related to the prescribed exercise. In addition, the exercise treatment providing system 1000 may analyze the exercise motion of the patient by determining whether the relative positional relationship between the key points satisfies the rule information.

The exercise treatment providing system 1000 extracts a relative positional relationship between related keypoints matched to a specific prescribed exercise from a plurality of continuous frames related to a specific prescribed exercise, and uses the extracted relative positional relationship for a specific prescribed exercise. The patient's joint range of motion may be obtained (or calculated).

Specifically, it is assumed that a motion image is composed of a plurality of frames having a first type corresponding to a first prescription exercise and a plurality of frames having a second type corresponding to a second prescription exercise.

The exercise treatment providing system 1000 uses keypoints extracted from a plurality of frames having a first type when analyzing a patient's exercise motion for a first prescription exercise among a first prescription exercise and a second prescription exercise, The patient's motion can be analyzed.

On the other hand, the exercise treatment providing system 1000, when analyzing the patient's motion for the second prescription exercise, analyzes the patient's motion using key points extracted from a plurality of frames having the second type. can

That is, the exercise therapy providing system 1000 may obtain (or calculate) a patient's motion range for a specific prescription exercise by analyzing keypoint positional relationships for consecutive motions (or postures).

Hereinafter, for convenience of explanation, a plurality of consecutive frames (ie, a plurality of frames having the first type) corresponding to a specific prescription exercise (ex: first prescription exercise) are selected according to the time before and after the frame is formed. They are named “first analysis target frame” and “second analysis target frame”.

Here, the first analysis object frame may be understood as a frame formed temporally before, and the second analysis object frame formed temporally after.

The exercise treatment providing system 1000 may extract key points from each of the first analysis target frame and the second analysis target frame.

In the first analyte frame. A first analysis target keypoint group corresponding to each of a plurality of joint points may be extracted, and a second analysis target keypoint group corresponding to each of a plurality of joint points may be extracted in a second analysis target frame.

The exercise treatment providing system 1000 analyzes the “first positional relationship” between the keypoints included in the first analysis target keypoint group, and performs a first motion analysis of the subject U included in the first analysis target frame. can In addition, the exercise treatment providing system 1000 analyzes the “second positional relationship” between the keypoints included in the second analysis target keypoint group, and analyzes the second motion of the subject U included in the second analysis target frame. can be done

The exercise treatment providing system 1000 may acquire (extract or calculate) a patient's joint motion range for a specific prescription exercise based on the first keypoint positional relationship and the second keypoint positional relationship.

In this case, the exercise treatment providing system 1000 refers to the user DB 30 and considers at least one of age information, gender information, height information, weight information, surgery history information, and musculoskeletal disease information of the patient. range of joint motion can be obtained.

Meanwhile, the exercise treatment providing system 1000 may determine whether the acquired exercise range of motion of the patient satisfies a reference joint motion range corresponding to rule information related to a specific prescription exercise. In the present invention, the rule-based analysis of the motion range of motion of the patient may be performed by the rule-based motion analysis unit 213 of the artificial intelligence server 200 (see FIG. 1), but the rule-based motion analysis unit (213) is not limited to what is analyzed.

In the present invention, rule information on reference joint movable ranges may exist for each of a plurality of exercise types. Such rule information may include information on different reference joint movable ranges according to age, gender, height, weight, and musculoskeletal disease.

The exercise treatment providing system 1000 compares the patient's joint range of motion for a specific prescription exercise with the reference joint range of motion for the specific prescription exercise included in rule information, so that the patient's joint range of motion is the reference range of motion for the joint. You can judge whether you are satisfied or not.

Based on the determination result, the exercise treatment providing system 1000 may provide an analysis result of the patient's exercise motion on the patient terminal 10 as feedback on the prescribed exercise.

Meanwhile, in the present invention, a process of transmitting the analysis result of the patient's exercise motion to the patient terminal may proceed (S360, see FIG. 3).

The exercise treatment providing system 1000 may provide motion analysis results in various ways so that the patient can intuitively recognize the analysis result of the exercise motion and increase the patient's compliance with the exercise.

The exercise treatment providing system 1000 may provide graphic objects corresponding to the key points P1 and P2 by overlapping the exercise images 300 in real time while being captured by the patient terminal 10. Yes (see FIG. 6).

In this case, the system for providing exercise therapy 1000 may place joint movable range information of the patient around the key points P1 and P2 related to the joint movable range.

Furthermore, the exercise therapy providing system 1000 is a keypoint graphic object having different visual appearances (or a positional relationship between keypoints) so that the patient can recognize whether the patient's joint movable range satisfies the reference joint movable range. A corresponding graphic object) may be overlapped with an exercise image and provided.

Furthermore, the visual appearance of the graphic object overlapping the motion image may be configured differently depending on whether the relative positional relationship between the extracted keypoints satisfies the rule information.

For example, when the patient's joint movable range satisfies the reference joint movable range, the graphic object A having the first visual appearance may overlap the exercise image 300 . On the other hand, when the patient's joint movable range does not satisfy the reference joint movable range, a graphic object B having a second visual appearance different from the first visual appearance may overlap the exercise image 300 .

Furthermore, the exercise treatment providing system 1000 provides the patient's evaluation score for the prescription exercise based on the key points extracted from each of a plurality of frames constituting the exercise image 300 (ex: “Kim Woo-young's squat posture is 70 point”) may be provided as a motion analysis result.

Meanwhile, in the present invention, the exercise therapy application 100 and the artificial intelligence server 200 installed in the patient terminal 10 may respectively analyze the exercise motion and generate a motion motion analysis result.

For example, the exercise therapy application 100 may create a first analysis result by overlapping graphic objects corresponding to the keypoints P1 and P2 on the exercise image in real time.

As another example, the artificial intelligence server 200 composed of a cloud server generates, as a second analysis result, an evaluation score of a patient for a prescription exercise based on a key point extracted from each of a plurality of frames constituting an exercise image. can do.

The exercise treatment providing system 1000 provides analysis results for the patient's exercise motion, including the first analysis result generated by the exercise therapy application 100 and the second analysis result generated by the artificial intelligence server 200, It can be provided on the terminal 10.

Meanwhile, the exercise treatment providing system 1000 may transmit the analysis result of the patient's exercise motion to the doctor terminal 20 . Both the first analysis result and the second analysis result may be provided to the doctor terminal 20 .

As described above, in the present invention, various user environments related to providing analysis results are provided so that the patient can intuitively recognize the analysis result of the exercise motion. A description of another embodiment related to providing analysis results will be described later.

On the other hand, as shown in FIG. 7 , the present invention analyzes the exercise motion of the patient U included in the exercise image 300 based on the exercise image 300 received from the patient terminal 10 and analyzes it. to provide results. In particular, the present invention relates to a method for processing and learning a learning data set centered on important joint points in order to analyze a patient's motor motion based on artificial intelligence.

Hereinafter, learning data learned by the posture estimation model in the present invention will be described in detail.

As shown in FIG. 7, the database 40 is a storage in which a learning data set is stored, and may be provided in the exercise therapy providing system 1000 itself according to the present invention or may be configured as an external storage (or external DB). there is. It can be understood that the database 40 according to the present invention suffices as long as the space in which the learning data set is stored, and there is no restriction on physical space.

At least one of a database (Data Base, 40), a posture estimation server 50, and an exercise treatment providing system 1000 may be included.

In the database 40, one learning data for learning the posture estimation model 52 may be stored as a learning data set.

As shown in FIG. 8B, the learning data set 400 in the present invention may be composed of a plurality of data groups 410 to 450 corresponding to different information properties 410a to 450a, respectively. Information included in each of the plurality of data groups 410 to 450 may be extracted and configured from the exercise image 300 including the subject U performing an exercise motion.

Here, “exercise image 300”, as shown in FIG. 8A, is a video (image or video) in which a process of a user performing an exercise motion is captured (included), and at least one of the user U's body some may be included.

In the present invention, a user object included in the exercise image 300 may be named and described as “subject U”. In the present invention, “subject U” may mean a user or a part of the user's body exercising in an exercise image. Accordingly, in the present invention, “subject” and “user” may be used interchangeably, and the same reference numeral “U” may be given for explanation.

Meanwhile, the “exercise image 300” described in the present invention may include “exercise image to be analyzed” and “exercise image to be learned”.

The “movement image to be analyzed” may be a motion image subject to analysis of the posture estimation of the subject U, and the “movement image to be learned” may be understood as a motion image subject to machine learning for the posture estimation model. Here, posture estimation analysis may refer to extracting a key point from an image.

The learning unit 51 may be configured to perform learning for a posture estimation model based on the training target exercise image 300 . The learning unit 51 may learn a posture estimation model using the learning data.

As shown in (a) of FIG. 8B, the learning unit 51 detects a subject U in the training target exercise image 300, and various learning data used for estimating the exercise posture from the detected subject U. can be extracted. Such learning data may be used interchangeably with “information” or “data” or “data value” or “data value”. On the other hand, extraction of learning data may also be performed by means other than the learning unit 51 .

The learning unit 51 may use various algorithms for object detection in order to detect the subject U from the training object exercise image 300 . For example, the learning unit 51 may use, for example, an algorithm (Weighted Box Fusion, WBF) that ensemble a plurality of bounding boxes. However, it goes without saying that the learning unit 51 is not limited to the above-described object detection algorithm, and can use various object detection algorithms capable of detecting an object corresponding to the subject U from the training target exercise image 300. .

The learning unit 51 may classify the extracted learning data into one of a plurality of data groups 410 to 450 corresponding to each of a plurality of different information attributes 410a to 450a.

A plurality of different information attributes 410a to 450a described in the present invention may be predefined and exist, as shown in (b) of FIG. 8B. Also, the plurality of data groups 410 to 450 corresponding to each of the plurality of information properties 410a to 450a may include learning data corresponding to predefined information properties.

For example, i) the data group 410 corresponding to the first information attribute 410a includes joint point location information of the subject U, and ii) the data group corresponding to the second information attribute 420a ( 420) may include information indicating whether the subject U's joint points are visible. And, iii) the data group 430 corresponding to the third information attribute 430a includes information on the photographing direction of the subject U, and iv) the data group 440 corresponding to the fourth information attribute 440a ) includes exercise code information for classifying the exercise action (or exercise type) performed by the subject U, and v) the data group 450 corresponding to the fifth information property 450a, the subject ( U) may include size and center location information of a bounding box.

Here, “joint points P1 and P2” may refer to a user's joint or a region corresponding to a joint of the subject U in the motion image 300 .

The learning unit 51 may create (construct) a data set for the exercise image 300 to be studied by linking the plurality of data groups 410 to 450 extracted from the exercise image 300 to be studied with each other. . Also, the learning unit 51 may store the generated learning data set 400 in the database 40 . The database 40 may be constructed as the database 40 for the posture estimation model 52 based on the storage of the learning data set 400 generated by the learning unit 51 .

Furthermore, the learning unit 51 may perform learning for the posture estimation model 52 based on the learning data set 400 existing in the database 40 . As described above, the training data set 400 may include location information of joint points.

The posture estimation model 52 is a posture estimation model learned using a learning data set including position information about joint points, and can estimate the motion posture of the subject U from the motion image to be analyzed. there is.

Meanwhile, the posture estimation model 52 uses the learning data set 400 generated by the learning unit 51 to extract key points corresponding to the joint points of the subject from the motion image 300, and the artificial intelligence motion analysis unit At least one of 122 and 212 and the rule-based motion analyzer 123 and 213 may analyze the motion motion of a subject in the motion image 300 using the extracted keypoint.

The motion posture of the subject U that can be estimated from the motion image 300 to be analyzed using the key points estimated from the posture estimation model 52 may vary. For example, at least one of the artificial intelligence motion analysis units 122 and 212 and the rule-based motion analysis units 123 and 213 determines i) the position of the joint point with respect to the subject U, ii) the joint movement of the joint point Information on at least one of a range, iii) a movement path of joint points, iv) a connection relationship between joint points, and v) a symmetric relationship between joint points may be estimated and analyzed.

In addition, the motion analysis units 122 and 212 determine the motion distance of the joint, the movement speed (or acceleration) of the joint, and the motion image to be analyzed from the key point or image 300 extracted from the motion image 300 to be analyzed. At least one of body balance, body balance, and body alignment (EX: axis alignment of legs, spine alignment, etc.) of the included subject (corresponding to the patient) may be analyzed.

In the present invention, it is also possible that the posture estimation model 52 includes a learning unit 51. Furthermore, contrary to this, the learning unit 51 may include the posture estimation model 52, and in this case, the learning unit 51 may learn the posture estimation model 52 to perform the posture estimation function. Accordingly, in the present invention, functions performed by the posture estimation model 52 may be described as being performed by the learning unit 51 .

On the other hand, the user terminals 10 and 20 send the user's motion motion analysis result (or motion motion analysis report), which is analyzed based on the keypoints extracted and estimated from the posture estimation model 52, to the user terminals 10 and 20. ) It may be made to perform the service provision as a result of the posture analysis provided to (see FIG. 1).

Here, the user terminals 10 and 20 may be at least one of the patient terminal 10 , the doctor terminal 20 , and a third party terminal.

The exercise therapy providing system 1000 may be configured to perform communication with the user terminals 10 and 20 . In the present invention, it may be understood that the exercise therapy providing system 1000 performing communication is performed by a communication unit of the exercise treatment providing system 1000 .

For example, the communication unit of the exercise treatment providing system 1000 may include Wireless LAN (WLAN), Wireless-Fidelity (Wi-Fi), Wireless Fidelity (Wi-Fi) Direct, Digital Living Network Alliance (DLNA), Wireless (WiBro) Broadband), World Interoperability for Microwave Access (WiMAX), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), 5G ( 5th Generation Mobile Telecommunication ), Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Wi-Fi Direct, It may be made to communicate with the user terminals 10 and 20 using at least one of wireless USB (Wireless Universal Serial Bus) technologies.

On the other hand, the user terminals 10 and 20 described in the present invention refer to electronic devices, such as smart phones, mobile phones, tablet PCs, kiosks, computers, laptops, digital broadcasting terminals, and PDAs ( Personal Digital Assistants), and PMP (Portable Multimedia Player). Furthermore, the user terminals 10 and 20 may be electronic devices to which a user account is logged in, connected to, or registered.

Here, the user account may mean an account previously registered in the exercise therapy providing system 1000 according to the present invention. This user account can be understood as a user ID (identification, identification number).

Meanwhile, in the present invention, a process of receiving an exercise image from the user terminals 10 and 20 may proceed. The exercise therapy providing system 1000 may receive an exercise image 300 in which a user performs an exercise operation is photographed through communication with the user terminals 10 and 20 .

In this case, the exercise image 300 received by the exercise treatment providing system 1000 from the user terminals 10 and 20 may be understood as a motion image to be analyzed, which is a subject of exercise motion analysis for the user.

The exercise treatment providing system 1000 may receive motion images to be analyzed from the user terminals 10 and 20 according to various viewpoints and paths.

For example, as shown in FIG. 1 , the exercise therapy providing system 1000 provides user terminals 10 and 20 based on a graphic object corresponding to “exercise start” being selected from the user terminals 10 and 20 . The camera state may be controlled to be active so that the camera 201 provided in ) captures a motion image to be analyzed. In addition, the exercise treatment providing system 1000 may receive an analysis target exercise image captured by the camera 201 from the user terminal 10 or 20 in real time or based on the user's exercise being completed.

Next, in the present invention, based on key points extracted from a posture estimation model learned using a learning data set including position information on joint points, related to a user's specific exercise motion included in an exercise image. A process of analyzing the exercise motion may proceed.

When the learning unit 51 receives the motion image to be analyzed from the user terminals 10 and 20, based on the posture estimation model 52 learned using the motion image to be studied 300, the motion image to be analyzed A keypoint corresponding to the joint point of the user U included in may be extracted. In addition, at least one of the artificial intelligence motion analysis units 122 and 212 and the rule-based motion analysis units 123 and 213 may analyze the motion motion of the subject in the motion image 300 using the extracted key points. there is.

The posture estimation information of the user U estimated by the learner 51 may include various types of information. For example, the learning unit 51 may estimate positional information of the joint points P1 and P2 of the subject U, and ii) joint movable range information (angle information) of the subject U.

Next, in the present invention, based on the completion of the analysis, a process of providing the user U's exercise motion analysis result related to a specific exercise motion to the user terminals 10 and 20 may proceed.

The exercise treatment providing system 1000 may generate an exercise motion analysis report by processing an analysis result of the user's motion motion. In addition, the exercise treatment providing system 1000 may provide an exercise motion analysis report to the user terminals 10 and 20 .

For example, as shown in FIG. 1 , the motion therapy providing system 1000, in the motion image of the user, at positions corresponding to the joint points P1 and P2 of the user U, the joint points P1, P2) A joint point graphic object corresponding to each may be rendered and provided. Also, the exercise treatment providing system 1000 may display the joint movable range information 221 of the specific joint point P1 around the specific joint point P1.

As such, the exercise treatment providing system 1000 according to the present invention can perform learning for the posture estimation model 52 using the database 40 built based on the exercise image 300 to be learned. . In addition, the user's exercise posture may be estimated using the posture estimation model 52, and a motion analysis result providing service may be performed based on the estimated posture.

The information included in the analysis result may vary. For example, in the analysis result, the range of motion of the joint, the motion distance, the motion speed (or acceleration) of the joint, analyzed from the extracted key point or image, and the body of the subject (corresponding to the patient) included in the motion image to be analyzed. Analysis information on at least one of balance, body balance, and body alignment (EX: leg alignment, spine alignment, etc.) may be included.

Furthermore, this analysis information may further include a score (SCORE), and this score may be an analysis score for the user's exercise motion (or posture). These analysis scores can be calculated based on various methods (EX: analysis based on rules based on predetermined criteria or analysis by artificial intelligence algorithms).

In the database 40 , the learning data set 400 extracted and generated from the training object exercise image 300 by the learning unit 51 may be stored and present.

Hereinafter, the learning data set 400 used for estimating the user's exercise posture will be described in detail.

As shown in FIG. 8A , the learning data set 400 may include data related to the subject U extracted from the training target exercise image 300 .

The learning unit 51 may compose the learning data set 400 by extracting data on the subject U from the training target exercise image 300 .

The learning data set 400 may include a plurality of sub data sets 401 to 403 . In the present invention, the learning data sets 400 and 300a may correspond to higher concepts, and the sub data sets 401 to 403 may be understood as data sets corresponding to lower concepts.

The learning unit 51 extracts data on the subject U from each of the reference frames 301 to 306 selected based on a predetermined criterion among a plurality of frames constituting the training target exercise image 300, , sub data sets 401 to 403 may be configured.

The learning unit 51 may select the reference frames 301 to 306 based on various criteria. The learning target exercise image 300 may be a video or a plurality of static images.

When the learning target exercise image 300 is a video, the learning unit 51 selects reference frames 301 to 306 based on a predetermined time interval (T) among a plurality of frames constituting the learning target exercise image 300 can do. For another example, the learning unit 51 may select the front and back frames as the reference frames 301 to 306 when the change in motion of the subject included in the front and back frames corresponds to a predetermined change amount or more.

The learning data included in the learning data set according to the present invention is obtained from each of the reference frames selected based on a predetermined criterion among a plurality of frames constituting the training object exercise image, and the subject included in the learning object exercise image. It can be composed of the extracted learning data centered on.

Hereinafter, for convenience of description, the “learning data set 400” and the “sub data sets 401 to 403” will not be separately distinguished, and will be described based on the “learning data set 400”. Information included in the learning data set 400 described below may be information included in the sub data sets 401 to 403 . In this case, it may be understood that the learning data set 400 according to the present invention includes a plurality of sub data sets 401 to 403 including information described below.

Meanwhile, as shown in (a) of FIG. 8B, the learning data set 400 may be composed of a plurality of data groups 410 to 460 corresponding to a plurality of different information properties.

The learning unit 51 extracts data corresponding to each of a plurality of information properties from the training target exercise image, classifies (or matches) data having the same information property of the extracted data into the same data group, and sets the learning data. (400) rules can be created.

Here, the plurality of information attributes 410a to 450a may be understood as criteria for distinguishing the type of information necessary for estimating the exercise posture of the subject U from the exercise image 300 to be studied. As shown in (b) of FIG. 8B, in the present invention, a plurality of different information properties (first to fifth information properties, 410a to 450a) may be predefined and present.

The learning unit 51 extracts learning data corresponding to each of the plurality of information properties 410a to 450a from the exercise image to be learned, and classifies the learning data corresponding to the same information property into the same data group for learning. A data set 400 may be created.

Further, the learning unit 51. Based on the association between the plurality of information attributes 410a to 450a, group-specific association between the plurality of data groups 410 to 460 is specified, and learning is performed on the learning data set 400 and the association by group. can do.

Hereinafter, a plurality of data groups and associations for each group will be described in detail.

As shown in FIG. 8C , the first data group 410 among the plurality of data groups 410 to 450 has a position relative to the joint points P1 and P2 of the subject U included in the motion image 300. Information 411 and 412 may be included.

As shown in (a) of FIG. 8C, in the present invention, the joint points P1 and P2 may mean a region of the subject U corresponding to the user's joints in the exercise image 300 to be learned. . And, as shown in (b) of FIG. 5, the location information (411, 412) for the joint points is understood as the location of the region where the joint points (P1, P2) are located in the exercise image 300 to be learned. It can be.

On the other hand, the human body is composed of about 200 bones, joints are parts where bones are connected, and the human body may be composed of a plurality of joints.

In the learning unit 51 , among a plurality of joint points of the subject U, joint points to be learned may be predefined and exist. That is, the “learning target joint point” described in the present invention may be understood as a joint point predefined for learning in the present invention among a plurality of joint points of a user.

As shown in FIG. 8D , in the database 40 , among a plurality of joint points, a joint point to be learned, which is a learning target of a posture estimation model, may be designated in advance and exist as reference information 500 . Also, in the reference information 500, an order for a plurality of joint points to be learned may be predefined.

The first joint point to be learned may be defined as a center of head. More specifically, the first joint point to be studied may be understood as a point that is inferred (predicted or corresponding) to the first cervical vertebrae level.

The second target joint point for learning may be defined as a center of neck. More specifically, the second learning target joint point is the C3-C4 level, which is the center of the Neck Lordotic Curve, and can be understood as the center point at the middle level of No. 1 and No. 3 from the front.

The third target joint point for learning may be defined as a lower end of the neck. More specifically, the third target joint point for learning is the C7-T1 level and can be understood as the center point of a line connecting both clavicle levels.

The fourth joint point to be learned may be defined as the center of the shoulder. More specifically, the fourth learning target joint point is the humerus head, and is a position corresponding to the center of rotation of the continuous motion in the rotational motion of abducting the arm to a position that is the central axis of the rotational motion of the shoulder joint. can be understood as In an image that is not a continuous rotational motion, a point corresponding to information about the predicted position of the center of the shoulder may correspond to the fourth target joint point for learning. And, the fourth joint point to be learned may exist at the center of the left shoulder and the center of the right shoulder, respectively.

The fifth learning target joint point may be defined as the center of the elbow. More specifically, the fifth learning target joint point is a region corresponding to the medial-lateral epicodyle center (Humerus Medial-Lateral Epicodyle) and can be understood as a center point at the level of the elbow. The fifth joint point to be learned may exist at the center of the left elbow and the center of the right elbow, respectively.

The sixth joint point to be learned may be defined as the center of the wrist. More specifically, the sixth joint point to be learned is the radius-ulnar styloid process, which can be understood as a central point at the level of the wrist. The sixth joint point to be learned may exist at the center of the left wrist and the center of the right wrist, respectively.

The seventh joint point to be learned may be defined around the hand. More specifically, the seventh joint point to be studied may be understood as a place corresponding to the 3rd metacarpal head, and may exist in the center of the left hand and the center of the right hand, respectively.

The eighth joint point to be learned may be defined as the center of the hip joint (the center of the Femoral Head). More specifically, the eighth joint point to be studied is a position that is the central axis of hip joint rotation motion and can be understood as a position corresponding to the rotation center of continuous motion in a rotation motion that abducts the leg. In an image that is not a continuous rotational motion, a point corresponding to the predicted location information of the center of the hip joint may be understood as an eighth joint point to be learned. The eighth joint point to be learned may exist at the center of the left hip joint and the center of the right hip joint, respectively.

The ninth learning target joint point can be understood as the center of the knee. More specifically, the ninth joint point to be studied is centered on the Femur Medial-Lateral Epicondyle and can be understood as a central point at the knee level. The ninth learning target joint point may exist at the center of the left knee and the center of the right knee, respectively.

The tenth joint point to be learned may be defined as the center of the ankle. More specifically, the 10th joint point to be studied may be understood as a central point at the ankle level centered on the Medial-Lateral Malleolus. The tenth joint point to be learned may be present at the center of the left ankle and the center of the right ankle, respectively.

The eleventh joint point to be learned may be defined as the center of the foot. More specifically, the 11th joint point to be studied corresponds to the 2nd metatarsal head, and may exist in the center of the left foot and the center of the right foot, respectively.

The twelfth joint point to be learned may be defined as the heel. More specifically, the twelfth joint point to be learned may be present at a level at which the heel touches the floor, respectively, on the left heel and the right heel. The twelfth joint point to be learned may not be visible when the subject U is standing completely frontally in the image, but may be visible when the foot is slightly twisted.

The thirteenth joint point to be studied may be defined as a lumbar curve start (SUP. END OF LORDOSIS). More specifically, the 13th joint point to be studied is a level of 8-10T Spine at the Zypoid Of Sternum level, and can be understood as the center point at the mid level of the average level of 4 times on both sides and the average level of 8 times on both sides.

The fourteenth joint point to be studied may be defined as the center of lordosis. More specifically, the 14th joint point to be studied is approximately at the L2-4 Spine level, and can be understood as a center point at the mid level of the 13th level and the average level of the 8th on both sides.

The fifteenth joint point to be studied may be defined as the end of the lumbar curve (INF. END OF LORDOSIS). More specifically, the 15th learning target joint point is approximately at the S1-2 spine level, and the center point can be understood at the mid level of the 14th level and the average level of the 8th on both sides.

Meanwhile, the first joint point to be studied (P1) may be defined as the center of the head (510), and the second joint point (P2) to be studied may be predefined as the center of the neck (520). In addition, a first order, which is the highest priority order, may be defined in the first joint point to be studied (P1), and a second order, which is subordinate to the second order, may be defined and present in the second target joint point (P2) to be studied. .

In this case, in the order of the joint points to be learned corresponding to the left and right sides of the subject U, the joint points to be learned corresponding to the first side (ex: left side) of the body are the second side (x: right side) of the body. ) may be prioritized over the learning target joint point corresponding to ). For example, the target joint point to be learned (P3) corresponding to the center of the left wrist 530 may be matched by defining a priority order over the joint point to be studied (P4) corresponding to the center of the right shoulder (540).

The learning unit 51 may extract coordinate information as positional information 411 and 412 of each of a plurality of previously designated joint points P1 and P2 to be studied from the motion image 300 to be studied.

Coordinate information may include at least one of 2D and 3D coordinates. When the 2D coordinate information is extracted, the learning unit 120 may extract x-axis and y-axis coordinate information of each of the plurality of target joint points P1 and P2 from the target exercise image 300 . In contrast, when 2-dimensional coordinate information is extracted, the learning unit 120 extracts the x, y, and z-axis coordinate information of each of the plurality of learning-target joint points P1 and P2 from the learning-target exercise image 300 can do.

Coordinate information may be extracted through various methods. In particular, the z-axis coordinate information may be extracted from a camera (ex: RGB camera, etc.) or various types of sensors (EX: distance measurement sensor, etc.). Furthermore, the z-axis coordinate information may be extracted from the learning target image 300 through artificial intelligence algorithms of various methods. When z-axis coordinate information is extracted through an artificial intelligence algorithm, the z-axis coordinate information may be expressed as “estimated” or “predicted”.

On the other hand, the learning unit 51 determines the first data group 410 based on the fact that the positional information 411 and 412 of each of the plurality of target joint points P1 and P2 corresponds to the first information attribute 410a. ) to generate a first data group 410, and to generate a learning data set 400 including the first data group 410.

Looking at the fact that two-dimensional coordinate information (x, y coordinate information) is extracted as an example, the learning unit 121 converts the positional information 411 and 412 of each of the plurality of joint points to be learned (P1 and P2) to the x-axis, The y-axis coordinate information can be extracted in a paired form. The learning unit 51 extracts the positional information “[599, 436]” of the first joint point to be studied (P1), and extracts the positional information “[599, 436]” of the second target joint point (P2) to be learned can do. Also, the learning unit 51 may generate a learning data set 400 composed of the first data group 410 including “[599, 436]” and “[599, 436]”.

The learning unit 51 is based on the positional information 411 and 412 of the joint points P1 and P2 to be studied constituting the first data group 410, the subject included in the exercise image 300 to be studied ( Learning to estimate the positions of the joint points P1 and P2 of U) may be performed.

On the other hand, as shown in (b) of FIG. 8C, the learning unit 51 transfers the positional information 411 and 412 of each of the plurality of subject joint points P1 and P2 to the plurality of subject joint points to be learned. Based on a predefined order between (P1 and P2), the learning data set 400 may be configured (created) by sequentially arranging them within the first data group 410.

As described above, the database 40 may have a predefined order for the plurality of joint points to be learned (P1, P2).

The learning unit 51 refers to the database 40 and provides positional information 411 and 412 of a plurality of joint points to be studied (P1 and P2) in the order corresponding to the joint points to be studied (P1 and P2). The training data set 400 may be created by sequentially arranging the first data group 410 . In addition, the learning data set 400 may be stored in the database 40 to construct the database 40 for posture estimation.

Specifically, as shown in (b) of FIG. 8C, the learning unit 51 performs the first position of the first target joint point P1 corresponding to the first order in the first data group 410. The information 411 may be arranged first, and the second location information 412 of the first joint point to be studied (P2) corresponding to the second order may be arranged following the first location information 411.

On the other hand, the learning unit 51 provides location information 411 and 412 of the joint points to be studied (P1 and P2) based on whether the joint points to be learned (P1 and P2) are visible in the motion image 300. ) may be extracted (or specified) according to any one of the first process and the second process.

In the present invention, whether or not the subject joint points to be studied may be understood to mean whether or not the subject joint points P1 and P2 are visible in the motion image 300 to be studied.

In the present invention, visible joint points in a motion image to be learned are named “visible joint points to be learned”, and invisible joint points in a motion image to be learned are designated as “invisible joint points to be learned”. can be named

The learning unit 51, if the learning target motion image 300 includes body parts of the subject U corresponding to the learning target joint points P1 and P2, the learning target movement image 300 It can be determined that the joint points are visible.

Based on the fact that the learning-target joint points P1 and P2 are visible in the learning-target motion image 300, the learning unit 51, according to the first process, from the learning-target motion image 300, the learning-target joint points ( Location information of the actual location where P1 and P2) are located can be extracted.

In the present invention, the location information of the joint points to be studied (P1, P2) extracted according to the first process will be described by naming “first type information (first type location information)” or “entity location information”. can

On the other hand, if the learning unit 51 does not include the body part of the subject U corresponding to the learning-target joint points P1 and P2 in the motion image 300, the learning object in the learning-target motion image 300 It may be determined that the joint points P1 and P2 are not visible.

Based on the fact that the learning target joint points P1 and P2 are not visible in the motion image 300, the learning unit 51 predicts the expected position of the learning target joint points P1 and P2 according to the second process, , it is possible to extract (or specify) predicted location information.

In the present invention, the location information of the joint points to be studied (P1, P2) extracted according to the second process will be described by naming “second type information (second type location information)” or “predicted location information”. can

As such, in the present invention, the plurality of positional information 411 and 412 included in the first data group 410 are visible ( visible), the extraction process and type information may be defined differently.

Meanwhile, the second process may include various data processing to extract (specify) expected position information of joint points to be learned (P1, P2) that are not visible in the motion image 300.

For example, the learning unit 51 that extracts expected position information according to the second process learns invisible from the exercise image 300 based on the actual position information of joint points to be learned seen from the exercise image 300. Predicted position information of the target joint points P1 and P2 may be predicted.

In this case, the learning unit 51 determines the relationship between the plurality of learning target joint points P1 and P2 visible in the motion image 300 and the learning target joint points P1 and P2 invisible in the motion image 300. Predicted location information may be specified by assigning a weight based thereon.

For example, the correlation between the subject joint points to be learned may be set higher as the order corresponding to the subject joint points P1 and P2 is closer. As for the correlation with the subject joint points to be learned corresponding to the third order, the subject joint points to be learned corresponding to the second order may be set higher than the subject joint points to be learned corresponding to the first order.

For another example, the correlation between the subject joint points P1 and P2 to be studied may be set to be the highest between the subject joint points P1 and P2 corresponding to the left and right sides of the subject U. For example, the correlation with the subject joint point P3 corresponding to the center of the left wrist may be set to the highest for the joint point P3 to be studied corresponding to the center of the right wrist (see (a) of FIG. 8C). .

Furthermore, the learning unit 51 determines joint points to be learned (P1, P2) that are not visible in the motion image 300 based on motion information of an exercise action performed by the subject U in the motion image 300. It is possible to extract the predicted position information of .

In the database 40, motion information including a movement path (eg, movement position, movement direction) of a body (or joint point) according to an exercise operation may be stored and present.

The learning unit 51 refers to the motion information of the database 40 and the location information of the joint points to be learned (P1, P2) visible in the motion image 300, and the joint to be learned that is not visible in the motion image 300. Estimated location information may be specified.

On the other hand, as shown in FIG. 8C , the second data group 420 among the plurality of data groups 410 to 450 is the learning target joint points P1 and P2 of the subject U included in the exercise image 300. ) may be made of data values (value, 421, 422) indicating whether or not to be visible.

As shown in (b) of FIG. 5, the data values of the data 421 and 422 included in the second data group 420 correspond to whether or not the joint points to be learned P1 and P2 are visible. It may consist of one of a first data value (ex: “1”) and a second data value (ex: “2”).

Data having a first data value (ex: “1”) is data indicating that the joint points to be learned (P1, P2) are visible in the exercise image 300, and is a position included in the first data group 410. It can be understood as information indicating that the information is of the first type (actual location information).

The learning unit 51 may extract the first type of location information (actual location information) of the joint points to be studied when the joint points to be studied are visible in the motion image 300 . The learning unit 51 has a first data value (ex: “1”) in the second data group 420 based on the extraction of the first type of location information (actual location information) from the exercise image 300. The training data set 400 can be created (constructed) by including data.

On the other hand, the second data value (ex: “2”) is data indicating that the joint points to be learned (P1, P2) are not visible in the exercise image 300, and is a position included in the first data group 410. It can be understood as information indicating that the information is of the second type (predicted location information).

The learning unit 51 may extract (or specify) the second type of location information (actual location information) of the joint point to be studied when the joint point to be studied is not visible in the motion image 300 . The learning unit 51 extracts (or specifies) the second type of location information (predicted location information) from the exercise image 300, and extracts data having a second data value (ex: “2”), The data set 400 may be created (configured) by being included in the second data group 420 .

On the other hand, as shown in (b) of FIG. 8C, the learning unit 51, the data (or data values, 421, 422) included in the second data group 420, a plurality of joint points to be learned ( P1, P2) are arranged in the second data group 420 in the same order as the predefined order in which the position information 411, 412 of each of the plurality of joint points to be learned is arranged to indicate whether each is visible. , it is possible to create (construct) a learning data set 400.

The learning unit 51 controls the data 421 and 422 indicating whether each of the plurality of joint points to be learned is visible, based on a predefined order among the plurality of joint points to be learned (P1 and P2). 2 can be arranged sequentially within the data group 420.

For example, as shown in (b) of FIG. 8C, the learning unit 51 determines the second data group 420 based on the fact that the first joint point P1 to be learned is visible in the motion image 300. ), the data 421 having the first data value (ex: “1”) may be arranged in the first order corresponding to the first joint point to be learned (P1).

In addition, although in (a) of FIG. 8C, the second learning target joint point P2 is shown as visible in the motion image 300, the second learning target joint point P2 is shown in the motion image 300. Let's assume you can't see it. The learning unit 51 determines the second data value in the second order corresponding to the second target joint point P2 based on the fact that the second target joint point P2 is visible in the motion image 300. Data 422 having (ex: “2”) can be arranged.

Meanwhile, in the present invention, it may be understood that the type of location information included in the first data group 410 is defined by a data value of data included in the second data group 420 .

As shown in (b) of FIG. 5, in the second data group 420, the data 421 arranged in the first order has a first data value (ex: “1”) and is in the second order. Assume that the arrayed data 422 has a second data value (ex: “2”).

In the present invention, based on the fact that the data 421 arranged in the first order in the second data group 420 has the first data value (ex: “1”), the first data group 410 The type of location information 411 arranged in the first order may be defined as the first type of location information (actual location information).

On the other hand, based on the fact that the data 422 arranged in the second order in the second data group 420 has the second data value (ex: “2”), the first data group 410 The type of location information 411 arranged in the order of 2 may be defined as the second type of location information (predicted location information).

On the other hand, the posture estimation model 52 according to the present invention, based on the data values included in the second data group 420, the position information of each of the plurality of joint points to be learned included in the first data group 410 Learning can be performed by setting different learning weights for (411, 412).

Specifically, when the data 421 arranged in the first order in the second data group 420 has a first data value (ex: “1”), the posture estimation model 52 calculates the first data value. Learning may be performed by setting a first learning weight for the location information 411 arranged in the first order within the group 410 .

On the other hand, when the data 422 arranged in the second order in the second data group 420 has a second data value (ex: “2”), the posture estimation model 52 performs the first data Learning may be performed by setting a second learning weight for the location information 412 arranged in the second order within the group 410 .

That is, the posture estimation model 52, based on the data values of the second data group 420, respectively, the first type of location information (actual location information) and the second type of location information (predicted location information) Learning can be performed by setting different learning weights.

In this case, the posture estimation model 52 may set the first learning weight higher than the second learning weight.

Meanwhile, as shown in FIG. 8B, the plurality of data groups 410 to 450 may further include a third data group 430 including information related to a photographing direction of a subject included in the motion image 300. can The third data group 430 may include data values indicating a photographing direction in which the subject U included in the exercise image 300 is photographed.

As shown in (a) of FIG. 8E, the subject U may be photographed in different photographing directions (eg, “front” or “side”).

Here, “capturing direction” may be understood as a direction of an axis of a camera (refer to reference numeral 201 in FIG. 1 ) with respect to the subject U. Here, the camera 201 may be understood as a camera 201 that captures a motion image 300 including the subject U. This camera 201 may include the camera 201 provided in the user terminals 10 and 20 .

The data values included in the third data group 430 shown in FIG. 8E (b) have different data values (ex: “0” or “1”) corresponding to the photographing direction of the subject U. can have Data values included in the third data group may have different data values according to a photographing direction of the subject based on a camera that captures the subject. Hereinafter, in order to avoid term confusion with data values included in the second data group 420, data values corresponding to photographing directions will be named and described as “data object values”.

Data having a first data object value (ex: “0”) may be understood as data representing a first direction (ex: front direction) in which the shooting direction of the exercise image 300 to be studied is (FIG. 8E). see (b)).

The learning unit 51 determines the third data group 430 based on the fact that the photographing direction of the subject U included in the learning target exercise image 300 corresponds to the first direction (ex: front direction). The training data set 400 may be created by including data having a first data object value (ex: “0”) in .

On the other hand, the second data object value (ex: “1”) may be understood as data indicating a second direction (ex: lateral direction) in which the direction of capturing the motion image 300 is different from the first direction. (See Fig. 8e (b)).

The learning unit 51 assigns a third data group 430 to the third data group 430 based on the fact that the photographing direction of the subject U included in the motion image 300 corresponds to the second direction (ex: lateral direction). The training data set 400 can be created by including data having 2 data object values (ex: “1”).

Furthermore, although not shown, a third data object value (ex: “2”) indicates that the shooting direction of the motion image 300 is in a third direction different from the first and second directions (ex: oblique direction). ) can be understood as data representing

The learning unit 51 assigns the third data group 430 based on the fact that the shooting direction of the subject U included in the training target exercise image 300 corresponds to the third direction (ex: oblique direction). Data having a third data object value (ex: “2”) may be included and stored in the database 40 .

Meanwhile, in the first to third directions described in the present invention, the angle between the subject U and the axis of the camera 201 relative to a preset direction (ex: clockwise direction) is within a first range to a third range. It can be understood as a case corresponding to each of the ranges.

For example, the first direction may be understood as corresponding to a second angle range in which an angle between the subject U and the axis of the camera 201 is greater than the first angle in the first angle. The second direction may be understood as corresponding to a third angle range in which an angle between the subject U and the axis of the camera 201 is greater than the second angle. Also, the third direction may be understood as corresponding to a third angle range in which an angle between the subject U and the axis of the camera 201 is greater than the second angle from the second angle.

Meanwhile, the learner 51 converts the angle (or angle value) between the subject U and the axis of the camera 201 based on a preset direction (ex: clockwise direction) to the third data group 430. It can be composed of data values of included data. For example, when the axes of the subject U and the camera 201 form a right angle with respect to the clockwise direction, the learning unit 51 configures the data value of “90” as the data of the third data group 430. can Meanwhile, the learning unit 51 may perform learning for estimating the posture of the subject U by linking the learning data sets 400 with different photographing direction information included in the third data group 430. .

For example, the data included in the third data group 430 includes a first training data set having a first data object value (ex: “0”) and a second data object value (ex: “1”). Assume that there exists a second training data set with . The learning unit 51 links the positional information of the first data group 410 included in the first data set and the positional information of the first data group included in the second data set to each other so as to determine the posture of the subject U. You can learn about estimation.

On the other hand, the learning unit 51, when estimating the exercise posture of the subject U from the motion image to be analyzed captured in the first direction, the learning target captured in the target learning exercise image 300 photographed in the first direction The motion posture of the subject U may be estimated based on the posture estimation model that has been trained on the motion image 300 .

The exercise therapy providing system 1000 according to the present invention is a posture estimation model learned using a learning data set 400 including photographing direction information (data object value or data value) corresponding to the photographing direction of an exercise image to be analyzed. Based on the key points extracted from the key points, it is possible to analyze the movement motion of the subject (U).

Meanwhile, when the motion analyzers 120 and 210 estimate the exercise posture of the subject U from the analysis target motion image 300 photographed in the first direction, the learning target photographed in the first and second directions, respectively. The motion motion of the subject U may be analyzed using posture estimation information estimated from a posture estimation model that has been trained on the motion image 300 .

That is, when the movement motion of the subject U is captured from the analysis target motion image 300 captured in the first capturing direction, the motion analyzer 120 or 210 sets a data object value corresponding to the first capturing direction. Posture estimation information estimated from a posture estimation model that has been trained on a first training data set including a first training data set and a second training data set including a data object value corresponding to a second photographing direction different from the first photographing direction. Using , it is possible to analyze the movement motion of the subject (U). In this case, the posture estimation model 122 may correspond to a posture estimation model in which learning is performed by setting weights for the first training data set.

In this way, the posture estimation model 52 may be learned by considering the photographing direction of the subject through the learning data set having different data values according to the photographing direction of the subject included in the learning target image. Furthermore, the user's motion motion analysis result may be a result of analyzing a specific motion motion of the user based on posture estimation information extracted by considering a photographing direction of the user included in the motion image in a posture estimation model. .

Meanwhile, as shown in FIG. 8B, the fourth data group 440 among the plurality of data groups is an exercise code (Code Code) matched to the exercise motion performed by the subject U included in the exercise image 300 to be learned. ) may be included.

As shown in FIG. 8F, in the database 40, different exercise codes (ex: “502”, “503”, “504”) are included in each of the plurality of different exercise motions 710, 720, and 730. Matching may exist.

The “exercise code” described in the present invention is a data value for distinguishing different exercise motions, and may be used interchangeably with “exercise key”, “action code”, and “action key”.

In the fourth data group 440, the learning unit 51 performs a specific exercise motion (ex: “Standing on one foot and stretching one foot forward”, 710) performed by the subject U included in the exercise image 300 to be studied. The learning data set 400 can be created by including the matched specific exercise code (“502”).

The learning unit 51 may perform learning for posture estimation by linking a plurality of learning data sets 400 including the same exercise code with each other.

For example, suppose that a first learning data set based on a first training object exercise image and a second learning data set based on a second learning object exercise image 300 exist. The learning unit 51 links the first training data set and the second training data set based on the fact that the exercise codes (ex: “502)” included in the first training data set and the second training data set are the same. , learning for posture estimation can be performed.

This exercise code is a fourth data group based on the information received from the user terminals 10 and 20, the exercise action performed by the subject U by at least one of the system manager or the learning unit 51 being specified. (440).

The learning unit 51 may specify an exercise operation performed by the subject U based on information received from the user terminals 10 and 20 . For example, the learning unit 51 moves the camera 201 provided in the user terminals 10 and 20 based on the selection of a graphic object corresponding to “exercise start” from the user terminals 10 and 20 . The state of the camera 201 may be controlled to an active state so as to capture an image.

In this case, the graphic object may correspond to a specific exercise motion, and the learning unit 51 determines that the subject U included in the exercise image 300 received from the user terminals 10 and 20 performs a specific exercise motion. can be judged to have been performed.

Furthermore, the learning unit 51 may specify an exercise operation performed by the subject U based on information input by the system manager.

Furthermore, the learning unit 51 may specify an exercise operation performed by the subject U based on positional information of the target joint of the subject U included in the motion image 300 . In this case, the exercise motion performed by the subject U may be specified by referring to the motion information on the exercise motion stored in the database 40 .

On the other hand, the learning unit 51 matches a plurality of training data sets 400 including the same exercise code based on the exercise codes included in the fourth data group 440 and stores them in the database 40. can

In this case, the learning unit 51 may divide and allocate the memory (or memory space) of the database 40 based on the learning code. In the present invention, dividing the memory (or memory space) of the database 40 can be understood as creating a folder on the database 40 based on exercise codes. Also, the learning unit 51 may store the learning data set 400 including a specific exercise code in a folder corresponding to the specific exercise code.

Meanwhile, the motion analysis units 120 and 210 are estimated from a posture estimation model learned using a learning data set including a motion code corresponding to a specific motion motion performed by the subject U in the motion image to be analyzed. An exercise motion may be analyzed using posture estimation information.

Furthermore, the motion analysis units 120 and 210 provide posture estimation information estimated from a posture estimation model learned based on the training target motion image 300 related to the same specific motion motion as the subject U included in the analysis motion target. It is possible to estimate the motion motion of the subject U for a specific motion motion by using.

Meanwhile, as shown in FIG. 8C, the fifth data group 450 among the plurality of data groups 410 to 450 is a bounding box for the subject U included in the exercise image 300 to be learned. , 301) and location information 452 of the center of the bounding box 301.

In the present invention, “size information” of the bounding box 301 may be used interchangeably with “scale”.

The learning unit 51 extracts size information 451 of the bounding box 301 corresponding to the subject U detected in the training target exercise image 300, and center position information 452 of the bounding box 301. ) is extracted and included in the fifth data group 450 to generate the training data set 400 .

As described above, the learning unit 51 may use various algorithms for object detection in order to detect the subject U from the training object exercise image 300 . For example, the learning unit 51 may use, for example, an algorithm (Weighted Box Fusion, WBF) that ensemble a plurality of bounding boxes. However, it goes without saying that the learning unit 51 is not limited to the above-described object detection algorithm, and can use various object detection algorithms capable of detecting an object corresponding to the subject U from the training target exercise image 300. .

Based on the object detection algorithm, the learning unit 51 extracts size information 451 and center location information 452 of the bounding box corresponding to the subject U from the learning target image 300, and the size information 451 and center location information 452 may be included in the fifth data group 450 to generate the learning data set 400 .

In this case, the learning unit 51 may extract the center position information 452 of the bounding box 301 in the form of a pair of x-axis and y-axis coordinate information.

Meanwhile, the learning unit 51 may configure the learning data set 400 by including image identification information of the training target exercise image 300 .

Here, “image identification information” is information for identifying the image 300 from which the information included in the training data set is extracted, for example, file name information and file format type information of the training target exercise image 300 (or extension information, ex: “JPG”, “TIF”).

In the present invention, the video identification information may be identified as a sixth data group corresponding to the sixth information attribute.

The learning unit 51 may generate the learning data set 400 by including a sixth data group composed of image identification information.

Meanwhile, in the present invention, based on the analysis of the patient's exercise motion being completed based on the motion analyzer 120 or 210, the exercise motion analysis result may be provided to the patient terminal 10. As shown in FIGS. 9A , 9B and 9C , the exercise motion analysis result may be provided through the exercise therapy application 100 installed on the patient terminal 10 .

As shown in (a) of FIG. 9A , the exercise therapy application 100 may provide a service page configured to enable access to each of a plurality of services provided by the present invention on the patient terminal 10 . For example, the service page includes i) an exercise guide page associated with a function of providing exercise guide information on an exercise plan assigned to a patient account, ii) an exercise page related to performing an exercise plan assigned to a patient account (FIG. 9A). (b)), iii) function evaluation page linked to function evaluation (see (c) of FIG. 9a), and iv) plan evaluation page linked to exercise plan evaluation function.

Furthermore, as shown in FIG. 9B , the exercise treatment application 100 may provide an exercise report page on the patient terminal 10 that provides an exercise report based on the exercise motion analysis result and the exercise performance result. For example, the exercise report page may include at least one of exercise performance rate information (see (a) and (b) of FIG. 9b) and exercise plan difficulty information (see (c) of FIG. 9b).

Furthermore, as shown in FIG. 9C , the motion therapy application 100 may provide the patient terminal 10 with a motion analysis result of the patient.

As shown in (a) of FIG. 9C, the exercise therapy application 100 analyzes the movement motion of a plurality of joint points of the subject U performing a specific prescription exercise (ex: “spread arms to the side”). Information can be provided on the patient terminal 10 . For example, exercise therapy application 100, the joint range of motion of the joint point located on the first side (ex: left) of the patient and the joint range of motion of the joint point located on the second side (ex: right) As a graph, it can be presented on the patient terminal 10 .

As shown in (b) of FIG. 9C , the exercise treatment application 100 may provide daily exercise motion analysis results of a patient who has performed a prescribed exercise according to an exercise plan for a certain period of time. For example, the exercise therapy application 100 may provide joint motion range information corresponding to the first exercise day and joint motion range information corresponding to the second exercise day. In addition, the exercise therapy application 100 may provide average ranges of joint motion on the first exercise day and the second exercise day.

As shown in (c) of FIG. 9C, the motion therapy application 100 renders graphic objects corresponding to the keypoints P1 and P2 to the motion image 300 and displays the results on the patient terminal 10. can provide In this case, the motion therapy application 100 may also provide joint motion range information (angle information) of the patient.

On the other hand, in the exercise treatment providing system 1000 according to the present invention, the exercise motion analysis result provided to the patient terminal 10 is provided to the doctor terminal 20 so that the doctor can monitor the execution of the patient's exercise plan. can also be provided.

As described above, the system and method for providing exercise treatment using an artificial intelligence posture estimation model and motion analysis model according to the present invention receive prescription information related to exercise for a patient from a doctor terminal, Based on the prescription information, an exercise plan including at least one prescription exercise may be assigned to the patient's account. Through this, even if there is no face-to-face meeting between a doctor and a patient for exercise treatment for musculoskeletal disorders, the doctor can prescribe to the patient and the patient is provided with an exercise plan according to the doctor's prescription, thereby providing a spatial and temporal improvement in exercise treatment. , can address economic constraints and increase access to exercise therapy.

Furthermore, a system and method for providing exercise therapy using an artificial, intelligent posture estimation model and a motion analysis model according to the present invention extracts key points corresponding to a plurality of preset joint points from the motion image, thereby providing exercise for musculoskeletal disorders. The user's motion may be analyzed by focusing on the joint required for treatment.

Furthermore, a system and method for providing exercise therapy using an artificial, intelligent posture estimation model and a motion analysis model according to the present invention is a posture estimation model learned using a learning data set including position information on joint points. Based on , an exercise motion related to a user's specific exercise motion included in the exercise image may be analyzed. Through this, in the present invention, it is possible to accurately analyze the patient's posture from the motion image, and in particular, it is possible to improve the quality of medical service by obtaining information on the patient's joint range of motion, alignment state, and deviation state. there is.

Furthermore, a system and method for providing exercise therapy using an artificial, intelligent posture estimation model and a motion analysis model according to the present invention transmits analysis results of a patient's exercise motion to the patient terminal, so that the patient is located at a long distance. Even if you do not visit the hospital directly, it is possible to improve the effect of exercise treatment by receiving feedback on the exercise image.

Meanwhile, the present invention described above may be implemented as a program that is executed by one or more processes in a computer and can be stored in a computer-readable medium (or a recording medium).

Furthermore, the present invention described above can be implemented as computer readable codes or instructions in a medium on which a program is recorded. That is, the present invention may be provided in the form of a program.

On the other hand, the computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. there is

Furthermore, the computer-readable medium may be a server or cloud storage that includes storage and can be accessed by electronic devices through communication. In this case, the computer may download the program according to the present invention from a server or cloud storage through wired or wireless communication.

Furthermore, in the present invention, the above-described computer is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit), and there is no particular limitation on its type.

On the other hand, the above detailed description should not be construed as limiting in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (10)

Receiving prescription information related to exercise for a patient from a doctor terminal;
allocating an exercise plan including at least one prescription exercise to the patient's account based on the prescription information;
receiving, from a patient terminal, an exercise image obtained by photographing an exercise according to the prescribed exercise;
extracting keypoints respectively corresponding to a plurality of preset joint points from the exercise image including the subject of the patient using an artificial intelligence posture estimation model learned based on a learning data set;
Analyzing a relative positional relationship between the key points through an artificial intelligence motion analysis model, and analyzing the motion motion of the patient for the prescribed exercise based on the analysis of the positional relationship; and
Transmitting an analysis result of the exercise motion of the patient to the patient terminal;
The posture estimation model learned with the learning data set,
The motion analysis model analyzes the motion motion of the patient based on both visible joint points and invisible joint points of the subject that are visible in the motion image, and the subject from the motion image. Each of the visible joint point and the non-visible joint point of is extracted as the key point,
The learning data set for which the posture estimation model learns,
It consists of a plurality of data groups each corresponding to different information properties,
In a first data group among the plurality of data groups, location information of each of a plurality of pre-designated joint points to be learned among joint points of a subject included in a target exercise image to be learned is defined as a predetermined value corresponding to each of the joint points to be learned. are arranged sequentially based on the order in which
In the second data group among the plurality of data groups, data values indicating whether each joint point to be learned is visible in the exercise image to be learned are displayed in the same order as the location information included in the first data group. are arranged,
The location information arranged in a specific order in the first data group is one of a first type and a second type different from the first type based on data values arranged in the specific order in the second data group. defined as location information,
The location information of the first type is actual location information of an area where the joint point to be studied is actually located in the exercise image to be studied,
The position information of the second type is predicted position information of the joint point to be learned predicted in the exercise image to be learned,
The posture estimation model sets different learning weights for the location information arranged in the specific order of the first data group based on the data values arranged in the specific order of the second data group. A method for providing exercise treatment using an artificial intelligence motion analysis model. According to claim 1,
outputting the exercise image to the patient terminal in real time in association with the motion image being captured by the patient terminal; and
A graphic object corresponding to the extracted key point is overlapped and provided in an area where a subject corresponding to the patient is located in the exercise image so that the patient can recognize the joint point at which the motion posture of the patient is analyzed. A method for providing exercise therapy using an artificial intelligence motion analysis model, further comprising the step of: According to claim 2,
In the step of extracting the keypoint,
Among the plurality of preset joint points, specifying visible joint points visible in the exercise image;
Method for providing exercise treatment using an artificial intelligence motion analysis model, characterized in that for predicting the non-visible joint points that are not visible in the motion image among the plurality of preset joint points. delete delete According to claim 2,
In the step of analyzing the patient's motor motion,
Based on the rule information related to the prescription exercise, analyzing the relative positional relationship between the key points,
A method for providing exercise treatment using an artificial intelligence motion analysis model, characterized in that for analyzing the motion motion of the patient by determining whether the relative positional relationship between the key points satisfies the rule information. According to claim 6,
The visual appearance of the graphic object overlapping the motion image is different depending on whether the relative positional relationship between the extracted keypoints satisfies the rule information. According to claim 7,
The analysis result of the patient's exercise behavior is,
A first analysis result of overlapping and providing the graphic object corresponding to the key point to the exercise image in real time with different visual appearances based on the rule information while the exercise image is being captured by the patient terminal; and
A second analysis result including an evaluation score of the patient for the prescription exercise based on a key point extracted from each of a plurality of frames constituting the exercise image;
The first analysis result is generated by a motion analysis model of an application installed in the patient terminal,
The second analysis result is made in a cloud server that works with the application,
Both the first analysis result and the second analysis result are transmitted to the doctor terminal. A communication unit for receiving prescription information related to exercise for a patient from a doctor terminal; and
A control unit allocating an exercise plan including at least one prescription exercise to the patient's account based on the prescription information;
The control unit,
Receiving an exercise image obtained by photographing an exercise according to the prescribed exercise from the patient terminal;
Using an artificial intelligence posture estimation model learned based on a learning data set, extracting key points respectively corresponding to a plurality of preset joint points from the motion image including the subject of the patient,
Analyzing the motion motion of the patient for the prescribed motion from the motion image using an artificial intelligence motion analysis model,
Transmitting an analysis result of the patient's motion to the patient terminal;
The posture estimation model learned with the learning data set,
The motion analysis model analyzes the motion motion of the patient based on both visible joint points and invisible joint points of the subject that are visible in the motion image, and the subject from the motion image. Each of the visible joint point and the non-visible joint point of is extracted as the key point,
The learning data set for which the posture estimation model learns,
It consists of a plurality of data groups each corresponding to different information properties,
In a first data group among the plurality of data groups, location information of each of a plurality of pre-designated joint points to be learned among joint points of a subject included in a target exercise image to be learned is defined as a predetermined value corresponding to each of the joint points to be learned. are arranged sequentially based on the order in which
In the second data group among the plurality of data groups, a data value representing whether each joint point to be learned is visible in the exercise image to be learned is included in the first data group, and are arranged in the same order,
The location information arranged in a specific order in the first data group is one of a first type and a second type different from the first type based on data values arranged in the specific order in the second data group. defined as location information,
The location information of the first type is actual location information of an area where the joint point to be studied is actually located in the exercise image to be studied,
The position information of the second type is predicted position information of the joint point to be learned predicted in the exercise image to be learned,
The posture estimation model sets different learning weights for the location information arranged in the specific order of the first data group based on the data values arranged in the specific order of the second data group. Exercise therapy providing system to be. A program that is executed by one or more processes in an electronic device and stored in a computer-readable recording medium,
said program,
Receiving prescription information related to exercise for a patient from a doctor terminal;
allocating an exercise plan including at least one prescription exercise to the patient's account based on the prescription information;
receiving, from a patient terminal, an exercise image obtained by photographing an exercise according to the prescribed exercise;
extracting keypoints respectively corresponding to a plurality of preset joint points from the exercise image including the subject of the patient using an artificial intelligence posture estimation model learned based on a learning data set;
Analyzing a relative positional relationship between the key points through an artificial intelligence motion analysis model, and analyzing the motion motion of the patient for the prescribed exercise based on the analysis of the positional relationship; and
Transmitting an analysis result of the exercise motion of the patient to the patient terminal;
The posture estimation model learned with the learning data set,
The motion analysis model analyzes the motion motion of the patient based on both visible joint points and invisible joint points of the subject that are visible in the motion image, and the subject from the motion image. Each of the visible joint point and the non-visible joint point of is extracted as the key point,
The learning data set for which the posture estimation model learns,
It consists of a plurality of data groups each corresponding to different information properties,
In a first data group among the plurality of data groups, location information of each of a plurality of pre-designated joint points to be learned among joint points of a subject included in a target exercise image to be learned is defined as a predetermined value corresponding to each of the joint points to be learned. are arranged sequentially based on the order in which
In the second data group among the plurality of data groups, data values indicating whether each joint point to be learned is visible in the exercise image to be learned are displayed in the same order as the location information included in the first data group. are arranged,
The location information arranged in a specific order in the first data group is one of a first type and a second type different from the first type based on data values arranged in the specific order in the second data group. defined as location information,
The location information of the first type is actual location information of an area where the joint point to be studied is actually located in the exercise image to be studied,
The position information of the second type is predicted position information of the joint point to be learned predicted in the exercise image to be learned,
The posture estimation model sets different learning weights for the location information arranged in the specific order of the first data group based on the data values arranged in the specific order of the second data group. A program stored on a computer-readable recording medium, characterized in that to.

Images