KR102071532B1 - An Apparatus and Method For Exercise Posture Correction Device Using Deep Learning - Google Patents

Abstract

The present invention relates to an exercise posture correction device using deep learning and a method therefor. The exercise posture correction device comprises: a receiving unit for receiving a skeleton-based image in real time from a motion sensor; and a processor configured to calculate a load applied to a joint based on the skeleton-based image received from the motion sensor, wherein the processor includes: a joint detector to detect a motion joint using a multilayer two-dimensional LSTM network from the skeleton-based image; a calculator for calculating a load on the motion joint; a load determination unit that determines the exercise posture by comparing a maximum load threshold with the calculated load; and an exercise posture coaching unit to generate correction information according to the exercise posture.

Description

An Apparatus and Method For Exercise Posture Correction Device Using Deep Learning}

The present invention relates to an apparatus and method for correcting exercise posture using deep learning, and more particularly, to an apparatus and method for correcting exercise posture using deep learning that can enhance individual exercise ability through exercise posture correction using artificial intelligence. It is about.

Recently, modern people use gym or home training to relieve the stress caused by busy life, lose weight, improve health, or correct posture. In particular, weight training, which consumes a lot of energy and does not have much space constraint, is a popular exercise for many people.

However, if you are a beginner using weight training equipment without the help of a professional, you are more likely to exercise in the wrong position, and you are more likely to exercise without knowing whether you are exercising while stimulating your target muscles in the correct posture. Human joints can move and remain stable under the action of surrounding muscles. Therefore, if you exercise for a long time in the wrong posture, the joints are overloaded, or the tissues around the joints may be damaged and the balance of the joints may be compromised, or muscle function may be reduced.

As such, the exercise using the weight training equipment should use the muscle suitable for the exercise type with the correct exercise posture so that the exercise can be performed effectively without any difficulty in the joint.

Therefore, beginners may use a method of personal training with the help of experts. However, even for professionals, it is not easy to correct posture while providing the optimal posture for members with different physical characteristics.

Therefore, there is a need to develop a technology that can help the user correct the exercise posture by analyzing the user's exercise posture and exercise habit using deep learning.

In addition, there is an urgent need to develop a technology that enables users with different physical characteristics to improve exercise ability by exercising in a correct exercise posture.

The problem to be solved by the present invention is to provide an apparatus and method for correcting exercise posture using deep learning capable of correcting the exercise posture according to the physical characteristics of each user.

Another problem to be solved by the present invention is to provide an apparatus and method for correcting exercise posture using deep learning which can minimize the damage to the joint and improve the exercise ability by correcting the wrong exercise posture in real time.

The objects of the present invention are not limited to the above-mentioned objects, and other objects that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, the exercise posture correcting apparatus using deep learning according to an embodiment of the present invention includes a receiver for receiving a skeleton-based image in real time from a motion sensor; And a processor configured to calculate a load applied to the joint based on the skeleton-based image received from the motion sensor, wherein the processor comprises: a joint detector to detect a motion joint using a multilayer two-dimensional LSTM network from the skeleton-based image; A calculator for calculating a load on the exercise joint; A load determination unit that determines an exercise posture by comparing a maximum load threshold with the calculated load; And an exercise posture coaching unit configured to generate correction information according to the exercise posture.

In this case, the multilayer two-dimensional LSTM network detects a motion joint by inputting the corrected joint coordinates from the first layer LSTM network that receives the skeletal-based image to generate corrected joint coordinates and the first layer LSTM network. A second layer LSTM network may be included.

The first layer LSTM network may include a spatial forgetting gate that determines whether the spatial coordinates of the joint coordinates are abnormal, and a temporal forgetting gate that determines whether the joint coordinates are temporally abnormal.

In addition, the second layer LSTM network may calculate the usability information and determine whether to have a motion joint based on the usability information.

The load calculator may determine that the joint is overloaded when the load on the exercise joint is greater than or equal to the maximum load threshold.

The load determination unit may receive a user's weight in advance and set the maximum load threshold differently for each weight.

The exercise posture correcting apparatus may further include an external output device, and the processor may transmit and output the calibration information to the external output device.

On the other hand, exercise posture correction method using deep learning according to an embodiment of the present invention comprises the steps of receiving a skeleton-based image from the motion sensor in real time; And a processing step of calculating a load applied to the joint based on the skeletal-based image, wherein the processing step includes: detecting a joint from the skeletal-based image using a multi-layer two-dimensional LSTM network; Calculating a load on the exercise joint; A load determination step of determining an exercise posture by comparing a maximum load threshold value with the calculated load; And an exercise posture coaching step of generating correction information according to the exercise posture.

Details of other embodiments are included in the detailed description and drawings.

The present invention calculates the torque and weight applied to the joints of the user by comparing and analyzing the model data and the user's data by adjusting the exercise posture so that the user can maintain the exercise posture that suits his / her physical condition You can correct it.

The present invention can minimize the damage to the joint and improve the exercise ability by providing a notification in real time for the wrong posture so that the user can exercise in the correct exercise posture.

The effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is a schematic diagram of an exercise posture correcting system according to an embodiment of the present invention.
2 is a block diagram of an exercise posture correcting apparatus according to an embodiment of the present invention.
Figure 3 is a flow chart for explaining the exercise posture correcting method according to an embodiment of the present invention.
4 is a block diagram illustrating a multilayer two-dimensional LSTM network according to an embodiment of the present invention.
5 is a view for explaining a joint indexing process according to an embodiment of the present invention.
6 is a view for explaining a method of calculating the torque and the angle by the calculating unit according to an embodiment of the present invention.
7 is a flowchart illustrating a method of detecting a motion joint according to an embodiment of the present invention.
8 is a view for explaining a process of obtaining the joint coordinates of the exercise posture correcting apparatus according to the embodiment of the present invention.
9 is a block diagram illustrating a cell of a multilayer two-dimensional LSTM network according to an embodiment of the present invention.
10 is a view for explaining a process of learning the exercise posture correcting apparatus according to an embodiment of the present invention.
FIG. 11 is a diagram for describing a multilayer two-dimensional LSTM network according to another embodiment of the present invention. FIG.
FIG. 11 is a diagram illustrating a result of detecting motion joints of a multilayer two-dimensional LSTM network according to an embodiment of the present invention. FIG.
13 is a view for explaining the detection of the motion joint according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art, although not explicitly described or illustrated herein, can embody the principles of the invention and invent various devices included in the concept and scope of the invention. In addition, all conditional terms and embodiments listed herein are in principle clearly intended to be understood only for the purpose of understanding the concept of the invention and are not to be limited to the specifically listed embodiments and states. do.

In addition, in the following description, ordinal expressions such as first and second are used to describe objects that are equivalent and independent of each other, and in order of the main / sub or master / slave. It should be understood as meaningless.

The above objects, features, and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby the technical spirit of the invention may be easily implemented by those skilled in the art. .

Each of the features of the various embodiments of the present invention may be combined or combined with each other, partly or wholly, various technically possible interlocking and driving as can be understood by those skilled in the art, each of the embodiments may be independently implemented with respect to each other It may be possible to carry out together in an association.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of the exercise posture correcting apparatus according to an embodiment of the present invention. 2 is a block diagram of the exercise posture correcting apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the workout posture correcting system 1000 includes a motion sensor 200, a workout posture correcting device 100, and a database 900. Preferably, the exercise posture correcting system 1000 may further include an external output device 300 such as a glasses display device or a wireless earphone.

The exercise posture correction system 1000 is based on a skeleton-based image obtained using the motion sensor 200 for the purpose of exercise correction and treatment using a long short term memory (LSTM) -based learning model. By calculating the load applied to the exercise joint, it is a system that can determine whether the preferred posture.

Referring to FIG. 1, the exercise posture correcting system 1000 may determine whether the exercise posture is based on a motion sensor 200 generating a skeleton based image in real time and a skeleton based image acquired through the motion sensor 200. The exercise posture correcting apparatus 100 and the database 900 may be included.

The motion sensor 200 is for acquiring a skeleton-based image. For example, the motion sensor 200 may use a Kinect of Microsoft Corporation that provides skeleton information. In this case, the motion sensor 200 may include an RGB camera, an infrared camera, and an infrared projector, and may utilize depth information sensed by using infrared light to images captured by the RGB camera through the infrared camera and the infrared projector. It is possible to obtain a skeleton-based image as shown in (a). However, the motion sensor 200 may provide only images captured from an RGB camera and an infrared camera, and the motion posture correcting apparatus 100 may obtain a skeleton-based image by processing it with an API provided by a motion sensor company. In the present specification, the skeleton-based image is expressed as 'acquisition' or 'receive', including the case where the motion sensor 200 receives the skeleton-based image and only the captured image is processed by the API.

However, due to the noise and occlusion of the skeletal part, the skeletal-based image is incorrectly acquired according to the photographing angle. In FIG. 1, the motion sensor 200 is illustrated as having a separate configuration from the exercise posture correcting apparatus 100, but the motion sensor 200 is physically included in one device with the exercise posture correcting apparatus 100, For example, the mobile phone corresponds to the exercise posture correcting apparatus 100 described above, and the motion sensor 200 according to the embodiment of the present invention may be implemented using the camera of the mobile phone.

The exercise posture correcting apparatus 100 obtains an accurate skeletal-based image and a motion joint corrected based on the obtained skeletal-based image, calculates a load applied to the athletic joint, and compares it with a maximum load threshold to determine whether it is a desirable exercise. Check the status, and print out the calibration information. The configuration and detailed operation of the exercise posture correcting apparatus 100 will be described later.

In addition, in this case, when the external output device 300 is wired at the time of exercise, it may interfere with the exercise, so that the above-described exercise posture correcting apparatus 100 is connected to the wireless communication means such as Bluetooth and WiFi. It is preferable.

On the other hand, the database 900 is a configuration for storing a variety of information from the exercise posture correcting apparatus 100, may store the body image and joint data (JD) received from the motion sensor 200. In addition, the database 900 may store various programs used in the exercise posture correction system 1000.

In this case, the database 900 may be embodied as one device physically with the above-described exercise posture correcting apparatus 100. For example, the exercise posture correcting apparatus 100 may be implemented in a PC, and the database 900 may be implemented in a storage device in the PC. Alternatively, the database 900 may be implemented as a server outside the exercise posture correcting apparatus 100.

Hereinafter, the exercise posture correcting apparatus 100 will be described in detail with reference to FIG. 2. As shown in FIG. 2, the exercise posture correcting apparatus 100 includes a receiver 110, a controller 120, a processor 130, and a display 140.

The receiver 110 may receive a skeleton-based image captured by the motion sensor 200.

The processor 131 may perform original image correction, motion joint detection, load calculation, state determination, and coaching guide generation based on the skeleton-based image received from the receiver 110 under the control of the controller 120. More specifically, referring to FIG. 2, the processor 131 may include a joint detector 130, a calculator 132, a load determiner 133, and a posture coach 134. The various information processed by the processor 131 is displayed and output through the display unit 140 or the external output device 300 of the exercise posture correcting apparatus 100, and the controller 120 corrects the exercise posture correcting apparatus 100. Each component of the can be controlled.

In this case, the joint detector 130 detects the motion joint from the skeleton-based image received from the receiver 110. The joint detector 130 detects the motion joint using, for example, a multilayer two-dimensional LSTM network as disclosed in FIG. 4. The motion joint detection operation of the joint detector will be described later.

The calculator 132 calculates the load on the exercise joint by calculating torque of the exercise joint based on the motion joint detected from the skeleton-based image, and the load determiner 133 compares the calculated exercise load with the maximum load. . In this case, when the calculated exercise load is greater than or equal to the maximum load, it may be determined that the exercise posture is incorrect.

The load state of the joint may be determined by the operation of the calculator 132. In addition, the posture coach may coach the exercise posture based on the load state of the user joint.

As described above, when the processing of the plurality of joint information is completed by the processor 131, the information on the exercise posture correction may be displayed through the display 140. Meanwhile, in order to deliver more vivid coaching information to the user, the exercise posture correcting apparatus 100 may transmit coaching information through the external output device 300. For example, when the external output device 300 is a glasses display, it may be output in the form of an image, and when the external output device 300 is a wireless earphone, it may be output in the form of sound. Or it may be output in a mixed form of the image and sound.

In general, in determining the exercise posture, people use a personal training method, which is coached by an expert with the help of a professional, but even an expert provides the optimal exercise posture for a large number of people with different physical characteristics. There was a difficulty in correcting posture.

The exercise posture correcting apparatus 100 according to an embodiment of the present invention may determine the load state of a joint through modeling of a body image photographed in real time and a pre-stored reference image, and display a correct exercise posture in real time through a display. You may be provided with guidance.

In addition, the exercise posture correcting apparatus 100 according to an embodiment of the present invention compares the body image photographed in real time from the motion sensor 200 with a reference image stored in advance in the database 900 to determine the joint state quickly. Exercise posture correction is possible.

In addition, the exercise posture correcting apparatus 100 according to an embodiment of the present invention can calculate the load by calculating the torque on the joint of a person's exercise to accurately measure the load applied to the exercise joint, and thus, the joint It can provide effective exercise method without overwhelming.

Hereinafter, to describe the operation of the processor 130 according to an embodiment of the present invention, reference is made to FIGS. 3 to 6 together.

Figure 3 is a flow chart for explaining the exercise posture correcting method according to an embodiment of the present invention. 4 is a block diagram illustrating a two-dimensional multilayer LSTM network included in the joint detector 131 according to an embodiment of the present invention. 5 is a view for explaining a joint index of the joint detector 131 according to an embodiment of the present invention. 6 is a view for explaining the operation of the load calculation unit according to an embodiment of the present invention.

First, the skeleton-based image captured in real time using the motion sensor 200 is assigned to the joint index, and input to the multi-layer two-dimensional LSTM network (S100).

The skeleton-based image is basically provided by a motion sensor as an image as shown in FIG. That is, the skeleton-based image includes a joint 510 and a joint connecting line 520, and the joint detecting unit 131 assigns an index 530 for each joint as shown in FIG. 4 (a).

The index 530 of FIG. 5 (a) is numbered in a tree structure as shown in FIG. 5 (b) in order to easily derive the relationship between the joints and the joints during the calculation. It may be indexed as shown in FIG. 5 (b), but in order to improve reliability, the same joints may be overlapped with each other as shown in FIG.

The skeleton-based image is input to the input layer of the first layer LSTM network 410 as a two-dimensional input as shown in FIG. 8.

On the other hand, based on the input two-dimensional input, using the two-dimensional multilayer LSTM network to detect the motion joint (S200). In this case, the two-dimensional multilayer LSTM network includes a first layer LSTM network 410 and a second layer LSTM network 420, and specific operations thereof will be described later.

On the other hand, after the motion joint is detected, the load calculation unit 132 calculates the load of the detected motion joint. (Step S300)

For example, referring to FIG. 5, FIG. 5 shows torques and weights applied to respective joints moving while the user moves. In this case, the load calculator 132 may calculate torques for the plurality of joint data JDs detected in the body image, respectively.

In this case, the load calculation unit 132 first calculates the reaction force at each joint as shown in Equation 1 below.

(Equation 1)

Where R _j is the reaction force at the j joint and W _L is the weight of each segment of the human body

On the other hand, the load calculation unit 132 calculates the load (torque) of each joint as shown in Equation 2 below.

(Equation 2)

Where M _j is the torque at the joint, jCM _L is the length from the joint to the center of gravity of the segment, θ _j is the angle with the segment from the horizontal plane of the joint to the right, and JJ-1 is the length of the segment. do.

After the calculation of the load calculator, the load determiner 133 compares the calculated load with the maximum load threshold. (Step S400)

The maximum load threshold is determined by the load computed based on skeletal-based images of a fitness trainer or exercise specialist who takes a good workout posture. First, the maximum exercise for each joint is calculated through exemplary exercise cases, and if the user enters the weight and then exercises, the user is warned to correct when the maximum load is over. In this way, heavier exercises such as deadlifts and bench presses can be corrected.

Finally, the posture coaching unit 134 generates the calibration information based on the determination result of the load determination unit 133, and transmits the generated calibration information to the display or external output device of the exercise posture correcting apparatus 100 and outputs the calibration information. Done. (Step S500) For example, when the external output device 300 is a spectacle display, it may be output in the form of an image, and in the form of sound when the external output device 300 is a wireless earphone. Or it may be output in a mixed form of the image and sound.

Therefore, according to the present invention, there is provided an exercise posture correcting apparatus and method using deep learning that can perform exercise posture correction according to physical characteristics for each user.

In addition, by correcting the wrong posture in real time to minimize the damage of the joint and at the same time can improve the exercise ability.

Hereinafter, a joint detecting operation according to an exemplary embodiment of the present invention will be described in detail with reference to FIGS. 7 to 10.

7 is a flowchart illustrating a joint detection operation according to an embodiment of the present invention. 8 is a block diagram illustrating a two-dimensional input according to an embodiment of the present invention. 9 is a diagram illustrating a cell structure of an LSTM network according to an embodiment of the present invention. 10 is a block diagram illustrating learning of an LSTM network according to an embodiment of the present invention.

을 구하고 전역 컨택스트 메모리에 업데이트 한다. (단계 S210)According to FIG. 7, in the joint detecting operation according to the embodiment of the present invention, first, the input two-dimensional input is input to the first layer LSTM network 410, and an initial value is obtained.

Obtains and updates the global context memory. (Step S210)

More specifically, the two-dimensional input is defined as shown in FIG. The horizontal axis, which is a spatial input, is determined according to an index given to a joint of a skeleton-based image, and the vertical axis, which is a temporal input, is determined according to a time frame of a skeleton-based image or a skeleton-based image of unit time. Accordingly, the two-dimensional inputs listed as shown in FIG. 8 are input to the first layer LSTM network 410.

On the other hand, each cell of the first layer LSTM network 410 has a structure as shown in Figure 9, if it is represented by the equations (3) to (5) below.

(Equation 3)

(Equation 4)

(Equation 5)

Here, the W function means an affine transformation of parameters, and the parameters are learned in a manner described later. And, ⊙ is a Hadamard Product operation that represents the product of each element, the input gate i _{j, t} ⊙u _{j, t} is a gate that serves to 'remember the current information'.

One of the features of the present invention, f ^T _{j, t} , f ^S _{j, t} is a gate for 'forgetting past information', and in the present invention, f ^T _{j, t} and temporal forgetting gate are spatial forget gates. We use two forget gates, f ^S _{j, t} (forget gate). In this case, by using two spatial and temporal forgetting gates, the current data in that case (in the image of the joint) is abnormal when the spatial relationship is abnormal or the temporal connection is abnormal in comparison with the previous step in space and time. Coordinates) are configured to be blocked without being output. Here, the space is defined based on the index 530 described above, and the time is defined based on the frame of the image.

Meanwhile, referring to FIG. 9, the difference between the prediction value of the next cell and the current input value is reflected in the cell. Specifically, p _{j, t} is a value of the activation function tanh as the prediction gate, and is a value of tanh of the current input value of x ' _{j, t} . Therefore, the confidence gate τ _{j, t} reflects the difference between the predicted value and the current input value. This is a feature of the present invention, and the value of the confidence gate is reflected by the forgetting gate and Hamadad product calculation, it is configured to determine whether to block based on the trust gate value.

p _{j, t} , x ' _{j, t} , τ _{j, t} can be found by the following equation.

(Equation 6)

(Equation 7)

(Equation 8)

M _x means an affine transformation, and the G function is an element-specific operation, which is calculated as in Equation 9 below.

(Equation 9)

Here, λ is an element for controlling the spread of the Gaussian function.

Finally, as represented by Equation 2 above, C is a variable representing a cell state in LSTM, including spatial and temporal forgetting gates, confidence gates and input values, spatial previous outputs (h _{j-1, t} ), The output value h _{j, t} is output based on the temporal previous output h _{j, t-1} .

는 j,t에서 예측한 움직임을 나타낸다.Finally, the first layer LSTM network 410 is trained to minimize the objective function. Here, the first layer LSTM network 410 trains the network to minimize the difference between the actual motion and the predicted motion by using a Softmax layer. The objective function (L) is determined by the following equation, y is calculated by averaging the movements of all points,

Denotes the motion predicted by j, t.

(Equation 10)

는 y와

의 차이를 측정하는 음의 로그우도 손실(negative log-likelihood loss)이다. 위와 같은 방법으로 h_j,t가 얻어지면, 이 결과 값들을 이용하여 아래의 초기값

을 구하고, 전역 컨텍스트 메모리(Global Context Memory, 420)에 초기값으로서 저장된다.At this time,

With y

Negative log-likelihood loss that measures the difference between If h _{j, t} is obtained in the same way as above _, then the initial value of

Is obtained and stored as an initial value in the global context memory 420.

(Equation 11)

On the other hand, h _{j, t} is input to the input layer of the second layer LSTM network 420. The second layer LSTM network 420 detects the joints of interest (motor joints) using the usability scores r ⁽ⁿ⁾ _{j, t} for each. (Step S220)

In this case, the usability score r ⁽ⁿ⁾ _{j, t} is found from the following equations (12) and (13).

(Equation 12)

(Equation 13)

The second layer LSTM network 420 is basically similar to the first layer LSTM network, but uses the usability score to update the cell state value as shown in Equation 14 below.

(Equation 14)

In this case, as can be seen from the above equation, the current data of interest (exercise occurring) is outputted by the usability score, and the current data of interest is blocked.

In addition, the global context memory 430 is updated with a value improved by Equation 15 below.

(Equation 15)

Here, F ⁽ⁿ⁾ is a value stored in the global context memory 430 which is updated when the operation of the n th layer LSTM network is repeated.

In the classifier 440 using the softmax layer as shown in Equation 14, the corresponding joint is classified as a moving joint. (Step S230)

(Equation 16)

Then, again, using Equation 10, the log likelihood loss function learns the parameters of the minimum value. On the other hand, as shown in Fig. 9, the improved F ⁽ⁿ⁾ is stored in the global context memory, and the calculated result value h _{j, t} is inputted to the second layer LSTM network 420 again so that the second layer LSTM network ( The learning of 420 is repeated.

At this time, it is natural and intuitive to directly optimize the entire network as shown in FIG. 10 (a), but it is difficult to implement in a situation where two modules interact with each other as in the embodiment of the present invention. Step by step optimize your network. Since the network is optimized in stages, in the n + 1 stage, the network up to n stages can be optimized more efficiently and effectively.

12 shows the results learned by the second layer LSTM network. If it is detected by the second layer LSTM network only once (Attention iteration # 1), it can be seen that the motion joint was detected by the joint of the leg in the operation of taking a self-camera of the mobile phone. It shows that the upper arm is represented by the motion joint.

In addition, it can be seen that the joint that moves accurately in the posture of placing the object as the motion joint. Finally, even in the kicking scene, the knees and legs can be accurately detected by the exercise joint. If the repetition continues, more accuracy can be improved, but it is desirable to repeat the appropriate number of times for efficiency.

Hereinafter, an example of determining by utilizing two streams will be described with reference to FIG. 11. In another embodiment, the motion joint was detected using two layers of LSTM networks. In this embodiment, one layer (the third layer LSTM network 450) is further added.

In this case, the operation of the third layer LSTM network as a whole is basically the same as the second layer LSTM network.

However, the difference is that when the body is divided into 5 parts (left and right arms / legs, torso, and JDG in FIG. 11) and the above Equation 10 is applied, the joints corresponding to each part are not represented without all the joints being substituted. The difference is that the values of interest (eg average coordinate values) are used to extract the joint of interest. The global context memory of the second layer LSTM network is referred to as the detailed context of context memory, and the global context memory of the third layer LSTM network is referred to as the schematic context of memory.

In this case, more accurate motion classification is possible by utilizing two streams of the second layer LSTM network as the detailed interest stream and the third layer LSTM network as the general interest stream.

Hereinafter, with reference to FIG. 13 will be described an example of the motion joint detection and load determination during the squat exercise.

Referring to FIG. 13, as a reference image having exemplary joint data (JD) for squat motion, a reference image is disposed on three-dimensional coordinates and then a skeleton image is detected. At this time, the weight information of the weight training equipment held by the user of the reference image is input, and the maximum load corresponding to the weight information is set. For example, assuming that the maximum load on the knee joint of a user holding 70 kg of equipment is 140 kg, the weight twice the weight of the user can be regarded as the threshold of the maximum load on the knee joint.

Thus, when the user enters the weight and exercise, the posture coaching part informs the user and corrects the posture when a load of more than twice the weight of the current equipment is applied to the knee joint. In FIG. 11, the exercise is limited to squats. However, if the exercise is performed using heavy equipment such as deadlifts and bench presses, the present invention may be variously applied.

Therefore, the exercise posture correcting apparatus 100 according to an embodiment of the present invention calculates torque and weight applied to a user's joint and compares and analyzes the exemplary data and the user's data to change the exercise according to the user's physical condition. Even if you can maintain the correct exercise posture to fit you can correct the posture.

In addition, the exercise posture correcting apparatus 100 according to an embodiment of the present invention provides a notification about a wrong posture in real time so that a user can exercise in a correct exercise posture, thereby minimizing damage to a joint and at the same time enhancing exercise ability. You can.

Further, the system according to the present invention can be embodied as computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. The computer readable recording medium may include a magnetic storage medium (eg, a ROM, a floppy disk, a hard disk, etc.) and an optical reading medium (eg, a CD-ROM, DVD, etc.). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications can be made without departing from the spirit of the present invention. . Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

1000: exercise posture correction system
100: exercise posture correction device
110: receiver
120: control unit
130: joint detection unit
131: processing unit
132: arithmetic unit
133: load determination unit
134: posture coaching unit
140: display unit
200: motion sensor
900: database
JC: Joint Coordinates
JD: Joint Data
JDG: Joint Data Group

Claims (15)

Receiving unit for receiving a skeleton-based image in real time from the motion sensor; And
It includes a processing unit for calculating the load applied to the joint based on the skeleton-based image received from the motion sensor,
The processing unit,
A joint detector for detecting a motion joint using a multilayer two-dimensional LSTM network from the skeletal-based image;
A load calculator configured to calculate a load on the exercise joint;
A load determination unit that determines an exercise posture by comparing a maximum load threshold with the calculated load; And
An exercise posture coaching unit for generating correction information according to the exercise posture,
The multilayer two-dimensional LSTM network receives a first layer LSTM network that receives the skeletal-based image to generate corrected joint coordinates, and a second that detects a motion joint by inputting the corrected joint coordinates from the first layer LSTM network. Including Layer LSTM Network,
Exercise posture correcting device using deep learning. delete The method of claim 1,
The first layer LSTM network includes a spatial forgetting gate that determines whether the spatial coordinates of the joint coordinates are abnormal and a temporal forgetting gate that determines whether the joint coordinates are temporally abnormal.
Exercise posture correcting device using deep learning. The method of claim 1,
The second layer LSTM network calculates usefulness information, and determines whether or not a motion joint is based on the usefulness information.
Exercise posture correcting device using deep learning. The method of claim 1,
The load determination unit, when the load on the exercise joint is greater than the maximum load threshold, determines that the overload occurs in the joint,
Exercise posture correcting device using deep learning. The method of claim 5,
The load determination unit,
Receiving the user's weight in advance and setting the maximum load threshold differently for each weight;
Exercise posture correcting device using deep learning. The method of claim 1,
The exercise posture correcting apparatus further includes an external output device, and the processor transmits and outputs the calibration information to the external output device.
Exercise posture correcting device using deep learning. Receiving a skeleton-based image in real time from a motion sensor; And a processing step of calculating a load applied to a joint based on the skeleton-based image.
The processing step may include: detecting a joint using a multi-layer two-dimensional LSTM network from the skeletal-based image;
A load calculation step of calculating a load on the exercise joint;
A load determination step of determining an exercise posture by comparing a maximum load threshold value with the calculated load; And
An exercise posture coaching step of generating correction information according to the exercise posture,
The multilayer two-dimensional LSTM network receives a first layer LSTM network that receives the skeletal-based image to generate corrected joint coordinates, and a second that detects a motion joint by inputting the corrected joint coordinates from the first layer LSTM network. Including Layer LSTM Network,
Exercise posture correction method using deep learning. delete The method of claim 8,
The first layer LSTM network includes a spatial forgetting gate that determines whether the spatial coordinates of the joint coordinates are abnormal and a temporal forgetting gate that determines whether the joint coordinates are temporally abnormal.
Exercise posture correction method using deep learning. The method of claim 8,
The second layer LSTM network calculates usefulness information, and determines whether or not a motion joint is based on the usefulness information.
Exercise posture correction method using deep learning. The method of claim 8,
The load determination step,
When the load on the exercise joint is greater than or equal to the maximum load threshold, it is determined that the overload occurs in the joint,
Exercise posture correction method using deep learning. The method of claim 12,
The load determination step,
Receiving the user's weight in advance and setting the maximum load threshold differently for each weight;
Exercise posture correction method using deep learning. The method of claim 8,
The processing step is to transmit and output the calibration information to an external output device,
Exercise posture correction method using deep learning. A computer-readable recording medium having recorded thereon a program for performing the method according to any one of claims 8 to 10.

Images