KR20220058790A - Exercise posture analysis method using dual thermal imaging camera, guide method for posture correction and computer program implementing the same method - Google Patents

Abstract

The present invention relates to a method for a computing device operated by at least one processor to analyze an exercise posture, comprising the steps of: receiving an image of a user's exercise motion through a dual thermal imaging camera; combining a confidence map generated based on the RGB channels of the dual thermal imaging camera and thermal image channels, and acquiring key joint position information of the user from the combined confidence map; extracting a joint region in which a temperature change estimated by the thermal image channel is greater than or equal to a temperature threshold value as a body stimulation region while a user performs an exercise operation; and matching main joint position information and body stimulation regions. The present invention is to provide an analysis method and apparatus for confirming a user's exercise posture and a used muscle part based on an RGB image and a thermal image of a dual thermal imaging camera.

Description

Exercise posture analysis method using dual thermal imaging camera, posture correction guide method using the same, and a computer program implementing the same

The present invention relates to a technique for providing an exercise posture analysis and posture correction guide.

Recently, as the importance of exercise posture has been emphasized, technologies for detecting a person with various equipment and estimating the posture to provide information about the correct exercise posture are being developed.

In detail, technologies are being developed that estimate a person's posture from an image captured by a camera, tell whether it is a correct exercise posture based on the estimated posture, or show a trainer's exercise posture.

However, the existing exercise system is only for external posture correction, and there is a limit in determining whether force is applied to the muscle part the user wants to exercise.

Therefore, there is a need for a technique capable of determining whether a load is accurately applied to an exercise part in order to prevent an exercise injury and to suggest and correct an accurate exercise posture to the user.

An object to be solved is to provide an analysis method and apparatus for identifying a user's exercise posture and muscle parts used based on the RGB image and thermal image of the dual thermal imaging camera.

An object to be solved is to provide a method and apparatus for identifying a user's exercise posture and body stimulation area, and inducing the user to an accurate exercise posture so that the body area according to the exercise posture is stimulated.

According to an embodiment of the present invention, there is provided a method of analyzing an exercise posture by a computing device operated by at least one processor. Combining the confidence map generated based on the RGB channel and the thermal image channel, obtaining the user's main joint position information from the combined confidence map, the temperature change estimated by the thermal image channel while the user performs an exercise motion is the temperature extracting a joint region equal to or greater than a threshold as a body stimulation region, and matching main joint position information with the body stimulation region.

In the obtaining of the main joint position information, each confidence map displayed as a heat map may be generated for each point estimated as the joint position by using the open pose API for the RGB channel and the thermal image channel, respectively.

The step of obtaining the main joint position information includes calculating the distance between the dual thermal imaging camera and the user based on the input key information of the user, and calculating the disparity for the RGB channel and the thermal image channel based on the distance. and applying a viewing interval to the estimated joint position of the laceration channel to move it to the RGB channel.

In the step of acquiring the main joint position information, the average is calculated by adding the brightness value of the joint position point of the thermal image channel moved to the RGB channel and the brightness value of the joint position point of the RGB channel at the same point, and it is added to the combined confidence map. can be displayed

In the matching step, when information about the movement is received, the position of the main joint, the target movement area (stimulation area), the movement amount of the joint, the distance between the joints, the ratio, the angle, and the center point of the user for each movement posture included in the movement movement. Match at least one or more of , and store it in an interworking database.

The method may further include inputting main joint position information including coordinates of joint points, distances, and angles between joints for each movement into the neural network to learn correct posture coordinates for each movement.

According to an embodiment of the present invention, as a program stored in a computer-readable storage medium and executed by a processor, an RGB image and a thermal image of a user's exercise posture while providing a target image for a selected exercise motion operation of collecting user's joint position information and body stimulation area based on RGB image and thermal image image, joint position information stored in the metadata of the target image with the obtained user's joint position information and body stimulation area and instructions for executing an operation that provides feedback for maintaining the exercise posture if they match compared to the body stimulation region, and provides feedback for changing the posture or body stimulation region if they do not match.

The operation to acquire is to generate a confidence map displayed as a heat map for each point estimated as a joint position using the open pose API for RGB image and thermal image image, respectively, and match the confidence map to the two-dimensional vector space to locate the joint position. information can be obtained.

The acquisition operation is to generate a confidence map for each RGB image and a thermal image, calculate the disparity for the RGB image and the thermal image, and apply the calculated viewing distance to combine them into one confidence map. can

The acquisition operation calculates the distance between the dual thermal imaging camera and the user based on the received user's key information, the focal length of the camera, and the size of the image formed on the sensor, and based on the distance, RGB according to the distance between the camera lenses and the focal length It is possible to calculate the viewing distance for the image and the thermal image.

The acquiring operation may include acquiring a joint region in which a temperature change estimated by a thermal image is greater than or equal to a temperature threshold while performing an exercise operation as a body stimulation region.

It trains a neural network based on continuous movement movements included in the target image, inputs the user's joint position information and body stimulation region to the learned neural network, and extracts the user's correct posture and body stimulation region for the next exercise operation. It may further include instructions for executing the operation.

The extraction operation compares the user's joint position information and the body stimulation area acquired in real time based on the user's correct posture and body stimulation area obtained through the neural network to calculate the accuracy of the exercise posture and the body stimulation area to provide feedback. can provide

According to the embodiment, by providing a posture correction guide by checking not only the correct exercise posture but also the muscle parts stimulated in the user's exercise movement, it is possible to increase the user's understanding of the exercise movement and maximize the effect obtained from the exercise movement.

According to an embodiment, by analyzing the user's movement motion using both the RGB channel and the thermal image channel, it is possible to improve the false positive rate for recognizing a hidden part or an external object similar to a human shape and improve the accuracy of recognizing a person. .

1 is a structural diagram of a system for providing guide information by analyzing an exercise posture according to an embodiment of the present invention.
2 is a block diagram illustrating a computing device according to an embodiment of the present invention.
3 is an exemplary diagram for explaining a process of combining a confidence map according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a confidence map for a person and an object other than a person according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a configuration for analyzing a body stimulation region based on a thermal image according to an embodiment of the present invention.
6 is a flowchart illustrating a process of analyzing an exercise posture according to an embodiment of the present invention.
7 is a flowchart illustrating a process of providing feedback on a user's exercise posture according to an embodiment of the present invention.
8 is an exemplary diagram illustrating a feedback providing screen according to an embodiment of the present invention.
9 is a hardware configuration diagram of a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as “…unit”, “…group”, and “module” described in the specification mean a unit that processes at least one function or operation, which is implemented by hardware or software or a combination of hardware and software. can be

The devices described in the present invention are composed of hardware including at least one processor, a memory device, a communication device, and the like, and a program to be executed in combination with the hardware is stored in a designated place. The hardware has the configuration and capability to implement the method of the present invention. The program includes instructions for implementing the method of operation of the present invention described with reference to the drawings, and is combined with hardware such as a processor and a memory device to execute the present invention.

As used herein, “transmission or provision” may include not only direct transmission or provision, but also transmission or provision indirectly through another device or using a detour path.

In this specification, expressions described in the singular may be construed in the singular or plural unless an explicit expression such as “a” or “single” is used.

As used herein, "exercise motion" may include not only an operation formed by a routine in which exercise postures are continuously followed, but also maintaining a specific exercise posture for a certain period of time.

1 is a structural diagram of a system for providing guide information by analyzing an exercise posture according to an embodiment of the present invention.

As shown in FIG. 1 , the dual thermal imaging camera 100 for photographing the user and the computing device 200 for analyzing the exercise posture from the captured images of the dual thermal imaging camera 100 and executing a program for providing guide information In addition, the user terminal 300 for displaying the exercise image and guide information is connected to each other through a network to transmit and receive data.

Here, the network may be composed of a wired network, a mobile communication network such as a long term evolution (LTE) network/5G network, various types of wireless networks such as a Wi-Fi network, and combinations thereof.

The dual thermal imaging camera 100 is a dual camera configured with a thermal imaging (IR) and RGB (EO) camera, and provides a thermal imaging image and an RGB image for a single shooting scene. In other words, the dual thermal imaging camera 100 provides a thermal image acquired through a thermal image channel and an RGB image acquired through an RGB channel for the same scene.

The dual thermal imaging camera 100 transmits a thermal imaging image and an RGB image captured by a real-time user on the basis of a photographing signal received by interworking with the computing device 200 or the user terminal 300 .

Then, the computing device 200 collects images taken from the dual thermal imaging camera 100, estimates the user's joint estimated position with a confidence map, and predicts the connection relationship between the joints to estimate the user's posture .

In addition, the computing device 200 may estimate, as a body stimulation region, a region in which a large temperature change appears from the thermal image according to the estimated posture.

In other words, the computing device 200 may estimate a muscle region stimulated according to a process in which the user maintains an exercise posture or repeatedly performs a predetermined posture, based on a change in temperature.

The computing device 200 may match the exercise posture and the body stimulation area estimated from the RGB image and the thermal image collected for a professional athlete such as a trainer, and store it in a separate database.

At this time, the exercise image, exercise posture analysis data, and body stimulation area for the trainer may be input to an artificial intelligence model for quick and accurate analysis later and used as learning data.

In addition, the computing device 200 may provide the user with a target image selected by the user based on the pre-stored exercise image, and may provide feedback information by analyzing the exercise posture through the user's thermal image and RGB image. Here, the feedback information includes various feedbacks such as posture correction, guidance of a body stimulation area, and posture maintenance.

The user terminal 300 has an interface capable of outputting video or audio to the outside, and provides guide information received from the computing device 200 through the interface.

The user terminal 300 may receive user information, such as height, weight, etc., to request a more accurate analysis of the exercise posture, and transmit it to the computing device 200 .

In detail, the user terminal 300 may be a terminal provided by a trainer for providing an exercise image and accurate information on an exercise posture, or a terminal provided by a user who receives an exercise image and requests a guide for the exercise posture.

The user terminal 300 is a tablet PC (Tablet PC), a laptop (Laptop), a personal computer (PC, Personal Computer), a smart phone (Smart Phone), a personal information terminal (PDA, Personal Digital Assistant), a mobile communication terminal (Mobile) Communication Terminal) and may include various terminals such as a notebook (NoteBook).

Meanwhile, in FIG. 1 , the dual thermal imaging camera 100 , the computing device 200 and the user terminal 300 are displayed as separate devices, but each of the dual thermal imaging camera 100 , the computing device 200 and the user terminal 300 may be implemented in one computing device.

Hereinafter, a computing device that analyzes images received from the dual thermal imaging camera 100 and provides guide information using FIG. 2 will be described in detail.

2 is a block diagram illustrating a computing device according to an embodiment of the present invention.

As shown in FIG. 2 , the computing device 200 includes an image collecting unit 210 , a joint position information generating unit 220 , a body stimulation region generating unit 230 , an exercise posture information generating unit 240 , and an exercise posture. It includes a matching unit 250 and a guide providing unit 260 .

For explanation, an image collection unit 210 , a joint position information generation unit 220 , a body stimulation region generation unit 230 , an exercise posture information generation unit 240 , an exercise posture matching unit 250 , and a guide providing unit ( 260), but these are computing devices operated by at least one processor. Here, the image collecting unit 210, the joint position information generating unit 220, the body stimulation region generating unit 230, the exercise posture information generating unit 240, the exercise posture matching unit 250 and the guide providing unit 260 are It may be implemented in one computing device or distributed in separate computing devices. When distributed in a separate computing device, the image collecting unit 210 , the joint position information generating unit 220 , the body stimulation region generating unit 230 , the exercise posture information generating unit 240 , the exercise posture matching unit 250 . ) and the guide providing unit 260 may communicate with each other through a communication interface. The computing device may be any device capable of executing a software program written to carry out the present invention, and may be, for example, a server, a laptop computer, or the like.

The image collecting unit 210 collects images of the subject's movement posture through two channels of the dual thermal imaging camera. Here, the two channels represent an RGB image and a thermal image, respectively.

The joint position information generator 220 extracts main joint position information from the collected RGB image and thermal image.

For example, the joint location information generating unit 220 may generate a confidence map using the open pose API for major joints that move according to the user's movement motion captured in the image.

In this case, the joint location information generating unit 220 generates a confidence map for each of the two channels, and combines the two confidence maps to generate one confidence map.

In the generated confidence map, a portion predicted to have a human key point is displayed as a heatmap. In addition, the joint position information generator 220 matches an affinity field that forms a two-dimensional vector space with respect to an input image and a key point of the confidence map to predict the connection relationship between the joints. Create joint position information.

And the joint position information generating unit 220 generates one confidence map through the following equations (1) and (2).

[Equation 1]

where Z is the distance from the trainer (user) to the object, H is the real world user height (in cm), F is the focal length of the RGB camera (focal length, in mm), and w is the captured RGB Indicates the user size in the image (object height in image sensor, in mm).

Here, w may be calculated as a value obtained by dividing the pixel coefficient per millimeter of the RGB image by the user size in the RGB image.

[Equation 2]

Here, d is a disparity value when looking at one object in two channels (thermal channel and RGB channel), B is the distance between the camera lenses (baseline), and F is the focal length of the RGB camera (focal). length ).

As such, the joint position information generator 220 may calculate the viewing distance d and apply it to the thermal image confidence map to project it to the RGB confidence map.

Meanwhile, the joint position information generator 220 may generate a Gaussian map by adding a Gaussian weight based on the joint point. In this case, an individual Gaussian map may be generated for each joint point.

The joint position information generator 220 finds a local maxima in the generated Gaussian map. At this time, if the local maxima is less than a certain reference value, it is removed, and the remaining local maxima can be calculated as the final joint.

Next, the body stimulation region generating unit 230 may extract a region having a large temperature change measured over a certain period of time from the thermal image and estimate the region as a body stimulation region.

The body stimulation region generating unit 230 extracts a change in brightness according to an exercise motion based on a thermal confidence map in which the user's joint region is displayed in brightness.

The body stimulation region generating unit 230 may estimate the change in brightness in the thermal confidence map as a change in temperature, and may estimate the stimulated region through this change.

For example, the size of 3x3 pixels is extracted by extending up, down, left and right based on the pixels corresponding to each joint position coordinate extracted from the RGB image, and the average value of 9 pixels corresponding to the extracted 3*3 pixel size is calculated as the joint position It can be calculated as a temperature value for Accordingly, when the corresponding temperature value changes by more than the threshold value, the corresponding joint location area may be estimated as the body stimulation area.

The body stimulation region generating unit 230 may take an image before the start of a specific exercise and an image until the end of the exercise, extract the joint portion with the greatest temperature change, and store it in the DB.

The exercise posture information generating unit 240 matches and stores the joint position information and the body stimulation region generated by each joint position information generating unit 220 and the body stimulation region generating unit 230 for each exercise posture of each exercise motion.

When the user is a trainer, the exercise posture information generator 240 may store joint position information and a body stimulation region together as metadata based on an exercise motion image.

In detail, the movement posture information generating unit 240 may store the position of the main joint, the target movement portion (stimulation region), the movement amount of the joint, the distance between the joints, the ratio, the angle, the center point of the user, and the like for each analyzed movement posture.

When the exercise posture matching unit 250 generates the user's joint position information and the exercise posture information corresponding to the body stimulation area, the joint position information of the trainer corresponding to the same exercise posture and the body stimulation area are compared with each other.

The exercise posture matching unit 250 first compares the user's joint position information to compare the external postures, and if the body stimulation area for a predetermined time is estimated, the corresponding body stimulation area is compared.

In addition, the guide providing unit 260 provides feedback for guiding the user to the correct exercise posture with respect to a portion different from the user's exercise posture and the trainer's exercise posture information.

Hereinafter, a process in which the computing device 200 analyzes the exercise motion of the trainer and builds a database using FIG. 3 will be described in detail.

3 is an exemplary diagram for explaining a process of combining a confidence map according to an embodiment of the present invention.

As shown in (a) of FIG. 3 , the dual thermal imaging camera 100 is a dual camera and a photographing module having an interval (B). When a thermal image and an RGB image are photographed, the computer device 200 ( Create the same confidence map as b).

The computer device 200 projects one channel to the other channel in consideration of the physical difference in order to match the viewpoint coordinates of the two cameras having different physical positions, and finds the coordinates having the highest confidence among the coordinates. .

In detail, the computing device 200 projects the point of coordinates (x:100, y:75) in the thermal image channel to the RGB channel by applying the calculated disparity D (Disparity).

Accordingly, the coordinates (x:100, y:75) in the thermal channel move to the coordinates (x:540, y:75).

4 is an exemplary diagram illustrating a confidence map for a person and a non-human object according to an embodiment of the present invention, and FIG. 5 is a configuration for analyzing a body stimulation area based on a thermal image according to an embodiment of the present invention. It is an example diagram shown.

As shown in (a) of FIG. 4 , when an object having a shape similar to a person is located adjacent to a person, as in (b), a head joint part is detected from both a head part of a person and an object similar to a person.

Looking at the generated RGB confidence map, it can be seen that a certain brightness value can be obtained even from an object similar to a person, but a relatively low brightness value is obtained corresponding to the brightness value of the actual human head.

Therefore, in order to eliminate false positives caused by human-like objects, a false positive threshold for the RGB confidence map is set. The false positive threshold can be set as an intermediate value between the brightness value obtained from an object and the brightness value obtained from a person.

However, since these brightness values change based on environmental changes or other external factors, it is inconvenient to readjust the false positive threshold each time to accurately remove false positives.

On the other hand, since non-human objects do not radiate heat in the same way as humans, they are not displayed in the thermal confidence map generated based on the thermal image.

Accordingly, the computer device 200 has a false positive rate that recognizes a human-like object like a human, but since it is formed based on the high-accuracy RGB confidence map and heat, the false-positive rate is low but the accuracy is somewhat lacking. Combined to improve accuracy while improving the false positive rate for recognizing non-human objects.

In detail, the computer device 200 moves the brightness values expressed in the thermal image confidence map to the RGB confidence map by utilizing the calculated disparity between the RGB channel and the thermal image channel.

Then, the computer device 200 adds the overlappingly expressed brightness values, calculates an average value, and displays the average value on the combined confidence map.

Through this, the brightness value for a person displayed in both the RGB confidence map and the thermal confidence map has a relatively larger value, but since the brightness value of an object is not displayed in the thermal confidence map, when the average value is calculated, a relatively lower value have

As in the combined confidence map in Fig. 4(b), it can be seen that the brightness value for a person has a relatively large value, but the brightness value for an object is lowered, so that the difference for each brightness value is large. there is.

Through this, the computer device 200 may increase the recognition rate and accuracy of a person by excluding points having a specific brightness value by using a false positive threshold indicating a constant brightness value without being affected by changes in the environment.

Fig. 5 (a) shows the human joint shape derived from the RGB image, and (b) is the human joint shape derived from the thermal image.

It can be seen that the degree of heat generation is different based on a person's posture in (b) of FIG. 5 . In general, when the human body is exerting force in a specific area, heat is generated due to energy consumption in the specific area.

Through this, it can be seen that the brightness value of the thermal image is strongly displayed in the area where the force is applied, which is the stimulation area.

Accordingly, in FIG. (b), it can be seen that stimulation is strongly applied to the joint portion of the ankle in the standing posture of the user.

By the brightness value of the thermal image, the computing device 200 can know the joint region to be stimulated, and when a large temperature change occurs in a part of the user's body as the repetition of the motion or the duration of the motion progresses, the body stimulates the body due to the motion. area can be estimated.

Hereinafter, a process of analyzing an exercise posture through an image captured by the dual thermal imaging camera 100 using FIG. 6 will be described in detail.

6 is a flowchart illustrating a process of analyzing an exercise posture according to an embodiment of the present invention.

As shown in FIG. 6 , the computing device 200 collects images of the trainer's exercise motions ( S110 ).

The computing device 200 collects a thermal image and an RGB image by photographing an exercise motion of the trainer with the dual thermal imaging camera 100 . In this case, the computing device 200 may collect basic information about the trainer. For example, a trainer name, career, height, weight, exercise motion name, effect of the corresponding motion, etc. may be collected.

Then, the computing device 200 generates a confidence map in which joint points are displayed in the RGB channel (S120). The computing device 200 generates a confidence map in which joint points are displayed in the thermal image channel (S130).

Steps S120 and S130 can be implemented at the same time, and depending on the situation, step S130 may be performed first and then step S120 may be performed.

Next, the computing device 200 combines the RGB confidence map and the thermal confidence map into one (S140).

At this time, due to the photographing module in a physically different position in one dual thermal imaging camera, a viewing distance between the RGB channel and the thermal imaging channel occurs.

Therefore, the computer device 200 may calculate a viewing interval for correcting the physical position of the camera, and project it onto each confidence map using the inconsistency to combine them.

In detail, the computing device 200 calculates the distance between the dual thermal imaging camera 100 and the trainer using the information of the dual thermal imaging camera itself (the focal length of the camera, the size of the image formed on the sensor, etc.) and the trainer's key information. Calculate.

In addition, the computing device 200 calculates a parallax with respect to where the coordinates (x1, y1) in the thermal image are located in the RGB image based on the distance between the dual thermal imaging camera 100 and the trainer.

For example, the brightness values of the thermal confidence map are moved to the RGB confidence map, the brightness values of the RGB confidence map and the brightness values of the thermal confidence map are summed, and the average is obtained, and the brightness value can be displayed on the combined confidence map.

As such, the computer device 200 may apply a viewing interval to the thermal image confidence map and combine it with the corresponding RGB confidence map.

And the computing device 200 acquires the main joint position information from the combined confidence map (S150). In the combined confidence map, it is possible to obtain the position of each joint, angle information according to the connection relationship between the joints, length degree, and the like.

The computing device 200 extracts a region in which the temperature change is equal to or greater than the threshold value for each continuous exercise posture in which the exercise operation is performed ( S160 ).

The computing device 200 extracts a region having the greatest temperature change from the captured thermal image. At this time, it is possible to check the temperature change corresponding to the joint area.

For example, the computing device 200 may extend the pixel corresponding to each joint position coordinate extracted from the RGB image by preset pixels to estimate the average value of the N pixels as the temperature value of the joint position coordinate. . (N is a natural number)

Here, if it is assumed that there is one preset pixel, the size of 3x3 pixels may be extended by one pixel at a time, up, down, left, right, and diagonally from the pixel for the joint position coordinates. In this case, the computing device 200 may estimate the average value of each pixel value constituting the 3x3 pixel as the temperature value of the joint position coordinates.

Here, the number of pixels or N pixels to be expanded can be easily set and changed by an administrator later. Therefore, the computer device 200 may define an area around a specific joint and obtain a temperature change value for the area around a specific joint. You can store it in a database.

Then, the computing device 200 matches and stores the coordinates of the joint points, the distance between the joints, the angle, and the stimulation area (S170).

The computing device 200 may match the main joint position information and the body stimulation area and store it based on the corresponding exercise motion information.

Accordingly, the computing device 200 matches the position of the main joint, the target movement region (stimulation region), the movement amount of the joint, the distance between the joints, the ratio, the angle, and the user's central point for each movement posture included in the movement motion, and is linked to the database can be stored in

Meanwhile, the computing device 200 may learn continuous movement postures and body stimulation regions by inputting coordinates of joint points, distances between joints, angles, and body stimulation regions to the neural network.

Accordingly, after learning the RNN network with a series of feature values extracted from the trainer image, when the feature values extracted from the user input image at time t are input to the learned neural network, the user's correct posture and body stimulation area at time t+1 can be extracted.

As such, the computing device 200 may check the posture accuracy by comparing the state value for the correct posture obtained from the learned neural network with the actual user input value.

Here, the neural network refers to a recurrent neural network or a recursive neural network, but is not limited thereto.

Here, as input data for learning the RNN network, the coordinates of the joint points extracted through the open pose API, the distances and angles between the joints are used.

The computing device 200 trains the RNN with the joint coordinates extracted from the trainer image exercising correctly with the training data. Through this, the RNN network learns the correct next coordinate for the current coordinate for a specific motion.

In detail, when the joint coordinates extracted at time t are given from new input data, the computing device 200 predicts the coordinates of the t+1 point of the corresponding coordinates through the RNN network.

In addition, when the distance between the predicted coordinates and the coordinates where the user is actually located is greater than the distance threshold, the computing device 200 provides the correct location for the corresponding part to the user terminal 300 .

7 is a flowchart illustrating a process of providing feedback on a user's exercise posture according to an embodiment of the present invention.

As shown in FIG. 7 , the computing device 200 receives and provides a target image for the exercise operation of the trainer ( S210 ).

When the computing device 200 receives an image for a condition input from the user terminal 300 , the computing device 200 may transmit a target image corresponding to the request and metadata about the target image to the user terminal 300 .

Here, the target image means an image according to a user's request among images classified into various conditions, such as a specific trainer, a specific exercise motion or exercise posture, an exercise effect to be obtained, and a body part to be strengthened.

In addition, the computing device 200 may receive the user's body information while receiving a request for the target image from the user terminal 300 .

For example, when receiving body information including user's unique ID, user's height, weight, athletic ability, etc., the computing device 200 stores the user's unique ID and unique information of the user terminal 300 in the database for the user. (Phone number, ID, etc.) can be matched and saved.

Later, when the user requests feedback on the exercise posture again for this information, the stored corresponding information may be called in without a separate input to perform exercise posture analysis based on the user information.

In addition, the computing device 200 may transmit such a target image to the user terminal 300 in real time or may provide a file in a download format.

Next, the computing device 200 collects images captured by the dual thermal imaging camera 100 of the user's exercise posture ( S220 ).

The computing device 200 collects an RGB image and a thermal image obtained by photographing a user's exercise posture with the dual thermal imaging camera 100 .

In addition, the computing device 200 analyzes the user joint position information and the body stimulation area from the image of the dual thermal imaging camera (S230).

In steps S220 and S230, the computing device 200 analyzes the user's exercise posture in the same manner as in steps S120 to S160 described above, and thus a redundant description will be omitted.

However, in the analysis, the computing device 200 may apply the user's key input to Equation 1 described above to perform posture analysis suitable for the user.

Next, the computing device 200 checks whether the user joint position information and the metadata of the target image match ( S240 ).

The computing device 200 compares it with the metadata of the target image based on the analyzed user posture analysis. The metadata represents posture analysis data for the trainer, and in detail, the position of the joint, the angle between the joints, and the body stimulation area are compared with each other.

First of all, the computing device 200 may check whether the joint position information matches the joint position information of the trainer for the same exercise posture based on the joint position information.

Here, the coincidence means that there is joint position information of the user and the joint position information of the trainer within the critical range set within the error range.

At this time, when the joint position information is different from the joint position information of the trainer included in the metadata, the computing device 200 provides feedback for changing the posture ( S250 ).

In detail, it is possible to provide a posture change that is induced to match the joint position information of the trainer based on the joint angle of the user's body, the direction according to the position of the joint, and the like.

After providing feedback on the posture change, the computing device 200 returns to step S220, collects images, and analyzes the user's exercise posture in real time.

On the other hand, when the joint position information matches the joint position information of the trainer included in the metadata, the computing device 200 checks whether the user body stimulation region matches the metadata of the target image ( S260 ).

In this case, the computing device 200 may provide a positive feedback for maintaining the posture.

And, in performing the exercise operation, the computing device 200 extracts a region with a large change in the user's joint region compared to the temperature sensed in the thermal image taken at a previous point in time to extract a region stimulated by the user performing an exercise motion. estimate

When the estimated body stimulation area is different from the body stimulation area of the trainer included in the metadata, the computing device 200 provides feedback for changing the body stimulation area.

For example, the muscles stimulated may vary depending on where the body's center of gravity is placed, even if the same external exercise posture is used.

Accordingly, the computing device 200 changes the body stimulation area, such as requesting to move the user's center of gravity so that the stimulation can go to the body stimulation area that leads to the correct exercise posture, or giving force to a specific body part such as the stomach or hip You can provide feedback as much as possible.

And, after providing feedback for changing the body stimulation area, the computing device 200 returns to step S220 to collect images and analyze the user's exercise posture and body stimulation area in real time.

Meanwhile, when the body stimulation area matches the body stimulation area of the trainer included in the metadata, the computing device 200 provides feedback for maintaining the posture.

The computing device 200 may provide positive feedback on posture maintenance based on the number of repetitions and the holding time set in the target image.

The computing device 200 may analyze and compare the user's exercise posture by repeating such a process based on a target image or one exercise motion or an exercise posture within the target image, and may provide feedback.

8 is an exemplary diagram illustrating a feedback providing screen according to an embodiment of the present invention.

As shown in FIG. 8 , the computing device 200 may provide both the user's exercise posture and the correct appearance of the exercise posture.

The computing device 200 may deliver information about which joint or muscle to use while providing an RGB image and a correct exercise posture together as an image captured by the user.

At this time, when the user performs an exercise motion with an incorrect posture, it is possible to set the content to be delivered in a certain step.

For example, if the user's exercise posture is different from the correct exercise posture in various joint areas, general information for implementing the exercise posture may be provided. Or, if the user's exercise posture differs only in the knee angle compared to the correct exercise posture, compare the difference in the knee angle formed in the current user's exercise posture by expanding only the information on the user's exercise posture and the knee angle in the correct exercise posture It can provide pictures or information about how the user should change the angle of the knee.

In addition, the computer device 200 may provide the user's exercise posture and correct exercise posture through 360-degree rotation, and may guide the type of pain caused by the user's incorrect exercise posture.

Such feedback information can be easily changed and designed by an administrator later.

9 is a hardware configuration diagram of a computing device according to an embodiment of the present invention;

As shown in FIG. 9 , the hardware of the computing device 400 may include at least one processor 410 , a memory 420 , a storage 430 , and a communication interface 440 , and may be connected through a bus. there is. In addition, hardware such as an input device and an output device may be included. The computing device 300 may be loaded with various software including an operating system capable of driving a program.

The processor 410 is a device for controlling the operation of the computing device 400 , and may be a processor 410 of various types that processes instructions included in a program, for example, a central processing unit (CPU), an MPU ( It may be a micro processor unit), a micro controller unit (MCU), a graphic processing unit (GPU), or the like. The memory 420 loads the corresponding program so that the instructions described to execute the operation of the present invention are processed by the processor 410 . The memory 320 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 430 stores various data, programs, etc. required for executing the operation of the present invention. The communication interface 440 may be a wired/wireless communication module.

According to the embodiment, by providing a posture correction guide by checking not only the correct exercise posture but also the muscle parts stimulated in the user's exercise movement, it is possible to increase the understanding of the user's exercise movement and maximize the effect obtained from the exercise movement.

According to an embodiment, by analyzing the user's movement motion using the RBG channel and the thermal image channel together, it is possible to improve the false positive rate for recognizing a hidden part or an external object similar to the shape of a person and improve the accuracy of recognizing a person. .

The embodiment of the present invention described above is not implemented only through the apparatus and method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also provided. is within the scope of the right.

Claims (13)

A method for a computing device operated by at least one processor to analyze an exercise posture, comprising:
Receiving an image of the user's exercise motion through the dual thermal imaging camera,
Combining the confidence map generated based on the RGB channel and the thermal image channel of the dual thermal imaging camera, and obtaining the main joint position information of the user from the combined confidence map;
extracting, as a body stimulation region, a joint region in which a temperature change estimated by the thermal image channel is greater than or equal to a temperature threshold while the user performs an exercise motion; and
Matching the main joint position information and the body stimulation area
Exercise posture analysis method comprising a. In claim 1,
The step of obtaining the main joint position information is,
An exercise posture analysis method for generating each confidence map displayed as a heat map for each point estimated as a joint position by using an open pose API for the RGB channel and the thermal image channel, respectively. In claim 2,
The step of obtaining the main joint position information is,
Calculating a distance between the dual thermal imaging camera and the user based on the received user's key information, and calculating a viewing distance for the RGB channel and the thermal imaging channel based on the distance; And
moving to the RGB channel by applying the viewing distance to the estimated joint position of the thermal image channel;
Exercise posture analysis method comprising a. In claim 3,
The step of obtaining the main joint position information is,
Motion posture analysis method of calculating an average of the sum of the brightness value of the joint position point of the thermal image channel moved to the RGB channel and the brightness value of the joint position point of the RGB channel at the same point and displaying the average on the combined confidence map. In claim 4,
The matching step is
Upon receiving the information on the movement motion, at least one of the position of the main joint, the target movement area (stimulation area), the movement amount of the joint, the distance between the joints, the ratio, the angle, and the center point of the user for each movement posture included in the movement motion. An exercise posture analysis method that matches the above and stores it in an interlocking database. In claim 5,
The exercise posture analysis method further comprising the step of inputting the main joint position information, including coordinates of joint points, distances and angles between joints, into a neural network for each movement movement to learn correct posture coordinates for each movement movement. A program stored in a computer-readable storage medium and executed by a processor, comprising:
The operation of collecting RGB images and thermal images of the user's exercise posture while providing the target image for the selected exercise motion;
obtaining joint position information and body stimulation region of the user based on the RGB image and the thermal image;
If the obtained user's joint position information and body stimulation region match with the joint position information stored in the metadata of the target image and the body stimulation region, feedback for maintaining the exercise posture is provided. Actions that provide feedback for area changes
A program containing instructions to execute In claim 7,
The action to obtain is
For the RGB image and the thermal image, each of the open pose APIs is used to generate a confidence map that is displayed as a heat map for each point estimated as a joint position, and matches the confidence map in a two-dimensional vector space to the joint position Programs to obtain information. In claim 8,
The action to obtain is
After generating a confidence map for the RGB image and the thermal image, respectively, calculating a disparity for the RGB image and the thermal image, applying the calculated viewing distance, and combining them into one confidence map program. In claim 9,
The action to obtain is
The distance between the dual thermal imaging camera and the user is calculated based on the received key information of the user, the focal length of the camera, and the size of the image formed on the sensor, and the RGB according to the distance between the camera lenses and the focal length based on the distance A program for calculating the viewing distance between the image and the thermal image. In claim 7,
The action to obtain is
A program for acquiring, as the body stimulation region, a joint region in which a temperature change estimated by the thermal image is equal to or greater than a temperature threshold while performing the exercise motion. In claim 7,
A neural network is trained on the basis of the continuous movement movements included in the target image, and the user's joint position information and body stimulation region are input to the learned neural network to determine the user's correct posture and body stimulation region for the next exercise operation. A program further comprising instructions for executing an extracting operation. In claim 12,
The extraction operation is
A program for providing feedback by comparing the user's joint position information and the body stimulation area acquired in real time based on the user's correct posture and body stimulation area obtained through the neural network to calculate the accuracy of the exercise posture and body stimulation area.

Images