KR102501300B1 - Method for motion evaluation - Google Patents

Abstract

The present invention relates to a motion evaluation method. According to an embodiment of the present invention, the motion evaluation method includes a motion evaluation method for evaluating a user motion followed while viewing a guide image in which an expert's guide motion is photographed. Comparison between the step of reproducing and photographing the user's motion to collect the user image, the expert of the first key frame predefined among the frames of the guide image, and the user of the second key frame corresponding to the first key frame of the frames of the user image Determining a frame coincidence rate according to , and determining a motion coincidence rate of a user motion with respect to a guide motion based on the determined frame coincidence rate. Through this, the present invention informs whether the user is properly following the expert's motion so that the user can home-train with the correct motion at home. In particular, by evaluating the user's motion through a simplified process, real-time user motion evaluation Quick evaluation can be performed without time delay, and operation evaluation can be performed using a relatively low-end device.

Description

Method for motion evaluation {Method for motion evaluation}

The present invention relates to a motion evaluation method, and more particularly, to collect user images that follow the expert's motions of the guide video while playing the guide video, and compare and determine the motion matching rate between the guide video and the user's video so that the user can properly perform the expert's motion. In particular, when comparing the experts in the guide video and the user in the user video, it is not possible to compare all frames, but rather to compare the key frames in which key poses are captured for each motion to determine the motion matching rate. It relates to a motion evaluation method capable of evaluating the motion of

While modern people's scientific and technological civilization has become highly developed and their living environment has become rich and convenient, obesity and overweight population are gradually increasing due to excessive nutrition and lack of exercise, and the need for modern people's health care is getting bigger and bigger.

Accordingly, interest in their own health has recently increased among healthy people, and the need for exercise for health promotion has been deeply recognized throughout society, and many people are actually exercising.

Most people use the gym a lot for health management. In general, the user signs up for a gym in the form of an annual membership or monthly membership, then visits the gym at a time when the user can use it and provides it to the gym. Exercise using the exercise equipment provided. In addition, in order to increase the exercise effect, the user can perform the exercise by making an exercise plan suitable for his/her exercise purpose with the help of the trainer by additionally registering personal lessons with the trainer registered in the gym.

Exercising at such a gym is expensive, and there is a limitation that you must visit the gym you have joined in order to exercise. In particular, in order to receive private lessons, expensive additional expenses must be paid, resulting in a greater economic burden.

In addition, when an epidemic caused by a virus is prevalent, there is also a problem that the risk of virus transmission is high in a gym where a large number of people gather in a small space.

In order to overcome the limitations and problems of training using gyms and private lessons, home training contents that can be exercised inexpensively and conveniently at home while watching expert exercise videos played through various devices have recently been provided.

However, all exercises must be performed with precise movements, but it is very important to exercise with accurate movements because you can see a great effect without risk of injury. However, since the currently provided home training service is unilateral and unidirectional like broadcasting, there is a problem in that it cannot be confirmed whether the user is exercising with an accurate motion.

Republic of Korea Patent Publication No. 10-2020-0129327 Republic of Korea Patent No. 10-2157322

The present invention is to solve the above-mentioned problems of the prior art, and an object of the present invention is to collect user images that follow the expert's motion of the guide image while playing the guide image, and to improve the motion coincidence rate between the guide image and the user image. Comparison and judgment is performed to inform whether the user is following the expert's motion properly. In particular, when comparing the expert in the guide video and the user in the user video, the motion is performed through comparison between key frames in which key poses are recorded for each motion, rather than comparing all frames. An object of the present invention is to provide a motion evaluation method capable of evaluating a user's motion through a simplified process by determining a match rate.

Another object to be achieved by the present invention is, when comparing the second key frame of the user image with the first key frame of the guide image, the matching rate between the frames based on the connection line between the limited joints of the expert and the user on the frame is determined. , to provide a motion evaluation method that can evaluate a user's motion through a simpler comparison process.

In the motion evaluation method according to an embodiment of the present invention, in the motion evaluation method of evaluating a user motion followed by watching a guide video in which an expert's guide motion is photographed, the user video is obtained by reproducing the guide video and photographing the user motion. Collecting, determining a frame matching rate according to comparison between an expert of a first key frame predefined among frames of a guide image and a user of a second key frame corresponding to the first key frame among frames of a user image, and and determining a motion coincidence rate of a user motion with respect to a guide motion based on the determined frame coincidence rate.

At this time, in the step of determining the frame coincidence rate, the frame coincidence rate is based on the frame coincidence rate between the expert skeleton image on the first key frame and the user skeleton image on the second key frame, and the expert skeleton image and the user skeleton image are one of the expert's or user's body. One or more joints and a connection line connecting one or more joints may be included.

In addition, in the step of determining the frame matching rate, the number of predefined first key frames among the frames of the guide image is one or more, and the expert skeleton image of each first key frame is a key in poses taken for each first key frame. Skeleton-limiting information for some joints and connection lines limited according to body parts is added, and the skeletal concordance rate may be determined based on some joints and connection lines limited and designated according to the bone-limiting information.

In addition, the step of determining the frame coincidence rate includes the steps of selecting, as a second key frame, one randomly selected frame among user video frames captured and collected for a time within a predetermined time from when the first key frame is reproduced; Creating and adding a user skeleton image to a second key frame, comparing the expert skeleton image of the first key frame and the user skeleton image of the second key frame to determine the skeleton matching rate of the second key frame, and A step of determining the skeleton coincidence rate of the second key frame as the frame coincidence rate may be included.

In addition, the step of determining the frame coincidence rate includes selecting one or more candidate frames captured during a predetermined time from the time when the first key frame is reproduced from among the frames of the collected user image, and selecting one or more candidate frames selected by the user. Creating and adding a skeleton image, comparing the user skeleton image of each candidate frame with the expert skeleton image of the first key frame to determine the skeleton matching rate of each candidate frame, and determining the skeleton matching rate of each candidate frame. A step of selecting a candidate frame having the highest matching rate as a second key frame and determining a frame matching rate of the selected second key frame as a frame matching rate.

In addition, the motion evaluation method further includes determining whether a photographing condition including at least one of whether a user's position for photographing a user's motion is within a predetermined position and whether or not the front of the user's body is photographed is satisfied, , In the step of collecting the user image, play of the guide image may be started when reproduction of the guide image is requested and shooting conditions are satisfied.

In addition, the motion evaluation method may further include counting the number of user motions by counting only when the motion coincidence rate is greater than or equal to a reference value.

According to the present invention, while playing the guide video, user images that follow the expert's motions are collected, and the motion matching rate between the guide video and the user's video is compared and judged to inform whether the user is properly following the expert's motions, so that the user can be at home. It allows you to do home training with the correct movement.

In addition, when the present invention compares an expert in a guide image and a user in a user image, instead of comparing all frames, the motion matching rate is determined through comparison between key frames in which key poses are captured for each motion, and the user's motion through a simplified process. By evaluating , fast evaluation can be performed without delay when evaluating real-time user motion, and motion evaluation can be performed using a relatively low-end device.

In addition, in the present invention, when comparing the second key frame of the user image of the first key frame of the guide image, the concordance rate between the frames is based on the connection line between the joint and some limitedly designated joints of the expert and the user on the frame, making the comparison simpler. The user's motion can be evaluated through the process.

1 is a diagram schematically illustrating a motion evaluation system to which a motion evaluation method according to an embodiment of the present invention is applied.
2 is a functional block diagram of a guide image generator according to an embodiment of the present invention.
3 is a diagram illustrating an example of creating an expert skeleton image in a guide image according to an embodiment of the present invention.
4 is a diagram showing an example of designating a first key frame in a guide image according to an embodiment of the present invention.
5 is a functional block diagram of an operation evaluation unit according to an embodiment of the present invention.
6 is a diagram illustrating an example of generating a user skeleton image from a user image according to an embodiment of the present invention.
7 is a diagram for explaining a process of selecting a second key frame in a user image according to an embodiment of the present invention.
8 is a diagram illustrating an example of comparison between an expert skeleton image and a user skeleton image according to an embodiment of the present invention.
9 is a flowchart illustrating a motion evaluation method according to an embodiment of the present invention.
10 is a flowchart illustrating an example of a process of determining a frame coincidence rate according to an embodiment of the present invention.
11 is a flowchart illustrating another example of a process of determining a frame coincidence rate according to an embodiment of the present invention.

It should be noted that technical terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in the present invention should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in the present invention, and are excessively inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in the present invention are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand.

Also, singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. In the present invention, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various elements or steps described in the invention, and some of the elements or steps are included. It should be construed that it may not be, or may further include additional components or steps.

In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

Hereinafter, a motion evaluation method according to the present invention will be described in more detail with reference to the accompanying drawings.

1 is a diagram schematically illustrating a motion evaluation system to which a motion evaluation method according to an embodiment of the present invention is applied.

The present invention relates to a motion evaluation method for determining and informing whether a user following a guide motion on a reproduced guide image is performing the motion correctly. An expert image collector 10 that collects, a guide image generator 30 that creates a guide image from the collected expert images, a guide image player 50 that plays back the generated guide image, and collects user images It may be configured to include a user image collection unit 70, and a motion evaluation unit 90 to evaluate the user's motion.

The expert image collecting unit 10 may collect expert images by capturing motions of the expert, and store the collected expert images. In this case, the expert motion may be various motions such as exercise and dance.

The guide image generator 30 may generate a guide image by processing the expert image provided by the expert image collection unit 10 . In detail, the guide image generation unit 30 may receive and edit the stored expert image to generate a guide image for one operation unit. In addition, the guide image generator 30 may add expert skeleton image information to the generated guide image. Here, the expert skeleton image is an image in which one or more joints and connection lines between the joints are included in the expert's body on the guide image, and a detailed description thereof will be described later with reference to FIG. 3 .

The guide image generated by the guide image generator 30 may be provided to the guide image reproducing unit 50 and the motion evaluation unit 90 . The guide image reproducing unit 50 may receive and reproduce the guide image generated by the guide image generating unit 30 . The guide image reproducing unit 50 may repeatedly reproduce one guide image or connect and reproduce different types of guide images. For example, the guide image reproducing unit 50 repeatedly reproduces the guide motion on the guide image 30 times, and the user can repeat the user motion 30 times according to the expert's guide motion while viewing it. If skeletal image information is added to the guide image, the guide image reproduced by the guide image reproducing unit 50 may be an image in which skeletal image information is deleted or hidden. As another example, if two guide operations correspond to each other in the left and right directions, the guide image reproducing unit 50 may sequentially and repeatedly reproduce the two guide operations.

The user can follow the expert's guide motion while viewing the guide video reproduced by the guide video reproducing unit 50 . In this case, an operation performed by the user while viewing the guide image corresponds to a user operation, and the user image collecting unit 70 may collect a user image by photographing a user following the guide operation.

Preferably, the guide image reproducing unit 50 may guide the user so that the user is located in the correct body direction or position (for example, the front or the center of the screen) for motion evaluation on the user image. Preferably, when reproduction of the guide image is requested from the user, the guide image reproducing unit 50 may provide an image guiding the correct body direction or position of the user before reproducing the guide image. After the guide image reproduction unit 50 provides an image for guiding the correct body direction or position, the user image collection unit 70 collects a preliminary image toward the user. The guide image reproducing unit 50 determines whether a shooting condition according to the direction or position of the user's body is satisfied based on the preliminary image collected by the user image collection unit 70, and if the shooting condition is met, an expert's guide video is displayed. Playback of the captured guide video may be started.

Specifically, the guide image reproducing unit 50 includes at least one condition among whether or not the user's position is within a predetermined position based on the preliminary image provided from the user image collecting unit 70 and whether or not the front of the user's body is captured. It is possible to determine whether the shooting condition to be satisfied. Preferably, the guide image reproducing unit 50 may determine that the photographing condition is satisfied only when the user's position is within a predetermined position and the front of the user's body is photographed at the same time. At this time, the user's location or body direction may be determined through deep learning technology.

The motion evaluation unit 90 compares the guide image reproduced by the guide image reproducing unit 50 and the user image collected by the user image collecting unit 70 to determine the motion matching rate between the guide motion and the user motion, so that the user can select the correct motion. is performed, and the evaluation result may be provided and output to the guide image reproducing unit 50 so that the user can check it immediately. The evaluation and output of the user's motion can be performed immediately whenever the user performs a user motion, and the motion evaluation result will be output together with the guide image that is repeatedly reproduced during repeated reproduction of the guide image by the guide image reproducing unit 50. can Accordingly, the user receives real-time feedback as to whether the user is performing the motion in the correct posture, and corrects the motion when the posture is wrong.

Here, the guide image playback unit 50, the user image collection unit 70, and the motion evaluation unit 90 according to the present embodiment may be configured as a single device such as a smart phone, but are not limited thereto.

2 is a functional block diagram of a guide image generating unit according to an embodiment of the present invention, FIG. 3 is a diagram showing an example of creating an expert skeleton image in a guide image according to an embodiment of the present invention, and FIG. It is a diagram showing an example of designating a first key frame in a guide image according to an embodiment of the present invention.

Looking more specifically at the guide image generator 30 with reference to FIG. 2 , the guide image generator 30 includes an image editor 31 that edits an expert video in units of one action, and an image edited in units of actions. It may include a first skeleton image generating unit 33 that generates and adds expert's skeleton image information and a skeleton image correction unit 35 that corrects the created expert's skeleton image information.

The image editing unit 31 may produce a guide image by editing the expert image provided from the expert image collection unit 10 in units of operation. For example, if the expert video is an image of a push-up motion, the video editing unit 31 may extract the push-up motion once and produce one guide image. Classification of one operation unit may be determined by a manager or an expert. Of course, the expert images collected by the expert image collecting unit 10 may be images collected in one operation unit, and even in this case, the image editing unit 31 may delete unnecessary frames from the expert image.

The first skeleton image generation unit 33 may receive a guide image edited and produced by the image editing unit 31, and create and add an expert's skeleton image on the provided guide image.

In detail with reference to FIG. 3 , the first skeleton image generating unit 33 extracts one or more joints jp1 from the expert's body on the provided guide image, and connects the extracted joints according to the structure of the body (connecting lines ( jl1) may be connected to create an expert skeleton image sk1. Here, generation of the skeletal image in the first skeletal image generator 33 may be performed in units of each frame of the guide image, or may be automatically performed through deep learning technology.

The bone image correction unit 35 may modify the bone image generated by the first bone image generator 33 . Specifically, the manager checks the guide image to which the expert skeleton image has been added through deep learning technology, selects joints that have not been extracted to the correct location in the frame where the expert skeleton image is incorrectly added, and requests to move them to the correct location. In addition, the skeleton image correction unit 35 may modify the expert skeleton image by moving the position of the joint and moving the position of the connecting line according to the request of the manager.

On the other hand, the motion evaluation system according to the present embodiment does not compare all frames when comparing the guide image and the user image, but determines the matching rate through comparison of key frames in which key poses are captured for each exercise, and through a simplified process, the motion evaluation system action can be evaluated.

To this end, the guide image generation unit 30 according to the present embodiment includes a first key frame designation unit 37 for designating one or more first key frames in the edited image, and skeleton limitation information in each first key frame. It may be configured to include a skeleton limitation information setting unit 39 for setting.

As shown in FIG. 4 , the first key frame designator 37 may designate a first key frame k1 in which a key pose of a corresponding motion is captured, among a plurality of frames constituting a guide image. Here, the core pose is the most important pose in the corresponding motion, and the designation of the core pose may be designated by a manager or an expert. For example, in the case of a motion of lifting one knee high to the side as in the guide motion of FIG. 4 , the pose at the time when the knee is lifted the most is the most important pose, and this may be selected and designated as a core pose.

At this time, although one frame may be designated as the first key frame k1, a plurality of frames may be designated. The number of first key frames k1 may vary according to a decision of a manager or an expert. For example, if the guide motion is a push-up motion, a pose at which the upper body is closest to the ground by bending the arm as much as possible and a pose when the upper body is farthest from the ground by fully extending the arm may be the core pose. In this case, two or three key frames (k1) may be designated.

Preferably, the first key frame specifying unit 37 may be set to extract a plurality of frames as the first key frame k1. In this way, when a plurality of frames are selected and set to be extracted as the first key frame (k1), when the manager selects one or more frames excluding the first frame and the last frame among all frames of the guide video, the first frame or the last frame of the guide video is automatically selected to designate two or more first key frames (k1). This is because in general, the first or last pose of most exercise motions is included in the core pose, and since the first or last pose is the same in most exercise motions, only one of the first or last poses is the core pose. It is preferable to include

The first key frame (k1) extracted in this way can be used in the motion evaluation unit 90 when comparing motion coincidence rates between a guide image and a user image, and a more detailed description thereof will be described later with reference to FIG. 5 .

Preferably, when a guide image is created in the image editing unit 31, the first key frame designation unit 37 may designate a first key frame before generating an expert's skeleton image in the first skeleton image generation unit 33. there is. The first skeleton image generation unit 33 generates the expert's skeleton image only for the first key frame specified by the first key frame designator 37, thereby minimizing the expert's skeleton image creation process.

The skeletal limitation information setting unit 39 may generate skeletal limitation information for some joints and connection lines limited and designated according to key body parts in a pose photographed on the first key frame, and add the skeletal limitation information to the first key frame. . If a plurality of first key frames are designated, the skeleton limitation information setting unit 39 may individually set skeleton limitation information for each of the plurality of first key frames.

Here, the skeleton limitation information may include joints and connection lines constituting key body parts in the expert's pose on the first key frame. For example, in a guide video for a push-up motion, if the first key frame is a frame in which the arm is maximally bent, the skeleton-limited information in the arm maximally bent pose is the shoulder joint, the elbow joint, the wrist joint, and the three It may be two connecting lines connecting a joint. At this time, if it is determined that the position of the buttocks is also important in the pose where the arms are maximally bent, the hip joint can be set by adding the skeletal limitation information. That is, the joints included in the skeletal limitation information do not necessarily have to be joints connected to each other.

The guide image to which the designated first key frame information, manager skeleton image generation information, and skeleton limitation information are added or set may be provided to the guide image reproducing unit 50 and motion evaluation unit 90 .

5 is a functional block diagram of a motion evaluation unit according to an embodiment of the present invention, FIG. 6 is a diagram showing an example of generating a user skeleton image from a user image according to an embodiment of the present invention, and FIG. 7 is a diagram illustrating the present invention. It is a diagram shown to explain a process of selecting a second key frame in a user image according to an embodiment of.

Looking more specifically at the motion evaluation unit 90 with reference to FIG. 5 , the motion evaluation unit 90 includes a second skeleton image generator 91 that generates a user skeleton image from a user image, and a first skeleton image generator 91 of a guide image. Based on the frame matching rate determination unit 93 for determining the frame matching rate of the key frame and the second key frame of the user image, the motion matching rate determining unit 95 for determining the motion matching rate between the guide image and the user image, and the determined motion matching rate It may be configured to include a counting unit 97 for counting the number of correct operations by the user.

The second skeletal image generator 91 may receive the user image collected by the user image collection unit 70, and may create and add the user's skeletal image jp2 on the provided user image.

In detail with reference to FIG. 6 , the second skeleton image generator 91 extracts one or more joints jp2 from the user's body on the provided user image, and connects the extracted joints according to the structure of the body. jl2) to create a user skeleton image (sk2). Here, generation of the skeleton image in the second skeleton image generator 91 may be performed in units of each frame of the user image, but may be performed limited to a selected specific frame. Also, the second skeletal image generation unit 91 may generate a user skeletal image sk2 through deep learning technology.

The frame matching rate determination unit 93 may determine frame matching rates between the first key frame of the user image and the second key frame of the guide image. In this case, the frame coincidence rate means the coincidence rate between the expert's pose on the first key frame and the user's pose on the second key frame. Specifically, the frame matching rate determination unit 93 may determine the frame matching rate between the expert skeleton image added to the first key frame and the user skeleton image of the second key frame as the frame matching rate. A detailed description of the process of determining the bone concordance rate will be described later with reference to FIG. 8 .

The frame coincidence rate determining unit 93 may select a second key frame among frames constituting the user image based on the first key frame designation information added to the reproduced guide image. In this case, the second key frame is a frame of a user image corresponding to the first key frame of the guide image, and may be a frame in which a user pose following an expert pose of the first key frame is captured. Therefore, the number of second key frames in the user image can be determined according to the number of first key frames included in the guide image, and each selected second key frame corresponds to each first key frame in a 1:1 relationship. .

The frame coincidence rate determination unit 93 may select a frame of the user image captured at the time when the first key frame is reproduced by the guide image reproducing unit 50 as a second key frame corresponding to the corresponding first key frame. However, if the user performs the user motion while already familiar with the guide motion, the pose on the first key frame can be performed at the same time as the first key frame is played (that is, the guide motion and the user motion are synchronized). However, when the user follows the guide motion, a predetermined time difference will occur until the user sees the pose on the first key frame and follows it. Therefore, although the frames collected at the time when the first key frame is reproduced in the frame coincidence rate determining unit 391 and the guide image reproducing unit 50 may be selected as the second key frame, a time point later than the time when the first key frame is reproduced The frames collected in may be selected as the second key frame.

In more detail with reference to FIG. 7 , the frame coincidence rate determination unit 391 captures images from the time point t1 when the first key frame is reproduced in the guide image reproducing unit 50 to the time point t2 after a predetermined time has elapsed. One of the frames of the user image may be selected as the first key frame.

Preferably, the frame coincidence rate determination unit 93 may select a frame taken at a point in time when a reference time elapses from the point in time t1 when the first key frame is reproduced in the guide image reproducing unit 50 as the first key frame. In this case, the reference time may be set according to the user's operational skill level, and the user's operational skill level may be selected by the user for each guide operation. For example, if the user selects a motion proficiency for a push-up motion as a beginner level, the reference time may be set to 2.0 seconds. On the other hand, if the user selects the motion proficiency for the push-up motion as an advanced level, the reference time may be set to 0.5 seconds. At this time, 0.0 second may also be included in an arbitrary time. For example, if the user selects the motion proficiency for the push-up motion as the expert level, the reference time is set to 0.0 second, and the frame coincidence rate determining unit 93 determines the frame captured at the time point t1 when the first key frame is reproduced. can be selected as the second key frame.

Here, the guide image reproducing unit 50 generates a first key frame reproducing signal at the time point t1 when the first key frame is reproduced based on the first key frame specifying information added to the guide image, and the frame matching rate determining unit ( 93), and the frame coincidence rate determination unit 93 may determine the time point t1 at which the first key frame was reproduced based on the received first key frame reproduction signal.

In addition, the frame matching rate determination unit 93 that has selected the second key frame compares the user skeleton image of the second key frame and the corresponding expert skeleton image of the first key frame to determine the skeleton matching rate, and determines the frame matching rate according to the determined skeleton matching rate. A frame coincidence rate between the first key frame and the second key frame may be determined.

At this time, the second skeleton image generation unit 91 generates a user skeleton image only for the selected second key frame among all frames of the user image, thereby simplifying the user skeleton image creation process as much as possible.

As described above, when the second key frame is selected, there is an advantage in that the process of selecting the second key frame can be simplified. However, even though the user accurately followed the pose of the first key frame, if the user's pose at the time of exactly following the pose of the first key frame was captured in a frame other than the second key frame, the frame matching rate was relatively low. There are problems that can be judged.

In order to solve this problem, preferably, the frame coincidence rate determination unit 93 includes one or more frames of the user image captured from the time point t1 when the first key frame is reproduced to the time point t2 after a predetermined time has elapsed. Among the frames, a frame in which a pose that most closely matches the pose of the first key frame is photographed may be selected as the second key frame.

Specifically, the frame coincidence rate determining unit 93 selects one or more frames captured from the time point t1 when the first key frame was reproduced to the time point t2 after a predetermined time has elapsed among the frames of the user image as candidate frames, , The skeleton matching rate of each candidate frame may be determined by comparing the user skeleton image of the selected candidate frame with the expert skeleton image of the first key frame. Further, the frame matching rate determining unit 93 may select a candidate frame having a skeleton matching rate determined to be the highest among one or more candidate frames as a second key frame. In this case, since the frame matching rate of the selected second key frame has already been determined during the selection process, the frame matching rate determination unit 93 does not additionally go through a separate skeleton matching rate determination process, and the frame matching rate is determined according to the frame matching rate of the selected second key frame. can determine

At this time, the second skeleton image generation unit 91 generates a user skeleton image only for the selected second key frame among all frames of the user image, compared to generating the user skeleton image for all frames of the user image. The image creation process can be relatively simplified.

Meanwhile, the motion coincidence rate determination unit 95 may determine a motion coincidence rate between the guide motion of the guide image and the user motion of the user image based on the frame coincidence rate between the first key frame and the second key frame.

Specifically, when the first key frame of the guide image is one, the motion coincidence rate determining unit 95 may determine the frame coincidence rate between the first key frame and the second key frame as the motion coincidence rate.

If there are multiple first key frames in the guide image, the motion consistency rate determination unit 95 may determine an average value of frame coincidence rates between each first key frame and a corresponding second key frame as the motion consistency rate.

In this way, the present invention, when determining the motion consistency rate between the expert motion of the guide image and the user motion of the user image, does not compare all frames, but compares the motion coincidence rate between the first key frame and the second key frame in which key poses are captured for each motion. Therefore, the user's motion can be evaluated through a simplified process. Therefore, in the present invention, when evaluating a user's motion in real time, quick evaluation can be performed without delay, and motion evaluation can be performed using a device with a relatively low specification.

Preferably, the user image collection unit 70 may collect user images for a time within a predetermined time from the time when the first key frame is reproduced in the guide image reproducing unit without capturing all of the user's motions. That is, the user image collecting unit 70 may collect the user image by capturing only the motion section used to determine the matching rate among user motions. Accordingly, the present invention can minimize unnecessary parts in the user image for determining the match rate, thereby reducing unnecessary memory usage and power consumption.

The motion coincidence rate information determined by the motion coincidence rate determining unit 95 may be provided to the guide image reproducing unit 50 . The guide image reproducing unit 50 may overlap and output the motion consistency rate information to the guide image being played so that the user can check the provided motion consistency rate information in real time. At this time, the guide image reproducing unit 50 may output the received motion agreement rate information as a digitized score or output different messages according to sections corresponding to the agreement rate. For example, output the word 'perfect' if the match rate is 90% or more, output the word 'very good' if the match rate is 80% or more and less than 90%, and output the word 'good' if the match rate is 60% or more and less than 80%. If the match rate is less than 60%, the word 'bad' may be output. Accordingly, the user can intuitively check whether he or she is performing the correct motion in real time while following the guide motion while viewing the guide image.

The counting unit 97 may count the number of user correct actions.

Specifically, the counting unit 97 counts the number of user actions once every time the guide image is reproduced once based on the reproduction information of the guide image reproducing unit 50, but the action determined by the motion matching rate determining unit 95 The user's motion may be counted only when the matching rate is greater than or equal to a reference value (eg, 60% or more). Therefore, the present invention can count only the user's motion that coincides with the guide motion more than a predetermined amount.

8 is a diagram showing an example of comparison between an expert skeleton image and a user skeleton image according to an embodiment of the present invention, and FIG. 9 is skeleton restriction information between a first key frame and a second key frame according to an embodiment of the present invention. It is a drawing showing an example of comparison.

As described above, the motion matching rate determination unit 95 compares the expert's skeleton image included in the first key frame with the user's skeleton image included in the corresponding second key frame or candidate frame to determine the skeleton matching rate. can

At this time, the motion matching rate determination unit 95 may determine the skeleton matching rate by comparing only a limited portion of the skeleton image without comparing all portions of the expert skeleton image with all portions of the user skeleton image.

Specifically, the motion matching rate determining unit 95 may determine the skeleton matching rate by comparing only a limited portion of the expert skeleton image and the user skeleton image according to skeleton limitation information set in the expert skeleton image of the first key frame.

Taking FIG. 8 as an example, looking more specifically, the joints lj limited and designated according to the skeleton limitation information set in the expert skeleton image sk1 of the first key frame are the left ankle joint, the left knee joint, the left hip joint, and the left elbow joint. , If the limitedly designated connection line ll is a connection line between the left ankle joint and the knee joint and a connection line between the left knee joint and the hip joint, the motion coincidence rate determining unit 95 determines the skeletal coincidence rate between the expert skeletal image and the corresponding user skeletal image, The skeleton congruence rate can be determined by mutually comparing the left ankle joint, the left knee joint, the left hip joint, the left elbow joint, the connection line between the left ankle joint and the knee joint, and the connection line between the left knee joint and the hip joint in the expert skeletal image and the user skeletal image. can

At this time, the motion matching rate determination unit 95 compares the distance l1 between two limitedly designated joints of the expert skeleton image sk1 and the distance l2 between two joints of the user skeleton image sk2 corresponding thereto, and compares the matching rate. Alternatively, the bone matching rate may be determined by comparing the angle a1 between the two limitedly designated connection lines of the expert skeleton image and the angle a2 between the two connection lines of the user skeleton image corresponding thereto.

Preferably, when generating the user skeleton image in the first key frame or candidate frame, the motion coincidence rate determination unit 95 determines the user skeleton image for the skeleton and connection line that are limited and designated based on the skeleton limitation information setting information set in the first key frame. can create That is, it is possible to generate only a user skeleton image for a specific body part without generating the entire shape of the user skeleton image, and through this, the user skeleton image creation process can be simplified as much as possible.

9 is a flowchart illustrating a motion evaluation method according to an embodiment of the present invention.

Looking at the motion evaluation procedure according to the present embodiment with reference to FIG. 9 , first, the guide image generation unit 30 generates a guide image by editing the expert image collected by the expert image collection unit 10 and adding various information. It can be done (S10).

When a user requests to play a guide image, the guide image reproducing unit 50 provides an image guiding the correct body direction or position of the user, and the user image collection unit 70 collects a preliminary image of the user. Yes (S20).

The guide image reproducing unit 50 determines whether the user's body direction or the user's location shooting conditions are satisfied based on the collected prior images (S30), and if the shooting conditions are met (S30-Y), the guide image is reproduced. The unit 50 may reproduce the guide image, and the user image collecting unit 70 may collect the user image (S40).

On the other hand, if the shooting condition is not satisfied (S30-N), the guide image reproducing unit 50 reproduces an image guiding the correct body direction or position of the user until the shooting condition is satisfied, and collects user images. The unit 70 may collect prior images of the user.

Meanwhile, when the guide image is reproduced from the side image reproducing unit 50 and the user image is collected from the user image collection unit 70, the frame matching rate determination unit 93 of the motion evaluation unit determines the first key frame of the guide image and the user image. It is possible to determine the frame coincidence rate between the second key frames of (S50).

When the frame matching rate is determined, the motion matching rate determining unit 95 of the motion evaluation unit may determine a motion matching rate between the guide image and the user image based on the determined frame matching rate.

The counter unit 97 of the motion evaluation unit may determine whether the motion coincidence rate determined by the motion coincidence rate determination unit 95 is greater than or equal to a reference value (S70). If the motion coincidence rate is greater than or equal to the reference value (S70-Y), the counter unit 97 counts the number of times for the corresponding user motion (S80), whereas if the motion coincidence rate is less than the reference value (S70-N), the counter unit ( 97) may not reflect the number of times of the user's motion to the count (S90).

In addition, the motion match rate determination result or the counted number of motions is provided to the guide image reproducing unit 50, and the guide image reproducing unit 50 outputs the motion coincidence rate determination result or the counted number of motions and provides the result to the user. (S100).

10 is a flowchart illustrating an example of a process of determining a frame coincidence rate according to an embodiment of the present invention.

Referring to FIG. 10, referring to an example of the process of determining the frame coincidence rate, the frame coincidence rate determiner 93 selects one randomly selected among the user's image frames collected and photographed for a period of time within a predetermined time from when the first key frame is reproduced. The frame of can be selected as the second key frame (S511).

When the second key frame is selected, the second skeleton image generation unit 91 may generate and add a user skeleton image to the second key frame selected by the frame coincidence rate determination unit 93 (S512).

When the user skeleton image is generated in the second key frame, the frame matching rate determination unit 93 compares the expert skeleton image in the first key frame with the user skeleton image in the second key frame to determine the skeleton matching rate in the second key frame. It can (S513).

Further, the frame matching rate determining unit 93 may determine the skeleton matching rate of the second key frame as the frame matching rate (S514).

11 is a flowchart illustrating another example of a process of determining a frame coincidence rate according to an embodiment of the present invention.

Referring to another example of the frame matching rate determination process with reference to FIG. 11 , the frame matching rate determining unit 93 includes one or more candidates captured for a predetermined time from the time when the first key frame is reproduced among the frames of the collected user image. A frame can be selected (S521).

When the candidate frames are selected, the second skeleton image generation unit 91 may generate and add a user skeleton image to each of the selected candidate frames (S522).

When a user skeleton image is added to each candidate frame, the frame matching rate determining unit 93 compares the user skeleton image of each candidate frame with the expert skeleton image of the first key frame to determine the skeleton matching rate of each candidate frame. Yes (S523).

Further, the frame matching rate determining unit 93 may select a candidate frame having the highest determined frame matching rate among the candidate frames as a second key frame, and determine the frame matching rate of the selected second key frame as the frame matching rate (S524). .

The above description is merely an example of the technical idea of the present invention, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

10: expert video collection unit
30: guide image generating unit
31: video editing department
33: first skeletal image generating unit
35: skeleton image correction unit
37: first key frame designation unit
39: skeleton limited information setting unit
50: guide video playback unit
70: user image collection unit
90: motion evaluation unit
91: second skeletal image generating unit
93: frame coincidence rate determination unit
95: operation coincidence rate determination unit
97: count unit

Claims (7)

A motion evaluation method for evaluating a user's motion followed by watching a guide video in which an expert's guide motion is filmed,
generating the guide image;
Collecting user images by reproducing the generated guide image and photographing the user's motion;
Determining a frame coincidence rate according to a comparison between an expert of a first key frame predefined among frames of the guide image and a user of a second key frame corresponding to the first key frame of frames of the user image; and
Determining a motion coincidence rate of the user motion with respect to the guide motion based on the determined frame coincidence rate;
The step of generating the guide image,
generating the guide image consisting of one motion by editing an expert image provided from an expert image collection unit; and
Designating one or more first key frames among frames constituting the guide motion and adding them to the guide image;
In the step of designating the first key frame and adding it to the guide image,
When a manager selects one or more frames excluding the first frame and the last frame among all the frames of the guide video to designate the first key frame, the first frame or the last frame is selected together with the frame selected by the manager. A motion evaluation method characterized in that it is specified as a key frame and added to a guide image.
2. The method of claim 1, wherein in the step of determining the frame coincidence rate
The frame coincidence rate is based on the skeleton coincidence rate between the expert skeletal image on the first key frame and the user skeletal image on the second key frame,
The motion evaluation method of claim 1 , wherein the expert skeleton image and the user skeleton image include at least one joint of the body of the expert or user and a connection line between at least one joint.
3. The method of claim 2, wherein in the step of determining the frame coincidence rate
The number of predetermined first key frames among the frames of the guide image is one or more;
The expert skeleton image of each of the first key frames is added with skeleton limitation information about some joints and connection lines defined according to key body parts in poses captured for each of the first key frames;
The motion evaluation method according to claim 1 , wherein the skeletal coincidence rate is determined based on some joints and connection lines limited and designated according to the skeletal limitation information.
3. The method of claim 2, wherein determining the frame coincidence rate
selecting, as the second key frame, one randomly selected frame among the collected image frames of the user photographed and collected for a time within a predetermined time from when the first key frame is reproduced;
generating and adding a user skeleton image to the selected second key frame;
comparing the expert skeletal image of the first key frame with the user skeletal image of the second key frame to determine a bone matching ratio of the second key frame; and
and determining the frame matching rate of the second key frame as the frame matching rate.
3. The method of claim 2, wherein determining the frame coincidence rate
selecting one or more candidate frames captured during a time within a predetermined time from a time point at which the first key frame was reproduced from among the collected frames of the user image;
generating and adding a user skeleton image to one or more selected candidate frames;
comparing the user skeleton image of each of the candidate frames with the expert skeleton image of the first key frame to determine a skeleton matching ratio of each of the candidate frames; and
and selecting a candidate frame having the highest determined frame matching rate among the candidate frames as the second key frame, and determining the frame matching rate of the selected second key frame as the frame matching rate. Assessment Methods.
The method of claim 1, wherein the motion evaluation method
Further comprising determining whether a photographing condition including at least one condition among whether a user's position for photographing the user's motion is within a predetermined position and whether the front of the user's body is photographed is satisfied;
In the collecting of the user image, reproduction of the guide image starts when reproduction of the guide image is requested and the photographing condition is satisfied.
The method of claim 1, wherein the motion evaluation method
and counting the number of user actions by counting only when the motion coincidence rate is greater than or equal to a reference value.

Images