KR20220057842A - Device for learning dance by using motion recognition - Google Patents

Abstract

According to an aspect of the present invention, a motion recognition dance learning device comprises: a model dance data storage unit for storing model dance data including a plurality of first skeleton coordinate values for each frame based on a dance image of a dancer; a learning dance data generation unit for generating learning dance data including a plurality of second skeleton coordinate values for each frame based on a learning image of a learner inputted in real time; a data comparison unit that compares the model dance data and the learning dance data to determine a similarity degree; and a score calculation unit for calculating a score based on the determined similarity degree.

Description

Motion recognition choreography learning device {Device for learning dance by using motion recognition}

The present invention relates to a choreography learning apparatus using motion recognition.

The conventional choreography learning apparatus is made in a form in which a user simply reproduces a choreography video and follows the reproduced choreography video. That is, the conventional choreography learning apparatus does not provide feedback on the user's choreography through unidirectional learning.

Such a conventional choreography learning apparatus has a problem in that, since the user cannot check the wrong action among his or her choreography actions, practice may be repeated with the wrong action, and subsequent choreography correction may be more difficult.

The present invention is to solve the above problems, and it is a technical task of the present invention to provide a motion recognition choreography learning apparatus capable of providing feedback on a learner's choreography.

A motion recognition choreography learning apparatus according to an aspect of the present invention for achieving the above object is an example for storing exemplary choreography data including a plurality of first skeleton coordinate values for each frame based on a choreography image of a choreographer A choreography data storage unit, a learning choreography data generation unit for generating learning choreography data including a plurality of second skeleton coordinate values for each frame based on a learner's learning image input in real time, the model choreography data and the learning choreography data It is characterized in that it comprises a data comparator that compares and determines the degree of similarity, and a score calculator that calculates a score based on the determined degree of similarity.

As described above, the motion recognition choreography learning apparatus according to the present invention may recognize the choreography movement of the learner and provide feedback on the choreography movement in real time. Since the present invention learns in both directions, it is possible to improve the learning effect.

In addition, according to the present invention, learning choreography data including a plurality of skeleton coordinate values or model choreography data is overlapped with the learner's learning image and output, so that the difference between the learner's motion and the choreographer's motion can be easily identified.

In addition, the present invention can provide an objective evaluation of the learner's own choreography by displaying the score for the learner's learning image.

In addition, the present invention can give a sense of achievement to the learner and maximize the motivation for learning by calculating a score for each section and allowing learning to proceed to the next section when the score is greater than or equal to a certain value.

1 is a diagram schematically showing the configuration of a motion recognition choreography learning apparatus according to an embodiment of the present invention.
2 is a diagram illustrating an example of a home screen displayed on a motion recognition choreography learning apparatus according to an embodiment of the present invention.
3 is a view showing an example of a setting screen displayed on the motion recognition choreography learning apparatus according to an embodiment of the present invention.
FIG. 4 is a block diagram showing the configuration of the exemplary choreography data generator shown in FIG. 1 .
5 is a diagram illustrating an example of a skeleton coordinate value.
FIG. 6 is a block diagram showing the configuration of the learning choreography data generating unit shown in FIG. 1 .
7 is a diagram illustrating an example of a screen in which second skeleton coordinate values are overlapped with a training image.

Like reference numerals refer to substantially identical elements throughout. In the following description, a detailed description of configurations and functions known in the art and cases not related to the core configuration of the present invention may be omitted. The meaning of the terms described in this specification should be understood as follows.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims.

The shapes, sizes, proportions, angles, numbers, etc. disclosed in the drawings for explaining the embodiments of the present invention are illustrative and the present invention is not limited to the illustrated matters. Like reference numerals refer to like elements throughout. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

When 'including', 'having', 'consisting', etc. mentioned in this specification are used, other parts may be added unless 'only' is used. When a component is expressed in the singular, cases including the plural are included unless otherwise explicitly stated.

In interpreting the components, it is construed as including an error range even if there is no separate explicit description.

In the case of a description of the positional relationship, for example, when the positional relationship of two parts is described as 'on', 'on', 'on', 'beside', etc., 'right' Alternatively, one or more other parts may be positioned between two parts unless 'directly' is used.

In the case of a description of a temporal relationship, for example, 'immediately' or 'directly' when a temporal relationship is described with 'after', 'following', 'after', 'before', etc. It may include cases that are not continuous unless this is used.

Although the first, second, etc. are used to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the spirit of the present invention.

The term “at least one” should be understood to include all possible combinations from one or more related items. For example, the meaning of “at least one of the first, second, and third items” means that each of the first, second, or third items as well as two of the first, second and third items are It may mean a combination of all items that can be presented from more than one.

Each feature of the various embodiments of the present invention can be partially or wholly combined or combined with each other, technically various interlocking and driving are possible, and each of the embodiments may be independently implemented with respect to each other or implemented together in a related relationship. may be

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram schematically showing the configuration of a motion recognition choreography learning apparatus according to an embodiment of the present invention, and FIG. 2 is an example of a home screen displayed on the motion recognition choreography learning apparatus according to an embodiment of the present invention 3 is a diagram showing an example of a setting screen displayed on a motion recognition choreography learning apparatus according to an embodiment of the present invention.

1 to 3 , the motion recognition choreography learning apparatus 100 according to an embodiment of the present invention provides feedback on a choreography image of a choreographer and a choreography action of a learner. To this end, the motion recognition choreography learning apparatus 100 may be installed with a choreography learning application (or choreography learning program). The choreography learning application (or choreography learning program) may be downloaded through an online market such as an app store and installed in the motion recognition choreography learning apparatus 100 or may be installed in the device when manufactured by a device manufacturer.

The motion recognition choreography learning apparatus 100 includes a camera 110 , a display unit 120 , a setting unit 130 , and a choreography learning unit 140 as shown in FIG. 1 .

The camera 110 captures the learner's choreography and acquires the learner's learning image data in real time. The camera 110 may be built in the motion recognition choreography learning apparatus 110, but is not necessarily limited thereto. The camera 110 may be a separate device connected to the motion recognition choreography learning apparatus 110 .

The display unit 120 displays a screen provided by the choreography learning application. Specifically, when the choreography learning application is executed by the learner, the display unit 120 may display the home screen. As an example, the home screen may display information on a plurality of choreography learning videos as shown in FIG. 2 . In this case, the information on the choreography learning video may include information about music and a choreographer 210 , information about user preference 220 , and a learning icon 230 . The learner may select any one of a plurality of choreography learning videos to start choreography learning.

Meanwhile, when any one of a plurality of choreography learning videos is selected, the display unit 120 may display a setting screen for the selected choreography learning video. For example, as shown in FIG. 3 , the setting screen may display speed icons 310 that can adjust the playback speed of the choreography image, for example, a 0.5x icon, a 0.7x icon, and a 1.0x icon. The learner may select any one of the plurality of speed icons 310 to adjust the playback speed of the choreography image.

Also, the display unit 120 may further display a section setting icon (not shown) for setting a section in one choreography image. The learner can set the learning section through the section setting icon and can also adjust the playback speed for the set section.

The display unit 120 may provide a learning screen on which at least one of a choreography image of a choreographer and a learning image of a learner is displayed. The learning screen will be described in more detail with reference to FIGS. 5 to 7 .

The setting unit 130 sets learning information. Specifically, the setting unit 130 may set information about a choreography image that the learner wants to learn, information about a section to be learned from the choreography image, and a playback speed.

The learning information may be set by the learner through the screen provided by the choreography learning application, but is not necessarily limited thereto. The learning information may have an initial setting value or a value set by a person who provides choreography learning. The setting unit 130 may change the learning information according to a learner's request even if the learning information has a preset value.

The choreography learning unit 140 provides the choreography image of the choreographer, and provides a learning result obtained by comparing the choreography image of the choreographer with the learning image of the learner. Specifically, the choreography learning unit 140 may receive the learner's learning image data from the camera 110 in real time. The choreography learning unit 140 may compare and analyze the learner's learning image data and the choreographer's choreography image data, and calculate a score for the learner's choreography based on the analyzed result. The choreography learning unit 140 may evaluate the choreography mastery level based on the learner's choreography score.

The choreography learning unit 140 may divide the choreography image of the choreographer into a plurality of sections, and evaluate the learner's choreography familiarity with respect to each of the plurality of sections. When the choreography mastery level for a specific section is equal to or greater than a predetermined value, the choreography learning unit 140 may evaluate the choreography mastery level for the next section by proceeding to the next section.

The choreography learning unit 140 includes a model choreography data storage unit 160 , a learning choreography data generation unit 170 , a data comparison unit 180 , and a score calculation unit 190 in order to evaluate the choreography mastery level. In an embodiment, the choreography learning unit 140 may further include a model choreography data generating unit 150 .

The model choreography data generator 150 generates model choreography data including a plurality of first skeleton coordinate values for each frame based on the choreography image of the choreographer.

In FIG. 1 , the choreography learning unit 140 includes the exemplary choreography data generating unit 150 , but the present invention is not limited thereto. When the motion recognition choreography learning apparatus 100 downloads a choreography learning application from an external choreography learning server (not shown) through a network, exemplary choreography data may also be downloaded along with the choreography image data of the choreographer. In this case, the motion recognition choreography learning apparatus 100 may not include a separate model choreography data generator 150 .

Hereinafter, the exemplary choreography data generating unit 150 will be described in detail with reference to FIGS. 4 and 5 .

4 is a block diagram showing the configuration of the exemplary choreography data generator shown in FIG. 1 , and FIG. 5 is a diagram showing an example of skeleton coordinate values.

4 and 5 , the exemplary choreography data generation unit 150 according to an embodiment of the present invention includes a first image center point determination unit 420 , a first skeleton coordinate value extraction unit 430 , and a first skeleton It includes a central coordinate determining unit 440 and a first skeleton coordinate value correcting unit 450 . In an embodiment, the model choreography data generation unit 150 may further include a first size adjustment unit 410 .

The first size adjustment unit 410 may adjust the size of the choreography image of the choreographer to a preset size. Specifically, the first size adjustment unit 410 may check the size of the choreography image of the choreographer. When the size of the choreography image is different from the preset size, the first size adjustment unit 410 may change the size of the choreography image to a preset size.

For example, the image size may be preset to 1280*720. When the size of the choreography image of the choreographer is 1920*1080, the first size adjuster 410 may resize the choreography image to 1280*720.

The motion recognition choreography learning apparatus 100 according to an embodiment of the present invention adjusts the size of the choreographer's choreography image through the first size adjustment unit 410, thereby scaling the choreographer's choreography image and the learner's learning image. ) can match the information. Through this, the motion recognition choreography learning apparatus 100 according to an embodiment of the present invention can prevent recognition of different motions depending on the size of the image despite the same motion.

In FIG. 4 , the exemplary choreography data generating unit 150 according to an embodiment of the present invention is illustrated as including the first size adjusting unit 410 , but the present invention is not limited thereto. The motion recognition choreography learning apparatus 100 according to an embodiment of the present invention may adjust the size of the learner's learning image to the size of the choreography image of the choreographer. In this case, in the exemplary choreography data generating unit 150 according to an embodiment of the present invention, the first size adjusting unit 410 may be omitted.

The first image center point determiner 420 determines a first image center point for each of the plurality of choreography images included in the choreography image of the choreographer.

Specifically, the choreography image of the choreographer may include a plurality of choreography images divided into frames. The first image center point determiner 420 may divide the choreography image of the choreographer into N frames per second. The N may represent a natural number greater than 0. For example, when the reproduction time of the choreography image A is 4 minutes and the image is divided into 15 frames per second, the first image center point determiner 420 may divide the choreography image A into 60 frames. A choreography image may be divided into 60 choreography images.

The first image center point determiner 420 determines a first image center point for each of the plurality of choreography images. The first image center point determiner 420 may determine the center point based on the size of the choreography image. For example, when the size of the choreography image is 1280*720, the size of the plurality of choreography images included in the choreography image is also 1280*720. The first image center point determiner 420 may determine 640, which is the middle of 1280, as the X-axis value, and may determine 360, which is the middle of 720, as the Y-axis value. As a result, the first image center point determiner 420 may determine (640, 360) as the first image center point.

The first skeleton coordinate value extraction unit 430 extracts a plurality of first skeleton coordinate values from each choreography image. The plurality of first skeleton coordinate values are information that may represent a characteristic of a person's motion, and may indicate a location of a specific body of a person. As an example, the plurality of first skeleton coordinate values may include location information of 17 keypoints as shown in FIG. 5 . Here, 17 keypoints are left eye, right eye, left ear, right ear, nose, left shoulder, right right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee (left knee), right knee (right knee), left ankle (left ankle), and may include a right ankle (right ankle).

The first skeleton coordinate value extractor 430 may extract a plurality of first skeleton coordinate values from each choreography image by using a deep learning algorithm. The deep learning algorithm may be a convolutional neural network (CNN) algorithm or a recurrent neural network (RNN). As an example, the deep learning algorithm may be a ResNet (Residual Network) algorithm.

The first skeleton center coordinate determiner 440 determines the first skeleton center coordinates based on the plurality of extracted first skeleton coordinate values. The first skeleton center coordinate determiner 440 may determine the first skeleton center coordinates by using at least two of the plurality of first skeleton coordinate values.

In one embodiment, the first skeleton central coordinate determining unit 440 determines the first skeleton central coordinates using the coordinate values of the left hip and the coordinate values of the right hip close to the center of the human body. can decide The first skeleton center coordinate determiner 440 may determine the center of the coordinate values of the left hip and the coordinate values of the right hip as the first skeleton center coordinates. The first skeleton center coordinate determiner 440 may calculate the first skeleton center coordinates using Equation 1 below.

The left hip may indicate a coordinate value of a left hip, and the right hip may indicate a coordinate value of a right hip. For example, when the coordinate value of the left hip is (640, 500) and the coordinate value of the right hip is (642, 300), the first skeleton center coordinate determiner 440 is (641) , 400) may be determined as the center coordinates of the first skeleton.

The first skeleton coordinate value correcting unit 450 corrects the plurality of first skeleton coordinate values so that the first image center point and the first skeleton center coordinate coincide. Specifically, the first skeleton coordinate value corrector 450 may calculate a distance between the first image center point and the first skeleton center coordinate. In an embodiment, the first skeleton coordinate value correcting unit 450 may calculate a distance value between the first image center point and the first skeleton center coordinate using Equation 2 below.

The distance value represents a difference between the center coordinates of the first skeleton and the center point of the first image, and may include an X-axis distance value and a Y-axis distance value.

The first skeleton coordinate value corrector 450 may correct the plurality of first skeleton coordinate values so that the first image center point and the first skeleton center coordinate coincide with each other using the calculated distance value.

Specifically, the first skeleton coordinate value corrector 450 may add or subtract the distance value to each of the plurality of first skeleton coordinate values. For example, when the X-axis value of the first skeleton center coordinate is greater than the X-axis value of the first image center point, the first skeleton coordinate value correction unit 450 subtracts the X-axis distance value from each of the plurality of first skeleton coordinate values. can Alternatively, when the X-axis value of the first skeleton center coordinate is smaller than the X-axis value of the first image center point, the first skeleton coordinate value correction unit 450 adds the X-axis distance value to each of the plurality of first skeleton coordinate values. can

And, when the Y-axis value of the first skeleton center coordinate is greater than the Y-axis value of the first image center point, the first skeleton coordinate value correction unit 450 may subtract the Y-axis distance value from each of the plurality of first skeleton coordinate values. there is. Alternatively, if the Y-axis value of the first skeleton center coordinate is smaller than the Y-axis value of the first image center point, the first skeleton coordinate value correction unit 450 adds the Y-axis distance value to each of the plurality of first skeleton coordinate values. can

Referring back to FIG. 1 , the model choreography data storage unit 160 stores model choreography data that is generated for each frame of a choreography image of the choreographer and includes a plurality of first skeleton coordinate values.

The model choreography data storage unit 160 may store the model choreography data generated by the model choreography data generator 150 , but is not limited thereto. The exemplary choreography data storage unit 160 may store exemplary choreography data provided from an external choreography learning server (not shown).

The learning choreography data generating unit 170 generates learning choreography data including a plurality of second skeleton coordinate values for each frame based on the learner's learning image input in real time through the camera 110 . Hereinafter, the learning choreography data generating unit 170 will be described in detail with reference to FIGS. 6 and 7 .

6 is a block diagram showing the configuration of the training choreography data generator shown in FIG. 1 , and FIG. 7 is a diagram showing an example of a screen in which second skeleton coordinate values are overlapped with a training image.

6 and 7 , the learning choreography data generating unit 170 according to an embodiment of the present invention includes a second image center point determiner 620 , a second skeleton coordinate value extractor 630 , and a second skeleton It includes a central coordinate determining unit 640 and a second skeleton coordinate value correcting unit 650 . In an embodiment, the learning choreography data generating unit 170 may further include at least one of the second size adjusting unit 610 and the correcting unit 660 .

The second size adjustment unit 610 may adjust the size of the learner's learning image to the same size as that of the choreographer's choreography image or to a preset size. Specifically, the second size adjustment unit 640 may check the size of the learner's learning image. When the size of the learning image of the learner is different from the size of the choreography image or a preset size, the second size adjusting unit 640 may change the size of the learning image to the size of the choreography image or a preset size.

For example, the image size may be preset to 1280*720. When the size of the learner's learning image is 1920*1080, the second size adjustment unit 610 may resize the learning image to 1280*720.

The motion recognition choreography learning apparatus 100 according to an embodiment of the present invention adjusts the size of the learner's learning image through the second size adjustment unit 610, thereby scaling the choreography image of the choreographer and the learning image of the learner. ) can match the information. Through this, the motion recognition choreography learning apparatus 100 according to an embodiment of the present invention can prevent recognition of different motions depending on the size of the image despite the same motion.

The second image center point determiner 620 determines a second image center point for each of the plurality of learning images included in the learner's learning image.

Specifically, the learning image of the learner may include a plurality of learning images divided into frames. The second image center point determiner 620 may divide the learner's learning image into M frames per second. M may represent a natural number greater than 0. For example, when the reproduction time of the learning image B is 2 minutes and the image is divided into 15 frames per second, the second image center point determiner 620 may divide the learning image B into 30 frames. The B training image may be divided into 30 training images.

Meanwhile, the choreographer's choreography video and the learner's learning video for the same choreography may have the same frame rate. Since the learner can learn section according to the selection, the playback time of the choreographer's choreography video and the learner's learning video may be different. However, even if the playback time is different, the frame rate of the choreography image of the choreographer and the learning image of the learner may be the same. In this case, the choreography image of the choreography image and the learning image of the learning image are matched one-to-one, so that it is easy to calculate the degree of similarity.

The second image center point determiner 620 determines a second image center point for each of the plurality of learning images. The second image center point determiner 620 may determine the center point based on the size of the training image. For example, when the size of the training image is 1280*720, the size of the plurality of training images included in the training image is also 1280*720. The second image center point determiner 620 may determine 640, which is the middle of 1280, as the X-axis value, and may determine 360, which is the middle of 720, as the Y-axis value. As a result, the second image center point determiner 620 may determine (640, 360) as the second image center point.

The second skeleton coordinate value extraction unit 630 extracts a plurality of second skeleton coordinate values from each training image. The plurality of second skeleton coordinate values are information that may represent characteristics of a person's motion, and may indicate a location of a specific body of a person. As an example, the plurality of second skeleton coordinate values may include location information of 17 keypoints, like the first skeleton coordinate values of the choreography image. Here, 17 keypoints are left eye, right eye, left ear, right ear, nose, left shoulder, right right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee (left knee), right knee (right knee), left ankle (left ankle), and may include a right ankle (right ankle).

The second skeleton coordinate value extractor 630 may extract a plurality of second skeleton coordinate values from each training image by using a deep learning algorithm. The deep learning algorithm may be a convolutional neural network (CNN) algorithm or a recurrent neural network (RNN). As an example, the deep learning algorithm may be a ResNet (Residual Network) algorithm.

The second skeleton center coordinate determiner 640 determines the second skeleton center coordinates based on the plurality of extracted second skeleton coordinate values. The second skeleton center coordinate determiner 640 may determine the second skeleton center coordinates by using at least two of the plurality of second skeleton coordinate values.

In one embodiment, the second skeleton central coordinate determiner 640 determines the second skeleton central coordinates using the coordinate values of the left hip and the coordinate values of the right hip close to the center of the human body. can decide The second skeleton center coordinate determiner 640 may determine the center of the coordinate value of the left hip and the coordinate value of the right hip as the second skeleton center coordinate. The second skeleton center coordinate determiner 640 may calculate the second skeleton center coordinates using Equation 1 above.

The second skeleton coordinate value correction unit 650 corrects the plurality of second skeleton coordinate values so that the second image center point and the second skeleton center coordinate coincide. Specifically, the second skeleton coordinate value corrector 650 may calculate a distance between the second image center point and the second skeleton center coordinates. In an embodiment, the second skeleton coordinate value corrector 650 may calculate a distance value between the first image center point and the first skeleton center coordinate using Equation 2 above.

The second skeleton coordinate value corrector 650 may correct the plurality of second skeleton coordinate values so that the second image center point and the second skeleton center coordinate coincide with each other using the calculated distance value.

Specifically, the second skeleton coordinate value corrector 650 may add or subtract the distance value to each of the plurality of first skeleton coordinate values. For example, if the X-axis value of the second skeleton center coordinate is greater than the X-axis value of the second image center point, the second skeleton coordinate value correction unit 650 subtracts the X-axis distance value from each of the plurality of second skeleton coordinate values. can Alternatively, the second skeleton coordinate value correcting unit 650 may add the X-axis distance value to each of the plurality of second skeleton coordinate values when the X-axis value of the second skeleton center coordinate is smaller than the X-axis value of the second image center point. can

And, when the Y-axis value of the second skeleton center coordinate is greater than the Y-axis value of the second image center point, the second skeleton coordinate value correction unit 650 may subtract the Y-axis distance value from each of the plurality of second skeleton coordinate values. there is. Alternatively, when the Y-axis value of the second skeleton center coordinate is smaller than the Y-axis value of the second image center point, the second skeleton coordinate value correction unit 650 adds the Y-axis distance value to each of the plurality of second skeleton coordinate values. can

The correcting unit 660 may correct the second skeleton coordinate values in consideration of the body shape of the choreographer or the shooting angle of the camera 110 .

If the body type of the choreographer and the learner are different, a large difference may occur between the skeleton coordinate values even if the same motion is performed. For example, when the choreographer is taller than the learner, the Y-axis average value of the first skeleton coordinate values may be greater than the Y-axis average value of the second skeleton coordinate values. As the height difference increases, the difference between the Y-axis value of the first skeleton coordinate values and the Y-axis value of the second skeleton coordinate values increases. In this case, the similarity between the first skeleton coordinate values and the second skeleton coordinate values is will be lowered

Also, when the shooting angle of the camera 110 is different from the shooting angle of the choreography image, a difference may occur between the skeleton coordinate values even if the same operation is performed.

In consideration of the foregoing, the learning choreography data generation unit 170 according to an embodiment of the present invention may correct the X-axis value and the Y-axis value of the second skeleton coordinate values. Specifically, the corrector 660 may include an X-axis corrector 662 and a Y-axis corrector 664 .

The X-axis corrector 662 may correct the X-axis values of the second skeleton coordinate values so that the distances between the left hip and the right hip in the choreography image and the learning image match.

In an embodiment, the X-axis compensator 662 determines that the distance on the X-axis of the coordinate value of the left hip and the coordinate value of the right hip among the second skeleton coordinate values is the first skeleton coordinate. The X-axis values of the second skeleton coordinate values may be corrected to be equal to the distance on the X-axis of the coordinate values of the left hip and the coordinate values of the right hip among the values.

The Y-axis correction unit 664 is a left shoulder, a right shoulder, a left eye, a right eye, a left ear, and a right in the choreography image and the learning image. The Y-axis value of the second skeleton coordinate values may be corrected so that the distance between any one of the right ear and any one of the left ankle and the right ankle matches.

In an embodiment, the Y-axis corrector 664 is a left shoulder coordinate value, a right shoulder coordinate value, and a left eye coordinate value among the second skeleton coordinate values. , the coordinate value of the right eye, the coordinate value of the left ear, the coordinate value of the right ear, the coordinate value of the left ankle, and the coordinate value of the right ankle of the first skeleton coordinate values, the maximum distance on the Y-axis is the coordinate value of the left shoulder, the coordinate value of the right shoulder, the coordinate value of the left eye, and the coordinate value of the right eye. The maximum distance on the Y axis of the coordinate values of , left ear, right ear, left ankle, and right ankle coordinates To be the same, the Y-axis values of the second skeleton coordinate values may be corrected.

For example, the Y-axis corrector 664 determines that the distance on the Y-axis of the coordinate value of the left shoulder and the coordinate value of the left ankle among the second skeleton coordinate values is the first skeleton coordinate values. The Y-axis value of the second skeleton coordinate values may be corrected so that the coordinate value of the middle left shoulder and the coordinate value of the left ankle are equal to the distance on the Y-axis.

For another example, the Y-axis corrector 664 may determine that the distance on the Y-axis of the coordinate value of the right eye and the coordinate value of the right ankle among the second skeleton coordinate values is the first skeleton coordinate value. Among them, the Y-axis value of the second skeleton coordinate values may be corrected so that the coordinate value of the right eye and the coordinate value of the right ankle are equal to the distance on the Y-axis.

The motion recognition choreography learning apparatus 100 according to an embodiment of the present invention may provide the learning choreography data generated by the learning choreography data generating unit 170 to the learner through the display unit 120 .

The display unit 120 may include a first area A1 that displays a learner's learning image and a second area A2 that displays a choreography image of a choreographer. In an embodiment, the motion recognition choreography learning apparatus 100 displays learning choreography data including a plurality of second skeleton coordinate values in the first area A1 of the display unit 120 as shown in FIG. 7 . It can be displayed by overlapping the learning image. Although not shown in FIG. 7 , in another embodiment, the motion recognition choreography learning apparatus 100 displays the model choreography data stored in the model choreography data storage unit 160 in the second area A2 of the display unit 120 . It can also be displayed by overlapping the choreography video. In another embodiment, the motion recognition choreography learning apparatus 100 may overlap and display the model choreography data and the learned choreography data on the learning image displayed in the first area A1 of the display unit 120 . In this case, in order to distinguish the model choreography data from the learned choreography data, the model choreography data may be displayed in a first color, and the learned choreography data may be displayed in a second color different from the first color.

The learner can easily recognize the difference between his or her own motion and that of the choreographer through the model choreography data or the learning choreography data displayed on the display unit 120 .

Referring back to FIG. 1 , the data comparison unit 180 compares the model choreography data with the learned choreography data to determine the degree of similarity. Specifically, the data comparison unit 180 may calculate a distance between the first skeleton coordinate values of the model choreography data and the second skeleton coordinate values of the learning choreography data.

In one embodiment, the data comparison unit 180 can calculate the distance between the first skeleton coordinate values of the exemplary choreography data and the second skeleton coordinate values of the learning choreography data by using the Euclidean Distance algorithm. there is. The data comparator 180 may determine the degree of similarity based on the Euclidean distance. The data comparison unit 180 may preset similarity according to the Euclidean distance. The similarity may have a higher value as the Euclidean distance is smaller, and conversely, may have a smaller value as the Euclidean distance is larger. For example, the similarity may be set to 1 when the Euclidean distance is 270 to 370, set to 2 when the Euclidean distance is 371 to 470, and set to 3 when the Euclidean distance is 471 to 570.

Meanwhile, the data comparison unit 180 may determine the similarity between the model choreography data and the learned choreography data for each frame. In this case, the data comparison unit 180 may match the model choreography data and the learning choreography data one-to-one and compare only the corresponding data to determine the similarity, but is not limited thereto.

In another embodiment, the data comparison unit 180 compares the model choreography data of one frame with each of a plurality of learning choreography data input in a predetermined section, and the similarity of the frame based on the learning choreography data having the highest similarity. can be decided

For example, the data comparison unit 180 compares the learning choreography data of the first frame, the second frame immediately before the first frame, and the third frame immediately after the second frame, with the model choreography data of the first frame. can be compared. That is, the data comparison unit 180 may calculate a first distance between the second skeleton coordinate values of the training choreography data of the first frame and the first skeleton coordinate values of the model choreography data of the first frame. The data comparison unit 180 may calculate a second distance between second skeleton coordinate values of the training choreography data of the second frame and first skeleton coordinate values of the model choreography data of the first frame. In addition, the data comparison unit 180 may calculate a third distance between the second skeleton coordinate values of the learning choreography data of the third frame and the first skeleton coordinate values of the model choreography data of the first frame.

The data comparator 180 may determine the similarity in the first frame based on the smallest value among the first distance, the second distance, and the third distance. As a result, the data comparison unit 180 determines the similarity of the frame having the highest similarity among the first frame, the second frame, and the third frame as the similarity between the model choreography data for the first frame and the learning choreography data. can

The motion recognition choreography learning apparatus 100 may acquire a learner's learning image in real time, and may generate learning choreography data for each frame. At this time, the motion recognition choreography learning apparatus 100 may cause a delay in the process of processing data within the choreography learning application or in the process of receiving data for each frame from the camera 110 .

In this case, the motion recognition choreography learning apparatus 100 needs to synchronize the model choreography data and the delayed learning choreography data. To this end, the motion recognition choreography learning apparatus 100 may determine the similarity with respect to the corresponding frame by using not only the corresponding frame but also the learning choreography data of the previous and subsequent frames. The motion recognition choreography learning apparatus 100 may select learning choreography data most in sync with model choreography data from among the learning choreography data of each of a plurality of frames, and determine the similarity with respect to the corresponding frame.

The score calculation unit 190 calculates a score based on the degree of similarity determined by the data comparison unit 180 . Specifically, the score calculator 190 may calculate a score by summing the similarities for each of the plurality of frames. When calculating a score for each section, the score calculator 190 may calculate a score by summing the similarities for each of a plurality of frames included in the corresponding section.

The score calculation unit 190 may provide the calculated score to the learner through the display unit 190 .

The motion recognition choreography learning apparatus 100 according to an embodiment of the present invention can provide an objective evaluation of the learner's own choreography by displaying the score for the learner's learning image.

In addition, the motion recognition choreography learning apparatus 100 according to an embodiment of the present invention calculates a score for each section, and when the score is greater than or equal to a certain value, the learning proceeds to the next section, thereby giving the learner a sense of achievement. there is.

Those skilled in the art to which the present invention pertains will understand that the above-described present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof.

For example, the functions performed by the motion recognition choreography learning apparatus 100 shown in FIG. 1 may be implemented in the form of an application or a program such as an agent. When the functions performed by the motion recognition choreography learning apparatus 100 are implemented as a program, codes for implementing a specific function may be implemented as one program, or a plurality of programs may be divided and implemented, and some of the programs is installed in the motion recognition choreography learning apparatus 100, and some of the programs may be installed in an external choreography learning server (not shown).

Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: motion recognition choreography learning device 110: camera
120: display unit 130: setting unit
140: choreography learning unit 150: model choreography data generation unit
160: model choreography data storage unit 170: learning choreography data generation unit
180: data comparison unit 190: score calculation unit

Claims (15)

a model choreography data storage unit for storing model choreography data including a plurality of first skeleton coordinate values for each frame of a choreography image of a choreographer;
a learning choreography data generation unit for generating learning choreography data including a plurality of second skeleton coordinate values for each frame based on a learner's learning image input in real time;
a data comparison unit that compares the model choreography data with the learned choreography data to determine a degree of similarity; and
A motion recognition choreography learning apparatus comprising a score calculator for calculating a score based on the determined similarity. The method of claim 1,
Motion recognition choreography learning apparatus further comprising a model choreography data generator for generating model choreography data including the plurality of first skeleton coordinate values based on the choreographer's choreography image provided from the outside. The method according to claim 2, wherein the model choreography data generation unit comprises:
a first image center point determiner configured to determine a first image center point with respect to a plurality of choreography images included in the choreography image of the choreographer;
a first skeleton coordinate value extraction unit for extracting the plurality of first skeleton coordinate values from each choreography image;
a first skeleton center coordinate determination unit for determining a first skeleton center coordinate using at least two of the plurality of first skeleton coordinate values; and
A motion recognition choreography learning apparatus comprising a first skeleton coordinate value correcting unit for correcting the plurality of first skeleton coordinate values so that the first image center point and the first skeleton center coordinate coincide. 4. The method of claim 3,
The plurality of first skeleton coordinate values include a right hip coordinate value and a left hip coordinate value of a choreographer,
The first skeleton center coordinate determining unit, Motion recognition choreography learning apparatus, characterized in that for determining the center between the right hip (hip) coordinate value and the left hip coordinate value as the first skeleton center coordinates. The method of claim 3, wherein the first skeleton coordinate value correction unit,
A motion recognition choreography learning apparatus, characterized in that by calculating a distance value between the first image center point and the first skeleton center coordinates, and adding or subtracting the calculated distance value to each of the plurality of first skeleton coordinate values. The method according to claim 3, wherein the model choreography data generation unit comprises:
Motion recognition choreography learning apparatus, characterized in that it further comprises a first size adjustment unit for adjusting the size of the choreography image of the choreographer to a preset size. The method of claim 1, wherein the data comparison unit,
Motion recognition choreography, characterized in that the similarity is calculated by comparing a plurality of learning choreography data input in a certain section and exemplary choreography data in units of frames, and determining the degree of similarity to the corresponding frame based on the learning choreography data with the highest similarity learning device. The method of claim 7, wherein the data comparison unit,
determining the similarity to the first frame by comparing the learning choreography data of the first frame, the second frame immediately before the first frame, and the third frame immediately after the first frame with the model choreography data of the first frame A motion recognition choreography learning device. The method of claim 1, wherein the learning choreography data generation unit,
a second image center point determining unit for determining a second image center point with respect to a plurality of learning images included in the learning image of the learner;
a second skeleton coordinate value extraction unit for extracting the plurality of second skeleton coordinate values from each training image;
a second skeleton center coordinate determination unit for determining second skeleton center coordinates using at least two of the plurality of second skeleton coordinate values; and
Motion recognition choreography learning apparatus comprising a second skeleton coordinate value correcting unit for correcting the plurality of second skeleton coordinate values so that the second image center point and the second skeleton center coordinates coincide. The method of claim 9, wherein the learning choreography data generation unit,
Motion recognition choreography learning apparatus, characterized in that it further comprises a second size adjusting unit for adjusting the size of the learning image of the learner to the same size as the size of the choreography image of the choreographer or a preset size. 10. The method of claim 9,
The plurality of first skeleton coordinate values include the choreographer's right hip coordinate value and the left hip coordinate value, and the plurality of second skeleton coordinate values include the learner's right hip coordinate value and the left hip coordinate value,
The learning choreography data generation unit,
the plurality of second skeleton coordinate values such that the distance on the X axis of the learner's right hip coordinate value and the left hip coordinate value has the same value as the distance on the X axis of the right hip coordinate value and the left hip coordinate value of the choreographer Motion recognition choreography learning apparatus, characterized in that it further comprises an X-axis correction unit for correcting each X-axis value. 10. The method of claim 9,
The plurality of first skeleton coordinate values include a choreographer's right eye coordinate value, left eye coordinate value, right shoulder coordinate value, left shoulder coordinate value, right ankle coordinate value, and left ankle coordinate value, and the plurality of second skeletons The coordinate values include the learner's right eye coordinate value, left eye coordinate value, right shoulder coordinate value, left shoulder coordinate value, right ankle coordinate value and left ankle coordinate value,
The learning choreography data generation unit,
The maximum distance on the Y axis of the learner's right eye coordinate value, left eye coordinate value, right shoulder coordinate value, left shoulder coordinate value, right ankle coordinate value, and left ankle coordinate value is the choreographer's right eye coordinate value, left eye coordinate value Correcting the Y-axis value of each of the plurality of second skeleton coordinate values to have the same value as the maximum distance on the Y-axis of the value, the right shoulder coordinate value, the left shoulder coordinate value, the right ankle coordinate value, and the left ankle coordinate value Motion recognition choreography learning apparatus, characterized in that it further comprises a Y-axis correction unit. The method of claim 1,
Motion recognition choreography learning apparatus, characterized in that the learning image of the learner and the choreography image of the choreographer have the same frame rate. The method of claim 1, wherein the data comparison unit,
Motion recognition choreography learning apparatus, characterized in that by using a Euclidean distance algorithm to calculate the distance between the model choreography data and the learning choreography data, and to determine the degree of similarity based on the calculated distance. The method of claim 1,
A display unit for displaying the learning image of the learner in a first area and displaying the choreography image of the choreographer in a second area,
The display unit, Motion recognition choreography learning apparatus, characterized in that the display overlapping the learning choreography data including the plurality of second skeleton coordinate values on the learning image of the learner.

Images