KR102258128B1 - User motion analysis method for dance training using ai-based image recognition - Google Patents

Abstract

The present invention relates to a user motion analysis method for dance training using artificial intelligence based image recognition and, more particularly, to a user motion analysis method for dance training, which analyzes a dance motion of a user performing dance training in real time, comprising the steps of: (1) receiving image data of a user, who performs dance training, taken by a camera; (2) recognizing the user from the received image data and separating a background from a user area; (3) extracting a skeleton from the separated user area based on recognition of body portions of the user by using joint information and estimating a pose by using the extracted skeleton; and (4) analyzing the user motion for the estimated pose, wherein in step (3), the pose is estimated by using a deep learning based pretrained pose estimation model. According to the user motion analysis method for dance training using the artificial intelligence based image recognition proposed in the present invention, the pose is estimated and the user motion is analyzed with the pose estimation model pre-trained on the basis of the deep learning by using the image data photographed by a general camera, thereby efficiently and accurately analyzing the motion of the user, who performs the dance training, even without a special camera like Kinect.

Description

User motion analysis method for dance training using artificial intelligence-based image recognition {USER MOTION ANALYSIS METHOD FOR DANCE TRAINING USING AI-BASED IMAGE RECOGNITION}

The present invention relates to a user motion analysis method for dance training, and more particularly, to a user motion analysis method for dance training using image recognition based on artificial intelligence.

In general, the Korean Wave refers to a phenomenon in which Korean popular culture is spread abroad and consumed by the public. The Korean Wave has passed the generation period of the Korean Wave (1997 to 2000) and the second stage of the deepening of the Korean Wave (mid-2000s), and is entering the third stage of Korean Wave diversification (after mid-2000). In particular, in the third period of the Korean Wave, the Korean Wave is evolving from a culture that consumes contents such as dramas, music, movies, and games to a culture that wants to directly participate or experience.

With the spread of the Korean Wave, in addition to the K-Pop Academy, users who want to directly train K-Pop dance using video streaming services such as YouTube are increasing. However, since the dance training method using such a video simply imitates the dance movement of an idol in the video uploaded by the user, there is a problem that proper dance training cannot be performed. This is because the user has to determine for himself how accurate the dance movement he is currently performing or which part does not match, while receiving video information only unilaterally through the video without any feedback.

In particular, in the dance training method using a video, there is a problem that it is very difficult for the user to continue the dance training because the user cannot receive any feedback on his/her movement. This is because, without a professional instructor helping to continue the dance training, the user has to repeat the specified movements while only watching the video, so they cannot feel the sense of accomplishment from the dance training at all.

In order to solve such a problem, Patent No. 10-1989447 (name of invention: dance motion feedback system that provides image feedback to users using augmented reality, registration date: June 10, 2019) was disclosed. have.

According to the prior art, feedback is provided as an image to the user through augmented reality, so that the user can follow the dance movement and learn the dance accurately, determine whether the dance movement the user is dancing is correct, and the user's motion image By providing feedback in real time to the user, it is possible to provide feedback in real time even while the user is dancing, and it is possible to support the user to correct the dance movement by himself using image information expressed in augmented reality.

However, in the prior art, since motion is detected using a motion recognition camera, a special camera such as Kinect must be used, and a separate software supporting this is also required. Therefore, in order to expand the scope of use, it is necessary to develop a technology that can more efficiently analyze user motion for dance training.

The present invention is proposed in order to solve the above problems of the previously proposed methods. Using image data captured by a general camera, the present invention estimates a pose using a pre-learned pose estimation model based on deep learning and analyzes user motion. Therefore, it is an object of the present invention to provide a user motion analysis method for dance training using artificial intelligence-based image recognition that can efficiently and accurately analyze the motion of a user performing dance training without a special camera such as Kinect.

A user motion analysis method for dance training using image recognition based on artificial intelligence according to a feature of the present invention for achieving the above object,

As a user motion analysis method for dance training,

Analyzing the dance motion of the user who is performing dance training in real time,

(1) receiving image data captured by a camera of the user performing the dance training;

(2) recognizing a user from the input image data and separating a background from a user region;

(3) extracting a skeleton using joint information based on the user's body part recognition in the separated user area, and estimating a pose using the extracted skeleton; And

(4) comprising the step of analyzing the user motion for the estimated pose,

In step (3), a pose is estimated using a pre-learned pose estimation model based on deep learning.

Preferably,

In the step (1), the two-dimensional image data captured by the camera is processed and converted into a depth map,

In step (2), the user region may be detected and separated from the depth map.

Preferably, the step (2),

(2-1) detecting a plurality of candidate regions including a user image from the input image data; And

(2-2) AI-based image recognition, comprising the step of specifying a user region of the current frame among the plurality of candidate regions and separating it from the background in consideration of the relationship with the user region of the previous frame. User motion analysis method for dance training using

More preferably, in the step (2-2),

Matching scores between the plurality of candidate regions and the user region of the previous frame may be calculated, and the candidate region for which the highest matching score was calculated may be specified as the user region of the current frame.

Preferably, in step (2),

If there are a plurality of users recognized in the input image data, the plurality of user regions may be separated from the background, and each user may be identified by tagging the separated user regions.

Preferably, the step (3),

(3-1) estimating a coordinate distribution map of each joint by using the user region as an input of the pose estimation model;

(3-2) obtaining the coordinates of each joint from the coordinate distribution map of the joint estimated in step (3-1);

(3-3) extracting a skeleton by connecting the coordinates of the joint obtained in step (3-2); And

(3-4) It may include the step of estimating a user pose from the extracted skeleton.

Preferably, in step (4),

The user motion may be analyzed by comparing the human body reference model with the estimated pose.

More preferably, in the step (4),

The similarity between the skeleton of the human body reference model and the skeleton extracted in step (3) can be measured and compared.

According to the user motion analysis method for dance training using image recognition based on artificial intelligence proposed in the present invention, a pose is estimated using a pre-learned pose estimation model based on deep learning using image data captured by a general camera. By analyzing user motion, it is possible to efficiently and accurately analyze the motion of the user who is performing dance training without a special camera such as Kinect.

1 is a diagram illustrating a method for analyzing a motion of a user for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention in comparison with the prior art.
2 is a diagram showing a system configuration for implementing a user motion analysis method for dance training using artificial intelligence-based image recognition according to an embodiment of the present invention.
3 is a diagram illustrating a flow of a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention.
4 is a diagram showing a detailed flow of step S200 in a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention.
5 is a diagram showing a detailed flow of step S300 in a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention.
6 is a diagram illustrating a process of steps S200 and S300 of estimating a pose from input image data in a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating, for example, image data collected to generate a pose estimation model in a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention.

Hereinafter, preferred embodiments will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, in describing a preferred embodiment of the present invention in detail, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. In addition, the same reference numerals are used throughout the drawings for parts having similar functions and functions.

In addition, in the entire specification, when a part is said to be'connected' with another part, it is not only'directly connected', but also'indirectly connected' with another element in the middle. Includes. In addition, "including" a certain component means that other components may be further included rather than excluding other components unless otherwise stated.

FIG. 1 is a diagram illustrating a method of analyzing a motion of a user for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention in comparison with the prior art. As shown in FIG. 1, the user motion analysis method for dance training using artificial intelligence-based image recognition according to an embodiment of the present invention uses complex and expensive equipment such as multiple sensors, 3D depth cameras, and 3D equipment. By overcoming the conventional technology of analyzing video and motion using video and motion analysis, and by incorporating AI technology to the video of a general camera, it is possible to efficiently and accurately analyze the motion of the user who performs dance training without special equipment such as depth camera and Kinect. It's about the method.

2 is a diagram illustrating a system configuration for implementing a user motion analysis method for dance training using artificial intelligence-based image recognition according to an embodiment of the present invention. As shown in FIG. 2, the user motion analysis method for dance training using artificial intelligence-based image recognition according to an embodiment of the present invention is a motion analysis that receives and analyzes a user's dance image from the camera 200. It can be implemented by the device 100. More specifically, the dance training system 10 providing dance training content may include a camera 200, a motion analysis device 100, and a dance training providing device 300, and a dance training providing device ( While 300) provides the dance training content to the user, the motion analysis apparatus 100 may provide various feedbacks based on the user's motion analyzed.

The present invention relates to a user motion analysis method for dance training using artificial intelligence-based image recognition, and may be composed of software recorded in hardware including a memory and a processor. For example, the user motion analysis method for dance training using artificial intelligence-based image recognition of the present invention may be stored and implemented in a personal computer, a notebook computer, a server computer, a PDA, a smart phone, a tablet PC, or the like. Hereinafter, for convenience of description, a subject performing each step may be omitted.

3 is a diagram illustrating a flow of a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention. As shown in FIG. 3, a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention is provided with dance training content through the dance training system 10 to perform dance training. Analyzing the dance motion of the user performing the dance in real time, but receiving the image data captured by the camera 200 of the user performing the dance training (S100), by recognizing the user from the received image data, the background and the user Separating the region (S200), extracting a skeleton using joint information based on the user's body part recognition from the separated user region, estimating a pose using the extracted skeleton (S300), and a user motion for the estimated pose It may be implemented including the step of analyzing (S400).

In step S100, image data captured by the camera 200 of a user performing dance training may be input. Here, the camera 200 is a camera 200 for photographing a general 2D image, and may include all of the cameras 200 provided in a webcam, a laptop computer, a mobile phone, and the like.

According to an embodiment, in step S100, the two-dimensional image data captured by the camera 200 may be processed and converted into a depth map. The depth map means one image containing information related to a distance from an observation point to an object surface in 3D computer graphics. In step S100, the image data captured by the camera 200 is converted to black and white. In addition, the depth map may be generated by converting so that the depth is differentiated and displayed according to the difference in brightness. It is possible to grasp a 3D user motion from 2D image data through the depth map.

In step S200, by recognizing a user from the input image data, the background and the user region may be separated. In this case, in step S200, a user may be recognized through an object recognition model pre-trained based on deep learning, and a background and a user region may be separated. Here, when the image data is converted into a depth map in step S100 and used, in step S200, a user region may be detected and separated from the depth map.

The pre-trained object recognition model in step S200 is a deep learning model trained with a large amount of training data to recognize people and classify them into bounding boxes. Users can be recognized and user domains can be separated using CNN, Mask R-CNN, etc.

In addition, in step S200, if there are a plurality of users recognized in the input image data, the plurality of user regions may be separated from the background, and each user may be identified by tagging the separated user regions. That is, a plurality of users may be photographed in the image data. In step S200, each user area may be separated for each user, and each user may be identified.

Hereinafter, a detailed flow of step S200 will be described in detail with reference to FIG. 6.

4 is a diagram illustrating a detailed flow of step S200 in a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention. As shown in FIG. 4, step S200 of the user motion analysis method for dance training using artificial intelligence-based image recognition according to an embodiment of the present invention includes a plurality of candidates including a user image from the input image data. It may be implemented including the step of detecting the region (S210) and the step (S220) of specifying the user region of the current frame among the plurality of candidate regions and separating it from the background in consideration of the relationship with the user region of the previous frame.

In step S210, a plurality of candidate regions including a user image may be detected from the input image data. Here, at least some of the plurality of candidate regions may overlap. When the user region is detected using the object recognition model, the candidate region is detected in the form of a bounding box, and a plurality of candidate regions having various sizes and shapes can be detected for one target user region. .

In step S220, in consideration of the relationship with the user region of the previous frame, the user region of the current frame among the plurality of candidate regions may be specified and separated from the background. More specifically, in step S220, matching scores between the plurality of candidate regions and the user region of the previous frame may be calculated, and the candidate region for which the highest matching score is calculated may be specified as the user region of the current frame.

According to an embodiment, in step S220, the feature map of each candidate region detected in step S210 and the feature map of the user region specified in the previous frame are used to compare the similarity, and the candidate region with the highest similarity is selected from the current frame. It can be specified as a user area. Therefore, it is possible to accurately track and analyze the appearance of a user performing dance training in a dance training image in which continuous image data is input in a frame unit.

In step S300, in the separated user region, a skeleton may be extracted using joint information based on the user's body part recognition, and a pose may be estimated using the extracted skeleton. Here, in step S300, a pose may be estimated using a pre-learned pose estimation model based on deep learning. In this case, the pose estimation model pre-trained based on deep learning may be Mask R-CNN, Faster R-CNN, MNC, or the like.

Hereinafter, a detailed flow of step S300 will be described in detail with reference to FIG. 5.

5 is a diagram illustrating a detailed flow of step S300 in a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention. As shown in FIG. 5, step S300 of the user motion analysis method for dance training using artificial intelligence-based image recognition according to an embodiment of the present invention includes a user region as an input of a pose estimation model, and each Estimating the coordinate distribution map of the joint (S310), obtaining the coordinates of each joint from the estimated coordinate distribution map of the joint (S320), extracting the skeleton by connecting the coordinates of the joint (S330), and the extracted It may be implemented including the step (S340) of estimating the user pose from the skeleton.

In step S310, the coordinate distribution map of each joint may be estimated by using the user region as an input of the pose estimation model. That is, a feature map is extracted from the user domain using a pose estimation model based on CNN or RNN that is pre-trained with a backbone network, and a coordinate distribution map of each joint in the user domain image is derived by connecting the inference model. can do.

In step S320, the coordinates of each joint may be obtained from the coordinate distribution map of the joint estimated in step S310. That is, in step S320, a plurality of prediction results (a plurality of prediction coordinates) for one joint may be derived as a coordinate distribution map of the joint. In the coordinate distribution map, one coordinate can be obtained for each joint by using the coordinate with the highest probability as the coordinate of the joint. Depending on the embodiment, one joint coordinate may be obtained by applying NMS (Non-Maximum Suppression) to the coordinate distribution map.

In step S330, a skeleton may be extracted by connecting the coordinates of the joint obtained in step S320. In step S320, coordinates as many as the number of joints may be output, and in step S330, a skeleton may be extracted by connecting them. For example, a skeleton can be extracted by connecting 18 joints including the head, neck, left and right shoulders, elbows, wrists, back of hands, chest, waist, pelvis, knees, and ankles in a straight line.

In step S340, a user pose may be estimated from the extracted skeleton. That is, the user's pose may be estimated from the shape of the skeleton extracted in step S330.

Meanwhile, steps S200 and S300 may be processed using a pre-learned pose estimation model based on deep learning. For example, Mask R-CNN is a pose estimation model capable of recognizing a user, specifying a user region, and estimating a user's pose, and may perform both steps S200 and S300.

6 is a diagram illustrating a process of steps S200 and S300 of estimating a pose from input image data in a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention. . As shown in FIG. 6, the user motion analysis method for dance training using artificial intelligence-based image recognition according to an embodiment of the present invention may perform steps S200 and S300 using Mask R-CNN. . More specifically, in step S200, a plurality of candidate regions including a user's appearance may be derived by using the ResNet50-FPN pre-learned to recognize the user.

In step S300, a feature map may be extracted from the candidate region derived in step S200 using RoIAlign. RoIAlign can generate a corrected feature map by accurately aligning the candidate regions (regions of interest, RoI) of the feature map using bilinear interpolation. Classify the class using the corrected feature map as an input to a CNN-based pose estimation model such as ResNet to specify the user area, detect the key-point of the user's body, that is, the coordinates of the joint, and connect the coordinates of each joint to extract the skeleton can do. Mask R-CNN can perform class classification and joint coordinate detection at the same time.

7 is a diagram illustrating, for example, image data collected to generate a pose estimation model in a user motion analysis method for dance training using image recognition based on artificial intelligence according to an embodiment of the present invention. In order to generate a pose estimation model used in a user motion analysis method for dance training using artificial intelligence-based image recognition according to an embodiment of the present invention, a large amount of training data including various postures is required. As shown in FIG. 7, image data of various postures that can be used as learning data for learning human body movements may be collected, classified by posture, and used as data for pre-learning a pose estimation model.

In step S400, a user motion for the estimated pose may be analyzed. That is, in step S400, the user's motion is analyzed using the estimated pose and the analysis result is transmitted to the dance training providing apparatus 300, thereby providing feedback on the user's dance training or providing various image effects. .

Here, in step S400, a user motion may be analyzed by comparing the human body reference model with the estimated pose. More specifically, in step S400, the similarity between the skeleton of the human body reference model and the skeleton extracted in step S300 may be measured and compared. That is, a skeleton-based human body reference model can be constructed to express the human body's dance movement, and the user's dance movement can be verified and evaluated by comparing it with the user's skeleton extracted in step S300.

According to an embodiment, the human body reference model may be configured as a 3D model, and the user pose may be reconstructed into 3D data using the result of comparing the skeleton extracted in step S300 with the human body reference model.

As described above, according to the user motion analysis method for dance training using image recognition based on artificial intelligence proposed in the present invention, a pose previously learned based on deep learning using image data captured by a general camera 200 By estimating the pose using the estimation model and analyzing the user motion, it is possible to efficiently and accurately analyze the motion of the user who performs dance training without a special camera 200 such as Kinect.

Meanwhile, the present invention may include a computer-readable medium including program instructions for performing operations implemented by various communication terminals. For example, a computer-readable medium includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD_ROMs and DVDs, and floptical disks. It may include a hardware device specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like.

Such a computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. At this time, the program instructions recorded in a computer-readable medium may be specially designed and configured to implement the present invention, or may be known to and usable by those skilled in computer software. For example, it may include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.

The present invention described above can be modified or applied in various ways by those of ordinary skill in the technical field to which the present invention belongs, and the scope of the technical idea according to the present invention should be determined by the following claims.

10: dance training system
100: motion analysis device
200: camera
300: device for providing dance training
S100: Step of receiving image data captured by a camera of a user performing dance training
S200: Recognizing a user from the input image data, and separating the background and the user area
S210: Detecting a plurality of candidate regions including a user image from the input image data
S220: In consideration of the relationship with the user area of the previous frame, specifying a user area of the current frame among a plurality of candidate areas and separating it from the background
S300: Extracting a skeleton using joint information based on the user's body part recognition in the separated user area, and estimating a pose using the extracted skeleton
S310: Estimating a coordinate distribution map of each joint by using the user region as an input of a pose estimation model
S320: Acquiring the coordinates of each joint from the estimated coordinate distribution map of the joint
S330: Step of extracting the skeleton by connecting the coordinates of the joint
S340: Estimating a user pose from the extracted skeleton
S400: Analyzing user motion for the estimated pose

Claims (8)

As a user motion analysis method for dance training,
Analyzing the dance motion of the user who is performing dance training in real time,
(1) receiving image data captured by the camera 200 of the user performing the dance training;
(2) recognizing a user from the input image data and separating a background from a user region;
(3) extracting a skeleton using joint information based on the user's body part recognition in the separated user area, and estimating a pose using the extracted skeleton; And
(4) comprising the step of analyzing the user motion for the estimated pose,
In step (3), a pose is estimated using a pre-learned pose estimation model based on deep learning,
In the step (1), the two-dimensional image data captured by the camera 200 is processed and converted into a depth map, but the two-dimensional image data is converted to black and white, and the depth is A depth map is generated by converting so that is displayed by being differentiated, so that a 3D user motion can be grasped from 2D image data through the depth map.
In the step (2), the user area is detected and separated from the depth map,
The step (2),
(2-1) Using ResNet50-FPN, which is pre-learned to recognize the user, a plurality of candidate regions including the user's appearance are detected from the input image data, but the size and shape of each target user region are Detecting a plurality of candidate regions that are different and at least partially overlap each other; And
(2-2) including the step of specifying the user area of the current frame among the plurality of candidate areas and separating it from the background in consideration of the relationship with the user area of the previous frame,
In step (2-2),
Each of the matching scores between the plurality of candidate regions and the user region of the previous frame is calculated, and the candidate region for which the highest matching score is calculated is specified as the user region of the current frame,
The step (3),
(3-1) Using the user region as an input of the pose estimation model, the coordinate distribution map of each joint is estimated, but the candidate region (region of interest, RoI) of the feature map is determined using bilinear interpolation. The corrected feature map is extracted using RoIAlign, which generates a corrected feature map to be aligned, and the corrected feature map is used as an input of a CNN or RNN-based pose estimation model pre-trained with a backbone network. Specifying a user area and connecting an inference model to derive a coordinate distribution map of each joint in the image of the user area;
(3-2) Obtain the coordinates of each joint from the coordinate distribution map of the joint estimated in step (3-1), but apply one for each joint by applying NMS (Non-Maximum Suppression) to the coordinate distribution map. Obtaining joint coordinates of;
(3-3) Connect the coordinates of the joints obtained in step (3-2) to extract the skeleton, but consist of head, neck, left and right shoulders, elbows, wrists, back of hands, chest, waist, pelvis, knees and ankles. Extracting a skeleton by connecting the 18 joints that are formed in a straight line; And
(3-4) including the step of estimating a user pose from the extracted skeleton,
In step (4),
The human body reference model and the estimated pose are compared, but the similarity between the skeleton of the human body reference model and the skeleton extracted in step (3) is measured and compared to analyze the user's motion, and the analysis result is transmitted to the dance training providing device (300). ) To
The human body reference model is composed of a 3D model, and a user pose is reconstructed into 3D data using a result of comparing the skeleton extracted in step (3) with the human body reference model. User motion analysis method for dance training using image recognition.
delete delete delete The method of claim 1, wherein in step (2),
When there are a plurality of users recognized in the input image data, the plurality of user areas are separated from the background, and each user is identified by tagging the separated user areas. User motion analysis method for dance training.
delete delete delete

Images