KR20220084642A - Method and Apparatus of human pose data 2D/3D conversion using deep learning for display and summarization - Google Patents

Abstract

This design uses deep learning technology from a single camera output, a 2D image, to determine the user's age, gender, body proportion, etc. As for the method, in the analysis of each joint (hereafter keypoint), it is possible to model the degree of uncertainty because the relationship between each pixel is preserved using heatmap, graph theory, etc.
In addition, since 3D stereoscopic images can be expressed in the form of coordinates and mesh on the 2D display, a separate device for the 3D display is not required.

Description

Method and method to display and summarize body pose from 2D image to 3D screen using deep learning technology {Method and Apparatus of human pose data 2D/3D conversion using deep learning for display and summarization }

The present invention converts and extracts motion data from a 2D image of the human body into 3D and summarizes the behavior. In detail, the user's information can be determined using deep learning technology in the input stage, It relates to a method of estimating body pose by connecting keypoints of the body after extraction and expressing it in 3D on a 2D display, and a device that summarizes and expresses body pose movements.

Recently, various methods have been used to convert 2D to 3D and display movement on the screen in 3D. Images input through a single camera, RGB-depth camera, and 3D camera are expressed in 3D. Among them, the conversion technology with the highest level of difficulty is a method of expressing an image input from a single camera in 3D. In the past, it was possible to express body movements by attaching marker-based input devices to input devices, or to display 3D images by integrating images input from multiple cameras. Of course, it can be seen that the cost for configuring the system has increased. Also, a separate display device is required for 3D images, and body movements input from RGB-depth camera and 3D camera are of course for 2D/3D conversion. Additional devices such as cost, time, and place were required, which returned to the burden of consumers. And when occlusion of the body occurred, the error increased or it was displayed in another 3D image. Even using deep learning technology, when recognizing body joints and expressing them in 3D, there was a disadvantage in that errors increased during rendering and back propagation. The 2D/3D conversion method using heatmap and graph used deep learning technology to improve the above shortcomings.

In the case of exercise or rehabilitation treatment, it is possible to check the parts that require correction of the 2D body pose, and the method of converting the body pose from a 2D image to a 3D screen and displaying and summarizing it on the display.

The present invention does not require an expensive RGB-depth camera or a system having multiple cameras for 3D images in the image capturing step for recognizing the movement of the human body, and is input from a single camera. It is based on 2D image. In addition, in the method of converting to 2D data or 3D data and displaying it on the 2D display, deep learning technology is used to compare the posture data of a professional athlete and the user's movement data stored in the storage device. Applied to rehabilitation treatment In this case, it is compared with the posture data of the general public.

Afterwards, it is a method that allows you to correct the movement of your posture through the process (including automatic recognition) to directly check the part where there is a lot of difference by looking at the table. It's about knowing which areas need to be corrected and how.

This design uses deep learning technology from a single camera output 2D image to determine the user's age, gender, body proportion, etc., set the user's joint area as a keypoint to detect it, and connect them to recognize a human pose. Regarding to, in the analysis of each joint (hereafter keypoint), it is possible to model the degree of uncertainty because the relationship between each pixel is preserved using heatmap, graph theory, etc.

In addition, since 3D stereoscopic images can be expressed in the form of coordinates and mesh on the 2D display, there is no need for a separate device for the 3D display.

In addition, it is possible to understand the overall posture of movement because the explanation is summarized using a natural language processing deep learning technique that applies a generative summary model that combines supervised learning and reinforcement learning.

By using a pose estimation model to detect and correct the wrong movement of a person exercising, it is possible to prevent injuries due to the wrong movement and to contribute to the effect of exercise and health promotion. In addition, it is possible to solve problems such as the risk of injury or reduction of the warming effect that may occur due to the absence of a professional trainer when the user is exercising. (Applicable to rehabilitation treatment) If you do not know the correct posture when viewing the desired exercise for the first time, you can check the correct posture through the present invention. And even when training at home, you can use your smartphone to check if your movements are correct, preventing injuries due to incorrect movements and bringing the effect of exercise.

Human pose recognition estimation technology has made a breakthrough in the past 5-6 years thanks to deep learning technology, and it is expected that new businesses in various application fields will be created according to the maturity of the technology.

In particular, explosive growth in AR/VR, robotics, and medical fields will occur, and it can be applied to all industries that require a monitor and can be applied to all images using 2D display.

1 is an overall conceptual diagram of a method of expressing a keypoint of a body in a 2D or 3D form on a 2D display after 2D/3D conversion according to the present invention.
2 is a conceptual diagram showing the configuration of a server for storing motion data of a professional player. 3 is a brief summary of motion data by applying natural language processing.

With respect to the embodiments of the present invention disclosed in the text, specific structural or functional descriptions are only exemplified for the purpose of describing the embodiments of the present invention, and the embodiments of the present invention may be implemented in various forms and the text It should not be construed as being limited to the embodiments described in . Since the present invention can have various changes and can have various forms, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Other expressions describing the relationship between elements, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly. The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or a combination thereof exists, but one or more other features or numbers , it should be understood that it does not preclude the presence or addition of steps, operations, components, parts, or combinations thereof.

1 is an overall conceptual diagram of a method of expressing a body keypoint in 2D or 3D form on a 2D display after 2D/3D conversion according to the present invention. The overall operation process is as follows. After the initial posture setting (100,250), the keypoint is extracted (110) by following the user's movement using deep learning technology. Afterwards, when the user wishes to view the 3D image (120), the extracted keypoint is displayed on the 2D display (300) (can also be displayed on the 3D display). If there is a professional player desired by the user (200, 210), the player's posture is started from the initial posture setting (100). After that, the part that is different from the user's movement is displayed on the 2D Dispaly in 3D form. (310) The initial posture setting 100 can perform face recognition and Input is possible using AI technology of pose estimation function. However, since the initial address posture means the starting point of each keypoint of the body, there is a need to match the start synchronization for the pose tracking of the movement compared to the professional athlete desired by the user. In addition, since each person's movement time is different, it is necessary to synchronize the movement of the player and the speed of the user's movement. If the user's movement is slow (it takes a lot of time), it reduces the time by selecting only the main frame using image processing techniques. If the user's starting point is wrong, subsequent motion recognition is not possible. Therefore, the user's posture is recognized at the starting point and posture recognition is started when the user's posture is within the error range. keypoint) is extracted (110). The pose estimation technique uses a mathematical technique to convert 2d to 3d. The methods used in this case are 1) generating a 2d predicted map to which a heatmap using Gaussian distribution is applied, 2) creating a map through 2D -> 3D pose projection, 3) predicting map There is a fused map generation by fusion of a projected map and 1)2)3) can be converted into a cascade. Also, graph theory can be used. Although general convolution does not utilize hierarchical information between keypoints, graph convolution has the advantage of being able to easily utilize it. The user can choose whether or not he wants to see his/her movement as a 3D image. (120) If If you want to see only the connection of keypoints in 2D form, the pose of the whole body is estimated by considering the relation between keypoints and connecting them (130). Furthermore, if the user wants to compare and confirm the movement with the desired professional player, he/she selects the answer to the question for which the user wishes to compare the movement with the professional player (210). After that, the motion data comparison unit 240 in the motion data preamble server 200 calculates the difference distribution between the 2D/3D storage 220 and 230 of each player's motion data and the user's motion data 110, and displays it in 3D data format. Display on the monitor or check the coordinates directly to understand the movement (310).

It summarizes and displays difference motion data by applying natural language processing deep learning technology. (400)

After analyzing the user's golf swing movement in 2d images or 2d images in sports, yoga, and rehabilitation that require body movement recognition, using deep learning techniques, we make data and compare it with data modeling the movements of professional players, the difference The corrective effect is excellent because it is possible to select a point in time when the s is large and express it in 3D so that it is possible to check where the user's body is in the wrong posture in 3D.

100: Initial posture setting (posture recognition starts when there is an error within the allowable range of the joint position of the basic posture)
110: Extracting joint parts from an image through a single camera (using deep learning pose estimation technology)
120: Step of receiving input from the user whether the 3D image is desired
130: Step of displaying the keypoint and connection part of the human body in 2D form
200: Pro player movement data storage server
210: Step of receiving input from the user on whether to compare the movement of the professional player
220: Each player movement 2D/3D data storage
230: player and user movement data comparison unit
240: Description of Default Adjustment/Frame Rate Synchronization
250: user movement data summary display (NLP Based)
300: 3D data displayed on 2D Dispaly Monitor
310: 2D Dispaly the part with a difference in movement in 3D data format
display on monitor
400: summary display of movement

Claims (7)

A display method and apparatus for converting 2D data in an image into 3D data and expressing it on a 2D display, comprising:
estimating user information about the user's initial posture using a deep learning technique;
Comparing the data on the user's initial posture with the basic data and the data of professional players, setting the motion recognition to be performed only within an error range;
extracting feature values of main points (skeleton, feature points) in the 2D image;
inputting an indication that the user wants to confirm the 3D shape;
converting the extracted 2D feature value into a 3D form;
Comparing the 2D/3D feature value with 2D/3D data of a professional player of a motion data storage server of the professional player;
displaying a color according to the degree to which the user needs to be corrected in the 2D and 3D form;
A method of converting the pose of a body from a 2D image to a 3D screen using deep learning technology, which comprises the step of summarizing and expressing the user's movement data using natural language processing technology, and a method of displaying and summarizing the body on a display The method of claim 1,
The step of estimating the user's information about the user's initial posture using the deep learning technique uses deep learning technology, which is characterized by automatically recognizing the matters that accompany changes in movement, such as the user's gender, age, and body size. How to convert body pose from 2D image to 3D screen and how to display and summarize on the display The method of claim 1,
When the recognition of the user's initial posture is within the error range when compared with the basic data or within the error range when compared with the initial posture of a professional athlete, the initial posture recognition is started. How to convert pose from 2D image to 3D screen and how to display and summarize on the display The method of claim 1,
The image is based on 2D and 3D images, and a method of converting body pose from a 2D image to a 3D screen using deep learning technology, characterized in that the body keypoints of the 2D and 3D forms are displayed on the 2D display and the display image How to mark and summarize in The method of claim 1,
Extracting keypoint feature values in 2D images is characterized by using CPM (Convolutional Pose Machine), Heatmap, Graph, etc., using deep learning technology to convert body poses from 2D images to 3D screens and display them on the display and how to summarize The method of claim 1,
The professional athlete's movement data storage server stores the 2D and 3D movement data for the professional athlete and compares the user's 2D and 3D movement data with the user's 2D and 3D movement data. How to convert to screen and how to display and summarize on display A method of converting a body pose from a 2D image to a 3D screen using deep learning technology, which is characterized in that the user's movements can be expressed in a summarized state by applying natural language processing technology, and a method to display and summarize on the display

Images