KR20180094532A - a method to learn and analyze motion of the user by using composed sequential depth images - Google Patents

Abstract

According to the present invention, a system for learning and analyzing a user motion by composing continuous depth images comprises a composite image generation module generating a second depth image by synthesizing first depth images obtained from a depth image camera; a general image analysis module analyzing the first depth image and outputting first motion analysis data corresponding to the first depth image; a composite image analysis module analyzing the second depth image and outputting second motion analysis data corresponding to the second depth image; and a posture analysis result generation module generating a posture analysis result by combining the first motion analysis data and the second motion analysis data.

Description

A method and system for learning and analyzing user motion by synthesizing continuous depth images and a system therefor,

The present invention relates to a method and system for learning and analyzing user actions by composing continuous depth images, and more particularly, to a computer vision and a machine learning technique for learning user's actions.

Computer Vision and Machine learning is a field that extracts meaningful data by analyzing incoming images from various sensors and is used in various industrial fields such as user attitude analysis, unmanned vehicle lane detection, user face recognition have. In addition, as the depth image sensor such as Microsoft Kinect is commercialized recently, more stable user attitude analysis becomes possible, and active research is being conducted on the algorithm for learning this.

FIG. 2 is a diagram illustrating an example in which the depth image and the motion capture data are compared with each other. In general, studies analyzing user attitudes are based on user attitude data from a depth image sensor (30 times per second) and a motion capture system (which can also operate over 120 times per second) to build data to learn user attitude In this case, it is only necessary to construct data based on a relatively slow depth image sensor, so that there is a problem in utilizing only 30 or less data per second. In the case of learning with insufficient data, there is a problem that the accuracy is low in fast pose analysis such as a golf swing or a baseball swing, and the user does not have data at a point where analysis is desired, so that the desired analysis result is not obtained.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and system for learning and analyzing user actions by synthesizing continuous depth images in order to solve the above problems.

Another object of the present invention is to provide a method of synthesizing a plurality of continuous depth images for user posture estimation and learning the depth images. It provides a method to synthesize continuous depth images, learn them by using them as data, and further analyze the user attitude. In general, for the user attitude learning data, a depth sensor operating at 30 times per second acquires a depth image, and at the same time acquires a ground-truth attitude value in a motion capture system capable of operating 120 times or more per second However, since existing methods use only acquired depth image for learning, there is a disadvantage that only 30 data per second is used for learning. In case of giving high speed user operation such as golf swing, there is not sufficient data, There is a limitation that this is not smooth.

 The present invention amplifies a depth image outputted at a speed of 30 times per second using a data synthesis method to generate data 60 times or more per second, and the result of the learning can be used for posture analysis of a sport including a high-speed operation.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to another aspect of the present invention, there is provided a system for synthesizing continuous depth images according to the present invention and learning and analyzing a user operation, the system comprising: a first depth image synthesizing unit for synthesizing first depth images obtained from a depth image camera, module; A general image analysis module for analyzing the first depth image and calculating first motion analysis data corresponding to the first depth image; A composite image analysis module for analyzing the second depth image and calculating second motion analysis data corresponding to the second depth image; And a posture analysis result generation module for generating a posture analysis result by combining the first motion analysis data and the second motion analysis data.

Preferably, the general video learning module; And a composite image learning module.

Wherein the general image learning module learns a relationship between a first depth image obtained from a depth image camera and first motion capture data corresponding to the first depth image, And a relationship between the synthesized second depth image and second motion capture data corresponding to the second depth image is learned.

The general image analysis module analyzes the image using the learning result of the general image learning module, and the composite image analysis module analyzes the result using the learning result of the composite image learning module.

A method for learning and analyzing a user operation by synthesizing continuous depth images according to another aspect of the present invention includes mapping a first depth image obtained from a depth image camera and first motion capture data obtained from a motion capture camera, A general image learning step; A synthetic image learning step of mapping the second depth image synthesized from the first depth image and the second motion capture data and learning the relationship; And an image analysis step of generating a user attitude analysis result from the temporally successive first depth images using the general image learning result and the composite image learning result.

Wherein the image analysis step synthesizes a second depth image from temporally successive first depth images using the general image learning result and the composite image learning result, analyzes the first depth image, Analyzing the second depth image to calculate second motion analysis data, and combining the first motion analysis data and the second motion analysis data to generate a user attitude analysis result .

Wherein the first depth image is obtained by calculating first motion analysis data using a result of learning in the general image learning step, and the first motion analysis data includes motion capture data generated by a motion capture system used in a learning step, Have a similar data structure.

Wherein the second depth image is used to calculate second motion analysis data using the result of learning in the synthetic image learning step and the second motion analysis data includes motion capture data generated by the motion capture system used in the learning step, Have a similar data structure.

According to the present invention, the depth images of the high frequency FPS are generated using the depth image data synthesized from the low frequency FPS depth images, and the mapping of the generated high frequency FPS depth images with the motion capture data, Based on the data of the learned high frequency FPS, precise posture analysis can be performed with precise time.

According to the present invention, the depth image of the low frequency FPS can be temporally amplified with the depth image of the high frequency FPS, and the posture of the user can be analyzed, so that the motion at high speed such as baseball and golf swing can be analyzed more finely.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an exemplary diagram for explaining a configuration of a computer system in which a method of learning and analyzing a user's operation by combining continuous depth images according to the present invention is implemented; FIG.
FIG. 2 is a diagram illustrating an example in which the depth image and the motion capture data are compared with each other. FIG.
FIG. 3 is an exemplary view for explaining a method of composing images using a depth image and motion capture data according to a specific embodiment of the present invention; FIG.
4 is a block diagram for explaining a learning method according to a partial embodiment of the present invention;
5 is a diagram illustrating an example of a depth image synthesis method according to an embodiment of the present invention.
FIG. 6 is an exemplary diagram for explaining a method of executing an attitude analysis module according to a partial embodiment of the present invention; FIG.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. As used herein, the terms " comprises, " and / or "comprising" refer to the presence or absence of one or more other components, steps, operations, and / Or additions.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is an exemplary diagram illustrating a configuration of a computer system in which a method of learning and analyzing a user's operation by combining continuous depth images according to the present invention is implemented.

Meanwhile, a method of synthesizing continuous depth images according to an embodiment of the present invention and learning and analyzing user actions may be implemented in a computer system or recorded on a recording medium. 1, a computer system includes at least one processor 110, a memory 120, a user input device 150, a data communication bus 130, a user output device 160, And may include a storage 140. Each of the above-described components performs data communication via the data communication bus 130. [

The computer system may further include a network interface 170 coupled to the network 180. The processor 110 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 130 and / or the storage 140.

The memory 120 and the storage 140 may include various forms of volatile or non-volatile storage media. For example, the memory 120 may include a ROM 123 and a RAM 126.

Accordingly, a method of synthesizing continuous depth images according to an embodiment of the present invention to learn and analyze user actions can be implemented in a computer-executable method. When a method of synthesizing continuous depth images according to an embodiment of the present invention and learning and analyzing a user's operation is performed in a computer device, computer-readable instructions can perform an operating method according to the present invention.

Meanwhile, the method of learning and analyzing the user operation by synthesizing the continuous depth images according to the present invention described above can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording media storing data that can be decoded by a computer system. For example, there may be a ROM (Read Only Memory), a RAM (Random Access Memory), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device and the like. The computer-readable recording medium may also be distributed and executed in a computer system connected to a computer network and stored and executed as a code that can be read in a distributed manner.

FIG. 2 is a diagram illustrating an example in which the depth image and the motion capture data are compared with each other.

Conventional depth image and user motion capture data analysis methods use only 30 or less data per second for learning by learning based on a depth image (for example, 30 FPS, frame per second) that is somewhat slower, In this way, it is possible to analyze attitude more precisely in time because it learns by generating more than 60 data per second at minimum and analyzes posture based on it.

Generally, studies analyzing user attitudes are generated by a depth image generated by a depth camera (30 FPS, Frame per second) and a motion capture system (possible over 120 FPS) to construct data for learning user attitude The learning step for analyzing the user's posture is performed using the motion capture data. For training, synchronization of the depth image with motion capture data is matched. For motion capture data of 120 FPS corresponding to depth image of 30 FPS, only 30 frames per second are selected to synchronize with depth image. That is, motion capture data of 90 frames per second is discarded. Conventional methods have shown that motion capture systems output more than 120 data per second, but only 30 data per second, which is a quarter of the depth sensor shooting speed limit.

The above-described numerical values are illustrative and do not limit the scope of the invention.

This problem is caused by the fact that the motion capture system is inconvenient to use the motion capture sensor to detect motion of a user and can generate motion data of a high frequency FPS with a relatively high price, Since it takes much time to synthesize images with parallax, in order to extract depth information, there is a difference in generating a depth image of low frequency FPS.

The present invention analyzes a depth image using a database obtained by mapping a depth image obtained by using a low-frequency FPS depth camera and motion capture data of a motion capture system, and calculates posture analysis data similar to motion capture data . If the learning data is sufficient, the posture analysis of the user can be performed only by the depth image obtained from the depth image camera.

In the case of Fig. 2, the motion capture data corresponding to the depth image is displayed. The top figure shows the depth image and the bottom figure shows the line connecting the feature points by visualizing the motion capture data. FIG. 2 assumes a depth image of 30 FPS and motion capture data of 120 FPS. It is not intended to limit the scope of the invention and is not limited to the numerical values set forth. The time interval between successive depth images is 1/30 second, and the motion capture data is displayed with a red line to display 120 FPS motion capture data. There is a problem that three frames between the motion capture data corresponding to the depth image are discarded during learning because there is no corresponding depth image.

In the case of FIG. 2, all depth images are used. The motion caption data in which there is no corresponding depth image is displayed as a line between the motion capture data to be used.

3 is a block diagram illustrating a system for learning and analyzing user activity using a continuous depth image according to the present invention.

The system for learning and analyzing a user operation using a continuous depth image according to the present invention includes a composite image learning module 300, a general image learning module 350, and a posture analysis module 800.

FIG. 3 shows a block diagram of the synthesis learning module and general image learning.

The data synthesis module 310 includes an image synthesis module 313 and a composite data mapping module 316.

4 is a diagram illustrating an exemplary method of synthesizing a depth image according to an embodiment of the present invention.

As described above, since the depth image is a low frequency FPS and the motion capture data has a high frequency FPS, the frames of the motion capture data are wasted, and the result of analyzing based on the low frequency FPS has a problem of low accuracy. To solve this problem, it is possible to generate depth images at an intermediate point by synthesizing depth images having a low frequency FPS.

For example, a depth image of 30 FPS generates an image of 30 frames per second, and a depth image of an intermediate point can be synthesized by using two depth images of successive frames. In FIG. 4, the left image and the right image represent the depth image directly generated by the depth image camera, and the middle image represents the newly generated depth image by using the left depth image and the right depth image. At this time, the depth image obtained from the depth image camera is referred to as a first depth image, and the newly generated depth image by synthesizing the first depth images is referred to as a second depth image.

For example, if a combined depth image (second depth image) between frames is generated using depth images of 30 FPS and combined with the first depth image, a depth image of 60 FPS can be obtained, and a motion of 120 FPS When synchronizing with the capture data, you will be able to learn by mapping 60 frames per second of motion capture data. Since the second depth image composes temporally successive depth images, the position and shape of the predicted object at the intermediate viewpoint can be determined using the position of the coincident object.

By creating a depth image by composing the continuous depth images, the depth image to be used for user attitude learning and analysis can be amplified. By mapping the motion capture data corresponding to the amplified depth image, the quality of user posture learning and analysis can be increased.

The image synthesis module 313 synthesizes a new depth image (second depth image) from the depth images (first depth images) obtained from the camera. For example, if there are 30 frames of 30 fpss depth image for 1 second and f1, f2, ..., f30 in order, depth images between f5 and f6 are synthesized using f5 and f6. Using the synthesized image between each frame, the depth image of 30 fps is amplified to a depth image of 60 fps.

However, since the first depth image obtained from the depth image camera and the second depth image synthesized from the first depth images have different methods and have different characteristics, when the same learning method and analysis method are used, And it was found that it was lowered somewhat. Thus, the first depth image is subjected to learning and analysis using a general image learning module and a general image analysis module, and the second depth image is subjected to learning and analysis using a composite image learning module and a composite image analysis module. The database is constructed by combining the learned and analyzed results.

FIG. 5 is an exemplary diagram illustrating a method of mapping a depth image and motion capture data according to an embodiment of the present invention. Referring to FIG.

4, it is possible to use more motion capture data. The motion capture data mapped to the synthesized depth image as well as the conventional depth image can be used. For example, if a depth image (first depth image) of 30 fps is amplified to a depth image of 60 fps (second depth image) by the image synthesizing module 313 and is mapped to 120 fps of motion capture data, 120 motion capture data 60 motion capture data is used per second, and unused motion capture data is reduced from 3/4 (90) to 1/2 (60).

Also, it can be amplified by a depth image of 120 FPS from a depth image of 30 FPS. The depth image can be amplified according to the FPS value of the learned motion capture data.

Also, one second depth image may be generated from three or more first depth images. In the case of generating one second depth image using two first depth images, speed and torque can be used. However, when three or more first depth images are used, acceleration and angular speed can be used, Can be predicted. However, there is a drawback that the calculation time increases.

The image synthesis module 313 synthesizes a plurality of depth images (first depth images) to generate one depth image (second depth image). A new depth image can be generated by synthesizing temporally successive depth images. The data synthesis module performs arithmetic operation between a plurality of pixel data included in the image or calculates synthesized pixel data using the maximum value or the minimum value of the pixel data.

FIG. 5 schematically shows a method of generating a video (central image) obtained by synthesizing temporally continuous images (left and right images) by a method of 'sum'. This is an example and is not intended to limit the method of synthesizing the first depth image.

When synthesizing the images, the image synthesis module 313 may synthesize the images or the pixel data included in each image by reflecting the weights. The weight value is reflected in each pixel data to affect the composite image data.

Generally, when synthesizing the depth image between f5 and f6 by combining the f5 frame and the f6 frame of the depth image, the depth image may be synthesized using only the f5 frame and the f6 frame, but it is possible to synthesize the depth image including f4 and f7 . the moving speed of the object is calculated using f4 and f5 and the moving speed of the object is calculated using f6 and f7 to predict the position of the object more accurately. In a similar manner, a composite depth image can be generated using more frames, and a more accurate synthetic depth image can be generated using more information.

At this time, the above-described functions can be implemented using an artificial neural network method. Since the depth image data is usually two-dimensional data and each node of the artificial neural network will be pixel data, each layer of the artificial neural network must be composed of two-dimensional data. Therefore, A forest method or a support vector mechanical method can be used.

The composite data mapping module 316 is a module for mapping the synthesized depth image and the best motion capture data, and estimates the point of time of the data currently synthesized from the viewpoints of the two synthesized data and maps the motion data with the motion data closest thereto do. For example, the image C obtained by combining the image A at 1 second and the image B at 1.1 seconds without weighting can be estimated to be 1.05 seconds, and the temporally closest motion capture data is regarded as a ground-truth value Mapping.

The synthetic data learning module 320 receives the depth information (second depth image) synthesized by the data synthesis module 310 as an input value and uses the motion capture data corresponding to the second depth image as an output value, It can be to learn by. The learning method is not limited to the use of the artificial neural network module, and the learning method may be varied by learning the relationship between the second depth image and the motion capture data. For example, the learning method may be a random forest method or a support vector machine method.

The general image learning module 350 includes a general data mapping module 360 and a general data learning module 370.

The general data learning module and the composite data learning module are respectively called the first depth image and the second depth image, and the output values are motion capture data corresponding to the first depth image and motion capture data corresponding to the second depth image, respectively There is a difference in point. The learning method itself may be the same, but when the learning method using the artificial neural network is used, the learned results (matrix between layers) will be different due to the difference in characteristics between the first depth image and the second depth image.

The general data mapping module 360 maps the motion capture data near the start point of the depth image obtained from the depth image camera. The general data learning module 370 is similar to the synthetic data learning module 320 of the composite image analysis module 300. The synthetic image learning module 300 may be different from the general image learning module and the learning method or may be the same. For example, a random forest method may be used to learn a composite image, and a support vector machine method may be used as a method of learning a general image

6 is an exemplary diagram for explaining an attitude analysis module without synthesizing a depth image.

The motion capture data 640 corresponding to the depth image 610 is extracted and the posture analysis module 620 calculates the posture analysis result 630 by associating the depth image 610 and the motion capture data 640 with each other.

The posture analysis method of FIG. 6 will be almost similar to the method used by the general image analysis module of FIGS.

7 is an exemplary diagram for explaining an execution method of the posture analyzing module according to a partial embodiment of the present invention.

8 is a block diagram specifically showing an execution method of the posture analyzing module according to the present invention.

The synthetic image learning module 300 and the general image learning module 350 learn the relationship between the depth image and the motion capture data. The posture analyzing module 800 analyzes the user's posture using the inputted continuous depth image as an input value based on the contents learned in the composite image learning module 300 and the general image learning module 350.

The analyzed user posture can be output in a data structure similar to motion capture data.

A system for synthesizing continuous depth images according to an embodiment of the present invention to learn and analyze a user operation includes a synthesized image generation module for synthesizing continuous depth images to generate a new depth image; A composite image analysis module for calculating and analyzing motion capture data corresponding to the generated depth image; A general image analysis module for calculating and analyzing motion capture data corresponding to a depth image obtained from a depth image camera; And a posture analysis result generation module for generating a posture analysis result by combining the analysis results of the composite image analysis module and the general image analysis module.

The posture analysis module 800 includes a composite image generation module 810, a general image analysis module 830, and a composite image analysis module 820.

The general image analysis module 830 uses the first depth image obtained from the depth image camera as an input value and calculates the posture analysis result 730 of the user using the result learned in the general image learning module 350 and the learning algorithm, .

The composite image generation module 810 has a function similar to that of the image synthesis module 313 of the composite image learning module 300. However, since the synthetic image generation module 810 uses the learning result and the learning algorithm of the synthetic image learning module 300, the synthetic quality can be improved compared to the image synthesis module 313 of the synthetic image learning module 300.

The composite image analysis module 820 receives the composite image 720 generated from the composite image generation module 810 and outputs the user attitude corresponding to the time of the input as an analysis result. The composite image analysis module 830 analyzes the second depth image using the mapping result of the composite data mapping module 316 of the composite image learning module 300 and the learning result of the composite data learning module 320, Results.

The result of analyzing in the general image analysis module 830 and the result of analyzing the synthesized depth image by the synthesized image analysis module 830 are integrated to generate the end user attitude analysis result. For example, a depth sensor having an input speed of 30 Hz can be used to generate a posture analysis result of 60 Hz or more.

The present invention can analyze the posture of the user more frequently over time and more accurately analyze motion at high speed such as baseball and golf swing.

9 is an exemplary diagram illustrating a system for learning and analyzing user actions by synthesizing continuous depth images according to an embodiment of the present invention.

FIG. 9 is a diagram created by combining the contents of FIG. 8 with the contents of FIG.

A system for synthesizing continuous depth images according to the present invention and learning and analyzing a user operation includes a composite image generation module for generating a second depth image by synthesizing first depth images obtained from a depth image camera; A general image analysis module for analyzing the first depth image and calculating first motion analysis data corresponding to the first depth image; A composite image analysis module for analyzing the second depth image and calculating second motion analysis data corresponding to the second depth image; And a posture analysis result generation module for generating a posture analysis result by combining the first motion analysis data and the second motion analysis data.

Preferably, the general video learning module; And a composite image learning module.

Wherein the general image learning module learns a relationship between a first depth image obtained from a depth image camera and first motion capture data corresponding to the first depth image, And a relationship between the synthesized second depth image and second motion capture data corresponding to the second depth image is learned.

The general image analysis module analyzes the image using the learning result of the general image learning module, and the composite image analysis module analyzes the result using the learning result of the composite image learning module.

FIG. 10 is a diagram for explaining a method of learning and analyzing a user operation by synthesizing continuous depth images according to the present invention.

A method for learning and analyzing a user operation by synthesizing continuous depth images according to another aspect of the present invention includes mapping a first depth image obtained from a depth image camera and first motion capture data obtained from a motion capture camera, A general image learning step; A synthetic image learning step of mapping the second depth image synthesized from the first depth image and the second motion capture data and learning the relationship; And an image analysis step of generating a user attitude analysis result from the temporally successive first depth images using the general image learning result and the composite image learning result.

Wherein the image analysis step synthesizes a second depth image from temporally successive first depth images using the general image learning result and the composite image learning result, analyzes the first depth image, Analyzing the second depth image to calculate second motion analysis data, and combining the first motion analysis data and the second motion analysis data to generate a user attitude analysis result .

Wherein the first depth image is obtained by calculating first motion analysis data using a result of learning in the general image learning step, and the first motion analysis data includes motion capture data generated by a motion capture system used in a learning step, Have a similar data structure.

Wherein the second depth image is used to calculate second motion analysis data using the result of learning in the synthetic image learning step and the second motion analysis data includes motion capture data generated by the motion capture system used in the learning step, Have a similar data structure.

While the present invention has been described in detail with reference to the accompanying drawings, it is to be understood that the invention is not limited to the above-described embodiments. Those skilled in the art will appreciate that various modifications, Of course, this is possible. Accordingly, the scope of protection of the present invention should not be limited to the above-described embodiments, but should be determined by the description of the following claims.

100: Computer system
110: Processor
120: Memory
123: ROM
126: RAM
130: Data communication bus
140: Store
150: User input device
160: User output device
170: Network interface
180: Network
300: Composite image learning module
310: Data Synthesis Module
313: image synthesis module
316: Composite Data Mapping Module
320: synthesis data learning module
350: general image learning module
360: Generic Data Mapping Module
370: General data learning module
610: Depth image frames obtained from a depth image camera
620: Posture Analysis Module
630: Posture analysis result
640: Motion capture data frames obtained from the motion capture camera
710: Depth image frames from depth camera
720: depth image frames including the synthesized depth image frame
730: Posture analysis result
800: Posture Analysis Module
810: Composite image generation module
820: Composite image analysis module
830: General image analysis module

Claims (1)

A composite image generation module for generating a second depth image by synthesizing first depth images obtained from the depth image camera;
A general image analysis module for analyzing the first depth image and calculating first motion analysis data corresponding to the first depth image;
A composite image analysis module for analyzing the second depth image and calculating second motion analysis data corresponding to the second depth image; And
A posture analysis result generation module for generating a posture analysis result by combining the first motion analysis data and the second motion analysis data;
The depth image is synthesized to learn and analyze the user's motion.

Images