KR102514697B1 - Apparatus and method for analyzing golf motion - Google Patents

Abstract

A golf motion analysis technique capable of efficiently implementing high-speed shooting and providing more accurate and diverse information to a user by analyzing motions in detail by extracting skeleton information is disclosed. To this end, a golf motion analysis method according to an embodiment of the present invention includes acquiring a two-dimensional image of a user's motion from an image sensor of a camera unit; obtaining a depth image including a depth value for each pixel of the 2D image from a depth sensor of a camera unit temporally alternately with obtaining a 2D image; a photographing speed increasing step of generating a corresponding depth image or a corresponding 2D image corresponding to a reference 2D image or a reference depth image acquired at a predetermined time; outputting the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image as output data for motion analysis; extracting user's skeleton information through analysis of the output data; and displaying the user's motion on a display unit based on the skeleton information.

Description

Apparatus and method for analyzing golf motion {APPARATUS AND METHOD FOR ANALYZING GOLF MOTION}

The present invention relates to an apparatus and method for analyzing golf motions, and in particular, the present invention effectively implements high-speed shooting and analyzes golf motions that can provide more accurate and diverse information to users by analyzing motions in detail by extracting skeleton information. It relates to an apparatus and method.

Golf is a sport where posture is more important than strength. The golf posture is internalized through muscle memory, and in order to accurately internalize the posture, the process of practice and correction must be constantly repeated.

On the other hand, recently introduced camera-based golf motion analysis systems are helping the correction process performed by coaching by experts to be performed by self-learning. Such a camera-based golf motion analysis system includes a camera unit for photographing golf motions and an analysis unit for analyzing captured images. However, there are several problems in the conventional camera-based golf motion analysis system.

First, some motions of the golf swing proceed at high speed, but the conventional camera-based golf motion analysis system has a problem in that some of the high-speed motions cannot be properly filmed due to the limitation of camera shooting speed. In order to improve these problems, a golf motion analysis system using a high-speed camera was developed, but this also had limitations in commercialization in terms of cost.

Problems with some high-speed motions in the golf swing also appeared in the analysis unit. In order to provide real-time information to the user, the analysis unit must have image processing performance equal to or higher than the shooting speed of the camera. However, since the computing equipment does not have such image processing performance, there is a problem in that the real-time information cannot be provided to the user. To improve these problems, a system including high-performance computing equipment has been proposed, but this also has limitations in commercialization in terms of cost and efficiency.

Conventional systems have limitations in the technique of analyzing captured images, because the information extracted from the images by the system is not suitable for motion analysis. For example, conventional systems analyze golf motions by recognizing outlines of the body, extracting body parts, and displaying lines or figures at positions of the body parts specified in this way. However, according to this method, there is a limitation in that the body part cannot be accurately separated from the part where the body overlaps (for example, the part where the arm and the upper body overlap in the address posture during frontal shooting) or the bent angle of the arm cannot be accurately recognized. there was.

In relation, Korean Patent Publication No. 2012-0021848 discloses a 'system for analyzing the posture of a golf swing'.

An object of the present invention is to provide a golf motion analysis system that efficiently implements high-speed shooting.

In addition, an object of the present invention is to provide more accurate and diverse information to the user by analyzing the user's motion in detail.

Another object of the present invention is to enable stable analysis of high-speed motion sections while providing real-time information to users in the analysis of golf swings including high-speed motions.

A golf motion analysis method according to an embodiment of the present invention for achieving the above object includes acquiring a two-dimensional image of a user's motion from an image sensor of a camera unit; acquiring a depth image including a depth value for each pixel of the 2D image from a depth sensor of the camera unit temporally alternately with acquiring the 2D image; a photographing speed increasing step of generating a corresponding depth image or a corresponding 2D image corresponding to a reference 2D image or a reference depth image acquired at a predetermined time; outputting the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image as output data for motion analysis; extracting skeleton information of the user through analysis of the output data; and displaying the motion of the user on a display unit based on the skeleton information.

In this case, the increasing of the photographing speed may include obtaining a first depth image at a first time by the depth sensor; obtaining, by the image sensor, a first 2D image as the reference 2D image at a second time after the first time; generating a second depth image of the first view by projecting the first depth image to the viewpoint of the image sensor; estimating a motion vector between the first 2D image and the second depth image; rendering a third depth image projected at the second viewpoint from the second depth image by using the motion vector; and projecting the third depth image to the depth sensor viewpoint to generate a fourth depth image at the second viewpoint using the corresponding depth image, wherein in the outputting, the first 2-dimensional image and the The fourth depth image may be output as output data.

At this time, the step of increasing the photographing speed may include acquiring a first 2-dimensional image at a first time by the image sensor; obtaining, by the depth sensor, a first depth image as the reference depth image at a second time after the first time; generating a second depth image by projecting the first depth image to the viewpoint of the image sensor; estimating a motion vector between the first 2D image and the second depth image; and rendering a second 2D image at the second viewpoint from the first 2D image to the corresponding 2D image using the motion vector, wherein in the outputting the first depth image and the corresponding 2D image The second 2D image may be output as output data.

At this time, the step of extracting the skeleton information may include extracting pixel data of the output data; a user areaization step of selecting a pixel of an area corresponding to the user based on the pixel data; Calculating a probability distribution value for each of the user-area pixels corresponding to a skeleton joint; defining predetermined pixels as skeleton joints according to the probability distribution values; defining a center point of predetermined pixels defined as the skeleton joint; and completing and extracting the skeleton information based on the center point.

At this time, the step of defining the skeleton joint and the step of defining the center point may include lowering the resolution of the pixel data; Defining skeleton joints and center points based on low resolution; increasing the resolution of the pixel data; setting an area of a predetermined distance from the center point derived based on the low resolution; and re-defining a skeleton joint and a center point based on pixels within the region.

In this case, the pixel data may be at least one of depth information, RGB values, and illuminance values of each pixel.

At this time, in the outputting step, the amount of output data for motion analysis may be adjusted according to the motion speed of the user.

At this time, in the outputting step, when the motion speed of the user is less than a preset value, motion analysis of all of the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image in real time When the motion speed of the user is equal to or greater than a predetermined value, the reference 2D image and the corresponding depth image or a part of the reference depth image and the corresponding 2D image are extracted and output for motion analysis data can be output.

At this time, in the outputting step, if the motion speed of the user is greater than or equal to a predetermined value, extracting the reference 2D image and the corresponding depth image or a part of the reference depth image and the corresponding 2D image to perform motion analysis In the case of outputting the output data for motion analysis, when the motion speed of the user is changed to less than a predetermined value after storing the remaining data, it may be output as output data for motion analysis.

At this time, in the displaying step, among the user's motions, a motion of the user different from a preset reference motion by a predetermined value or more may be displayed.

In addition, a golf motion analysis method according to an embodiment of the present invention for achieving the above object includes acquiring a two-dimensional image of a user's golf motion from an image sensor of a camera unit; obtaining a depth image including a depth value for each pixel of the 2D image from a depth sensor of the camera unit; extracting skeleton information of the user through analysis based on the 2D image and the depth image; and displaying the user's joint position on a display unit based on the 2D image and the skeleton information.

At this time, in the step of obtaining the depth image, the depth image is obtained alternately with acquiring the 2D image, and a reference 2D image obtained at a predetermined time or a corresponding depth image corresponding to the reference depth image Alternatively, a photographing speed increasing step of generating a corresponding 2D image may be further included.

At this time, in the extracting step, the user's skeleton information includes skeleton head information and is extracted, and in the displaying step, head movement from the address section to the impact section of the user's golf motion may be displayed. .

At this time, in the step of extracting, a value corresponding to the foot interval is calculated based on the position value of the ankle joint of the user through analysis of the output data, and in the step of displaying, in the golf motion of the user The foot spacing may be displayed.

At this time, in the step of extracting, the coordinates of the center of gravity of the user and the coordinates of the joint of the right or left foot are calculated, and in the step of displaying, the center of gravity and The positions of the joints of the right foot or the left foot may be displayed.

At this time, in the step of extracting, a value corresponding to the arm angle is calculated based on the coordinates of the joints of the shoulder, elbow, and hand, and in the step of displaying, the shoulder, elbow, and hand in the impact section of the user's golf motion. A line connecting the joints can be displayed.

At this time, in the step of extracting, a value corresponding to the inverted spine angle is calculated based on the coordinates of the head and the left and right pelvis, and in the step of displaying, the left foot or right foot and the left and right pelvis in the backswing section of the user's golf motion A line connecting the midpoint of can be displayed.

In addition, an apparatus for analyzing golf motion according to an embodiment of the present invention for achieving the above object is an image sensor that acquires a two-dimensional image of a user's motion. and a camera unit including a depth sensor which obtains a depth image including a depth value for each pixel of the 2D image alternately with the 2D image in time; a control unit generating a corresponding depth image or a corresponding 2-dimensional image corresponding to a reference 2-dimensional image or a reference depth image acquired by the camera unit at a predetermined time; an output unit outputting the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image as output data; an extractor configured to extract skeleton information of the user through analysis of the output data; and a display unit displaying the motion of the user based on the skeleton information.

According to the present invention, it is possible to provide a golf motion analysis system that efficiently implements high-speed shooting.

In addition, the present invention can provide more accurate and diverse information to the user by extracting skeleton information and analyzing motions in detail.

In addition, in the analysis of a golf swing including a high-speed motion, the present invention enables stable analysis of a high-speed motion section while providing real-time information to the user.

1 is a flowchart for explaining a golf motion analysis method according to an embodiment of the present invention.
2 is a diagram for explaining image acquisition timing of a depth sensor and an image sensor of a camera unit in a golf motion analysis method according to an embodiment of the present invention.
3 is an embodiment of a flowchart for further explaining a step for increasing a shooting speed in a golf motion analysis method according to an embodiment of the present invention.
4 is another embodiment of a flowchart for further explaining a step for increasing a shooting speed in a golf motion analysis method according to an embodiment of the present invention.
5 is a flowchart for explaining in detail the skeleton information extraction step in the golf motion analysis method according to an embodiment of the present invention.
6 and 7 are diagrams for explaining a user areaization step in the skeleton information extraction step.
8 is a diagram for explaining a method of calculating a probability distribution in a skeleton information extraction step.
9 is a diagram for explaining a multi-scale technique for reducing the amount of calculation in a skeleton information extraction step.
10 to 12 are diagrams for explaining the mode change according to the user's operating speed according to the present invention.
13 is a diagram illustrating an example of display on a display unit in a golf motion analysis method according to an embodiment of the present invention.
14 to 16 illustrate examples of display on the display unit when the analyzed user's motion matches the reference motion or when it does not.
17 is an example of displaying the foot spacing in the displaying step.
18 and 19 illustrate an example of determining a Sway through analysis of a user's motion.
20 and 21 illustrate an example of determining hanging back through analysis of a user's motion.
22 and 23 show an example of determining a chicken wing through motion analysis of a user.
24 and 25 illustrate an example of determining a reverse spine angle through motion analysis of a user.
26 illustrates an example of determining a slide through analysis of a user's motion.
27 and 28 illustrate an example of determining a flat shoulder plane through motion analysis of a user.
29 illustrates an example of determining whether a user's head is fixed through motion analysis.
30 is a block diagram showing the configuration of a golf motion analysis device according to an embodiment of the present invention.

The present invention will be described in detail with reference to the accompanying drawings. Here, repeated descriptions, well-known functions that may unnecessarily obscure the subject matter of the present invention, and detailed descriptions of configurations are omitted. Embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clarity.

Hereinafter, a golf motion analysis method according to an embodiment of the present invention will be described.

1 is a flowchart for explaining a golf motion analysis method according to an embodiment of the present invention.

Referring to FIG. 1 , in the golf motion analysis method according to an embodiment of the present invention, first, a two-dimensional image of a user's motion is obtained from an image sensor of a camera unit (S100).

In addition, a depth image including a depth value for each pixel of the 2D image is acquired from the depth sensor of the camera part alternately with the acquisition of the 2D image in step S100 (S200). 2 is a diagram for explaining image acquisition timing of a depth sensor and an image sensor of a camera unit in a golf motion analysis method according to an embodiment of the present invention. In FIG. 2 , each arrow represents a time point for acquiring an image, and the concept of obtaining an image by temporally alternating the depth sensor and the image sensor through steps S100 and S200 is illustrated. For example, if a 2D image is obtained at time t-1, a depth image is obtained at time t, and if a depth image is acquired at time t-1, a 2D image may be obtained at time t.

Thereafter, a corresponding depth image or a corresponding 2D image corresponding to the reference 2D image or the reference depth image acquired at a predetermined time is generated (S300). That is, in step S300, the camera unit synthesizes the 2D image obtained from the image sensor, for example, the RGB image and the depth image obtained from the depth sensor, and increases the frame rate (frame rate per hour) of each image. there is. The camera unit synthesizes the second depth image and the second 2D image corresponding to each of the first 2D image and the first depth image alternately obtained by alternate shooting, and outputs them together, thereby doubling the actual shooting speed. can make it

After step S300, the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image are output as output data for motion analysis (S400). At this time, in step S400, the amount of output data for motion analysis may be adjusted according to the user's motion speed. Details thereof will be described later along with FIGS. 10 to 12 .

Thereafter, the skeleton information of the user is extracted through analysis of the output data of step S400 (S500). In this case, the skeleton information may include skeleton joint information and skeleton head information. Skeleton joint information may include, for example, information on the neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. In addition, skeleton information may be extracted using a randomized decision forest (RDF) technique. Skeleton joints in which the background area and the user area are set based on the depth information value from the pixel data, and the pixels in the user area set based only on the pixel data in which the data range is reduced within the set user area correspond to the skeleton joints in the user's body. It is possible to estimate a pixel corresponding to a specific position in the user's body by deriving a probability distribution for , and classifying a skeleton joint with a high probability distribution value as a region of a pixel.

In step S500, head information is extracted as the user's skeleton information, and a value corresponding to the foot interval can be calculated based on the position value of the user's ankle joint through analysis of the output data. And, in step S500, the coordinates of the user's center of gravity and the coordinates of the right or left foot joints are calculated, the value corresponding to the arm angle is calculated based on the coordinates of the shoulder, elbow, and hand joints, and the coordinates of the head and left and right pelvis Based on this, the value corresponding to the inverted spinal angle can be calculated. In addition, in step S500, the angle between the line connecting the head and the pelvis and the line connecting both shoulders is calculated based on the coordinates of the head, left and right pelvis, and left and right shoulders, and a 3D point cloud is generated based on the pixel data of the output data. can be calculated.

Then, based on the skeleton information, the user's motion is displayed on the display unit (S600). In step S600, among the user's motions, the user's motions different from the preset reference motions by a predetermined value or more may be displayed on the display unit. In step S600, head movement from the address section to the impact section during the user's golf action may be displayed, and the foot distance may be displayed during the user's golf action. In addition, in step S600, the center of gravity and the positions of the right or left foot joints may be displayed in the backswing section or downswing section of the user's golf motion, and the shoulder, elbow, and hand joints in the impact section of the user's golf motion may be displayed, and a line connecting the midpoint of the left or right foot and the left and right pelvis in the backswing section of the user's golf motion may be displayed. In addition, in step S600, a line connecting the head and the pelvis and a line connecting both shoulders may be displayed in the backswing top section of the user's golf motion, and the skeleton information may be displayed as a 3D point cloud on the display unit. .

The golf motion analysis method according to an embodiment of the present invention may be performed including steps S100, S200, S500, and S600 of FIG. 1 . That is, a golf motion analysis method according to another embodiment of the present invention includes acquiring a 2D image of a user's golf motion from an image sensor of a camera unit, and obtaining a depth value for each pixel of the 2D image from a depth sensor of the camera unit. Obtaining a depth image including a depth image, extracting user's skeleton information through analysis based on the two-dimensional image and the depth image, and displaying the user's joint position on the display unit based on the two-dimensional image and the skeleton information steps may be included.

Hereinafter, a method for increasing the shooting speed according to step S300 of FIG. 1 will be described in more detail.

3 is an embodiment of a flowchart for further explaining a step for increasing a shooting speed in a golf motion analysis method according to an embodiment of the present invention.

Referring to FIG. 3 , in an embodiment of step S300, first, the depth sensor acquires a first depth image at a first time point (S311), and the image sensor converts a reference 2D image into a reference 2D image at a second time point after the first time point (S311). A first 2-dimensional image is acquired (S312). The first depth image and the first 2D image obtained in steps S311 and S312 may be the depth image and the 2D image obtained through steps S100 and S200.

After step S312, a first depth image is projected to the viewpoint of the image sensor to generate a second depth image of the first viewpoint (S313).

Then, a motion vector between the first 2D image and the second depth image is estimated (S314).

Thereafter, a third depth image projected at the second viewpoint is rendered from the second depth image using the motion vector (S315).

Then, a third depth image is projected to the depth sensor viewpoint, and a fourth depth image at the second viewpoint is generated as the corresponding depth image (S316). In step S400 of FIG. 1 , the first 2D image and the fourth depth image may be output as output data.

4 is another embodiment of a flowchart for further explaining a step for increasing a shooting speed in a golf motion analysis method according to an embodiment of the present invention.

Referring to FIG. 4 , in another embodiment of step S300, first, an image sensor acquires a first 2D image at a first time point (S321), and a depth sensor converts a reference depth image to a second time point after the first time point (S321). A first depth image is acquired (S322). The first 2D image and the first depth image obtained in steps S321 and S322 may be the depth image and the 2D image obtained through steps S100 and S200.

After step S322, a second depth image is generated by projecting the first depth image to the viewpoint of the image sensor (S323).

Then, a motion vector between the first 2D image and the second depth image is estimated (S324).

Thereafter, a second 2D image at the second viewpoint is rendered from the first 2D image to the corresponding 2D image using the motion vector (S325). In step S400 of FIG. 1 , the first depth image and the second 2D image may be output as output data.

Steps S313 and S323 may be a method of calculating 3D coordinates corresponding to each pixel of the depth image by using internal parameters of the depth sensor. In this case, 3D coordinates are projected onto the image plane of the image sensor by utilizing external parameters between the image sensor and the depth sensor.

In steps S314 and S324, motion information between the first 2D image and the projected second depth image may be estimated. In an embodiment, a motion vector may be calculated by matching each pixel of the first RGB image with pixels existing in a search area of the depth image. Since the modality of the two images is different, a corresponding relationship can be found using the mutual information between the two matching patches:

.

.

In steps S315 and S325, the image at time t can be generated through rendering using the image and motion information at time t-1:

.

.

where x is the pixel coordinate and m(x) is the motion vector at pixel x .

Hereinafter, the step of extracting skeleton information according to step S500 of FIG. 1 will be described in more detail.

5 is a flowchart for explaining in detail the skeleton information extraction step in the golf motion analysis method according to an embodiment of the present invention.

Referring to FIG. 5 , pixel data is extracted from output data acquired through S400 of FIG. 1 (S511). In this case, the pixel data may be at least one of depth information, RGB values, and illuminance values of each pixel.

Then, based on the pixel data, a pixel of an area corresponding to the user is selected (S512). Step S512 is a step of user domain. Referring to FIGS. 6 and 7 , examples of user domains are shown. Step S512 may be implemented in various ways. For example, a distribution between a maximum value and a minimum distance value of each pixel may be obtained, and a group of learned values of a specific distribution may be estimated as a user area.

After step S512, a probability distribution value for whether each of the user-area pixels corresponds to a skeleton joint is calculated (S513). A method of obtaining a probability distribution value in step S513 may be based on a Randomized Decision Forest (RDF) technique. The probability distribution value based on the RDF is obtained by traversing one or more decision trees and summing the probability distribution values stored in leaf nodes. The decision tree includes a root node and child nodes classified in several steps in the same manner as a first child node first classified from the root node and a second child node classified from the first child node in a tree form. do. A child node located at the bottom is defined as a leaf node. Separation of an upper node into a lower node is performed by a node separation variable including a feature variable based on one or more pixel information and a predefined threshold variable. 8 is a diagram for explaining a method of calculating a probability distribution, and shows a process of obtaining a probability distribution through a decision tree when an input depth image and a test pixel are given.

Assuming that a depth image I and a pixel x are given to a decision tree, the following weak classification rule is repeatedly applied to each node starting from the root node until reaching the leaf node to construct the tree traverse:

As a feature, we use a simple depth comparison feature:

Here, θ = (u, v) means a feature parameter composed of two offset vectors. We obtain a discrete probability distribution, Pt(c|I,x), over the joint classes at the leaf nodes of tree t. By combining these distributions over all trees in the random forest, we get the following final Calculate the distribution:

The algorithm to learn the random forest is as follows. For each tree, new training image data is constructed by uniformly sampling with replacement from existing training image data. Using these new training data, the tree is extended by recursively applying the following procedure until certain stopping criteria are satisfied:

a. A set of node separation variables φ = (θ, τ) composed of feature variables θ and threshold variables τ is randomly generated.

b. For each split variable φ, split the training data Q = {(I, x)} into left/right sets as follows:

c. Calculate the following information gain for the left/right set corresponding to the split variable:

Here, H(Q) represents the Shannon entropy of the normalized histogram PQ of the set Q.

d. Select the node separation variable with the largest amount of information increment and assign it to the current node:

e. If the maximum information increment is less than a predefined threshold or if the depth of the current node in the tree reaches the maximum value, the algorithm stops and the tree is returned. Otherwise, apply steps a-d above recursively to the left and right nodes.

According to the probability distribution value calculated through step S513, predetermined pixels are defined as skeleton joints (S514).

Then, the center point of predetermined pixels defined as the skeleton joint is defined (S515).

Steps S514 and S515 include lowering the resolution of the pixel data, defining a skeleton joint and a center point based on the lower resolution, increasing the resolution of the pixel data, and a center point derived based on the lower resolution. The step of setting a region by a predetermined distance and the step of re-defining a skeleton joint and a center point based on pixels within the region may be performed to define the skeleton joint and center point. 9 is a diagram for explaining a multi-scale technique for reducing the amount of calculation in a skeleton information extraction step, and shows an example of a step of defining skeleton joints and center points by adjusting resolution. A method of finding the center point by the RDF method based on a small number of pixels at low resolution and finding the center point by the RDF method at high resolution only for a certain area from the center point. can increase

On the other hand, in order to increase the computational efficiency of RDF, the controller no longer classifies nodes in which each probability distribution value in the nth child node does not exceed a predetermined value, and is a method for classifying leaf nodes in all decision trees. A method of deriving a leaf node value can be applied only to a node with a high probability value rather than obtaining a value.

More specifically, the role of the random forest in the skeleton extraction algorithm is to calculate a probability value corresponding to a specific skeleton joint for each pixel of the input image. This is obtained by traversing each decision tree and summing the probability values stored in the leaf nodes. These probability values mean the contribution of each 3D point to the probability density function for each joint. The basic idea of the cascade method is to ignore points with small contributions. For points of such small contribution, it is not necessary to traverse the decision tree completely, so the random forest computation can be terminated before reaching a leaf node. To implement that idea, we recursively define an auxiliary probability vector Ptn for each node n in decision tree t as follows:

Here, Pt is the probability distribution stored in the leaf node of the original random forest classifier, and P _t ^l and P _t ^r represent auxiliary probability vectors for the two child nodes of node n, respectively. Assuming that P _t ⁿ (c) = ρ after reaching node n during the process of calculating the random forest for input depth I and pixel x, this means that whatever path is selected from the current node n, the probability Pt(c| I,x) does not exceed ρ. Therefore, if ρ is smaller than the predefined small threshold value ρ _node , the cascade algorithm returns a probability value of 0 without further traversing the tree to reduce the amount of computation.

On the other hand, the controller additionally defines a rejection tree that classifies all skeleton joint classes into joint or non-joint classes by combining them into one joint class, uses the rejection tree as an introduction tree of the RDT, and output probability according to the rejection tree. When this value is greater than a predetermined value, a method of proceeding with RDT can be applied.

Specifically, the above cascade idea is expanded to a tree level. The basic idea is to define a new classifier that serves as a quick exit check and place it at the front of the random forest. In order to reduce the additional computational burden due to the new classifier, such a classifier should have a simple structure and be able to be tested quickly. To this end, we define a randomized decision tree that classifies input pixels into joint or non-joint classes, and we define this as a rejection tree. This binary decision tree can be learned by combining all skeleton joint classes into one joint class in the training data introduced for random forest in the previous section. Let P ₀ (c|I,x) be the output probability of the rejection tree. The developed tree cascade method performs random forest calculation only when P ₀ (c|I,x) is greater than the small threshold ρ _tree . For many pixels far from the skeleton joint points, this probability is small enough to ignore these pixels in the further random forest calculation.

Then, based on the center point, the skeleton information is completed and extracted (S516).

Hereinafter, in the golf motion analysis method according to an embodiment of the present invention according to the user's motion speed, a mode change when output data is output will be described.

10 to 12 are diagrams for explaining the mode change according to the user's operating speed according to the present invention.

Referring to FIG. 10 , when the motion speed of the user is less than a predetermined value, all of the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image may be output as output data for motion analysis in real time. . In this real-time motion recognition mode, the user can be photographed at n hz through the camera unit, and the photographed information can be analyzed in real time through the controller. Therefore, image storage memory or result storage memory is not used. The advantage of this mode is that the user's motion can be checked in real time. The control unit of the present invention analyzes the user's motion in the real-time motion recognition mode and can check in real time whether or not the user is ready to take a high-speed motion. As a more specific example, before a golf swing, which is one of the high-speed motions, takes an address posture in which the club is grasped with both hands together. It can be predicted that high-speed operation will take place.

Referring to FIG. 11 , when the user's motion speed is equal to or greater than a preset value, a reference 2D image and a corresponding depth image or a part of the reference depth image and the corresponding 2D image may be extracted and output as output data for motion analysis. . This is because in order for the image captured in the high-speed motion recognition mode to operate in real time, it is necessary to apply both image processing and motion recognition within the high-speed recording cycle time and output the result. A part of it is used for real-time motion recognition result analysis. For example, when the user is performing a high-speed operation, the camera unit takes pictures at a faster cycle (2n hz) than the real-time mode and stores them in an image storage memory. The control unit may perform image processing and motion recognition by extracting only odd-numbered images from the stored image storage memory. The motion recognition result may be stored in the result storage memory and outputted in real time by the display unit. The difference from the first mode is that high-speed images are stored in the image storage memory for later analysis, and the results of analyzing some of the stored high-speed images are output and stored in the result memory at the same time. . The advantage of the high-speed motion recognition mode is, firstly, it checks whether the user has finished the high-speed motion, and if it has ended, the high-speed motion recognition mode is terminated so that the camera can efficiently capture only the necessary part at high speed, and secondly, the image is saved. Memory and result storage Even in the memory, only the high-speed operation part is stored, so the memory can be managed efficiently.

Referring to FIG. 12, when the user's motion speed is equal to or greater than a predetermined value, a reference 2D image and a corresponding depth image or a part of the reference depth image and the corresponding 2D image are extracted and output as output data for motion analysis. After storing the residual data, when the user's motion speed is changed to less than a predetermined value, it may be output as output data for motion analysis. This analyzes the entire high-speed motion photographed through the camera unit in the remaining high-speed motion analysis mode, stores the result of analyzing the entire high-speed motion in the result storage memory, and outputs the analyzed result. In the remaining high-speed motion analysis mode, the display unit may analyze the remaining high-speed motion after the user's finish posture indicating that the golf swing is over and reproduce and provide the user's swing as a slow screen. In addition, the display unit can reproduce a model golf swing on a slow screen, and can also show tips related to golf. Through such a display, the user's perceived time for the waiting time until the remaining high-speed motions are analyzed can be reduced.

Hereinafter, in the golf motion analysis method according to an embodiment of the present invention, the step of displaying the user's motion on the display unit based on the skeleton information will be described in detail.

13 is a diagram illustrating an example of display on a display unit in a golf motion analysis method according to an embodiment of the present invention.

Referring to FIG. 13 , skeleton information may be displayed on a two-dimensional image of the user's action at each step (ADDRESS, TAKE-BACK, BACKSWING TOP, DOWNSWING, IMPACT, FOLLOW THROUGH, FINISH) of the user's golf motion on the display unit. . In addition, it is possible to indicate a change in head position in motion, that is, whether the head is fixed or not and the angle of the leading arm. In addition, the main points in golf motion (ex; foot width, head fixation, balance) and points to pay attention to (ex; sway, hanging back, reverse spine angle, slide), that is, the analysis results for golf motion correction are displayed on the display. It can be. For example, the control unit may extract information on the skeleton joint and the skeleton head, and the display unit may display the location of the skeleton joint as a point on a 2D image. Also, the display unit may display the location of the skeleton head as a circle.

Also, depending on embodiments, lines connecting two or more skeleton joints or skeleton heads may be displayed on a 2D image. For example, the display unit may display lines or figures connecting points of both shoulder joints, both arms joints, and wrist joints on a 2D image. When the controller classifies the golf motion for each section, the display unit may display lines or figures connecting points of both shoulders, arms, and wrist joints on the two-dimensional image obtained in the address section or the impact section, which is the address section. And in the impact section, the angle formed by both shoulders and arms serves as a guide line to form an inverted triangle of a certain shape.

The position or shape of the line or figure connected between the skeleton joint points may be compared with a criterion set in advance by the controller. According to the comparison result, the display unit may display the color or expression method of the line or figure differently.

For example, the controller may calculate an angle formed by the shoulder-elbow-wrist based on the elbow. At this time, if the calculated angle is within a reference range, the line connecting the corresponding skeleton joint may be displayed in green. Otherwise, the controller may display the corresponding line in yellow or red depending on the degree of deviation from the reference range.

Regarding the circle indicating the position of the skeleton head, if the position of the circle is within the standard range, the color of the circle representing the skeleton head is displayed in green. can be displayed 14 to 16 illustrate examples of display on the display unit when the analyzed user's motion matches the reference motion or when it does not. In FIG. 14, since the position of the user's head is within the normal range, it is represented by a solid circle, and in FIGS. 15 and 16, since the position of the user's head is out of the normal range, it is represented by a dotted circle. When the controller classifies the golf motion for each section, the display unit may display a positional change amount (movement distance) of the skeleton joint and the skeleton head between each section. For example, the display unit may display a position change amount of the skeleton head from the address section to the impact section. On the other hand, since the control unit can extract 3D location information from skeleton information, such a location change amount may be a 3D location change amount.

17 is an example of displaying the foot spacing in the displaying step. In the golf motion, the width of the foot is expressed because it is basically required to maintain the width of the shoulder, and the calculation is made based on the following.

,

- 3D coordinates of the joints of both feet:

,

,

- 3D coordinates of both shoulder joints:

,

- discriminant:

18 and 19 illustrate an example of determining a Sway through analysis of a user's motion. Sway means that the lower body moves excessively from the target point during the backswing and the center of gravity moves outward behind the foot. Fig. 18 shows the right posture and Fig. 19 shows the sway. The calculation is based on the following.

- Point cloud:

- 3D coordinates of the center of gravity:

- 3D coordinates of the right foot joint:

- discriminant:

20 and 21 illustrate an example of determining hanging back through analysis of a user's motion. Hanging back means not sending your weight to the lead on the downswing. Fig. 20 shows the right posture and Fig. 21 expresses the hanging bag, and the calculation is made based on the following.

- Point cloud:

- 3D coordinates of the center of gravity:

- 3D coordinates of the joint of the left foot:

- discriminant:

22 and 23 show an example of determining a chicken wing through motion analysis of a user. Chicken wings mean the elbow doesn't straighten at impact. Figure 22 is an expression of the right posture, Figure 23 is a chicken wing, the calculation is made based on the following.

- 3D coordinates of the left shoulder:

- 3D coordinates of the left elbow:

- 3D coordinates of the left hand:

- The angle formed by the shoulder-elbow straight line and the elbow-hand straight line:

- discriminant:

- Repeat the above process for the right side

24 and 25 illustrate an example of determining a reverse spine angle through motion analysis of a user. A reverse vertebral angle means that the upper body is excessively deflected backward during the backswing. Figure 24 shows the upright posture and Figure 25 represents the inverted spine angle, and the calculation is made based on the following.

- 3D coordinates of the head:

- 3D coordinates of the left pelvis:

- 3D coordinates of the right pelvis:

- 3D coordinates of the center of the pelvis:

- discriminant:

26 illustrates an example of determining a slide through analysis of a user's motion. Slide refers to excessive lateral movement of the lower body in the target direction during impact. Figure 26 is a representation of a slide, and calculations are made based on the following.

- 3D coordinates of the left pelvis:

- 3D coordinates of the right pelvis:

- 3D coordinates of the center of the pelvis:

- 3D coordinates of the joint of the left foot:

- discriminant:

27 and 28 illustrate an example of determining a flat shoulder plane through motion analysis of a user. Flat shoulders mean the shoulders are parallel to the ground at the top of the backswing. Fig. 27 shows the right posture, and Fig. 28 shows the flat shoulders, and the calculation is made based on the following.

- 3D coordinates of the head:

- 3D coordinates of the center of the pelvis:

- 3D coordinates of the left shoulder:

- 3D coordinates of the right shoulder:

- The angle formed by the head-pelvic straight line and the shoulder straight line:

- discriminant:

29 illustrates an example of determining whether a user's head is fixed through motion analysis. During the process of address-backswing-backswing-top-downswing-impact during golf motions, the position of the head should not change as much as possible. In FIG. 29, the change of the head position can be displayed on the user's head position, and the calculation is made based on the following.

- 3D coordinates of the head at address:

- 3D coordinates of the head at the top of the backswing:

- 3D coordinates of the head at the time of impact:

- discriminant:

The golf motion analysis method according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes all types of hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, etc. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to act as one or more software modules to perform the operations of the present invention, and vice versa.

The teachings of the principles of the present invention may be implemented as a combination of hardware and software. Also, the software may be implemented as an application program actually implemented on the program storage unit. An application program may be uploaded to and executed by a machine having any suitable architecture. Preferably, the machine may be implemented on a computer platform having hardware such as one or more central processing units (CPU), a computer processor, random access memory (RAM), and input/output (I/O) interfaces. . Additionally, the computer platform may include an operating system and microinstruction code. The various processes and functions described herein may be part of microinstruction code or part of an application program, or any combination thereof, which may be executed by various processing units, including a CPU. Additionally, various other peripherals may be connected to the computer platform, such as additional data storage and printers.

It should be noted that, as some of the constituent system components and methods shown in the accompanying drawings are preferably implemented in software, the actual connections between system components or process function blocks may vary depending on how the principles of the present invention are programmed. need to be further understood. Given the teachings herein, those skilled in the art will be able to contemplate these and similar implementations or configurations of the principles of the present invention.

Hereinafter, the configuration and operation of the apparatus for analyzing golf motions according to an embodiment of the present invention will be described.

30 is a block diagram showing the configuration of a golf motion analysis device according to an embodiment of the present invention.

Referring to FIG. 30 , the golf motion analysis apparatus 100 according to an embodiment of the present invention includes a camera unit 110, a control unit 120, an output unit 130, an extraction unit 140, and a display unit 150. can be configured. This golf motion analysis device 100 shares the technical contents described with FIGS. 1 to 29, and will be described below with a focus on the main technical contents.

The camera unit 110 acquires an image sensor that acquires a 2D image of a user's motion and a depth image including a depth value for each pixel of the 2D image alternately with the acquisition of the 2D image in time It includes a depth sensor that

The controller 120 generates a corresponding depth image or a corresponding 2D image corresponding to the reference 2D image or the reference depth image acquired by the camera unit 110 at a predetermined time.

The output unit 130 outputs the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image as output data.

The extraction unit 140 extracts the user's skeleton information through analysis of the output data.

The display unit 150 displays the user's motion based on the skeleton information.

On the other hand, the display unit 150 may display a continuous motion-video-of the user's golf motion by continuously displaying the 2D image obtained from the camera unit 110.

The controller 120 may extract a 3D point cloud from the depth image obtained from the camera unit 110, and the display unit 150 may output the 3D point cloud on a screen.

The depth image is a two-dimensional function called D(x,y) and can be defined as a function corresponding to a physical distance from a pixel coordinate called (x,y) to a real object corresponding to the pixel called D.

At this time, if a camera parameter for the depth image is known, a 3D coordinate of a 3D point corresponding to each pixel may be calculated using this.

Assuming a simple pinhole camera model, if the focal length of the camera is f and the pixel coordinates of the principal point are (px, py), a 3-dimensional point (X, Y, Z ) and the two-dimensional pixel coordinates (x, y) projected onto the depth image can be expressed as follows.

This is summarized in the following two equations.

Therefore, if the depth D of the pixel (x, y) of the depth image is given, the coordinates of the 3D point corresponding to the pixel are calculated as follows.

When the control unit 120 extracts the 3D point cloud through this process and transmits it to the display unit 150, the display unit 150 outputs the 3D point cloud to the screen. In this case, the display unit 150 may differently control the display time point of the 3D point cloud according to the user's manipulation input (eg, mouse input).

In the analysis of golf motion, information such as bending the waist and maintaining the angle of the arms is also important, and these are factors that are not easy to determine with only the two-dimensional image of the front. According to the present invention, since the control unit 120 calculates a 3D point cloud and the display unit 150 visualizes it, the above information can be easily conveyed to the user.

As described above, the golf motion analysis method and apparatus according to the present invention are not limited to the configuration and method of the embodiments described above, but the above embodiments can be modified in various ways. All or part of them may be selectively combined.

100; Golf motion analysis device
110; camera part
120; control unit
130; output
140; extraction unit
150; display part

Claims (20)

obtaining a 2D image of a user's motion from an image sensor of a camera unit;
acquiring a depth image including a depth value for each pixel of the 2D image from a depth sensor of the camera unit temporally alternately with acquiring the 2D image;
a photographing speed increasing step of generating a corresponding depth image or a corresponding 2D image corresponding to a reference 2D image or a reference depth image acquired at a predetermined time;
outputting the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image as output data for motion analysis;
extracting skeleton information of the user through analysis of the output data; and
and displaying the motion of the user on a display unit based on the skeleton information. The method of claim 1,
In the step of increasing the shooting speed,
obtaining, by the depth sensor, a first depth image at a first time;
obtaining, by the image sensor, a first 2D image as the reference 2D image at a second time after the first time;
generating a second depth image of the first view by projecting the first depth image to the viewpoint of the image sensor;
estimating a motion vector between the first 2D image and the second depth image;
rendering a third depth image projected at the second viewpoint from the second depth image by using the motion vector; and
Projecting the third depth image to the depth sensor viewpoint to generate a fourth depth image at the second viewpoint using the corresponding depth image;
In the outputting step, the first two-dimensional image and the fourth depth image are output as output data. The method of claim 1,
In the step of increasing the shooting speed,
obtaining, by the image sensor, a first 2-dimensional image at a first time;
obtaining, by the depth sensor, a first depth image as the reference depth image at a second time after the first time;
generating a second depth image by projecting the first depth image to the viewpoint of the image sensor;
estimating a motion vector between the first 2D image and the second depth image; and
rendering a second 2-dimensional image at the second point of view from the first 2-dimensional image to the corresponding 2-dimensional image using the motion vector;
In the outputting step, the first depth image and the second two-dimensional image are output as output data. The method of claim 1,
The step of extracting the skeleton information,
extracting pixel data of the output data;
a user areaization step of selecting a pixel of an area corresponding to the user based on the pixel data;
Calculating a probability distribution value for each of the user-area pixels corresponding to a skeleton joint;
defining predetermined pixels as skeleton joints according to the probability distribution values;
defining a center point of predetermined pixels defined as the skeleton joint; and
and completing and extracting the skeleton information based on the center point. The method of claim 4,
The step of defining the skeleton joint and the step of defining the center point,
lowering the resolution of the pixel data;
Defining skeleton joints and center points based on low resolution;
increasing the resolution of the pixel data;
setting an area of a predetermined distance from the center point derived based on the low resolution; and
and re-defining a skeleton joint and a central point based on pixels within the region. The method of claim 4,
Wherein the pixel data is at least one of depth information, RGB values, and illuminance values of each pixel. The method of claim 1,
In the output step,
The method of analyzing golf motion, characterized in that adjusting the amount of output data for motion analysis according to the motion speed of the user. The method of claim 1,
In the outputting step,
When the motion speed of the user is less than a predetermined value, outputting the reference 2D image and the corresponding depth image or all of the reference depth image and the corresponding 2D image as output data for motion analysis in real time;
When the motion speed of the user is equal to or greater than a predetermined value, extracting the reference 2D image and the corresponding depth image or a part of the reference depth image and the corresponding 2D image and outputting the result as output data for motion analysis. Golf motion analysis method. The method of claim 8,
In the outputting step,
When the user's motion speed is equal to or greater than a predetermined value, the reference 2D image and the corresponding depth image or a part of the reference depth image and the corresponding 2D image are extracted and output as output data for motion analysis. A golf motion analysis method characterized in that after storing the data, when the user's motion speed is changed to less than a preset value, outputting the data as output data for motion analysis. The method of claim 1,
In the display step,
A method for analyzing golf motion, characterized in that displaying a motion of the user that is different from a preset reference motion among the motions of the user by a predetermined value or more. obtaining a two-dimensional image of a user's golf motion from an image sensor of a camera unit;
obtaining a depth image including a depth value for each pixel of the 2D image from a depth sensor of the camera unit;
extracting skeleton information of the user through analysis of output data based on the 2D image and the depth image; and
Displaying the user's joint position on a display unit based on the two-dimensional image and the skeleton information;
In the step of acquiring the depth image,
It proceeds to acquire the depth image alternately with acquiring the two-dimensional image,
The golf motion analysis method further comprising a photographing speed increasing step of generating a corresponding depth image or a corresponding 2D image corresponding to a reference 2D image or a reference depth image obtained at a predetermined time. delete The method of claim 11,
In the step of extracting, the user's skeleton information includes skeleton head information and is extracted;
In the displaying, head movement from an address section to an impact section of the golf motion of the user is displayed. The method of claim 11,
In the extracting step, a value corresponding to the foot distance is calculated based on the position value of the user's ankle joint through analysis of the output data;
The method of analyzing golf motion, characterized in that in the displaying, the foot distance is displayed in the golf motion of the user. The method of claim 11,
In the extracting step, the coordinates of the center of gravity of the user and the coordinates of the right or left foot joint are calculated,
In the displaying, the center of gravity and the positions of the joints of the right foot or the left foot are displayed in a backswing section or a downswing section of the user's golf motion. The method of claim 11,
In the extracting step, a value corresponding to the arm angle is calculated based on the coordinates of the shoulder, elbow, and hand joints,
In the displaying, a line connecting the shoulder, elbow, and hand joints in an impact section of the golf motion of the user is displayed. The method of claim 11,
In the extracting step, a value corresponding to the inverted spine angle is calculated based on the coordinates of the head and the left and right pelvis,
In the displaying step, a line connecting the left or right foot and the midpoint of the left and right pelvis in the backswing section of the user's golf motion is displayed. The method of claim 11,
In the extracting step, an angle between a line connecting the head and the pelvis and a line connecting both shoulders is calculated based on the coordinates of the head, left and right pelvis, and left and right shoulders,
In the displaying, a line connecting the head and the pelvis and a line connecting both shoulders are displayed in the backswing top section of the user's golf motion. The method of claim 11,
In the extracting step, a 3D point cloud is calculated based on pixel data of the output data;
In the displaying, the skeleton information is displayed as a 3D point cloud on a display unit. An image sensor for acquiring a 2D image of a user's motion and a depth sensor for acquiring a depth image including a depth value for each pixel of the 2D image alternately and temporally with the acquisition of the 2D image camera unit;
a control unit generating a corresponding depth image or a corresponding 2-dimensional image corresponding to a reference 2-dimensional image or a reference depth image acquired by the camera unit at a predetermined time;
an output unit outputting the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image as output data;
an extractor configured to extract skeleton information of the user through analysis of the output data; and
and a display unit for displaying the motion of the user based on the skeleton information.

Images