KR20160121379A - Apparatus and method for analyzing golf motion - Google Patents

Abstract

Disclosed are a technology for analyzing a golf motion, which can implement high-speed photographing and can provide a user with more accurate and various information by minutely analyzing a motion through extraction of skeleton information. To this end, according to an embodiment of the present invention, a method for analyzing a golf motion comprises: a step of obtaining a 2D image of a users motion by an image sensor of a camera unit; a step of obtaining a depth image including a depth value of each pixel of the 2D image by a depth sensor of the camera unit temporarily and alternately with the obtaining of the 2D image; a photographing speed increase step of generating a corresponding depth image or a corresponding 2D image corresponding to a reference 2D image or a reference depth image obtained at a certain time; a step of outputting the reference 2D image and the corresponding depth image, or the reference depth image and the corresponding 2D image as output data for analyzing motion; a step of extracting the users skeleton information through the analysis on the output data; and a step of displaying the users motion on a display unit based on the skeleton information.

Description

[0001] APPARATUS AND METHOD FOR ANALYZING GOLF MOTION [0002]

The present invention relates to an apparatus and a method for analyzing a golf motion, and more particularly, to an apparatus and method for analyzing a golf motion that can efficiently provide high-speed shooting and provide more accurate and versatile information to a user by analyzing operations by skeleton information extraction Apparatus and method.

Golf is a sport where posture is more important than strength. Golf's attitude is internalized through Muslce Memory, and the process of practice and correction must be repeated in order to accurately internalize the posture.

On the other hand, the camera-based golf motion analysis systems that have been recently introduced to help the calibration process performed by the coaches of the prior arts to be performed by self-learning. The camera-based golf motion analyzing system includes a camera unit for photographing a golf operation and an analyzing unit for analyzing the photographed image. The conventional camera-based golf motion analyzing system has some problems.

First, some operations of the golf swing proceed at a high speed. However, the conventional camera-based golf motion analysis system has a problem in that it can not photograph some high-speed operations properly due to the limitation of the camera shooting speed. In order to solve this problem, a golf motion analysis system using a high-speed camera has been developed, but it has also been limited in terms of cost.

The problem with some high-speed motion in the golf swing also appeared in the analysis department. In order to provide real-time information to the user, the analysis unit must have image processing performance higher than the photographing speed of the camera. However, since the computing equipment does not have such image processing capability, the user can not provide real-time information. In order to improve these problems, a system including high-performance computing equipment has been proposed, but this has also been limited in terms of cost and efficiency.

Conventional systems also have limitations in techniques for analyzing photographed images because the information that the system extracts from the images is not suitable for motion analysis. For example, the conventional system recognizes the outline of the body, extracts the body part, and analyzes the golf operation in such a manner that a line or a figure is displayed at the position of the specified body parts. However, according to this method, there is a limitation that the body part can not be accurately separated from the overlapping part of the body (for example, the part where the upper body overlaps with the arm at the address posture in the frontal photographing) there was.

Korean Patent Publication No. 2012-0021848 discloses a 'golf swing posture analysis system'.

SUMMARY OF THE INVENTION An object of the present invention is to provide a golf motion analysis system that efficiently implements high-speed shooting.

In addition, the present invention aims to provide more precise and diverse information to the user by finely analyzing the operation of the user.

It is another object of the present invention to provide stable analysis of a high-speed operation section while providing real-time information to a user in the analysis of a golf swing including a high-speed operation.

According to another aspect of the present invention, there is provided a golf motion analyzing method including: obtaining a two-dimensional image of a motion of a user in an image sensor of a camera; Acquiring the two-dimensional image and a depth image including a depth value of each pixel of the two-dimensional image in a temporally alternate depth sensor of the camera unit; An imaging speed increasing step of generating a corresponding depth image or a corresponding two-dimensional image corresponding to a reference two-dimensional image or a reference depth image acquired at a predetermined time; Outputting the reference two-dimensional image and the corresponding depth image or the reference depth image and the corresponding two-dimensional image as output data for motion analysis; Extracting skeleton information of the user through analysis of the output data; And displaying the operation of the user on the display unit based on the skeleton information.

At this time, the photographing speed increasing step may include: the depth sensor obtaining a first depth image at a first time; The image sensor acquiring a first two-dimensional image as the reference two-dimensional image at a second time after the first time; Generating a second depth image of the first time by projecting the first depth image to a viewpoint of the image sensor; Estimating a motion vector between the first two-dimensional image and the second depth image; Rendering the projected third depth image at the second time from the second depth image using the motion vector; And projecting the third depth image to the depth sensor point to generate a fourth depth image at the second time with the corresponding depth image, and in the outputting step, the first two- The fourth depth image can be output as output data.

At this time, the photographing speed increasing step may include: obtaining the first two-dimensional image at the first time by the image sensor; The depth sensor acquiring a first depth image with the reference depth image at a second time after the first time; Generating a second depth image by projecting the first depth image to a viewpoint of the image sensor; Estimating a motion vector between the first two-dimensional image and the second depth image; And rendering the second two-dimensional image at the second time from the first two-dimensional image to the corresponding two-dimensional image using the motion vector, wherein in the outputting step, And output the second two-dimensional image as output data.

The extracting of the skeleton information may include extracting pixel data of the output data; A user region selection step of selecting a pixel in an area corresponding to the user based on the pixel data; Estimating a probability distribution value for each of the user-domain pixels as to whether the pixel corresponds to a skeleton joint; Defining a skeleton joint for predetermined pixels according to the probability distribution value; Defining a center point at predetermined pixels defined by the skeleton joint; And completing and extracting the skeleton information based on the center point.

In this case, the step of defining the skeleton joint and the step of defining the center point may include: lowering the resolution of the pixel data; Defining a skeleton joint and a center point based on a lower resolution; Increasing the resolution of the pixel data; Setting an area of a predetermined distance at a center point derived based on the lower resolution; And redefining the skeleton joint and the center point based on the pixels in the region.

At this time, the pixel data may be at least one of depth information, RGB value, and illuminance value of each pixel.

At this time, in the outputting step, the amount of output data for operation analysis can be adjusted according to the operation speed of the user.

At this time, in the outputting step, if the operation speed of the user is less than a predetermined value, motion analysis is performed on all of the reference two-dimensional image and the corresponding depth image, the reference depth image, Dimensional image, the corresponding depth image, the reference depth image, and the corresponding two-dimensional image when the operation speed of the user is equal to or greater than a predetermined value, and outputs an output Data can be output.

At this time, in the outputting step, the operation speed of the user is not less than a predetermined value, and a part of the reference two-dimensional image and the corresponding depth image, the reference depth image, and the corresponding two- The output data for the operation analysis can be output when the operation speed of the user is changed to less than the preset value after storing the remaining data.

At this time, in the displaying step, an operation of the user that is different from a preset reference operation by a predetermined value or more may be displayed.

According to another aspect of the present invention, there is provided a golf motion analyzing method including: obtaining a two-dimensional image of a user's golf motion in an image sensor of a camera; Acquiring a depth image including a depth value of each pixel of the two-dimensional image from a depth sensor of the camera unit; Extracting skeleton information of the user through analysis based on the two-dimensional image and the depth image; And displaying the joint position of the user on the display unit based on the two-dimensional image and the skeleton information.

At this time, in the step of acquiring the depth image, the depth image is acquired to acquire the depth image alternately with acquiring the two-dimensional image, and the corresponding depth image corresponding to the reference two- Or a shooting speed increasing step of generating a corresponding two-dimensional image.

At this time, in the extracting step, skeleton head information is extracted as the skeleton information of the user, and in the displaying step, head movement from the address section to the impact section during the golf operation of the user can be displayed .

At this time, in the extracting step, a value corresponding to the foot interval is calculated based on the position value of the ankle joint of the user through analysis of the output data, and in the displaying step, The foot interval can be displayed.

In the extracting step, the coordinates of the center of gravity of the user and the coordinates of the right foot or the left foot joint are calculated. In the displaying step, in the backswing section or the downswing section of the user's golf operation, The position of the right foot or the left foot joint can be displayed.

At this time, in the extracting step, a value corresponding to the octagon is calculated based on the coordinates of the shoulder, the elbow and the hand joint, and in the displaying step, the shoulder, elbow, You can mark the line after the joint.

At this time, in the extracting step, a value corresponding to the inverse vertebral angle is calculated based on the coordinates of the head and the left and right pelvis, and in the displaying step, in the backswing section of the user's golf, Can be displayed.

According to another aspect of the present invention, there is provided an apparatus for analyzing a golf motion according to an embodiment of the present invention. The apparatus includes an image sensor for acquiring a two- And a depth sensor for acquiring the depth image including acquiring the two-dimensional image and containing a depth value for each pixel of the two-dimensional image alternately in time; A control unit for generating a corresponding depth image or a corresponding two-dimensional image corresponding to a reference two-dimensional image or a reference depth image acquired at a predetermined time by the camera unit; An output unit for outputting the reference two-dimensional image and the corresponding depth image, the reference depth image, and the corresponding two-dimensional image as output data; An extracting unit for extracting the skeleton information of the user through analysis of the output data; And a display unit for displaying an operation of the user based on the skeleton information.

According to the present invention, it is possible to provide a golf motion analysis system that efficiently implements high-speed shooting.

In addition, the present invention extracts skeleton information and analyzes operations in detail, thereby providing more accurate and diverse information to the user.

In addition, the present invention enables stable analysis of a high-speed operation section while providing real-time information to a user in analysis of a golf swing including a high-speed operation.

1 is a flow chart for explaining a golf operation analysis method according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining image acquisition timings of a depth sensor and an image sensor of a camera unit in a golf operation analysis method according to an embodiment of the present invention.
FIG. 3 is a flow chart for further illustrating a step for increasing a photographing speed in a golf operation analyzing method according to an embodiment of the present invention.
4 is another embodiment of a flowchart for further illustrating a step for increasing a photographing speed in a golf operation analyzing method according to an embodiment of the present invention.
FIG. 5 is a flow chart for explaining the skeleton information extracting step in more detail in the golf operation analyzing method according to the embodiment of the present invention.
FIGS. 6 and 7 are views for explaining the user regionization step in the skeleton information extraction step.
FIG. 8 is a diagram for explaining a method of calculating a probability distribution in the skeleton information extracting step. FIG.
9 is a view for explaining a multi-scale technique for reducing the amount of calculation of the skeleton information extraction step.
10 to 12 are diagrams for explaining the mode change of the present invention according to the operation speed of the user.
13 is a diagram showing an example of a display on a display unit in the golf operation analysis method according to the embodiment of the present invention.
Figs. 14 to 16 show examples of display on the display unit when the analyzed user's operation is not suitable for the reference operation or not.
FIG. 17 shows an example of displaying the foot interval in the displaying step.
FIGS. 18 and 19 show an example of determining a Sway through an operation analysis of a user.
20 and 21 show an example of determining hanging back by analyzing the operation of the user.
22 and 23 illustrate an example of determining a chicken wing through an analysis of a user's action.
FIGS. 24 and 25 show an example in which the reverse spine angle is discriminated by analyzing the motion of the user.
FIG. 26 shows an example of determining a slide through analysis of a user's action.
27 and 28 illustrate an example of discriminating a flat shoulder plane through an analysis of a user's action.
FIG. 29 illustrates an example of determining whether a head is fixed through an operation analysis of a user.
30 is a block diagram showing a configuration of a golf motion analyzing apparatus according to an embodiment of the present invention.

The present invention will now be described in detail with reference to the accompanying drawings. Hereinafter, a repeated description, a known function that may obscure the gist of the present invention, and a detailed description of the configuration will be omitted. Embodiments of the present invention are provided to more fully describe the present invention to those skilled in the art. Accordingly, the shapes and sizes of the elements in the drawings and the like can be exaggerated for clarity.

Hereinafter, a golf operation analysis method according to an embodiment of the present invention will be described.

1 is a flow chart for explaining a golf operation analysis method according to an embodiment of the present invention.

Referring to FIG. 1, a golf motion analyzing method according to an embodiment of the present invention acquires a two-dimensional image of a user's operation in an image sensor of a camera unit (S100).

In step S200, a depth image including a depth value for each pixel of the two-dimensional image is acquired from the depth sensor of the camera unit in order to obtain a two-dimensional image in step S200. FIG. 2 is a diagram for explaining image acquisition timings of a depth sensor and an image sensor of a camera unit in a golf operation analysis method according to an embodiment of the present invention. In FIG. 2, each arrow represents a time point at which an image is acquired, and the concept of acquiring an image by mutually alternating between the depth sensor and the image sensor in steps S100 and S200 is shown. For example, if a two-dimensional image is acquired at time t-1, a depth image is acquired at time t, and if a depth image is acquired at time t-1, a two-dimensional image can be acquired at time t.

Thereafter, a corresponding depth image or a corresponding two-dimensional image corresponding to the reference two-dimensional image or the reference depth image acquired at a predetermined time is generated (S300). That is, in step S300, the camera unit synthesizes a two-dimensional image obtained from the image sensor, for example, an RGB image and a depth image acquired from the depth sensor, thereby increasing the frame rate of each image have. The camera unit synthesizes the second depth image and the second two-dimensional image corresponding to the first two-dimensional image and the first depth image, which are alternately obtained by the alternate photographing, and outputs them together, thereby doubling the actual photographing speed .

After step S300, the reference two-dimensional image and the corresponding depth image or the reference depth image and the corresponding two-dimensional image are output as output data for operation analysis (S400). At this time, in step S400, the amount of output data for operation analysis can be adjusted according to the operation speed of the user. Specific details thereof will be described later in conjunction with FIG. 10 to FIG.

Thereafter, the skeleton information of the user is extracted through analysis of the output data of step S400 (S500). At this time, the skeleton information may include skeleton joint information and skeleton head information. The skeleton joint information may include, for example, information about the neck, left and right shoulders, left and right elbows, left and right wrists, right and left unilateral, left and right knees, and left and right ankles. Then, the skeleton information can be extracted using RDF (Randomized Decision Forest). A method of setting a background region and a user region from pixel data based on a depth information value and setting a pixel in a user region set based only on pixel data whose data range is reduced within a set user region to be a skeleton joint And classifies the skeleton joint of the high probability distribution value into the area of the pixel to estimate a pixel corresponding to a specific position in the user's body.

In step S500, the skeleton information of the user is extracted including the head information, and the value corresponding to the foot interval can be calculated based on the position value of the ankle joint of the user through analysis of the output data. In step S500, the coordinates of the center of gravity of the user and the coordinates of the right foot or the left foot joint are calculated. Based on the coordinates of the shoulder, elbow, and hand joints, values corresponding to the octagon are calculated. The value corresponding to the inverse vertebra angle can be calculated. In step S500, the angle between the line connecting the head and the pelvis to the line connecting the head and the pelvis based on the coordinates of the head, the left and right pelvis, and the right and left shoulders is calculated. Based on the pixel data of the output data, Can be calculated.

Then, the operation of the user is displayed on the display unit based on the skeleton information (S600). In operation S600, the operation of the user, which is different from the predetermined reference operation by a predetermined value, may be displayed on the display unit. In step S600, the head movement from the address section to the impact section during the golf operation of the user can be displayed, and the foot interval can be displayed in the user's golf operation. In step S600, the center of gravity and the position of the right or left foot joint may be displayed in the backswing section or the downswing section of the user's golf operation. In the impact section of the user's golf operation, the shoulder, elbow, And a line connecting the center of the left foot or the right foot and the center of the left and right pelvis in the backswing section during the golfing operation of the user can be displayed. In step S600, a line connecting the head and the pelvis and a line connecting both shoulders in the back swing top section of the user's golf swing can be displayed, and the skeleton information can be displayed on the display section in a three-dimensional point cloud .

The method for analyzing a golf operation according to an embodiment of the present invention may be performed by including step S100, step S200, step S500, and step S600 of FIG. That is, the golf motion analyzing method according to another embodiment of the present invention includes the steps of acquiring a two-dimensional image of a user's golf operation in an image sensor of a camera unit, acquiring a depth value of each pixel of the two- Extracting the skeleton information of the user through analysis based on the two-dimensional image and the depth image, and displaying the joint position of the user on the display unit based on the two-dimensional image and skeleton information Step < / RTI >

Hereinafter, a method for increasing the photographing speed according to step S300 of FIG. 1 will be described in more detail.

FIG. 3 is a flow chart for further illustrating a step for increasing a photographing speed in a golf operation analyzing method according to an embodiment of the present invention.

Referring to FIG. 3, in operation S300, the depth sensor acquires a first depth image at a first time (S311), and the image sensor acquires a reference two-dimensional image at a second time after the first time And acquires the first two-dimensional image (S312). The first depth image and the first two-dimensional image acquired in steps S311 and S312 may be a depth image and a two-dimensional image obtained through steps S100 and S200.

After step S312, the first depth image is projected to the viewpoint of the image sensor to generate a second depth image of the first view (S313).

Then, a motion vector between the one-dimensional image and the second depth image is estimated (S314).

Thereafter, the projected third depth image at the second time is rendered from the second depth image using the motion vector (S315).

Then, a third depth image is projected to the depth sensor view to generate a fourth depth image at the second time with the corresponding depth image (S316). In step S400 of FIG. 1, the first two-dimensional image and the fourth depth image may be output as output data.

4 is another embodiment of a flowchart for further illustrating a step for increasing a photographing speed in a golf operation analyzing method according to an embodiment of the present invention.

Referring to FIG. 4, in another embodiment of S300, the image sensor acquires a first two-dimensional image at a first time (S321), and the depth sensor acquires a reference depth image at a second time after the first time And acquires the first depth image (S322). The first two-dimensional image and the first depth image acquired in steps S321 and S322 may be the depth image and the two-dimensional image obtained through steps S100 and S200.

After step S322, the first depth image is projected to the viewpoint of the image sensor to generate a second depth image (S323).

Then, a motion vector between the first two-dimensional image and the second depth image is estimated (S324).

Thereafter, the second two-dimensional image at the second time is rendered from the first two-dimensional image to the corresponding two-dimensional image using the motion vector (S325). In step S400 of FIG. 1, the first depth image and the second two-dimensional image may be output as output data.

Steps S313 and S323 may be a method of calculating three-dimensional coordinates corresponding to each pixel of the depth image using internal parameters of the depth sensor. In this case, the 3D coordinates are projected onto the image plane of the image sensor using the external parameters between the image sensor and the depth sensor.

In step S314 and step S324, motion information between the first two-dimensional image and the projected second depth image may be estimated. In one embodiment, motion vectors may be calculated by matching pixels in the search region of the depth image with respect to each pixel of the first RGB image. Since the modalities of the two images are different, we can use mutual information between two matching patches to find the correspondence:

.

.

In steps S315 and S325, the image at time t may be generated through rendering using the image and motion information at time t-1:

.

.

Where x is the pixel coordinate and m (x) is the motion vector at pixel x .

Hereinafter, the step of skeleton information extraction according to step S500 of FIG. 1 will be described in more detail.

FIG. 5 is a flow chart for explaining the skeleton information extracting step in more detail in the golf operation analyzing method according to the embodiment of the present invention.

Referring to FIG. 5, pixel data is extracted from the output data acquired through S400 of FIG. 1 (S511). At this time, the pixel data may be at least one of depth information, RGB value, and illuminance value of each pixel.

Then, based on the pixel data, a pixel of an area corresponding to the user is selected (S512). Step S512 is a user zoning step, and referring to Figs. 6 and 7, examples of user zoning are shown. Step S512 may be implemented by various methods. For example, a distribution between the maximum value and the minimum value of the distance value of each pixel can be obtained, and the group of values of the learned specific distribution can be estimated as the user area.

After step S512, a probability distribution value of whether each of the user-domain pixels corresponds to a skeleton joint is calculated (S513). The method of obtaining the probability distribution value in step S513 can be performed by the Randomized Decision Forest (RDF) technique. The RDF-based probability distribution value is obtained by summing the probability distribution values stored in the leaf node by traversing one or more decision trees. The decision tree includes a root node and child nodes classified into a plurality of stages in the same manner as a first child node first classified from the root node and a second child node classified from the first child node in a tree format do. The child node at the bottom is defined as a leaf node. The separation of an ancestor into a child node is by a node separation variable that includes a feature variable based on one or more pixel information and a predefined threshold variable. FIG. 8 is a diagram for explaining a method of calculating a probability distribution. FIG. 8 illustrates a process of obtaining a probability distribution through a decision tree when an input depth image and a test pixel are given.

Assuming that the depth image I and pixel x are given in the decision tree, we apply the following weak classification rule to each node, starting from the root node until reaching the leaf node, Traverse:

We use the following simple depth comparison feature as a feature:

Here, θ = (u, v) denotes a feature parameter consisting of two offset vectors. We obtain the discrete probability distribution Pt (c | I, x) for the joint classes at the leaf node of the tree t, combining these distributions for all the trees in the random forest, Calculate the distribution:

The algorithm for learning the random forest is as follows. For each tree, new learning image data is composed by uniformly sampling with existing replacement training image data with replacement. Using this new learning data, the following procedure is recursively applied to extend the tree until certain stopping criteria are satisfied:

a. We randomly generate a set of node separation variables φ = (θ, τ) consisting of a feature variable θ and a threshold variable τ.

b. For each separate variable φ, we separate the learning data Q = {(I, x)} into a left / right set as follows:

c. The following information gain is calculated for the left / right set corresponding to the separation variable:

Where H (Q) represents the Shannon entropy of the normalized histogram PQ of set Q.

d. Select a node separation variable with the maximum amount of information increase and assign it to the current node:

e. If the maximum amount of information increase is less than the predefined threshold or the depth for the current node in the tree reaches the maximum value, the algorithm is aborted and the tree is returned. Otherwise, apply steps a-d above recursively to the left and right nodes.

In step S514, the predetermined pixels are defined as skeleton joints according to the calculated probability distribution values in step S513.

Then, a center point in predetermined pixels defined by the skeleton joint is defined (S515).

Steps S514 and S515 may include: lowering the resolution of the pixel data; defining a skeleton joint and a center point based on a lower resolution; increasing a resolution of the pixel data; Establishing an area of a predetermined distance, and defining a skeleton joint and a center point based on the pixels in the area, to define a skeleton joint and a center point. FIG. 9 is a view for explaining a multi-scale technique for reducing the amount of calculation in the skeleton information extracting step, and shows an example of a step of defining a skeleton joint and a center point by resolution adjustment. A method of finding the center point by the RDF method based on a small number of pixels at a low resolution and finding the center point by the RDF method at a high resolution only for a certain region from the center point. Can be increased.

In order to increase the computation efficiency of the RDF, the control unit does not classify a node whose probability distribution value in the n-th child node does not exceed a predetermined value, It is possible to apply a method of deriving a leaf node value only from a node having a high probability value rather than obtaining a value.

More specifically, the role of the random forest in the skeleton extraction algorithm is to calculate a probability value corresponding to a particular skeleton joint for each pixel of the input image. This is obtained by traversing each decision tree and summing the probability values stored in the leaf nodes. This probability value represents the contribution of each of the three-dimensional points to the joint density function. The basic idea of the cascade technique is to ignore points of small contribution. For such small contribution points, it is not necessary to traverse the decision tree completely, so it is possible to terminate the random forest calculation before reaching the leaf node. To implement such an idea, we recursively define an auxiliary probability vector Ptn for each node n of the decision tree t as follows:

Where Pt is the probability distribution stored in the leaf node of the original random forest classifier and P _t ^l and P _t ^r are the auxiliary probability vectors for the two child nodes of node n, respectively. Assuming that P _t ⁿ (c) = ρ is reached at node n during the random forest computation for input depth I and pixel x, this means that no matter what path is selected at current node n, the probability Pt (c | I, x) does not exceed ρ. Thus, if ρ is smaller than the predefined small threshold ρ _node , the cascade algorithm will return a probability of zero without further processing the tree traversal to reduce the amount of computation.

Meanwhile, the control unit further defines a rejection tree for classifying all the skeleton joint classes into one joint class, and classifies the rejection tree as a joint or non-joint class. The rejection tree is used as an introduction tree of the RDT, and the output probability Is greater than the predetermined value, the method of proceeding with RDT can be applied.

Specifically, the above-described cascade idea is extended to a tree level. The basic idea is to define a new classifier that serves as a quick end check and place it at the front end of the random forest. In order to reduce the additional computational burden of the new classifier, this classifier should have a simple structure and also be capable of quick testing. To do this, we define a randomized decision tree that classifies input pixels into joint or non-joint classes and defines them as rejection trees. This binary decision tree can be learned by combining all the skeleton joint classes into one joint class in the training data introduced for the random forest in the previous section. Let the output probability of the rejection tree be P ₀ (c | I, x). The developed tree cascade technique performs random forest computation only when P ₀ (c | I, x) is larger than the small threshold ρ _tree . For many pixels far away from the skeleton joint point, this probability value is small enough to ignore these pixels in an additional random forest calculation.

Thereafter, the skeleton information is completed and extracted based on the center point (S516).

Hereinafter, the mode change when outputting the output data in the golf operation analyzing method according to the embodiment of the present invention will be described according to the operation speed of the user.

10 to 12 are diagrams for explaining the mode change of the present invention according to the operation speed of the user.

Referring to FIG. 10, when the operation speed of the user is less than a preset value, all of the reference 2D image and the corresponding depth image or the reference depth image and the corresponding two-dimensional image can be output as output data for motion analysis in real time . This is a real-time motion recognition mode in which the user is photographed at n hz through the camera unit and the photographed information can be directly analyzed through the control unit. Therefore, no image storage memory or result storage memory is used. The advantage of this mode is that it can check the user's operation in real time. The controller of the present invention analyzes the user's operation in the real-time operation recognition mode and can confirm in real time whether or not the user is ready to take a high-speed operation. As a more specific example, before the golf swing, which is one of the high-speed operations, is performed, the two hands are gathered to take an address posture holding the club. When the control section analyzes the address posture in real time and the user takes an address posture It can be predicted that a high-speed operation will be performed.

11, when the operation speed of the user is equal to or greater than a preset value, a part of the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image may be extracted and output as output data for operation analysis . In order to operate an image photographed in the high-speed motion recognition mode in real time, it is necessary to apply the image processing and motion recognition within the high-speed shooting cycle time and output the result. Some of them are used to analyze real-time motion recognition results. For example, when the user is taking a high-speed operation, the camera section shoots at a faster period (2n hz) than the real-time mode and stores it in the image storage memory. Only the odd-numbered images among the stored image storage memories are extracted and the controller can perform image processing and motion recognition. The motion recognition result may be stored in the result storage memory and output by the display unit in real time. The difference from the first mode is that a high-speed image is stored in an image storage memory for later analysis, and a result of analyzing a part of the stored high-speed images is output and stored in a result memory . The advantage of the high-speed motion recognition mode is that the user can first check whether or not the high-speed motion has been completed, and if it is finished, end the high-speed motion recognition mode, Memory and result storage memory are stored only in the high-speed operation part, so that the memory can be efficiently managed.

12, when the operation speed of the user is equal to or greater than a predetermined value and a part of the reference 2D image and the corresponding depth image or the reference depth image and the corresponding 2D image are extracted and output as output data for operation analysis, After storing the remaining data, if the operation speed of the user is changed to less than the preset value, it can be outputted as output data for motion analysis. It analyzes the entire high-speed operation taken through the camera unit in the remaining high-speed operation analysis mode, and stores the result of analyzing the entire high-speed operation in the result storage memory, and can output the analyzed result. The display unit analyzes the remaining high-speed motion after the finishing posture in which the user informs that the golf swing is over in the remaining high-speed motion analysis mode, and at the same time, the user can reproduce and provide the slow swing of the user's swing. In addition, the display unit can reproduce the exemplary golf swing on a slow screen, and can display tips related to golf. Through such a display, the user can reduce the time taken to wait for the remaining high-speed operations to be analyzed.

Hereinafter, the step of displaying the user's action on the display unit based on the skeleton information will be described in detail in the golf operation analysis method according to the embodiment of the present invention.

13 is a diagram showing an example of a display on a display unit in the golf operation analysis method according to the embodiment of the present invention.

Referring to FIG. 13, skeleton information may be displayed on a two-dimensional motion picture taken by a user at each step of the user's golf operation (ADDRESS, TAKE-BACK, BACKSWING TOP, DOWNSWING, IMPACT, FOLLOW THROUGH, FINISH) . In addition, it is possible to display the change of the head position in the operation, that is, whether the head is fixed and the angle of the leading arm. In addition, the analysis results for calibration of the golf operation, that is, points (ex: swing, hanging back, inverted vertebra, slide) . For example, the control unit can extract skeleton joint and skeleton head information, and the display unit can display the position of the skeleton joint as a point (point) on the two-dimensional image. Further, the display unit can display the position of the skeleton head as a circle.

Also, according to an embodiment, a line connecting two or more skeleton joints or skeleton heads may be displayed on a two-dimensional image. For example, the display unit may display a line or a graphic on a two-dimensional image that connects both shoulder joints, both arm joints, and wrist joint points. When the control unit distinguishes the golf operation by intervals, the display unit may display a line or a figure connecting the two shoulders, the arms and the wrist joint points on the two-dimensional image obtained in the address period or the impact period, And serves as a guide line for forming an inverted triangle having a constant angle formed by both shoulders and arms in the impact section.

The position or shape of the line or figure connected between the skeleton joint points can be compared with a preset reference by the control unit. According to the comparison result, the display unit can display the color or the expression of the line or the figure differently.

For example, the controller can calculate the angle formed by the shoulder-elbow-wrist with respect to the elbow. If the calculated angle is within the reference range, the line connecting the skeleton joints can be displayed in green. Otherwise, the control unit can display the line in yellow or red, depending on the degree of deviation from the reference range.

For the circle indicating the position of the skeleton head, if the circle position is within the reference range, the color of the circle representing the skeleton head is indicated in green. Otherwise, the corresponding line is changed to yellow or red Can be displayed. Figs. 14 to 16 show examples of display on the display unit when the analyzed user's operation is not suitable for the reference operation or not. In Fig. 14, the head position of the user is represented by a solid line circle because it is a normal category, and in Fig. 15 and Fig. 16, the head position of the user is out of the normal category. In the case where the control unit distinguishes the golf operation by intervals, the display unit can display the position change amount (moving distance) of the skeleton joint and the skeleton head between the respective intervals. For example, the display unit can display a change amount of the position of the skeleton head from the address section to the impact section. On the other hand, since the control unit can extract the three-dimensional position information in the skeleton information, this position change amount can be a three-dimensional position change amount.

FIG. 17 shows an example of displaying the foot interval in the displaying step. In the golf operation, the foot width is basically represented because it is required to maintain the shoulder width, and the calculation is based on the following.

,

- 3D coordinates of both joints:

,

,

- 3D coordinates of both shoulder joints:

,

- Discrimination formula:

FIGS. 18 and 19 show an example of determining a Sway through an operation analysis of a user. The swing means that during the backswing the lower body moves excessively from the target point and the center of gravity goes out the back of the foot. Fig. 18 shows the right posture, and Fig. 19 shows the sway. The calculation is performed based on the following.

- Point cloud:

- 3D coordinates of the center of gravity:

- 3D coordinates of the right foot joint:

- Discrimination formula:

20 and 21 show an example of determining hanging back by analyzing the operation of the user. A hanging bag means you can not send your weight to the lead when you are doing a downswing. Fig. 20 shows a right posture. Fig. 21 shows a hanging bag. The calculation is made based on the following.

- Point cloud:

- 3D coordinates of the center of gravity:

- 3D coordinates of the left foot joint:

- Discrimination formula:

22 and 23 illustrate an example of determining a chicken wing through an analysis of a user's action. The chicken wing means that the elbow does not stretch at impact. Fig. 22 shows the right posture, and Fig. 23 shows the chicken wing. The calculation is made based on the following.

- 3D coordinates of left shoulder:

- 3D coordinates of left elbow:

- 3D coordinates of the left hand:

- Shoulder - Elbow straight and elbow - Angle of hand straight line:

- Discrimination formula:

- Repeat the above procedure for the right

FIGS. 24 and 25 show an example in which the reverse spine angle is discriminated by analyzing the motion of the user. An inverted vertebral angle means that the upper body is overturned backwards during the backswing. Fig. 24 shows the right posture, and Fig. 25 shows the inverse vertebra angles. The calculation is made based on the following.

- 3D coordinates of head:

- 3D coordinates of the left pelvis:

- 3D coordinates of the right pelvis:

- 3D coordinates of pelvis center:

- Discrimination formula:

FIG. 26 shows an example of determining a slide through analysis of a user's action. The slide means that the lower body is excessively moved laterally in the target direction at the time of impact. Fig. 26 represents a slide, and calculation is performed based on the following.

- 3D coordinates of the left pelvis:

- 3D coordinates of the right pelvis:

- 3D coordinates of pelvis center:

- 3D coordinates of the left foot joint:

- Discrimination formula:

27 and 28 illustrate an example of discriminating a flat shoulder plane through an analysis of a user's action. A flat shoulder means that the shoulders are parallel to the ground at the top of the backswing. Fig. 27 shows the right posture, and Fig. 28 shows the flat shoulder. The calculation is made based on the following.

- 3D coordinates of head:

- 3D coordinates of pelvis center:

- 3D coordinates of left shoulder:

- 3D coordinates of the right shoulder:

- The angle between the head - pelvic straight line and both shoulder lines:

- Discrimination formula:

FIG. 29 illustrates an example of determining whether a head is fixed through an operation analysis of a user. Address - Backswing - Backswing Top - Downswing during Golfing - The position of the head during the course of the impact should not change. In FIG. 29, a change in head position can be displayed at the head position of the user, and the calculation is made based on the following.

- 3D coordinates of head at address:

- 3D coordinates of the head at the top of the backswing:

- 3D coordinates of the head at impact time:

- Discrimination formula:

The golf operation analysis method according to the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Includes all types of hardware devices that are specially configured to store and execute magneto-optical media and program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include machine language code such as those generated by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. Such a hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The teachings of the principles of the present invention may be implemented as a combination of hardware and software. In addition, the software can be implemented as an application program that is actually implemented on the program storage unit. The application program can be uploaded to and executed by a machine that includes any suitable architecture. Advantageously, the machine may be implemented on a computer platform having hardware such as one or more central processing units (CPUs), a computer processor, a random access memory (RAM), and input / output (I / . In addition, the computer platform may include an operating system and microinstruction code. The various processes and functions described herein may be part of microcommand codes or a portion of an application program, or any combination thereof, and they may be executed by various processing devices including a CPU. In addition, various other peripheral devices such as additional data storage and printers may be connected to the computer platform.

It is to be understood that the actual connections between system components or process functional blocks may vary depending on how the principles of the present invention are programmed, as some of the constituent system components and methods illustrated in the accompanying drawings are preferably implemented in software It should be further understood. Given the teachings herein, those skilled in the relevant art (s) will be able to contemplate these and similar implementations or configurations of the principles of the invention.

Hereinafter, the configuration and operation of a golf game analyzing apparatus according to an embodiment of the present invention will be described.

30 is a block diagram showing a configuration of a golf motion analyzing apparatus according to an embodiment of the present invention.

30, the apparatus 100 for analyzing a golf operation according to an embodiment of the present invention includes a camera unit 110, a control unit 120, an output unit 130, an extraction unit 140, and a display unit 150 . The golf motion analyzing apparatus 100 of the present invention shares the description with reference to FIG. 1 to FIG. 29, and will be described below with reference to main technical contents.

The camera unit 110 includes an image sensor for acquiring a two-dimensional image of a user's operation and a depth image including a depth value for each pixel of the two-dimensional image alternately in time And a depth sensor for measuring the temperature of the substrate.

The control unit 120 generates a corresponding depth image or a corresponding two-dimensional image corresponding to the reference two-dimensional image or the reference depth image acquired at a predetermined time by the camera unit 110.

The output unit 130 outputs the reference two-dimensional image and the corresponding depth image, the reference depth image, and the corresponding two-dimensional image as output data.

The extraction unit 140 extracts the skeleton information of the user through analysis of the output data.

The display unit 150 displays the operation of the user based on the skeleton information.

On the other hand, the display unit 150 can continuously display the two-dimensional images obtained from the camera unit 110, thereby continuously displaying the user's golf motion.

The control unit 120 extracts a 3D point cloud from the depth image acquired from the camera unit 110 and the display unit 150 outputs the 3D point cloud to the screen.

The depth image can be defined as a function that maps the physical distance from the pixel coordinate (x, y) as D (x, y) to the actual object corresponding to the pixel.

At this time, if the camera parameter for the depth image is known, it is possible to calculate the three-dimensional coordinates of the three-dimensional point corresponding to each pixel.

Assuming a simple pinhole camera model and assuming that the focal length of the camera is f and the pixel coordinates of the principal point are (px, py), three-dimensional points (X, Y, Z ) And the two-dimensional pixel coordinates (x, y) projected from the depth image can be expressed as follows.

This is summarized in the following two equations.

Therefore, given the depth D of the pixel (x, y) of the depth image, the coordinates of the three-dimensional point corresponding to the pixel are calculated as follows.

When the control unit 120 extracts the three-dimensional point cloud through the process and transmits the extracted three-dimensional point cloud to the display unit 150, the display unit 150 outputs the three-dimensional point cloud to the screen. At this time, the display unit 150 may control the display time of the three-dimensional point cloud differently according to a user's operation input (for example, a mouse input).

It is also important to know the angle of the arm and bend the waist in golf motion analysis, which is not easy to judge by the two-dimensional image of the front face alone. According to the present invention, the control unit 120 calculates a three-dimensional point cloud, and the display unit 150 visualizes the three-dimensional point cloud, so that the above information can be easily transmitted to the user.

As described above, the method and apparatus for analyzing golf motion according to the present invention are not limited to the configuration and method of the embodiments described above, but the embodiments may be modified in various ways, All or some of them may be selectively combined.

100; Golf motion analyzer
110; Camera part
120; The control unit
130; Output portion
140; The extraction unit
150; Display portion

Claims (20)

Acquiring a two-dimensional image of a user's operation in an image sensor of a camera unit;
Acquiring the two-dimensional image and a depth image including a depth value of each pixel of the two-dimensional image in a temporally alternate depth sensor of the camera unit;
An imaging speed increasing step of generating a corresponding depth image or a corresponding two-dimensional image corresponding to a reference two-dimensional image or a reference depth image acquired at a predetermined time;
Outputting the reference two-dimensional image and the corresponding depth image or the reference depth image and the corresponding two-dimensional image as output data for motion analysis;
Extracting skeleton information of the user through analysis of the output data; And
And displaying the operation of the user on the display unit based on the skeleton information. The method according to claim 1,
Wherein the photographing speed increasing step comprises:
The depth sensor acquiring a first depth image at a first time;
The image sensor acquiring a first two-dimensional image as the reference two-dimensional image at a second time after the first time;
Generating a second depth image of the first time by projecting the first depth image to a viewpoint of the image sensor;
Estimating a motion vector between the first two-dimensional image and the second depth image;
Rendering the projected third depth image at the second time from the second depth image using the motion vector; And
And projecting the third depth image to the depth sensor point to generate a fourth depth image at the second time with the corresponding depth image,
Wherein the outputting step outputs the first two-dimensional image and the fourth depth image as output data. The method according to claim 1,
Wherein the photographing speed increasing step comprises:
The image sensor acquiring a first two-dimensional image at a first time;
The depth sensor acquiring a first depth image with the reference depth image at a second time after the first time;
Generating a second depth image by projecting the first depth image to a viewpoint of the image sensor;
Estimating a motion vector between the first two-dimensional image and the second depth image; And
And rendering the second two-dimensional image at the second time from the first two-dimensional image to the corresponding two-dimensional image using the motion vector,
Wherein the outputting step outputs the first depth image and the second two-dimensional image as output data. The method according to claim 1,
The step of extracting the skeleton information includes:
Extracting pixel data of the output data;
A user region selection step of selecting a pixel in an area corresponding to the user based on the pixel data;
Estimating a probability distribution value for each of the user-domain pixels as to whether the pixel corresponds to a skeleton joint;
Defining a skeleton joint for predetermined pixels according to the probability distribution value;
Defining a center point at predetermined pixels defined by the skeleton joint; And
And completing and extracting the skeleton information based on the center point. The method of claim 4,
Wherein defining the skeleton joint and defining the center point comprise:
Lowering the resolution of the pixel data;
Defining a skeleton joint and a center point based on a lower resolution;
Increasing the resolution of the pixel data;
Setting an area of a predetermined distance at a center point derived based on the lower resolution; And
And redefining the skeleton joint and the center point based on the pixels in the region. The method of claim 4,
Wherein the pixel data is at least one of depth information, RGB value, and illuminance value of each pixel. The method according to claim 1,
In the outputting step,
Wherein the amount of output data for operation analysis is adjusted according to the operation speed of the user. The method according to claim 1,
In the outputting step,
Dimensional image, the corresponding depth image, the reference depth image, and the corresponding two-dimensional image as output data for motion analysis in real time when the operation speed of the user is less than a predetermined value,
And extracts a part of the reference two-dimensional image and the corresponding depth image or the reference depth image and the corresponding two-dimensional image to output as output data for operation analysis, when the operation speed of the user is not less than a predetermined value, Of the golf course. The method of claim 8,
In the outputting step,
When the operation speed of the user is not less than a preset value and a part of the reference two-dimensional image and the corresponding depth image, the reference depth image, and the corresponding two-dimensional image are extracted and output as output data for operation analysis, And outputting the output data for operation analysis when the operation speed of the user is changed to less than a predetermined value after storing the data. The method according to claim 1,
In the displaying step,
Wherein the operation of the user is different from the predetermined reference operation during the operation of the user. Acquiring a two-dimensional image of a user's golf motion in an image sensor of a camera unit;
Acquiring a depth image including a depth value of each pixel of the two-dimensional image from a depth sensor of the camera unit;
Extracting skeleton information of the user through analysis based on the two-dimensional image and the depth image; And
And displaying the joint position of the user on the display unit based on the two-dimensional image and the skeleton information. The method of claim 11,
In the step of acquiring the depth image,
Dimensional image and acquiring the depth image alternately with acquiring the two-dimensional image,
Further comprising a shooting speed increasing step of generating a corresponding depth image or a corresponding two-dimensional image corresponding to the reference two-dimensional image or the reference depth image acquired at a predetermined time. The method of claim 11,
Wherein the extracting includes extracting skeleton head information including the skeleton information of the user,
Wherein the displaying step displays the head movement from the address section to the impact section during the golf operation of the user. The method of claim 11,
Calculating a value corresponding to a foot interval based on a position value of the ankle joint of the user through analysis of the output data,
Wherein the displaying step displays the foot spacing in the golf operation of the user. The method of claim 11,
In the extracting step, the coordinates of the center of gravity of the user and the coordinates of the right foot or left foot joint are calculated,
Wherein the displaying step displays the position of the center of gravity and the position of the right foot or the left foot joint in a backswing section or a downswing section of the golf operation of the user. The method of claim 11,
In the extracting step, a value corresponding to the octagon is calculated based on the coordinates of the shoulder, the elbow, and the hand joint,
Wherein the displaying step displays a line connecting the shoulder, the elbow, and the hand joint in the impact period of the golf operation of the user. The method of claim 11,
In the extracting step, a value corresponding to an inverse vertebra angle is calculated based on the coordinates of the head and the left and right pelvis,
Wherein the displaying step displays a line connecting a center point of the left foot or the right foot and the left and right pelvis in the backswing section of the golf operation of the user. The method of claim 11,
In the extraction step, an angle between a line connecting the head and the pelvis and a line connecting the both shoulders is calculated based on the coordinates of the head, the left and right pelvis, and the left and right shoulders,
Wherein the displaying step displays a line connecting the head and the pelvis and a line connecting both shoulders in the backswing top section of the user's golf operation. The method of claim 11,
Wherein the outputting step calculates a three-dimensional point cloud based on the pixel data of the output data,
Wherein the displaying step displays the skeleton information in a three-dimensional point cloud on the display unit. An image sensor for acquiring a two-dimensional image of the user's motion, and a depth sensor for acquiring the two-dimensional image and acquiring a depth image including a depth value for each pixel of the two- A camera section;
A control unit for generating a corresponding depth image or a corresponding two-dimensional image corresponding to a reference two-dimensional image or a reference depth image acquired at a predetermined time by the camera unit;
An output unit for outputting the reference two-dimensional image and the corresponding depth image, the reference depth image, and the corresponding two-dimensional image as output data;
An extracting unit for extracting the skeleton information of the user through analysis of the output data; And
And a display unit for displaying an operation of the user based on the skeleton information.

Images