KR20200115729A - Method and apparatus of analyzing golf motion - Google Patents

Abstract

The present invention relates to a golf motion analysis method for analyzing golf motion of a user. The method comprises the steps of: training an artificial neural network model in which golf motion is learned based on a first training golf image obtained by photographing golf motion and a second training golf image which is a skeleton image generated based on the first training golf image; receiving a user′s first golf image obtained by photographing user′s golf motion desired to be analyzed; generating the second golf image which is a skeleton image from the first golf image based on the artificial neural network model in which the golf motion is learned; and analyzing the user′s golf motion based on the second golf image. According to the present invention, the golf motion analysis method is capable of analyzing a mutual location relation between the user′s body and a golf club.

Description

Golf motion analysis method and apparatus {METHOD AND APPARATUS OF ANALYZING GOLF MOTION}

The present invention relates to a golf motion analysis method and apparatus, and more particularly, a golf motion analyzing the user's golf motion by extracting the skeleton information on the user's body and the skeleton information on the golf club using deep learning technology. It relates to an analysis method and apparatus.

Golf is a game of every movement, from address to take-back, backswing top, downswing, impact, follow through and finish. As an exercise requiring accuracy of posture, numerous exercises and repetitions of correction processes are required in order for the user to acquire the correct posture.

In this regard, in recent years, various systems for analyzing golf motions have been introduced to replace the correction process performed by conventional coaching by self-learning. In particular, recently, from images acquired using a depth camera Various systems are emerging that extract depth information and analyze the user's golf motion.

For example, in the case of Korean Patent Publication No. 10-1394274 (B1) (hereinafter referred to as Patent Document 1), a user part is extracted from a depth image acquired from a depth camera, and through the upper or lower region of the user part. Disclosed is a method of analyzing a user's golf motion by detecting body parts such as a user's head, lower body, torso, and arms. In addition, in the case of Unexamined Patent Publication No. 10-2018-0062069 (hereinafter, Patent Document 2), the user's skeleton image is acquired by photographing the user's golf motion using a depth camera, and the obtained skeleton image is applied to each joint of the user. In order to extract the corresponding coordinates and analyze the user's golf motion more precisely during the golf swing, an inertial sensor for detecting the user's hand motion is additionally installed on the user's hand. Disclosed is to install a camera that photographs a user's golf swing at a high frame in order to detect the rapid movement of the club.

However, in the case of Patent Documents 1 and 2, commercialization is limited in terms of cost and efficiency in that a system including high-performance computing equipment is required to implement this, and prior learning or a predefined body shape is used. Because the analysis is based on the user's image captured by the depth camera, not the body part (e.g., the part where the arm and upper body overlap in the address posture when shooting in front), the body part cannot be accurately separated or the arm There was a limit in not being able to accurately recognize the bent angle of

Registered Patent Publication No. 10-1394274 B1 Unexamined Patent Publication No. 10-2018-0062069 A

The present invention is to solve the above problems, and an object of the present invention is to extract the skeleton information on the golf club together with the skeleton information on the user's body by using a deep learning technology. It is to provide a golf motion analysis method for analyzing motion.

One object of the present invention is to provide an apparatus for performing the method.

In order to achieve the above object, in the golf motion analysis method according to an embodiment of the present invention, a first learning golf image photographing a golf motion and a second learning that is a skeleton image generated based on the first learning golf image Based on the golf image, an artificial neural network model in which golf motion is learned is trained. The user's first golf image, which has captured the user's golf motion to be analyzed, is input. A second golf image, which is a skeleton image, is generated from the first golf image based on the artificial neural network model from which the golf motion is learned. The user's golf motion is analyzed based on the second golf image.

In order to achieve the above object, the golf motion analysis apparatus according to an embodiment of the present invention is based on an artificial neural network model in which a golf motion using deep learning is learned, and at least one device photographing a golf motion of a user to be analyzed. 1 A skeleton image generator configured to generate a second golf image, which is a skeleton image, from a golf image, and extracts a joint to be analyzed among the joints displayed in the second golf image, and uses the extracted coordinate values of the joints. It includes a motion analysis unit configured to analyze the user's golf motion.

In the golf motion analysis method and apparatus according to an embodiment of the present invention as described above, since the golf motion of the user is analyzed based on a pre-learned golf swing model using deep learning technology, the body Even for body parts that are overlapped and do not appear in the image, coordinates corresponding to each joint of the skeleton can be extracted.

In addition, in the golf motion analysis method and apparatus according to an embodiment of the present invention, the skeleton coordinate information of the user's body and the skeleton coordinates of the golf club are determined based on a pre-learned golf swing model using deep learning technology. By extracting, in the user's golf motion, it is possible to analyze the mutual positional relationship between the user's body and the golf club and the motion of the golf club.

1 is a conceptual diagram illustrating a golf motion analysis apparatus for analyzing a user's golf motion according to the present invention.
FIG. 2 is a block diagram illustrating an apparatus for analyzing a golf motion illustrated in FIG. 1.
3 is an exemplary diagram of a predefined joint on a golf club.
FIG. 4 is a block diagram illustrating training of an apparatus for converting a user's first golf image of FIG. 2 into a second golf image, which is a skeleton image, using deep learning technology.
5 is a diagram illustrating a frame image for each division operation of the second golf image of FIG. 2.
6 is a flow chart showing a golf motion analysis method according to an embodiment of the present invention.
7 is a block diagram illustrating a computing system including a golf motion analysis apparatus according to embodiments of the present invention.

With respect to the embodiments of the present invention disclosed in the text, specific structural or functional descriptions are exemplified only for the purpose of describing the embodiments of the present invention, and the embodiments of the present invention may be implemented in various forms, It should not be construed as being limited to the embodiments described in the text.

Since the present invention can apply various changes and have various forms, specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific form of disclosure, it is to be understood as including all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. Should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle. Other expressions describing the relationship between components, such as "between" and "just between" or "adjacent to" and "directly adjacent to" should be interpreted as well.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of a set feature, number, step, action, component, part, or combination thereof, and one or more other features or numbers It is to be understood that the possibility of addition or presence of, steps, actions, components, parts, or combinations thereof is not preliminarily excluded.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning of the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. .

Meanwhile, when a certain embodiment can be implemented differently, a function or operation specified in a specific block may occur differently from the order specified in the flowchart. For example, two consecutive blocks may actually be executed at the same time, or the blocks may be executed in reverse depending on a related function or operation.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same elements in the drawings, and duplicate descriptions for the same elements are omitted.

1 is a conceptual diagram illustrating a golf motion analysis apparatus for analyzing a user's golf motion according to the present invention. FIG. 2 is a block diagram illustrating an apparatus for analyzing a golf motion illustrated in FIG. 1.

1 and 2, the golf motion analysis apparatus 100 according to the present embodiment is based on the user's golf motion based on at least one golf image of the user received from the camera unit 10 connected by wire or wirelessly. Is configured to analyze. The golf motion analysis apparatus 100 is configured to analyze a user's golf motion based on an artificial neural network model in which the golf motion is learned.

The camera unit 10 is a terminal configured to capture a user's golf motion as a video, and may be configured with a general camera, a depth camera capable of capturing a depth image, or a high frame camera capable of high-speed shooting. In addition, the camera unit 10 is composed of a single camera, is configured to transmit one golf image captured at the arranged position to the golf motion analysis apparatus 100, or is composed of a plurality of cameras, It may be configured to transmit a plurality of golf images photographing the user's golf motion at different view points to the golf motion analysis apparatus 100.

In addition, the camera unit 10 is various types of terminals supporting camera functions, for example, smartphones, tablet PCs, mobile phones, desktop PCs, laptop PCs, netbook computers, workstations, servers, PDAs, PMPs. It may be at least one of a (portable multimedia player), an MP3 player, or a wearable device.

The golf motion analysis apparatus 100 is configured to analyze a user's golf motion based on an artificial neural network model in which a two-dimensional or three-dimensional golf motion is learned. For example, when the golf motion analysis apparatus 100 is based on an artificial neural network model in which a two-dimensional golf motion is learned, the camera unit 10 corresponds to a time point of the golf motion in which the artificial neural network model is trained. It is necessary to photograph the user's golf motion at a specific point in time. In this case, the camera unit 10 may be a fixed terminal configured to be fixed at a specific position to photograph the user's golf motion. Alternatively, the camera unit 10 may be a mobile terminal, and in this case, when photographing the user's golf motion, the camera unit 10 includes a line or outline of a guide line for the shooting point on the display. It is configured to be displayed as so that the photographer can take a picture at the specific point in time. However, when the camera unit 10 is configured with a depth camera, since it is possible to photograph a three-dimensional golf motion, it is possible to convert it into a two-dimensional golf motion at the specific point of view, even if photographed at any point in time. Accordingly, in this case, the arrangement position of the camera unit 10 is not limited to a specific position, and it is not required to be fixed even while photographing the user's golf motion.

In addition, the golf motion analysis apparatus 100 may be configured to analyze a user's golf motion based on a plurality of artificial neural network models in which two-dimensional golf motions at different viewpoints are learned. In this case, the camera unit 10 may include a plurality of cameras, and may be configured to photograph a user's golf motion at each corresponding viewpoint. That is, the camera unit 10 is configured to transmit a plurality of golf images of the user at different viewpoints to the golf motion analysis apparatus 100. However, even in this case, when the camera unit 10 is configured as a depth camera, the three-dimensional golf motions photographed can be converted into a plurality of two-dimensional golf motions corresponding to each viewpoint, so that the camera unit ( 10) can be composed of a single camera.

Alternatively, the golf motion analysis apparatus 100 may be configured to analyze a user's golf motion based on an artificial neural network model in which a three-dimensional golf motion is learned. In this case, the camera unit 10 photographs The viewpoint is not limited to a specific viewpoint. That is, the arrangement position of the camera unit 10 is not limited to a specific position, and it is not required to be fixed even while photographing the user's golf motion. This is the same even when the camera unit 10 is configured as a depth camera.

The golf motion analysis apparatus 100 includes an input buffer 110, at least one processing element 120, and an output buffer 130. In addition, the golf motion analysis apparatus 100 may further include a parameter buffer 140 and a memory 150.

The input buffer 110 receives at least one first golf image VIMG1 in the golf motion of the user to be analyzed. For example, the input buffer 110 may include at least one frame buffer. The first golf image VIMG1 input to the input buffer 110 is a video including a user's golf motion to be processed by the processing element 120, and includes a plurality of frame images, each frame image It is composed of images of a specific time.

In addition, the golf motion analysis apparatus 100 converts the user's golf image received from the camera unit 10 before the input buffer 110 into an image in a suitable format so that the processing element 120 can process it. It may further include an image conversion unit (not shown) configured to perform. For example, when the processing element 120 is configured to analyze a golf image based on an artificial neural network model in which a two-dimensional golf motion at a specific point of view is learned, the user's golf output from the camera unit 10 There is no problem when the image is two-dimensional (in this case, the golf image of the user output from the camera unit 10 and the first golf image VIMG1 are the same.), but in the case of a three-dimensional image, for effective analysis The two-dimensional golf motion needs to be converted into a two-dimensional first golf image VIMG1 at a specific point of view from which it is learned.

Alternatively, when the processing element 120 is configured to analyze a golf image based on a plurality of artificial neural network models in which two-dimensional golf motions at different viewpoints are learned, output from a plurality of cameras at each corresponding viewpoint There is no problem in the case where the plurality of golf images are two-dimensional (in this case, the golf image of the user output from the camera unit 10 and the first golf image VIMG1 are the same.), the camera unit 10 ), for effective analysis, a two-dimensional golf motion at different viewpoints needs to be converted into a two-dimensional first golf image VIMG1 at each learned viewpoint. There is.

Alternatively, if the processing element 120 is configured to analyze a golf image based on an artificial neural network model in which a 3D golf motion is learned, the golf image of the user output from the camera unit 10 is 3D. In the case of, there is no problem (in this case, the golf image of the user output from the camera unit 10 and the first golf image VIMG1 are the same), but in the case of two-dimensional, a three-dimensional first for effective analysis. 1 It needs to be converted to a golf video (VIMG1). In this case, a 3D first golf image VIMG1 is generated from a plurality of 2D golf images at different viewpoints using techniques such as stitching, or a 3D golf image using deep learning technology. A 3D first golf image VIMG1 may be generated by inferring a depth value from a 2D golf movement and rendering it in 3D through the artificial neural network model in which the golf movement is learned. Since the above description is outside the scope of the present invention, a detailed description thereof will be omitted.

Therefore, the image conversion unit (not shown) differs in a two-dimensional or three-dimensional format between the golf image of the user output from the camera unit 10 and the golf image analyzed by the processing element 120 as described above. If there is, it is configured to convert the user's golf image output from the camera unit 10 into a first golf image VIMG1 in a suitable format for effective analysis.

That is, in each frame image of the first golf image VIMG1 input to the input buffer 110, according to a 2D or 3D format of the user's golf image processed by the processing element 120, the user In the golf motion of the user, coordinate information [f(x, y)] of the x-coordinate and y-coordinate of the user's body and golf club is included, or coordinate information [f(x, y, z) that further includes the z-coordinate )] may be included. In addition, each frame image of the first golf image VIMG1 may include information on a gray scale value (or R, G, and B values in the case of a color image).

In the present embodiment, it has been described as an example that the golf image of the user output from the camera unit 10 before the input buffer 110 is converted into an image in a suitable format so that the processing element 120 can process it. Not limited. For example, the processing element 120 is configured to include the image conversion unit (not shown), and the first golf image VIMG1 is converted to a golf image in a suitable format for effective analysis in the processing element 120 It may be configured to convert to.

The processing element 120 is configured to analyze the user's golf motion based on the artificial neural network model in which the golf motion is learned. Specifically, the processing element 120 generates a second golf image VIMG2, which is a skeleton image corresponding to the user's golf motion, from the received first golf image VIMG1 based on the artificial neural network model. The coordinates corresponding to each joint for the body and golf club are extracted, and from address, take-back, backswing top, downswing, impact, follow-through It is configured to track the movement of the user's body and the golf club and the position of the coordinates in each division movement of the golf movement until the (follow through) and the finish (finish).

Specifically, the processing element 120 includes a skeleton image generation unit 121 and a motion analysis unit 123.

The skeleton image generator 121 includes a skeleton corresponding to the user's golf motion from the at least one first golf image VIMG1 received from the input buffer 110 based on the artificial neural network model in which the golf motion is learned. It is configured to generate a second golf image VIMG2, which is an image, extract each joint of the user's body and golf club, and display it on the second golf image VIMG2. For example, when a plurality of first golf images VIMG1 at different viewpoints are received, the skeleton image generation unit 121, based on a plurality of artificial neural network models in which golf motions at each viewpoint are learned, A plurality of second golf images VIMG2 having different viewpoints may be generated.

Specifically, the skeleton image generation unit 121 extracts characteristics of each of a plurality of frame images of the first golf image VIMG1 to generate feature maps, calculates them, and calculates the respective It is configured to infer coordinates of not only the 20 joints of the user's body in the golf motion from the frame images but also the joints predefined on the golf club. Here, the predefined joint will be described later.

On the other hand, the generation and calculation of the feature map is a layer implemented based on a convolution neural network (CNN) and/or a recurrent neural network RNN, a rectified linear unit (RELU) layer, It can be implemented by a computational model including various computational layers such as a pooling layer, a bias add layer, and a softmax layer.

Here, an artificial neural network (ANN) refers to a computational model implemented in software or hardware that mimics the computational power of a biological system using a large number of artificial neurons connected by connecting lines. In the artificial neural network, artificial neurons that have simplified the functions of biological neurons are used. In addition, human cognitive actions or learning processes are performed by connecting them to each other through a connection line having a connection strength. That is, the computational model used in the skeleton image generator 121 may be a computational model in a state that is previously learned using the artificial neural network.

The motion analysis unit 123 extracts a joint to be analyzed from among the joints on the user's body and golf club displayed in the second golf image VIMG2, and uses the extracted coordinate values of the corresponding joint. It may be configured to analyze the user's golf motion. For example, the motion analysis unit 123 pre-stored information about a joint that is a basis for analysis for each divisional motion of golf motion from address to takeback, backswing top, downswing, impact, follow-through and finish. With reference to these, it may be configured to extract a related joint during a corresponding division operation. In addition, the motion analysis unit 123 uses the extracted coordinate values of the corresponding joint to determine the absolute position of the corresponding joint in each classification operation, for example, a classification operation indicating a state such as address, impact, and finish. And it can be configured to analyze the relative position with other joints, and analyze the movement direction, movement speed, and trajectory of movement of the corresponding joint in the divisional movements indicating motion such as takeback, backswing top, down swing, and follow-through. have. However, it is not limited thereto.

Meanwhile, the joints displayed on the user's body and golf club among the golf movements of the second golf image VIMG2 are based on an artificial neural network model in which golf movements using deep learning have been learned, and each frame image with continuity is Since the position or movement of each joint is inferred based on the position of each joint displayed, the position can be extracted even for the joint of the part where the body overlaps and does not appear in the image during the golf motion of the second golf image VIMG2. have. Accordingly, the processing element 120 according to an embodiment of the present invention can track a joint of a part that is covered by the user's body and does not appear in the image, and in particular, a follow-through operation photographed at a back view point of view. In this case, a trajectory of a hand or a trajectory of a golf club that is covered by the user's body and does not appear in the image may be traced.

In addition, the processing element 120 according to an embodiment of the present invention, based on the artificial neural network model in which golf motion is learned, coordinates of a joint predefined for a golf club together with coordinate information of a joint for a user's body By extracting the information, it is possible to analyze the mutual positional relationship between the user's body and the golf club and the movement of the golf club in the user's golf motion. In this regard, the predefined joint will be described with reference to FIG. 3.

3 is an exemplary diagram of a predefined joint on a golf club.

3, a first joint (c1) is defined on the golf club at a point of the handle where the user's golf grip ends, a second joint (c2) is defined at a central position in the longitudinal direction of the shaft, and golf A third joint (c3) is defined at the head of the club. That is, based on the coordinates of the first to third joints c1 to c3, the user's golf motion, specifically the motion of the golf club, may be analyzed. For example, in order to measure the angle between the wrist and the shaft in each division operation, the coordinates of the first joint c1, the coordinates of both wrist joints, and the coordinates of the second joint c2 may be used. In addition, the position, speed, and trajectory of the head may be analyzed using the coordinates of the third joint c3.

In addition, in order to analyze the movement of the golf club, a fourth joint c4 may be further defined at an end point of the shaft on the golf club toward the head. For example, in order to measure the degree of bending of the shaft in each divisional motion, in particular, a downswing or impact motion, the coordinates of the first joint c1, the coordinates of the second joint c2, and the fourth joint The coordinates in (c4) can be used. Specifically, by measuring the direction and distance of the second joint (c2) away from the straight line formed by the first joint (c1) and the fourth joint (c3), it is possible to measure the moving direction and the degree of bending of the golf club. have. Through this, it is possible to determine whether the strength of the shaft of the golf club used by the user is appropriate based on the user's golf motion.

In addition, in order to measure whether the club head hits the ball in the open or closed state in the impact motion, the coordinates of the third joint c3 and the coordinates of the fourth joint c4 may be used. For example, measuring the relative position of the third joint (c3) from the fourth joint (c4), or the coordinates of the third joint (c3) and the fourth joint (c4) in an address operation and an impact operation. By comparing the coordinates of ), it is possible to analyze the state of the club head in the impact operation.

That is, in the present invention, in converting the first golf image VIMG1 to the second golf image VIMG2, in generating an artificial neural network model in which basic golf motions are learned, a plurality of golf images for learning By learning not only about the 20 joints of the body but also the predefined joints on the golf club for each of them, in analyzing the user's golf motion, the relationship between the user's body and the golf club's mutual position and golf You can also analyze the movement of the club.

In an embodiment of the present invention, the processing element 120 includes a central processing unit (CPU), a graphic processing unit (GPU), and a neural processing unit to perform the above-described operations. It may be implemented by including at least one of various processing devices such as a unit (NPU), a digital signal processor (DSP), an image signal processor (ISP), and the like. Depending on the embodiment, the processing element 120 may be implemented by including a plurality of homogeneous processing devices among the above-described processing devices or a plurality of heterogeneous processing devices. .

In one embodiment of the present invention, the processing element 120 may be implemented by including a plurality of processor cores to parallelize the above-described operations.

The output buffer 130 stores the second golf image VIMG2 generated from the first golf image VIMG1 as a result of an operation based on an artificial neural network model in which a golf motion is learned by the processing element 120 And can be output. For example, the output buffer 130 may include at least one register.

The parameter buffer 140 may store a plurality of parameters and/or a plurality of hyper parameters used by the processing element 120 to perform the above-described operations. For example, the parameter buffer 140 may store parameters of an artificial neural network model learned by a learning process.

The memory 150 may temporarily or continuously store data processed or to be processed by the processing element 120. For example, the memory 150 is a volatile memory such as dynamic random access memory (DRAM), static random access memory (SRAM), and the like, and flash memory, phase change random access memory (PRAM), and resistance (RRAM). random access memory), nano floating gate memory (NFGM), polymer random access memory (PoRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), and the like. According to an embodiment of the present invention, the memory 150 may be implemented in the form of a mass storage device such as a solid state drive (SSD), an embedded multimedia card (eMMC), and a universal flash storage (UFS).

Although not shown, the golf motion analysis apparatus 100 according to the present embodiment may further include a control unit for controlling the overall operation of components, a task manager for managing assignment of specific tasks, and the like.

FIG. 4 is a block diagram illustrating training of an apparatus for converting a user's first golf image of FIG. 2 into a second golf image, which is a skeleton image, using deep learning technology.

Referring to FIG. 4, the processing element 120 may further include a joint extraction model learning unit 125. The joint extraction model learning unit 125 includes a plurality of first training golf images (TVIMG1; TVIMG11, TVIMG12, ...) and a plurality of second training golf images generated using each of the first training golf images (TVIMG1). It is configured to train the artificial neural network model by using an image (TVIMG2; TVIMG21, TVIMG22, ...).

The first golf image for learning TVIMG1 is a video image of a golf motion, which is an object to be captured, and is composed of a plurality of frame images. For example, when it is desired to train an artificial neural network model in which a two-dimensional golf motion is learned, the first training golf image TVIMG1 may be, for example, a golf image photographed at a predetermined specific viewpoint. Alternatively, in the case of training a plurality of artificial neural network models in which two-dimensional golf motions at different viewpoints are learned, the first training golf image TVIMG1 is, for example, golf photographed at different predetermined viewpoints. It can be images. In addition, when it is desired to train an artificial neural network model in which 3D golf movements are learned, the first training golf image TVIMG1 is, for example, a golf image including 3D golf movements photographed with a depth camera. , Golf images acquired by stitching using a plurality of two-dimensional golf images captured at different viewpoints, or through an artificial neural network model in which three-dimensional golf motions using deep learning technology are learned, in two-dimensional golf motions. It may be a golf image generated by inferring a depth value and rendering it in 3D.

The second golf image for learning TVIMG2 is a skeleton image generated using the first golf image for learning TVIMG1, and each frame image contains not only 20 joints of the body in golf motion, but also on a golf club. Information of the predefined joint is included. For example, in generating each frame image of the second learning golf image (TVIMG2), a joint of the body and a predefined joint on the golf club are tracked based on the first golf image for learning (TVIMG1). It can be extracted by applying software.

That is, the joint extraction model learning unit 125 sets a plurality of first training golf images TVIMG1 and a plurality of second training golf images TVIMG2 corresponding to each of the first training golf images TVIMG1 as a training data set. Using as, supervised learning can be performed. Learning data sets may be repeatedly obtained for a plurality of golf images, and learning may be performed by repeating them. Therefore, according to the 2D or 3D format of the training data set, it is possible to train an artificial neural network model in which 2D or 3D golf movements are learned, and also 2D golf movements at a specific point in time are learned. In order to train the artificial neural network model, the training data set is required to be a two-dimensional golf image at a specific point in time.

In addition, the joint extraction model learning unit 125, in training the artificial neural network model, from an address to a take-back, a backswing top, and a downswing ), impact, follow through, and finish. That is, in configuring the first learning golf image (TVIMG1) and the second learning golf image (TVIMG2) as a learning data set, a learning data set can be configured by classifying a plurality of frame images for each classification operation of a golf motion. have. Accordingly, it is possible to generate an artificial neural network model that has been learned for each classification operation, and based on this, the second golf image VIMG2 is generated from the first golf image VIMG1 to correspond to each classification operation of the user's golf movement. It is possible to track the movement of the user's body and the golf club and the position of the coordinates of the joint.

5 is a diagram illustrating a frame image for each division operation of the second golf image VIMG2 of FIG. 2.

5A to 5G, the user's body and the joints of the golf club in each division operation of takeback, backswing top, downswing, impact, follow-through, and finish from address are displayed. The second golf image VIMG2 of FIG. 5 shows a viewpoint of a user looking at the right side of the user's golf motion as a photographing object, but is not limited thereto, and the photographing time of the golf image obtained by training the artificial neural network model Accordingly, it may be output as a view point of a back side, a left side, and the like of the golf motion.

6 is a flow chart showing a golf motion analysis method according to an embodiment of the present invention.

Referring to FIG. 6, training an artificial neural network model (S100), inputting a user's first golf image (S200), and a second skeleton image from the first golf image based on the artificial neural network model. Generating a golf image (S300), and analyzing a user's golf motion based on the second golf image (S400).

In the learning step (S100) of training the artificial neural network model, a plurality of first training golf images (TVIMG1) photographing golf motions and a plurality of second training images generated using each of the first training golf images (TVIMG1) It is configured to train the artificial neural network model by using the golf image TVIMG2.

The second golf image for learning TVIMG2 is a skeleton image generated using the first golf image for learning TVIMG1, and each frame image contains not only 20 joints of the body in golf motion, but also on a golf club. Information of the predefined joint is included.

In addition, in the training of the artificial neural network model (S100), a plurality of first training golf images TVIMG1 and a plurality of second training golf images TVIMG2 corresponding to each of the first training golf images TVIMG1 are provided. Using the learning data set, supervised learning can be performed. Learning data sets may be repeatedly obtained for a plurality of golf images, and learning may be performed by repeating them. Therefore, according to the 2D or 3D format of the training data set, it is possible to train an artificial neural network model in which 2D or 3D golf movements are learned, and also 2D golf movements at a specific point in time are learned. In order to train the artificial neural network model, the training data set is required to be a two-dimensional golf image at a specific point in time.

In addition, in the step of training the artificial neural network model (S100), in training the artificial neural network model, from an address, a take-back, a backswing top, and a downswing (downswing), impact (impact), follow through (follow through) and finish (finish) are configured to learn golf movements for each divisional movement. That is, in configuring the first learning golf image (TVIMG1) and the second learning golf image (TVIMG2) as a learning data set, a learning data set is formed by classifying a plurality of frame images for each classification operation of golf movements, The artificial neural network model can be trained.

In step S200 of inputting the user's first golf image VIMG1, the first golf image VIMG1 of the user's golf motion to be analyzed is input. The first golf image VIMG1 is a video including a user's golf motion to be processed by the processing element 120, and includes a plurality of frame images, and each frame image is composed of an image of a specific time.

For example, when analyzing a golf motion based on an artificial neural network model in which a two-dimensional golf motion is learned, the first golf image VIMG1 is a user at a specific point of view corresponding to the point of view of the learned artificial neural network model. It may be a two-dimensional golf image. In addition, when analyzing a golf motion based on a plurality of artificial neural network models in which two-dimensional golf motions at different points of view are learned, the first golf image VIMG1 corresponds to each point of view of the learned artificial neural network models. These may be 2D golf images of the user at different viewpoints. In addition, when a golf motion is analyzed based on an artificial neural network model in which a three-dimensional golf motion is learned, the first golf image VIMG1 may be a three-dimensional golf image.

Meanwhile, when analyzing a golf motion based on an artificial neural network model in which a two-dimensional golf motion at a specific viewpoint is learned, the inputted first golf image VIMG1 may be a three-dimensional golf image of the user, In this case, the first golf image VIMG1 may be converted into a two-dimensional golf image of the user at the specific viewpoint. In addition, when analyzing a golf motion based on a plurality of artificial neural network models in which two-dimensional golf motions at different viewpoints are learned, the inputted first golf image VIMG1 may be a three-dimensional golf image of the user. In this case, the first golf image VIMG1 may be converted into two-dimensional golf images of the user at different viewpoints. In addition, when analyzing a golf motion based on an artificial neural network model in which a three-dimensional golf motion is learned, the input first golf image VIMG1 may be a two-dimensional golf image of the user, and in this case, stitching ), etc., from a plurality of two-dimensional golf images at different viewpoints to a three-dimensional golf image, or through an artificial neural network model in which three-dimensional golf motions using deep learning technology are learned, 2 By inferring a depth value from a 3D golf motion and rendering it in 3D, it can be converted into a 3D golf image.

In addition, in each frame image of the first golf image VIMG1, according to a two-dimensional or three-dimensional format of the user's golf image processed by the processing element 120, the user's body and Coordinate information [f(x, y)] of the x-coordinate and y-coordinate of the golf club may be included, or coordinate information [f(x, y, z)] further including the z-coordinate may be included. In addition, each frame image of the first golf image VIMG1 may include information on a gray scale value (or R, G, and B values in the case of a color image).

In the step (S300) of generating a second golf image, which is a skeleton image, from the first golf image based on the artificial neural network model, the first golf image VIMG1 ), a skeleton image corresponding to the user's golf motion, a second golf image VIMG2 is generated, and each joint for the user's body and golf club is extracted and displayed on the second golf image VIMG2. . For example, when a plurality of first golf images VIMG1 at different viewpoints are received, a plurality of second golf images at different viewpoints based on a plurality of artificial neural network models in which golf motions at each viewpoint are learned (VIMG2) can be created.

Specifically, in generating the second golf image VIMG2, feature maps are generated by extracting characteristics of each of a plurality of frame images of the first golf image VIMG1, and calculating these , From each of the frame images, it is configured to infer coordinates of not only 20 joints of the user's body in golf motion, but also joints predefined on the golf club.

In the step of analyzing the user's golf motion based on the second golf image (S400), among the joints on the user's body and golf club displayed in the second golf image VIMG2, a joint to be analyzed is selected. It is configured to extract and analyze the user's golf motion by using the extracted coordinate values of the corresponding joint.

For example, by referring to pre-stored information on the joints that are the basis for analysis for each classification movement of golf movements from address to takeback, backswing top, downswing, impact, follow-through and finish, when the corresponding classification operation , Related joints are extracted, and the absolute position of the joint, relative position with other joints, movement direction, movement speed, and movement trajectory are analyzed.

Meanwhile, the joints displayed on the user's body and golf club among the golf movements of the second golf image VIMG2 are based on an artificial neural network model in which golf movements using deep learning have been learned, and each frame image with continuity is Since the position or movement of each joint is inferred based on the position of each joint displayed, the position can be extracted even for the joint of the part where the body overlaps and does not appear in the image during the golf motion of the second golf image VIMG2. have.

Accordingly, in the golf motion analysis method according to an embodiment of the present invention, it is possible to track joints in portions that are covered by the user's body and not appear in the image. In particular, in the follow-through operation, the body is covered by the user's body. You can track the trajectory of the hand that does not appear or the trajectory of the golf club.

In addition, in the golf motion analysis method according to an embodiment of the present invention, based on the artificial neural network model in which the golf motion is learned, coordinate information of a joint predefined for a golf club together with coordinate information of a joint for a user's body By extracting, in the user's golf motion, it is possible to analyze the mutual positional relationship between the user's body and the golf club and the movement of the golf club.

7 is a block diagram illustrating a computing system including a golf motion analysis apparatus according to embodiments of the present invention.

Referring to FIG. 7, the golf motion analysis method and apparatus according to embodiments of the present invention may be implemented in the form of a product including a computer-readable program code stored in a computer-readable medium. The computer-readable program code may be provided by processors of various computers or other data processing devices. The computer-readable medium may be a computer-readable signal medium or a computer-readable recording medium. The computer-readable recording medium may be any tangible medium that can store or contain a program in or connected to an instruction execution system, equipment, or device.

The computing system 1000 includes a processor 1010 and a golf motion analysis apparatus 100. The computing system 1000 may further include a connectivity unit 1020, a storage unit 1030, a user interface 1050, and a power supply unit 1060.

Processor 1010 may perform various computing functions, such as specific calculations or tasks. For example, the processor 1010 may be an arbitrary processor such as a CPU, a microprocessor, or an application processor (AP). The processor 1010 may execute an operating system (OS) for driving the computing system 1000, and may execute various applications that provide Internet browsers, games, videos, cameras, and the like.

The golf motion analysis apparatus 100 is controlled by the processor 1010. The golf motion analysis apparatus 100 may be the device of FIG. 2, and may operate based on the method described above with reference to FIGS. 1 to 6 and form a motion characteristic network system. Depending on the embodiment, a part of the golf motion analysis apparatus 100 may be included in the processor 1010 and/or the storage unit 1030.

The communication unit 1020 may communicate with an external device. For example, the communication unit 1020 is a universal serial bus (USB) communication, Ethernet communication, near field communication (NFC), radio frequency identification (RFID) communication, mobile Communication (mobile telecommunication), memory card communication, etc. may be performed.

The storage unit 1030 may store data processed by the processor 1010 or may operate as a working memory. The storage unit 1030 is connected to a boot image for booting the computing system 1000, a file system related to the operating system for driving the computing system 1000, and the computing system 1000 A device driver related to an external device to be used, the application executed in the computing system 1000, and the like may be stored. For example, the storage unit 1030 may include at least one volatile memory such as DRAM, SRAM, etc., and EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, PRAM, RRAM, NFGM, PoRAM, MRAM, It may also include at least one nonvolatile memory such as FRAM.

The user interface 1050 may include one or more input devices such as keypads, buttons, microphones, and touch screens, and/or one or more output devices such as speakers and display devices. The power supply unit 1060 may supply an operating voltage of the computing system 1000.

In one embodiment, the computing system 1000 includes a computer, a laptop, a cellular, a smart phone, an MP3 player, a personal digital assistant (PDA), a portable multimedia player (PMP), Digital TV, digital camera, portable game console, navigation device, wearable device, internet of things (IoT) device, internet of everything (IoE) device, e-book ), a virtual reality (VR) device, an augmented reality (AR) device, or the like.

The golf motion analysis method and apparatus according to embodiments of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

In addition, the golf motion analysis method and apparatus according to embodiments of the present invention may be implemented as a computer program (or computer program product) including instructions executable by a computer. The computer program includes programmable machine instructions processed by a processor, and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, or a machine language. . Further, the computer program may be recorded on a tangible computer-readable recording medium (eg, memory, hard disk, magnetic/optical medium or solid-state drive (SSD), etc.).

Accordingly, the golf motion analysis method and apparatus according to the embodiments of the present invention may be implemented by executing the computer program as described above by the computing device. The computing device may include at least some of a processor, a Mary, a storage device, a high speed interface connected to the memory and a high speed expansion port, and a low speed interface connected to the low speed bus and the storage device. Each of these components is connected to each other using a variety of buses, and can be mounted on a common motherboard or in other suitable manner.

Here, the processor can process commands within the computing device, such as displaying graphic information for providing a GUI (Graphic User Interface) on an external input or output device, such as a display connected to a high-speed interface. For example, instructions stored in memory or storage devices As another embodiment, multiple processors and/or multiple buses can be used with multiple memories and memory types as appropriate. Also, the processor may be implemented as a chipset formed by chips including a plurality of independent analog and/or digital processors.

Also, the memory stores information within the computing device. As an example, the memory may be composed of volatile memory units or a set of them. As another example, the memory may be composed of a nonvolatile memory unit or a set of them. Also, the memory may be another type of computer-readable medium such as a magnetic or optical disk.

In addition, the storage device may provide a large amount of storage space to the computing device. The storage device may be a computer-readable medium or a configuration including such a medium, for example, devices in a storage area network (SAN) or other configurations, a floppy disk device, a hard disk device, an optical disk device, Or it may be a tape device, a flash memory, or another semiconductor memory device or device array similar thereto.

Although described above with reference to the preferred embodiments of the present invention, those skilled in the art will be able to variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. You will understand that you can.

10: camera unit
100: golf motion analysis device
110: input buffer
120: processing element
121: skeleton image generator
123: motion analysis unit
125: joint extraction model learning unit
130: output buffer
140: parameter buffer
150: memory

Claims (17)

In the golf motion analysis method for analyzing the user's golf motion,
Learning an artificial neural network model in which a golf motion is learned based on a first golf image for training photographing a golf motion and a second golf image for training, which is a skeleton image generated based on the first golf image for training;
Inputting a user's first golf image, which has captured the user's golf motion to be analyzed;
Generating a second golf image, which is a skeleton image, from the first golf image based on the artificial neural network model in which the golf motion is learned; And
Golf motion analysis method comprising the step of analyzing a user's golf motion based on the second golf image. The method of claim 1,
The second golf image for learning includes, in each of the frame images, not only a joint of a body in a golf motion, but also information of a joint predefined on a golf club. The method of claim 1,
A golf motion analysis apparatus in which a joint of a body and a predefined joint on a golf club are displayed on the second golf image. The method according to claim 2 or 3,
The predefined joint,
A first joint defined at a point of a handle on the golf club at which a golf grip ends;
A second joint defined at a central position in the longitudinal direction of the shaft of the golf club; And
Golf motion analysis method including a third joint defined in the head of the golf club. The method of claim 4,
The predefined joint further includes a fourth joint defined at a head end point of the shaft on the golf club. The method of claim 1,
In the step of training the artificial neural network model, each image frame of the first training golf image and the second training golf image is processed from an address of a golf motion to a take-back and a backswing. top), downswing, impact, follow through, and finish. Golf motion analysis method for training the artificial neural network model by classifying each motion. The method of claim 1,
The artificial neural network model is learned based on a two-dimensional golf motion at a specific view point, and the input first golf image is a golf motion, which is a two-dimensional golf image of the user photographed at the specific point of view. Analysis method. The method of claim 1,
The artificial neural network model is composed of a plurality of pieces learned based on two-dimensional golf motions at different viewpoints, and the input first golf image is a plurality of two-dimensional images of the user captured at each viewpoint at the different viewpoints. Golf motion analysis method of golf images. The method of claim 1,
The artificial neural network model is learned based on a three-dimensional golf motion, and the input first golf image is a three-dimensional golf image of a user. The method of claim 1,
The artificial neural network model is learned based on a two-dimensional golf motion at a specific point in time, and when the input first golf image is a user's three-dimensional golf image, the second golf image from the first golf image Before generating the golf motion analysis method of converting the first golf image into a two-dimensional golf image of the user at the specific viewpoint. A skeleton image generator configured to generate a second golf image, which is a skeleton image, from at least one first golf image photographing a golf motion of a user to be analyzed, based on an artificial neural network model in which golf motions using deep learning are learned; And
A golf motion analysis apparatus comprising a motion analysis unit configured to extract a joint to be analyzed from among the joints displayed in the second golf image, and analyze a user's golf motion using the extracted coordinate values of the joint. The method of claim 11,
It is configured to train the artificial neural network model in which golf motions are learned based on a first golf image for training photographing a golf motion and a second golf image for training, which is a skeleton image generated based on the first golf image for training. Golf motion analysis apparatus further comprising a joint extraction model learning unit. The method of claim 12,
The joint extraction model learning unit generates the artificial neural network model on the basis of the second golf image for training, which includes information on a joint predefined on a golf club as well as a joint of a body in a golf motion in each frame image. Golf motion analysis device configured to learn. The method of claim 11,
A golf motion analysis apparatus including a joint of the body and a pre-defined joint on a golf club among the user's golf motions, which are displayed on the second golf image. The method of claim 13 or 14,
The predefined joint,
A first joint defined at a point of a handle on the golf club at which a golf grip ends;
A second joint defined at a central position in the longitudinal direction of the shaft of the golf club; And
Golf motion analysis apparatus including a third joint defined in the head of the golf club. The method of claim 15,
The predefined joint further comprises a fourth joint defined at a head end point of the shaft on the golf club. The method of claim 12,
The joint extraction model learning unit includes a take-back, a backswing top, and a down of each image frame of the first training golf image and the second training golf image from an address of a golf motion. A golf motion analysis apparatus configured to train the artificial neural network model by classifying it by divisional motions such as downswing, impact, follow through, and finish.

Images