KR20230081949A - System and method for artificial intelligence golf swing analysis/correction based on 3D character retargeting - Google Patents

Abstract

Disclosed are a system and method for artificial intelligence golf swing analysis and correction based on 3D character retargeting. The systems and method comprise extracting a key point including a joint position from an image of golf swing motion, tracking the positional movement of the key points to extract a sequence of golf swing motion, comparing correction points with the motion or posture of professional athletes in the extracted sequence and analyzing the same, and generating a 3D image by retargeting a 3D character. A user's body type is measured, to suggest swing motion suitable for the user, and a 3D image to intuitively understand correction details can be provided.

Description

System and method for artificial intelligence golf swing analysis/correction based on 3D character retargeting}

An image processing technology, in particular, a computer vision technology for recognizing a human motion in golf swing images is disclosed.

As the standard of living improves, golf, which was considered a luxury sport in the past, is becoming increasingly popular. Accordingly, the number of people who want to learn golf is also increasing, and demand for golf clubs, golf lessons, and golf driving ranges is also increasing.

In order to learn golf, you must first learn the correct swing motion or posture. However, it is difficult to check your posture by yourself, so you need the help of an expert such as an instructor. On the other hand, if it is difficult to receive help from an expert, the swing motion may be corrected using an image recording system.

Publicated Patent Publication No. 10-2021-0045148 published on April 26, 2021 discloses a method for estimating information about a golf swing. When a photographed image of the user's golf swing is acquired, at least one joint of the user is detected in the photographed image using an artificial neural network model, and the position of the at least one detected joint is referred to to determine the comparison target of the user's golf swing. Compare golf swings to each other. A method of estimating information about a user's golf swing based on a comparison result is provided.

Registered Patent No. 10-2231202 published on March 24, 2021 discloses a golf swing machine through an around view. By providing an around photographing unit, a swing analysis unit, a data storage unit, an augmented reality data generation unit, and an augmented reality output unit that perform specific roles, it is possible to analyze the golfer's movement during the golf swing motion in real time, and to display the analyzed results. It is expressed in three dimensions, so that golfers can check their own posture in three dimensions, and provide a golf swing analyzer that can suggest a correct swing posture correction method.

Although technologies for recognizing golf swing motions or postures and suggesting corrections have been developed, most applications stop at recording or analyzing postures or motions and do not suggest corrections suitable for a user's personal body shape.

An object of the proposed invention is to provide information on a part to be corrected in a golf swing motion by recognizing a golf swing motion.

Furthermore, the proposed invention aims at recognizing a user's body shape and providing correction information suitable for the user's body shape.

Furthermore, the proposed invention aims to provide correction information of a style preferred by a user.

According to one aspect of the proposed invention, an artificial intelligence golf swing analysis and correction system based on 3D character retargeting includes an input module, a keypoint extraction module, a sequence extraction module, a correction analysis module, and a visualization module. The input module receives an image of a user's golf swing motion. The keypoint extraction module extracts keypoints including joint positions from an image input through an input module. The sequence extraction module extracts a sequence of a user's golf swing motion by tracking the movement of key points. The correction analysis module compares and analyzes a part corresponding to a correction point in the motion or posture of the user's golf swing extracted from the sequence extraction module with the motion or posture of a professional player. The visualization module generates a 3D image by retargeting the 3D character to the motion or posture analyzed by the calibration analysis module.

According to an additional aspect, the keypoint extraction module includes a keypoint recognizer learned by artificial intelligence using a 2D pose recognition algorithm, and posture characteristics of a user taking a golf swing movement (distance of two hands close, two legs attached to the ground) A primary keypoint compensator reflecting the first keypoint corrector, a keypoint collector that collects keypoints corrected by the primary keypoint corrector, and swing movement characteristics (hand movement draws an arc, using past keypoint information collected in the keypoint collector) It may include a secondary keypoint compensator that reflects the movement of the upper body is not large, etc.).

According to a further aspect, the sequence extraction module utilizes a plurality of vectors (positions between the wrists and positions between the ankles, etc.) that characterize the distance relationship therebetween, based on the keypoint coordinates corrected by the secondary keypoint corrector. A sequence classifier that classifies the swing sequence of the frame, a sequence collector that collects a sequence of keypoints for each sequence classified by the sequence classifier, and a central optimal sequence from continuous sequence information collected by the sequence collector, may include a sequence extractor that determines whether the sequence of golf swings has been completed.

According to an additional aspect, the system may further include a camera module that captures a plurality of videos including front and side views of a user standing during a golf swing and transfers the captured videos to the input module.

According to an additional aspect, the system determines that the golf swing is completed and increases the number of swings when the angle at which the arm and the golf club are positioned around the central point of the key points of the user's two shoulders in the input image rotates beyond a predetermined range. A swing count module may be further included.

According to an additional aspect, the system includes a body shape input module that receives body data including height and weight from a user, takes a picture of the user's body, and converts the data of the body shape input module and the captured 2D image into an artificial intelligence model. It may further include a 3D body shape measuring module that sets the user's body shape by generating 3D body shape information by inputting the input to the body shape.

According to an additional aspect, the system matches a professional player whose body shape is most similar to that of the user set in the 3D body shape measurement module, and compares the swing posture of the user with the swing posture or motion of the matched professional player. A calibration module may be further included.

According to an additional aspect, the system matches a style input module that receives a preferred golf style from the user with a professional player corresponding to the golf style selected in the style input module, and matches the user's swing posture with the matched professional player's swing. A posture correction module for each style that compares the posture or motion may be further included.

According to an additional aspect, the system may further include a 3D angle manipulation module for changing a viewing direction of the 3D character generated by the visualization module.

According to the proposed invention, by analyzing the golf swing motion by an automated system and analyzing the motion state for each joint, it is possible to self-correct by comparing the motion with that of professional players, and in particular, it can be corrected according to an individual's body shape and style.

1 shows the overall configuration of a golf swing analysis/correction system according to an embodiment.
2 is a block diagram showing the main components of a golf swing analysis/correction system according to an embodiment.
3 is a block diagram showing a detailed configuration of a golf swing analysis/calibration system according to an embodiment.
4 is an example output screen according to the swing count module of the golf swing analysis/correction system according to an embodiment.
5 is an example output screen according to a correction analysis module of a golf swing analysis/correction system according to an embodiment.
6 is an example output screen showing how a swing motion is reproduced in the golf swing analysis/correction system according to an embodiment.
7 is an output example screen showing head-up correction according to the correction analysis module of the golf swing analysis/correction system according to an embodiment.
8 is a photograph showing a criterion for classifying a posture in a golf swing analysis/correction system according to an embodiment.
9 is a flowchart illustrating a swing analysis method of a golf swing analysis/correction system according to an embodiment.

The foregoing and additional aspects are embodied through embodiments described with reference to the accompanying drawings. It is understood that the elements of each embodiment can be combined in various ways within one embodiment or with elements of another embodiment without contradiction with each other or other references. Based on the principle that the inventor can properly define the concept of terms in order to explain his/her invention in the best way, the terms used in this specification and claims have meanings consistent with the description or proposed technical idea. and should be interpreted as a concept. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 shows the overall configuration of a golf swing analysis/correction system according to an embodiment.

According to one aspect of the proposed invention, the artificial intelligence golf swing analysis and correction system 100 based on 3D character retargeting includes a main server 110, a camera module 130, and an output module 170.

According to an additional aspect, the camera module 130 may capture a plurality of videos including front and side views of a user standing during a golf swing, and transmit the captured videos to the input module.

The camera module 130 includes a front camera 133 that captures the front of the user's chest and a side camera 137 installed vertically with the front camera. Additional cameras can be installed as needed. The camera module 130 may be installed in the main server 110 except for the camera.

Images of the user 150 captured by the front camera 133 and the side camera 137 are transmitted to the main server 110 through the camera module 130 . The main server 110 extracts key points and sequences from the captured golf swing image, and analyzes the user's swing motion or posture to be corrected. A 3D character is retargeted to the analyzed content and visualized as a 3D image, and the visualized image is displayed through the output module 170. As the output module 170, various display devices connected to the main server 110 through a wired or wireless network, or a tablet, laptop, or mobile communication terminal may be used.

2 is a block diagram showing the main components of a golf swing analysis/correction system according to an embodiment.

According to one aspect of the proposed invention, the main server 110 of the 3D character retargeting-based artificial intelligence golf swing analysis and correction system 100 includes an input module 210, a keypoint extraction module 230, and a sequence extraction module 250 ), a calibration analysis module 270, and a visualization module 290. Each module of the main server 110 can be implemented using a storage device such as a memory, HDD, SSD, etc. in which instructions executable by a computer are stored, and a processor that executes these instructions, or an ASIC, FPGA, and electronic circuit.

The input module 210 receives an image of a user's golf swing motion.

The keypoint extraction module 230 extracts keypoints including joint positions from an image input through an input module.

According to an additional aspect, the keypoint extraction module includes a keypoint recognizer learned by artificial intelligence using a 2D pose recognition algorithm, and posture characteristics of a user taking a golf swing movement (distance of two hands close, two legs attached to the ground) A primary keypoint compensator reflecting the first keypoint corrector, a keypoint collector that collects keypoints corrected by the primary keypoint corrector, and swing movement characteristics (hand movement draws an arc, using past keypoint information collected in the keypoint collector) It may include a secondary keypoint compensator that reflects the movement of the upper body is not large, etc.).

The sequence extraction module 250 extracts a sequence of a user's golf swing motion by tracking the movement of key points.

According to a further aspect, the sequence extraction module utilizes a plurality of vectors (positions between the wrists and positions between the ankles, etc.) that characterize the distance relationship therebetween, based on the keypoint coordinates corrected by the secondary keypoint corrector. A sequence classifier that classifies the swing sequence of the frame, a sequence collector that collects a sequence of keypoints for each sequence classified by the sequence classifier, and a central optimal sequence from continuous sequence information collected by the sequence collector, may include a sequence extractor that determines if has completed a series of golf swing sequences.

The correction analysis module 270 compares and analyzes a part corresponding to a correction point in the motion or posture of the user's golf swing extracted from the sequence extraction module with the motion or posture of a professional player.

The visualization module 290 generates a 3D image by retargeting the 3D character to the motion or posture analyzed by the calibration analysis module.

3 is a block diagram showing a detailed configuration of a golf swing analysis/calibration system according to an embodiment.

According to an additional aspect, the system 100 determines that the golf swing is completed when the angle at which the arm and the golf club are positioned around the center point of the user's two shoulder key points rotates beyond a certain range in the input image, and swings A swing count module 310 that increases the number of times may be further included.

According to an additional aspect, the system 100 includes a body shape input module 320 that receives body data including height and weight from a user and takes a picture of the user's body, A 3D body shape measuring module 330 configured to set the user's body shape by generating 3D body shape information by inputting the 2D image into the artificial intelligence model may be further included.

According to an additional aspect, the system 100 matches a professional player whose body shape is most similar to that of the user set in the 3D body shape measurement module, and compares the swing posture of the user with the swing posture or motion of the matched professional player. A posture correcting module 340 for each body type may be further included.

According to an additional aspect, the system 100 matches the style input module 350 that receives the user's preferred golf style with a professional player corresponding to the golf style selected in the style input module, and determines the user's swing posture. It may further include a posture correction module 360 for each style that compares the swing posture or motion of the matched professional player.

According to an additional aspect, the system 100 may further include a 3D angle manipulation module 370 for changing the viewing direction of the 3D character generated by the visualization module.

4 is an example output screen according to the swing count module of the golf swing analysis/correction system according to an embodiment.

When the user's practice swing is recognized and the target swing posture is completed, the number of times is counted. Recognizes 2D images of the front and side and extracts key points and 3D poses using artificial intelligence learning models such as Media Pipe. Key points of the user can be displayed as red dots so that the user can grasp the core of the action himself.

5 is an example output screen according to a correction analysis module of a golf swing analysis/correction system according to an embodiment.

By analyzing the sequence of user key points, corrective feedback is possible for common mistakes such as setup, sway, head-up, reverse pivot, forward lunge, etc., and memory, HDD. Save to SSD etc. The stored content is later output as a 3D image in the visualization module 190.

In the visualization module, the analyzed 3D pose of the user retargets the 3D character so that a 3D image suitable for the user's body shape can be output. In the drawing, the posture of the user is shown on the left, and the posture of a professional player corresponding to the user posture is shown on the right. It displays problems in the user's posture so that the user can correct them himself.

6 is an example output screen showing how a swing motion is reproduced in the golf swing analysis/correction system according to an embodiment.

(1) The user is displayed on the left, and the character of a professional player is displayed on the right. (2) Simultaneously reproduce the swing motions of the user and professional players so that they can be compared. Key points representing joints and swing trajectories can be graphically displayed on the character. (3) Swing motion can be played or stopped. (4) Swing motion can be checked step by step.

7 is an output example screen showing head-up correction according to the correction analysis module of the golf swing analysis/correction system according to an embodiment.

This is a screen that displays the correction contents for the head-up among the user correction items. (1) The head position that is the standard for setup is marked in orange. (2) If there is a change in the y value at the head position during setup, the current position is displayed in red. (3) The axis of rotation during setup is displayed in orange. (4) Displayed when there is a difference compared to the axis of rotation during setup. (5) The standard head position and rotation axis of professional players are displayed in blue.

8 is a photograph showing a criterion for classifying a posture in a golf swing analysis/correction system according to an embodiment.

Based on the center point (clavicle) of the two shoulder key points, the area is divided into lines with a specific angle. Depending on which area the golf club and arm are located, frames of the captured image may be distinguished. In the drawing, the frame is top, mid-backswing or mid-downswing, toe-up, address or impact, mid-follow-through -follow-through), finish, etc.

9 is a flowchart illustrating a swing analysis method of a golf swing analysis/correction system according to an embodiment.

According to another aspect of the proposed invention, an artificial intelligence golf swing analysis and correction method based on 3D character retargeting includes an input step of receiving an image of a user's golf swing motion (S910), and an image input in the input step A keypoint extraction step (S920) of extracting keypoints including joint positions from the keypoints (S920), a sequence extraction step (S930) of extracting a sequence of the user's golf swing motion by tracking the location movement of the keypoints, and a sequence extraction step (S930). A correction analysis step (S940) of comparing and analyzing the part corresponding to the correction point in the user's golf swing motion or posture with the motion or posture of a professional player, and retargeting the 3D character to the motion or posture analyzed in the correction analysis step and a visualization step (S950) of generating a 3D image.

In the above, the present invention has been described through embodiments with reference to the accompanying drawings, but is not limited thereto, and should be interpreted to cover various modifications that can be obviously derived by those skilled in the art. The claims are intended to cover these variations.

110: main server (110)
130: camera module 130
150: user (150)
170: output module 170
210: input module 210
230: keypoint extraction module (230)
250: sequence extraction module 250
270: calibration analysis module 270
290: visualization module (290)
310: swing count module 310
320: body shape input module 320
330: 3D body shape measurement module (330)
340: posture correction module for each body type (340)
350: style input module (350)
360: correction module for each style (360)
370: 3D angle manipulation module (370)

Claims (16)

an input module that receives an image of a user's golf swing motion;
a keypoint extraction module for extracting keypoints including joint positions from the image input through the input module;
a sequence extraction module for extracting a sequence of a user's golf swing motion by tracking the positional movement of the keypoint;
a correction analysis module that compares and analyzes a part corresponding to a correction point in the user's golf swing motion or posture extracted from the sequence extraction module with the motion or posture of a professional player; and
a visualization module generating a 3D image by retargeting a 3D character to the motion or posture analyzed by the calibration analysis module;
Including, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. The method according to claim 1, wherein the keypoint extraction module:
A keypoint recognizer trained by artificial intelligence using a 2D pose recognition algorithm;
a primary key point compensator reflecting the user's posture characteristics taking a golf swing movement;
a keypoint collector that collects keypoints calibrated by the primary keypoint corrector; and
a secondary keypoint compensator reflecting swing movement characteristics using past keypoint information gathered in the keypoint collector;
Including, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. The method according to claim 2, wherein the sequence extraction module:
a sequence classifier for classifying a swing sequence of a frame using a plurality of vectors characterized by a distance relationship between them based on the keypoint coordinates corrected by the secondary keypoint corrector;
a sequence collector for collecting a sequence of keypoints for each sequence classified by the sequence classifier; and
a sequence extractor for finding an optimum sequence that is the center of the continuous sequence information collected by the sequence collector and determining whether the user has completed a series of golf swing sequences;
Including, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. The system of claim 1 , wherein the system:
a camera module that captures a plurality of videos including front and side views of a user standing during a golf swing and transfers the captured videos to the input module;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. The system of claim 1 , wherein the system:
a swing count module determining that a golf swing is completed and increasing the number of swings when an angle at which the arm and the golf club are positioned around the central point of the user's two shoulder key points in the input image rotates within a predetermined range;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. The system of claim 1 , wherein the system:
a body shape input module that receives body data including height and weight from a user; and
a 3D body shape measurement module that takes a picture of the user's body, inputs data from the body shape input module and the captured 2D image into an artificial intelligence model to generate 3D body shape information, and sets the body shape of the user;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. 7. The method of claim 6, wherein the system:
a posture correction module for each body type that matches a professional player whose body shape is most similar to that of the user set in the 3D body shape measurement module and compares the swing posture of the user with the swing posture or movement of the matched professional player;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. The system of claim 1 , wherein the system:
a style input module that receives a preferred golf style from a user; and
a posture correction module for each style that matches a professional player corresponding to the golf style selected in the style input module and compares a user's swing posture with the matched professional player's swing posture or motion;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. The system of claim 1 , wherein the system:
a 3D angle manipulation module for changing the viewing direction of the 3D character generated by the visualization module;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. A camera module that captures a plurality of videos including front and side views of a user standing during a golf swing;
an input module that receives an image of a user's golf swing motion captured by the camera module;
a keypoint extraction module for extracting keypoints including joint positions from the image input through the input module;
a sequence extraction module for extracting a sequence of a user's golf swing motion by tracking the positional movement of the keypoint;
a correction analysis module that compares and analyzes a part corresponding to a correction point in the user's golf swing motion or posture extracted from the sequence extraction module with the motion or posture of a professional player;
a visualization module generating a 3D image by retargeting a 3D character to the motion or posture analyzed by the calibration analysis module;
a 3D angle manipulation module for changing the viewing direction of the 3D character generated by the visualization module; and
an output module displaying the 3D character generated by the visualization module;
Including, 3D character retargeting-based artificial intelligence golf swing analysis and correction system. an input step of receiving an image of a user's golf swing motion;
a keypoint extraction step of extracting keypoints including joint positions from the image input in the inputting step;
a sequence extraction step of extracting a sequence of a user's golf swing motion by tracking the positional movement of the keypoint;
a correction analysis step of comparing and analyzing a part corresponding to a correction point in the user's golf swing motion or posture extracted in the sequence extraction step with the motion or posture of a professional player; and
a visualization step of generating a 3D image by retargeting a 3D character to the motion or posture analyzed in the correction analysis step;
Including, 3D character retargeting-based artificial intelligence golf swing analysis and correction method. The method according to claim 11, wherein the keypoint extraction step:
A keypoint recognition step of learning with artificial intelligence using a 2D pose recognition algorithm;
A first key point correction step reflecting the user's posture characteristics taking a golf swing movement;
a keypoint collection step of collecting keypoints corrected in the primary keypoint correction step; and
a secondary keypoint correction step in which swing movement characteristics are reflected using past keypoint information collected in the keypoint collection step;
Including, 3D character retargeting-based artificial intelligence golf swing analysis and correction method. The method according to claim 12, wherein the sequence extraction step:
a sequence classification step of classifying a swing sequence of a frame using a plurality of vectors characterized by a distance relationship between them based on the keypoint coordinates corrected in the secondary keypoint correction step;
a sequence collection step of collecting a sequence of keypoints for each sequence classified in the sequence classification step; and
a sequence extraction step of finding an optimal sequence that is the center of the continuous sequence information collected in the sequence collection step and determining whether the user has completed a series of golf swing sequences;
Including, 3D character retargeting-based artificial intelligence golf swing analysis and correction method. 12. The method of claim 11, wherein the method:
A swing count step of determining that a golf swing is completed and increasing the number of swings when the angle at which the arm and the golf club are positioned around the central point of the user's two shoulder key points in the input image rotates within a predetermined range;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction method. 12. The method of claim 11, wherein the method:
a body type input step of receiving body data including height and weight from a user;
a 3D body shape measurement step of taking a picture of the user's body and setting the user's body shape by generating 3D body shape information by inputting the data and the captured 2D image input in the body shape input step to an artificial intelligence model; and
a posture correction step for each body type of matching a professional player whose body shape is most similar to that of the user set in the 3D body shape measuring step and comparing a swing posture of the user with a swing posture or motion of the matched professional player;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction method. 12. The method of claim 11, wherein the method:
a style input step of receiving a preferred golf style from a user; and
a posture correction step for each style of matching a professional player corresponding to the golf style selected in the style input step and comparing a swing posture of the user with a swing posture or motion of the matched professional player;
Further comprising, 3D character retargeting-based artificial intelligence golf swing analysis and correction method.

Images