KR20230092165A - Ai based swing video analyzing apparatus and method thereof - Google Patents

Abstract

The present invention relates to an AI based swing video analyzing apparatus and a method thereof. The AI based swing video analyzing apparatus according to an embodiment of the present invention comprises: a frame extracting unit which extracts a preset number of frames from a user's swing image; a swing trajectory generating unit which extracts at least one joint point for the frame by using trained image analysis, and generates a swing trajectory by using the coordinate value displacement of the joint point; and a composite image generating unit which generates a composite image that overlaps the swing trajectory with the swing image.

Description

AI-based swing video analysis device and analysis method {AI BASED SWING VIDEO ANALYZING APPARATUS AND METHOD THEREOF}

The present invention relates to an AI-based swing image analysis device and analysis method, and more particularly, a technique for learning a user's swing image to compare a swing posture and recommending a user-customized sporting goods is disclosed.

Golf club fitting can be defined as fitting the golf club to your body, not fitting your body to the club. Existing golf club fittings largely consist of expertise that can faithfully fulfill the role of fitter/assembler, accurate measuring equipment, and good fitting parts. However, there are limitations such as the ability of experts and subjective opinions according to the golf club fitting situation, and expensive measuring equipment that reduces accessibility and usability for users who want golf club fitting.

In particular, since golf club fitting is directly linked to the financial part of club rental/purchase, some are selling expensive fitting parts for the purpose of selling golf club fitting rather than custom golf club fitting for skill improvement. Sometimes. The user needs to review the results of golf club fitting several times, and convert them into data for reference in the next golf club fitting, personal practice, screen golf, field activity, and the like.

Among the prior art, Korean Patent Registration No. 10-2012963 (published on August 21, 2019) relates to a golf swing coaching system and golf swing coaching method using a stretch sensor, which senses detailed movements of joints and uses them to A technique for enabling coaching of a golf swing, sensing a joint motion at a specific point in time of a golf swing, and applying the same to golf coaching is disclosed.

However, since the conventional technology senses movement while the user is wearing the sensor, there is a limitation in not being able to utilize an image taken while the user is not wearing the sensor.

The technical problem to be solved by the present invention is to provide an AI-based swing image analysis device and analysis method in which a user can correct his or her own posture by comparing swing trajectories while watching swing images of himself and others.

In addition, it is to provide an AI-based swing image analysis device and analysis method that allows users to receive correction guidance on their swing posture without directly meeting the instructor.

In addition, it is possible to receive correction information from foreign leaders abroad, and to provide an AI-based swing video analysis device and analysis method that can minimize language inconvenience.

In addition, it is to provide an AI-based swing image analysis device and analysis method that can increase the user's participation by analyzing and compensating for the change in the user's swing posture through the reference swing image.

An AI-based swing image analysis device according to an embodiment of the present invention includes a frame extractor for extracting a predetermined number of frames from a user's swing image, and at least one joint point for the frame using learned image analysis. and a swing trajectory generating unit for extracting and generating a swing trajectory using the coordinate value displacement of the joint points, and a synthetic image generating unit for generating a synthesized image obtained by overlapping the swing trajectory with the swing image.

In addition, a comparison information generation unit for generating comparison information by comparing the swing trajectory for each joint point among the plurality of synthetic images may be further included.

In addition, it may further include a correction information providing unit that provides correction information in the form of graphics or audio input by a leader to the synthesized video.

The sporting goods recommendation unit may further include extracting the user's body information and swing pattern information from the swing image, and matching and recommending at least one or more sports goods according to the user's correction information.

In addition, the swing image may include an image of a swing using sporting goods including a golf club, a baseball bat, a tennis racket, a badminton racket, and a table tennis racket, or a motion of a naked body without a tool captured by a user terminal.

In addition, by comparing the swing trajectory difference between the swing image and the reference swing image captured at the first time point and the swing trajectory difference between the swing image and the reference swing image captured at the second point in time, a predetermined rate of change is obtained. If it exceeds, a compensation unit for paying a preset compensation means to the account of the user may be further included.

On the other hand, the analysis method of the AI-based swing image analysis device according to another embodiment of the present invention includes a frame extraction step of extracting a predetermined number of frames from a user's swing image, and the frame by using the learned image analysis. A swing trajectory generation step of extracting at least one joint point for the joint point and generating a swing trajectory using the coordinate value displacement of the joint point, and a synthetic image generation step of generating a synthetic image overlapping the swing trajectory with the swing image. includes

Accordingly, the user can correct his/her own posture by comparing swing trajectories while watching swing images of himself and others.

In addition, the user can receive correction guidance on his/her swing posture without directly meeting the instructor.

In addition, correction information can be provided from foreign leaders abroad, and inconvenience caused by language can be minimized.

In addition, the user's participation can be increased by analyzing and compensating for the change in the user's swing posture through the reference swing image.

1 is a configuration diagram of an AI-based swing image analysis device according to an embodiment of the present invention.
2 is a flowchart of an analysis method of the AI-based swing image analysis device according to FIG. 1;
3 is an exemplary view for explaining that the AI-based swing image analysis device according to FIG. 1 acquires a swing image from a user terminal.
FIG. 4 is an exemplary diagram for explaining setting joint points for each frame in a swing trajectory generating unit among the AI-based swing image analysis apparatus according to FIG. 1 .
FIG. 5 is an exemplary view for explaining that a synthetic image generating unit of the AI-based swing image analysis device according to FIG. 1 synthesizes and provides a swing trajectory to a swing image.
6 is an exemplary diagram for explaining the comparison of a plurality of synthesized images in a comparison information generating unit of the AI-based swing image analysis device according to FIG. 1 .
FIG. 7 is an exemplary diagram for explaining the provision of correction information by the correction information providing unit in the AI-based swing image analysis device according to FIG. 1 .
8 and 9 are exemplary diagrams for explaining that a sporting goods recommendation unit analyzes a swing image and recommends sporting goods among the AI-based swing image analysis apparatus according to FIG. 1 .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The terms used are terms selected in consideration of functions in the embodiment, and the meaning of the terms may vary depending on the intention of a user or operator or a precedent. Therefore, the meaning of terms used in the embodiments to be described later, when specifically defined in the present specification, follows the definition, and when there is no specific definition, it should be interpreted as a meaning generally recognized by those skilled in the art.

1 is a configuration diagram of an AI-based swing image analysis device according to an embodiment of the present invention, FIG. 2 is a flowchart of an analysis method of the AI-based swing image analysis device according to FIG. 1, and FIG. 3 is an AI-based analysis method according to FIG. It is an exemplary view for explaining that a swing image analysis device acquires a swing image from a user terminal, and FIG. 4 is an example of setting joint points for each frame in a swing trajectory generator among AI-based swing image analysis devices according to FIG. 1 FIG. 5 is an exemplary view for explaining that a synthetic image generating unit of the AI-based swing image analysis device according to FIG. 1 synthesizes and provides a swing trajectory to a swing image.

1 to 5, the AI-based swing image analysis device 100 according to an embodiment of the present invention includes a frame extractor 110 and a swing trajectory generator 120 and a synthetic image generating unit 130.

The frame extraction unit 110 extracts a predetermined number of frames 112 from the user's swing image 111 (S110). The frame extraction unit 110 may receive a user's swing image 111 from a plurality of user terminals 10 . The swing image 111 includes swings using sporting goods such as golf clubs, baseball bats, tennis rackets, badminton rackets, table tennis rackets, and the like, as well as motions performed with the naked body without tools. In this case, it is also possible that an application for guiding the user's location when capturing the swing image 111 is installed in the user terminal 10 .

Also, the frame extraction unit 110 may extract different numbers of frames 112 from the same swing image 111 . For example, the frame extraction unit 110 may also adjust the number of frames 112 according to the reproduction time of the swing image 111 . The frame extraction unit 110 may increase the number of frames 112 when the playback time of the swing video 111 is shorter than a reference value, and may reduce the number of frames 112 when longer than the reference value. This is to extract the optimal frame 112 by actively adjusting the number of frames 112 according to the reproduction time of the swing image 111 .

In addition, the frame extraction unit 110 may analyze sound information from the swing image 111 to extract a reference frame of timing for hitting a ball with a sporting goods. This is to more accurately extract the posture at the moment of impact when hitting the ball. The frame extraction unit 110 may extract a frame by searching for a hitting sound having a preset intensity or higher, and set the frame as a reference frame. The frames before and after the reference frame can be analyzed by dividing them into before and after hitting. Accordingly, the swing image may be analyzed based on the hitting sound of hitting the ball.

The swing trajectory generator 120 extracts at least one joint point 121 for the frame 112 using the learned image analysis (S120). For example, the swing trajectory generator 120 may extract the joint points 121 based on differences in pixel values in the frame 112 . The swing trajectory generator 120 may extract the joint points 121 using an artificial intelligence analysis algorithm such as pose estimation, but is not necessarily limited thereto. The swing trajectory generator 120 may extract a plurality of joint points 121 for each frame 112 of the swing image 111 . For example, the knee, wrist, ankle, elbow, neck, head, waist, etc. may be extracted as the joint points 121, but are not necessarily limited thereto.

In addition, the swing trajectory generator 120 generates the swing trajectory 122 using the coordinate value displacement of the joint point 121 . The swing trajectory generator 120 tracks the coordinate value displacement of the joint points 121 in a plurality of frames 112 using the coordinate values of the joint points 121 of each frame 112 . In other words, the swing trajectory generator 120 may generate a swing trajectory for each joint point 121 . In this case, the swing trajectory generator 120 may set different color values according to the swing trajectories of the plurality of joint points 121 . This is to allow the user to visually identify the screen.

The synthesized image generator 130 generates a synthesized image 131 in which the swing trajectory 122 is overlapped with the swing image 111 (S130). The synthesized image 131 means that the swing trajectory 122 in graphic form is displayed on the swing image 111 . In this case, it is also possible to simultaneously display a plurality of swing trajectories 122 or selectively display only some swing trajectories 122 in the synthesized image 131 . The user may download the synthesized video 131 through the user terminal 10 or receive it in a streaming form. Accordingly, the user can correct his or her own posture by comparing swing trajectories while watching the swing images 111 of the user and others.

In addition, the synthesized image generator 130 may include detailed information such as the user's location, gender, age, body size, and sports equipment as classification information in the synthesized image 131 . Such additional information may be added in the form of a hashtag, but is not necessarily limited thereto. This is to facilitate conditional search when searching for or analyzing another person's synthesized image 131 . For example, when searching for a synthesized image 131 of a man in his 40s who is 175 cm tall, the related synthesized image 131 can be quickly searched through additional information of the synthesized image 131 .

6 is an exemplary diagram for explaining the comparison of a plurality of synthesized images in a comparison information generating unit of the AI-based swing image analysis device according to FIG. 1 .

1 to 6, the AI-based swing image analysis device 100 according to an embodiment of the present invention may further include a comparison information generator 140.

The comparison information generation unit 140 compares swing trajectories 122 for each joint point 121 between a plurality of synthesized images 131 to generate comparison information 141 . For example, the comparison information generation unit 140 may generate the comparison information 141 by digitizing the difference between the angle, speed, length, and start point and end point of the swing trajectory 122 . The comparison information generating unit 140 may generate comparison information 141 by comparing the synthesized image 131 of a professional player with the synthesized image 131 of a user as a reference image. Accordingly, even in the case of a beginner, the posture can be corrected by oneself while checking the difference from the swing trajectory 122 of a professional player.

In addition, the comparison information generation unit 140 may compare the swing patterns of the swing trajectories 122 to generate comparison information 141 between synthesized images 131 having a predetermined similarity or higher. For example, in the case of the synthesized image 131 in which the user is left-handed, comparison information 141 may be generated between the synthesized images 131 of the left-handed user. Accordingly, it is possible to more effectively correct the posture by obtaining comparison information 141 for the synthesized image 131 of the user similar to the user's own swing pattern.

Also, the comparison information generation unit 140 may generate the comparison information 141 while managing the synthesized images 131 of the same user for each timeline. The comparison information generating unit 140 may accumulate comparison information 141 of the user's swing trajectory 122 and display a degree of change. This is to determine whether the user is correcting or maintaining his or her swing posture. For example, if the user is a beginner in exercise and the degree of change of the comparison information 141 of the swing trajectory 122 exceeds a preset value, it can be seen that the posture is being corrected. If the user is a professional player and the degree of change of the comparison information 141 of the swing trajectory 122 is less than a preset value, it can be seen that the user maintains the posture.

FIG. 7 is an exemplary diagram for explaining the provision of correction information by the correction information providing unit in the AI-based swing image analysis device according to FIG. 1 .

1 to 7, the AI-based swing image analysis device 100 according to an embodiment of the present invention may further include an correction information providing unit 150.

The correction information providing unit 150 provides correction information 151 in the form of graphics or audio input by the leader to the synthesized video 131 . This correction information 151 is stored separately from the synthesized image 131, and additional information may be provided together with the synthesized image 131 upon a user's request. For example, the correction information provision unit may provide a graphic interface so that the instructor can input graphic correction information 151 in the form of handwriting while viewing the synthesized image 131 . The correction information providing unit 150 may record and provide the instructor's voice. Accordingly, the user can receive correction guidance on his or her swing posture without directly meeting the instructor.

In addition, if the correction information 151 of the instructor is audio, the correction information providing unit 150 may convert it into text and provide it. For example, the correction information providing unit 150 may automatically convert voice into text using STT (Speech to Text) technology. The cheomseok information providing unit compares the language used by the user and the language used by the instructor, and in the case of different languages, it is also possible to translate and provide the voice. Accordingly, it is possible to receive correction information 151 from foreign instructors abroad, and inconvenience caused by language can be minimized.

8 and 9 are exemplary diagrams for explaining that a sporting goods recommendation unit analyzes a swing image and recommends sporting goods among the AI-based swing image analysis apparatus according to FIG. 1 .

1 to 8 , the AI-based swing image analysis device 100 according to an embodiment of the present invention may further include a sporting goods recommendation unit 160.

The sporting goods recommendation unit 160 extracts the user's body information and swing pattern information from the swing image 111 . Here, the user's body information may include height, body shape, leg length, arm length, and the like. Swing pattern information may include swing speed, swing tempo, swing direction, etc., but is not necessarily limited thereto. The sporting goods recommendation unit 160 may extract swing pattern information by analyzing the plurality of swing images 111 . In this case, the sporting goods recommendation unit 160 may also receive additional user information from the user. The user information may include age, weight, glove size, ball power, driving distance, average score, etc., but is not necessarily limited thereto.

Also, the sporting goods recommendation unit 160 may match and recommend at least one or more sporting goods according to the user's correction information. Here, the correction information is target information set by the user. For example, in the case of golf, an increase in flight distance or an increase in accuracy may be set as correction information. The sporting goods recommendation unit 160 matches the optimal sporting goods using swing pattern information and correction information. In this case, it is preferable that characteristic information for each sporting goods is previously stored in the sporting goods recommendation unit 160 . For example, in the case of a golf club, shaft weight, flex, grip size, loft angle, and the like may be set.

In addition, the sporting goods recommendation unit 160 may provide a purchase link or purchase page for sporting goods to the user terminal 10 . This is to allow the user to immediately purchase the sporting goods recommended to him or her. In this case, the user can also purchase sports equipment used in another person's swing video or synthesized video. The sporting goods recommendation unit 160 may analyze and identify sporting goods among swing images. The sporting goods recommendation unit 160 may receive user reviews and swing images using the recommended sporting goods when the user purchases the recommended sporting goods. This is to check changes in the swing trajectory before and after purchasing sporting goods.

Meanwhile, the AI-based swing image analysis device 100 according to an embodiment of the present invention may further include a compensation unit 170.

The compensation unit 170 pays a preset compensation means to the user's account. Here, the compensation means may be points, coupons, gift certificates, virtual currency, and the like. For example, the compensation unit 170 compares the swing trajectory 122 difference between the user's swing image and the reference swing image. Here, the reference swing image may be a swing image of a professional player or a swing image desired by the user.

The compensation unit 170 compares the difference between the swing trajectory 122 between the swing image captured at the first point in time and the reference swing image, and the difference between the swing image captured at the second point in time and the reference swing image. In this case, if the difference exceeds the preset rate of change, a preset compensation means is provided to the user's account. This is to compensate for the change in the user's constant swing posture. Accordingly, the user's participation can be increased by analyzing and compensating for the change in the user's swing posture through the reference swing image.

In addition, the compensation unit 170 may provide a compensation means when the user purchases and authenticates sports goods recommended by the AI-based swing image analysis device 100. For example, when the compensation unit 170 receives input of the user's payment details and authentication photos from the user terminal 10, it may authenticate them and provide a preset compensation means. When a user uploads a swing video captured using a recommended sporting goods, the compensating unit 170 identifies and authenticates the sporting goods in the corresponding Swee video through image analysis, and provides an additional compensation means. This is to induce the user who has purchased the recommended sporting goods to upload a swing video using it.

In the above, the present invention has been described with reference to the preferred embodiments described with reference to the drawings, but is not limited thereto. Therefore, the present invention should be interpreted by the description of the claims intended to cover obvious modifications that can be derived from the described examples.

100: AI-based swing video analysis device
110: frame extraction unit
111: swing video
112: frame
120: swing trajectory generating unit
121: joint point
122: swing trajectory
130: synthetic image generating unit
131: composite video
140: comparison information generating unit
141: comparative information
150: correction information provision department
151: correction information
160: sporting goods recommendation department
170: compensation unit

Claims (7)

A frame extraction unit for extracting a predetermined number of frames from the user's swing image;
a swing trajectory generating unit extracting at least one joint point for the frame using learned image analysis and generating a swing trajectory using coordinate value displacement of the joint points; and
AI-based swing image analysis device comprising a synthetic image generator for generating a synthetic image in which the swing trajectory is overlapped with the swing image. According to claim 1,
AI-based swing image analysis device further comprising a comparison information generator for generating comparison information by comparing the swing trajectory for each joint point between the plurality of synthetic images. According to claim 1 or 2,
AI-based swing video analysis device further comprising a correction information providing unit for providing correction information in the form of graphics or audio input by the instructor to the synthesized image. According to claim 1 or 2,
and an AI-based swing image analysis device further comprising a sporting goods recommendation unit that extracts the user's body information and swing pattern information from the swing image, matches and recommends at least one sporting goods according to the user's correction information. According to claim 1 or 2,
The swing image is an AI-based swing image analysis including an image of a swing using sporting goods including a golf club, a baseball bat, a tennis racket, a badminton racket, and a table tennis racket or a motion of a naked body without a tool taken by a user terminal. Device. According to claim 1 or 2,
When the difference in the swing trajectory between the swing image captured at the first time point and the reference swing image is compared with the difference in the swing trajectory between the swing image captured at the second point in time and the reference swing image, the change rate exceeds a preset change rate. AI-based swing video analysis device further comprising a compensation unit for providing a preset compensation means to the user's account. In the analysis method of the AI-based swing image analysis device,
A frame extraction step of extracting a predetermined number of frames from the user's swing image;
A swing trajectory generating step of extracting at least one joint point for the frame using the learned image analysis and generating a swing trajectory using the coordinate value displacement of the joint point; and
An AI-based swing image analysis method comprising a synthetic image generating step of generating a synthesized image in which the swing trajectory is overlapped with the swing image.

Images