KR102631752B1 - Ai based swing video analyzing apparatus and method thereof - Google Patents

Abstract

The present invention relates to an AI-based swing image analysis device and analysis method. The AI-based swing image analysis device according to an embodiment of the present invention includes a frame extractor that extracts a preset number of frames from the user's swing image, and a learning method. A swing trajectory generator that extracts at least one joint point for the frame using image analysis and generates a swing trajectory using the coordinate value displacement of the joint point, and overlaps the swing trajectory in the swing image. It includes a composite image generator that generates a composite image.

Description

AI-based swing video analysis device and analysis method {AI BASED SWING VIDEO ANALYZING APPARATUS AND METHOD THEREOF}

The present invention relates to an AI-based swing image analysis device and analysis method. More specifically, a technology is disclosed that learns a user's swing image, compares the swing posture, and recommends customized sports equipment to the user.

Golf club fitting can be defined as fitting a golf club to your body, rather than fitting your body to a club. Existing golf club fittings largely consist of professional technology that can faithfully perform the role of a fitter/assembler, accurate measuring equipment, and good fitting parts. However, subjective opinions based on the expert's capabilities and golf club fitting situation, and expensive measuring equipment have limitations, such as reducing accessibility and usability for users seeking golf club fitting.

In particular, since golf club fitting is directly connected to the financial aspect of club rental/purchase, some people conduct golf club fitting for sales purposes rather than custom golf club fitting to improve skills for the purpose of selling expensive fitting parts. There are also cases. Users need to review the golf club fitting results multiple times and dataize them for reference during the next golf club fitting, personal practice, screen golf, and field activities.

Among the conventional technologies, Republic of Korea Patent Publication No. 10-2012963 (announced on August 21, 2019) relates to a golf swing coaching system and golf swing coaching method using an elastic sensor, which senses the detailed movement of the joints and uses this to A technology has been disclosed that allows golf swing coaching, senses joint movements at specific points in the golf swing, and applies this to golf coaching.

However, because the conventional technology senses movement while the user is wearing the sensor, there is a limitation in that it cannot utilize images captured while the user is not wearing the sensor.

The technical problem to be solved by the present invention is to provide an AI-based swing image analysis device and analysis method that allows users to correct their own posture by comparing swing trajectories while watching the swing images of oneself and others.

In addition, the purpose is to provide an AI-based swing video analysis device and analysis method that allows users to receive corrective guidance on their swing posture without having to face the instructor directly.

In addition, the purpose is to provide an AI-based swing video analysis device and analysis method that can provide editing information from foreign coaches overseas and minimize inconvenience caused by language.

In addition, the purpose is to provide an AI-based swing image analysis device and analysis method that can increase user participation by analyzing and compensating for changes in the user's swing posture through the reference swing image.

The AI-based swing image analysis device according to an embodiment of the present invention includes a frame extractor that extracts a preset number of frames from the user's swing image, and at least one joint point for the frame using learned image analysis. It includes a swing trajectory generator that extracts and generates a swing trajectory using the coordinate value displacement of the joint point, and a composite image generator that generates a composite image that overlaps the swing trajectory with the swing image.

In addition, it may further include a comparison information generator that generates comparison information by comparing the swing trajectory for each joint point between the plurality of composite images.

In addition, it may further include a correction information providing unit that provides correction information in the form of graphics or voice input by the leader to the composite image.

In addition, it may further include a sporting goods recommendation unit that extracts the user's body information and swing pattern information from the swing image, and matches and recommends at least one sporting goods according to the user's correction information.

Additionally, the swing video may include a swing using sporting goods including a golf club, a baseball bat, a tennis racket, a badminton racket, and a table tennis racket, or an image of a motion performed with one's body without tools, captured by a user terminal.

In addition, the difference in the swing trajectory between the swing image captured at a first time point and the reference swing image is compared with the difference in the swing trajectory between the swing image captured at a second time point and the reference swing image to determine a preset change rate. If it exceeds the limit, it may further include a compensation unit that pays a preset compensation method to the user's account.

Meanwhile, the analysis method of the AI-based swing image analysis device according to another embodiment of the present invention includes a frame extraction step of extracting a preset number of frames from the user's swing image, and the frames using learned image analysis. A swing trajectory generation step of extracting at least one joint point and generating a swing trajectory using the coordinate value displacement of the joint point, and generating a composite image of overlapping the swing trajectory with the swing image. Includes.

Accordingly, users can correct their own posture by comparing swing trajectories while watching swing videos of themselves and others.

Additionally, users can receive corrective guidance on their swing posture without having to face the instructor directly.

Additionally, you can receive correction information from foreign leaders overseas and minimize inconveniences due to language.

In addition, the user's participation can be increased by analyzing and compensating for changes in the user's swing posture through the reference swing video.

Figure 1 is a configuration diagram of an AI-based swing image analysis device according to an embodiment of the present invention.
Figure 2 is a flowchart of the analysis method of the AI-based swing image analysis device according to Figure 1.
FIG. 3 is an example diagram illustrating how the AI-based swing image analysis device according to FIG. 1 acquires a swing image from a user terminal.
FIG. 4 is an example diagram illustrating setting joint points for each frame in the swing trajectory generator of the AI-based swing image analysis device according to FIG. 1.
FIG. 5 is an example diagram illustrating how the synthetic image generator of the AI-based swing image analysis device according to FIG. 1 synthesizes and provides a swing trajectory to a swing image.
FIG. 6 is an example diagram illustrating comparison of a plurality of composite images in the comparison information generation unit of the AI-based swing image analysis device according to FIG. 1.
FIG. 7 is an example diagram illustrating the provision of correction information by the correction information providing unit of the AI-based swing image analysis device according to FIG. 1.
Figures 8 and 9 are exemplary diagrams to explain how the sporting goods recommendation unit among the AI-based swing image analysis devices according to Figure 1 analyzes swing images and recommends sporting goods.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. The terms used are terms selected in consideration of their functions in the embodiment, and the meaning of the terms may vary depending on the intention of the user or operator or precedents. Therefore, the meaning of terms used in the embodiments described below, if specifically defined in the present specification, will follow the definition, and if there is no specific definition, the meaning will be interpreted as generally recognized by those skilled in the art.

FIG. 1 is a configuration diagram of an AI-based swing image analysis device according to an embodiment of the present invention, FIG. 2 is a flowchart of an analysis method of the AI-based swing image analysis device according to FIG. 1, and FIG. 3 is an AI-based swing image analysis device according to FIG. 1. It is an example diagram to explain how a swing image analysis device acquires a swing image from a user terminal, and FIG. 4 illustrates setting joint points for each frame in the swing trajectory generator of the AI-based swing image analysis device according to FIG. 1. FIG. 5 is an example diagram to explain how the synthetic image generator of the AI-based swing image analysis device according to FIG. 1 synthesizes and provides a swing trajectory to a swing image.

Referring to Figures 1 to 5, the AI-based swing image analysis device 100 according to an embodiment of the present invention includes a frame extractor 110 and a swing trajectory generator 120. and a composite image generator 130.

The frame extractor 110 extracts a preset number of frames 112 from the user's swing image 111 (S110). The frame extractor 110 may receive the user's swing image 111 from a plurality of user terminals 10. The swing video 111 includes swings using sporting goods such as golf clubs, baseball bats, tennis rackets, badminton rackets, and table tennis rackets, as well as motions performed with the bare body without tools. In this case, it is possible to install an application on the user terminal 10 to guide the user's position when shooting the swing image 111.

Additionally, the frame extractor 110 is also capable of extracting different numbers of frames 112 from the same swing image 111. For example, the frame extractor 110 can also adjust the number of frames 112 according to the playback time of the swing video 111. The frame extractor 110 can increase the number of frames 112 when the playback time of the swing video 111 is shorter than the standard value, and decrease the number of frames 112 when the playback time of the swing video 111 is longer than the standard value. This is to extract the optimal frame 112 by actively adjusting the number of frames 112 according to the playback time of the swing video 111.

In addition, the frame extractor 110 can analyze sound information in the swing video 111 to extract a reference frame for timing when hitting a ball with a sports product. This is to more accurately extract the posture at the moment of impact when hitting the ball. The frame extractor 110 may search for a hitting sound of a preset intensity or higher to extract a frame and set it as a reference frame. Frames before and after this reference frame can be analyzed by dividing them into before and after hitting. Accordingly, the swing video can be analyzed based on the sound of hitting the ball.

The swing trajectory generator 120 extracts at least one joint point 121 for the frame 112 using learned image analysis (S120). For example, the swing trajectory generator 120 may extract the joint point 121 based on the difference between pixel values in the frame 112. The swing trajectory generator 120 may extract the joint points 121 using an artificial intelligence analysis algorithm such as pose estimation, but is not necessarily limited to this. The swing trajectory generator 120 may extract a plurality of joint points 121 for each frame 112 of the swing image 111. For example, knees, wrists, ankles, elbows, neck, head, waist, etc. can be extracted as joint points 121, but are not necessarily limited to this.

Additionally, the swing trajectory generator 120 generates the swing trajectory 122 using the coordinate value displacement of the joint point 121. The swing trajectory generator 120 uses the coordinate value of the joint point 121 of each frame 112 to track the displacement of the coordinate value of the joint point 121 in the plurality of frames 112. In other words, the swing trajectory generator 120 can generate a swing trajectory for each joint point 121. In this case, the swing trajectory generator 120 is also capable of setting different color values according to the swing trajectories of the plurality of joint points 121. This is to enable users to visually identify it.

The composite image generator 130 generates a composite image 131 that overlaps the swing trajectory 122 with the swing image 111 (S130). The composite image 131 refers to a swing trajectory 122 displayed in graphic form on the swing image 111. In this case, it is possible to simultaneously display a plurality of swing trajectories 122 in the composite image 131 or to selectively display only some of the swing trajectories 122. The user can download the composite video 131 through the user terminal 10 or receive it in streaming form. Accordingly, the user can correct his or her own posture by comparing the swing trajectories while watching the swing images 111 of oneself and another person.

In addition, the synthetic image generator 130 can include classification information such as detailed information such as the user's location, gender, age, body size, and sports equipment in the synthetic image 131. This additional information may be added in the form of a hashtag, but is not necessarily limited to this. This is to facilitate conditional search when searching or analyzing another person's composite image 131. For example, if you want to search for a composite image 131 of a man in his 40s with a height of 175 cm, the related composite image 131 can be quickly searched through the additional information of the composite image 131.

FIG. 6 is an example diagram illustrating comparison of a plurality of composite images in the comparison information generation unit of the AI-based swing image analysis device according to FIG. 1.

Referring to FIGS. 1 to 6, the AI-based swing image analysis device 100 according to an embodiment of the present invention may further include a comparison information generator 140.

The comparison information generator 140 generates comparison information 141 by comparing the swing trajectories 122 for each joint point 121 between the plurality of composite images 131. For example, the comparison information generator 140 may generate comparison information 141 by quantifying differences in the angle, speed, length, start point, and end point of the swing trajectory 122. The comparison information generator 140 may use the professional athlete's composite image 131 as a reference image and compare it with the user's composite image 131 to generate comparative information 141. Accordingly, even beginners can correct their posture by checking the difference from the swing trajectory 122 of a professional player.

In addition, the comparison information generator 140 can compare the swing patterns of the swing trajectories 122 and generate comparison information 141 between the composite images 131 with a degree of similarity greater than or equal to a preset level. For example, in the case of a composite image 131 where the user is left-handed, comparison information 141 can be generated between the composite images 131 of the left-handed user. Accordingly, posture correction can be performed more effectively by obtaining comparative information 141 for the user's composite image 131 similar to the user's swing pattern.

Additionally, the comparison information generator 140 may generate comparison information 141 while managing the composite images 131 of the same user by timeline. The comparison information generator 140 may display the degree of change by accumulating the comparison information 141 of the user's swing trajectory 122. This is to help the user determine whether he or she is correcting or maintaining his or her swing posture. For example, if the user is a beginner in exercise and the degree of change in the comparison information 141 of the swing trajectory 122 exceeds the preset value, it can be seen that the user is correcting the posture. If the user is a professional athlete and the degree of change in the comparison information 141 of the swing trajectory 122 is less than the preset value, it can be seen that the user is maintaining the posture.

FIG. 7 is an example diagram illustrating the provision of correction information by the correction information providing unit of the AI-based swing image analysis device according to FIG. 1.

Referring to FIGS. 1 to 7 , the AI-based swing image analysis device 100 according to an embodiment of the present invention may further include an editing information providing unit 150.

The correction information providing unit 150 provides correction information 151 in the form of graphics or voice input by the leader to the composite image 131. This editing information 151 is stored separately from the composite image 131, and additional information may be provided along with the composite image 131 upon the user's request. For example, the correction information provision unit may provide a graphic interface so that the instructor can input graphic correction information 151 in the form of handwriting while viewing the composite image 131. The editing information provision unit 150 is also capable of recording and providing the leader's voice. Accordingly, users can receive corrective guidance on their swing posture without having to face the instructor directly.

In addition, if the leader's correction information 151 is voice, the correction information providing unit 150 can convert it into text and provide it. For example, the editing information providing unit 150 can automatically convert voice to text using STT (Speech to Text) technology. The cornerstone information service department compares the language used by the user and the language used by the leader, and if the languages are different, it is also possible to provide interpretation and translation of the voice. Accordingly, correction information 151 can be provided from foreign leaders overseas, and inconveniences due to language can be minimized.

Figures 8 and 9 are exemplary diagrams to explain how the sporting goods recommendation unit among the AI-based swing image analysis devices according to Figure 1 analyzes swing images and recommends sporting goods.

Referring to FIGS. 1 to 8 , the AI-based swing image analysis device 100 according to an embodiment of the present invention may further include a sporting goods recommendation unit 160.

The sporting goods recommendation unit 160 extracts the user's body information and swing pattern information from the swing image 111. Here, the user's physical information may include height, body shape, leg length, arm length, etc. Swing pattern information may include swing speed, swing tempo, swing direction, etc., but is not necessarily limited thereto. The sporting goods recommendation unit 160 may extract swing pattern information by analyzing a plurality of swing images 111. In this case, the sporting goods recommendation unit 160 is also capable of receiving additional user information from the user. User information may include, but is not limited to, age, weight, glove size, driving power, driver distance, average score, etc.

Additionally, the sporting goods recommendation unit 160 may match and recommend at least one sporting goods according to the user's calibration information. Here, correction information is target information set by the user. For example, in the case of golf, an increase in distance or an increase in accuracy can be set as correction information. The sporting goods recommendation unit 160 matches the optimal sporting goods using swing pattern information and correction information. In this case, it is desirable that the sporting goods recommendation unit 160 stores characteristic information about each sporting goods in advance. For example, in the case of a golf club, the weight, flex, grip size, loft angle, etc. of the shaft may be set.

Additionally, the sporting goods recommendation unit 160 may provide a purchase link or purchase page for sporting goods to the user terminal 10. This is to allow users to immediately purchase sports equipment recommended to them. In this case, users can also purchase sports equipment used in other people's swing videos or composite videos. The sporting goods recommendation unit 160 is also capable of identifying sporting goods among swing videos by analyzing images. The sporting goods recommendation unit 160 can receive user reviews and swing videos using the recommended sports goods when the user purchases the recommended sports goods. This is to check changes in swing trajectory etc. before and after purchasing sports equipment.

Meanwhile, the AI-based swing image analysis device 100 according to an embodiment of the present invention may further include a compensation unit 170.

The compensation unit 170 pays a preset compensation method to the user's account. Here, the compensation means may be points, coupons, gift certificates, virtual currency, etc. For example, the compensation unit 170 compares the difference in swing trajectory 122 between the user's swing image and the reference swing image. Here, the reference swing image may be a professional player's swing image or a swing image desired by the user.

The compensation unit 170 compares the difference in swing trajectory 122 between the swing image captured at the first time point and the reference swing image and the difference between the swing image captured at the second time point and the reference swing image. In this case, if the difference exceeds the preset rate of change, the preset compensation method is paid to the user's account. This is to compensate for when the user changes a certain swing posture. Accordingly, the user's participation can be increased by analyzing and compensating for changes in the user's swing posture through the reference swing image.

In addition, the compensation unit 170 may provide compensation when the user purchases and authenticates sporting goods recommended by the AI-based swing image analysis device 100. For example, when the compensation unit 170 receives the user's payment details and authentication photo from the user terminal 10, it can authenticate them and pay a preset compensation method. When a user uploads a swing video captured using recommended sports equipment, the compensation unit 170 can identify and authenticate the sports equipment in the swing video through image analysis and provide additional compensation means. This is to encourage users who have purchased recommended sports equipment to upload swing videos using them.

In the above, the present invention has been described focusing on preferred embodiments described with reference to the drawings, but is not limited thereto. Accordingly, the present invention should be construed by the recitation of the claims, which are intended to cover obvious modifications that may be derived from the described embodiments.

100: AI-based swing video analysis device
110: frame extraction unit
111: Swing video
112: frame
120: Swing trajectory generation unit
121: joint point
122: swing orbit
130: composite image generator
131: Composite video
140: comparative information generation unit
141: Comparative information
150: Editing information provision department
151: Editing information
160: Sports goods recommendation department
170: Compensation unit

Claims (7)

A frame extractor that extracts a preset number of frames from the user's swing video;
a swing trajectory generator that extracts at least one joint point for the frame using learned image analysis and generates a swing trajectory using coordinate value displacement of the joint point;
a composite image generator that generates a composite image that overlaps the swing trajectory with the swing image;
A correction information providing unit that provides correction information in the form of graphics or voice input by an instructor to the composite image;
a sporting goods recommendation unit that extracts the user's body information and swing pattern information from the swing image, matches and recommends at least one sporting goods according to the user's correction information; and
If the difference in the swing trajectory between the swing image captured at a first time point and the reference swing image is compared with the difference in the swing trajectory between the swing image captured at a second time point and the reference swing image and exceeds a preset change rate, A preset compensation means is paid to the user's account, and when the user uploads the swing video filmed using the recommended sports product, the sports product recommended in the swing video is identified and authenticated through image analysis, It includes a compensation unit that additionally pays the above compensation means,
The frame extractor,
An AI-based swing video analysis device that analyzes sound information in the swing video and extracts a reference frame for timing when hitting the ball with the sports equipment. According to paragraph 1,
An AI-based swing image analysis device further comprising a comparison information generator that generates comparison information by comparing the swing trajectory for each joint point between the plurality of composite images. delete delete According to claim 1 or 2,
The swing video is an AI-based swing video analysis that includes images of swings using sporting goods including golf clubs, baseball bats, tennis rackets, badminton rackets, and table tennis rackets, or videos of motions performed with the bare body without tools, recorded using a user terminal. Device. delete In the analysis method of the AI-based swing image analysis device,
A frame extraction step of extracting a preset number of frames from the user's swing video;
A swing trajectory generation step of extracting at least one joint point for the frame using learned image analysis and generating a swing trajectory using the coordinate value displacement of the joint point;
A composite image generating step of generating a composite image that overlaps the swing trajectory with the swing image;
A correction information providing step of providing correction information in the form of graphics or voice input by a leader to the composite image;
A sporting goods recommendation step of extracting the user's body information and swing pattern information from the swing image, matching and recommending at least one sporting goods according to the user's correction information; and
If the difference in the swing trajectory between the swing image captured at a first time point and the reference swing image is compared with the difference in the swing trajectory between the swing image captured at a second time point and the reference swing image and exceeds a preset change rate, A preset compensation means is paid to the user's account, and when the user uploads the swing video filmed using the recommended sports product, the sports product recommended in the swing video is identified and authenticated through image analysis, Including a compensation step of additionally paying the above compensation means,
The frame extraction step is,
An AI-based swing video analysis method that analyzes sound information in the swing video and extracts a reference frame for timing when hitting the ball with the sports equipment.