KR20220129762A - Server, method and computer program for analyzing swing motion - Google Patents

Abstract

A server for analyzing swing motion comprises: a joint extraction unit that extracts n joints of a human body from a captured image using a deep learning network and extracts coordinate values of the n joints for each frame; a conversion unit that converts the coordinate values of the n joints into joint feature vectors; a motion extraction unit that extracts a specific motion section among preset swing motion from the captured image; a motion feature value calculation unit that calculates a motion feature value on the basis of the joint feature vectors in the specific motion section; and a similarity computation unit that analyzes a similarity between specific motion and the preset swing motion on the basis of the motion feature value, and computes a motion similarity score.

Description

Swing motion analysis server, method and computer program {SERVER, METHOD AND COMPUTER PROGRAM FOR ANALYZING SWING MOTION}

The present invention relates to a swing motion analysis server, method and computer program.

Golf is a sport in which the golf club is rotated to hit the ball accurately. For golfers, it is very important to take an accurate posture for each major section of the golf swing.

In general, the golf swing can be largely divided into four movements. For example, a golf swing can be divided into an address where the feet are positioned to swing and the club face is aimed at the ball, the back swing that lifts the club backwards, the impact of the club head hitting the ball, and the finish, which is the closing position of the swing. have. In order to improve the skill of the golf swing, it is necessary to train each of the above-mentioned four movements to correctly take a posture.

A conventional technique for analyzing a person's posture and motion measures and detects a major part (eg, a joint, etc.) according to a person's movement by attaching a sensor to the human body.

However, in the prior art, only an analysis of tracking a person's motion is performed, and an analysis of how much skill is improved by comparing the previous motion with the current motion is insufficient. Also, there is no method for identifying a problem to be improved in performing a specific operation.

Korea Patent Publication No. 2020-0022788 (published on March 4. 2020) Korea Patent Publication No. 2016-0121379 (published on October 19, 2016)

The present invention is to solve the problems of the prior art described above, a swing motion analysis server capable of detecting the posture and motion of an athlete and comparing the previous motion with the current motion to analyze how much the athletic ability has improved. would like to provide

In addition, an object of the present invention is to provide a swing motion analysis server that can identify and improve deficiencies in performing a specific swing motion.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention, in a swing motion analysis server, extracts n joints for a human body from a captured image using a deep learning network, and the n a joint extraction unit for extracting joint coordinate values for each frame; a transformation unit that converts the n joint coordinate values into joint feature vectors; a motion extraction unit for extracting a specific motion section from a preset swing motion from the captured image; an operation characteristic value calculating unit for calculating operation characteristic values based on the joint characteristic vector in the specific operation section; It is possible to provide a swing motion analysis server comprising a similarity calculator configured to analyze a similarity with a specific motion among the preset swing motions in a reference image based on the motion feature value and calculate a motion similarity score.

Another embodiment of the present invention provides a swing motion analysis method, comprising: extracting n joints of a human body from a captured image using a deep learning network, and extracting the n joint coordinate values for each frame; converting the n joint coordinate values into joint feature vectors; extracting a specific motion section from a preset golf swing motion from the captured image; calculating a motion feature value based on the joint feature vector in the specific motion section; It is possible to provide a swing motion analysis method comprising analyzing a similarity with a specific motion among the preset golf swing motions in a reference image based on the motion characteristic value and calculating a motion similarity score.

Another embodiment of the present invention, in a computer program stored in a computer-readable recording medium including a sequence of instructions for analyzing a swing motion, when the computer program is executed by a computing device, a deep learning network in the captured image extracting n joints for the human body using the A motion section is extracted, a motion characteristic value is calculated based on the joint feature vector in the specific motion section, and a similarity with a specific motion among the preset swing motions is analyzed in a reference image based on the motion feature value. A computer program stored in a computer-readable medium including a sequence of instructions for calculating a similarity score may be provided.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, a joint coordinate value is detected from an image of a golfer's swing motion, a joint feature vector is extracted based on the detected joint coordinate value, and a feature according to a frame change By analyzing the change pattern between vectors, it is possible to compare the golfer's previous motion with the current motion. This makes it possible to accurately analyze how much an athlete has improved.

In addition, by comparing and analyzing specific swing motions and continuous swing motions with motions performed by experts, deficiencies can be identified and improved.

1 is a block diagram of a swing motion analysis system according to an embodiment of the present invention.
2 is a configuration diagram of a swing motion analysis server according to an embodiment of the present invention.
3 is an exemplary view for explaining a joint according to an embodiment of the present invention.
4 is an exemplary view for explaining a main posture of a golf swing operation according to an embodiment of the present invention.
5 is an exemplary diagram for explaining a flowchart of a swing motion analysis server according to an embodiment of the present invention.
6 is an exemplary diagram for explaining a flowchart of a conversion unit according to an embodiment of the present invention.
7 is an exemplary diagram for explaining a flowchart of an operation feature value calculating unit according to an embodiment of the present invention.
8 is a flowchart of a swing motion analysis method according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the corresponding server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Herein, golf will be described as an example for convenience of description, but the present invention may be applied to any sport having a swing motion other than golf. For example, it is applicable to sports such as baseball, tennis, and badminton.

1 is a block diagram of a swing motion analysis system according to an embodiment of the present invention. Referring to FIG. 1 , a swing motion analysis system 1 may include a photographing device 110 and a swing motion analysis server 120 . However, the above components 110 and 120 are merely illustrative of components that can be controlled by the swing motion analysis system 1 .

The photographing device 110 is a device for photographing a swing motion of a golfer. For example, the photographing apparatus 110 may include a camera and a user terminal including a camera. The photographing device 110 may photograph the swing motion of the golfer including both the golfer and the golf club in the frame of the image, and transmit the captured image to the swing motion analysis server 120 . For example, the photographing apparatus 110 may acquire an image including image frames in which the front and side surfaces of a swing motion are continuous, and may transmit the acquired image to the swing motion analysis server 120 .

The swing motion analysis server 120 detects the golfer's joint coordinate value from the golfer's swing image received from the photographing device 110 , extracts a joint feature vector based on the detected joint coordinate value, and features according to frame change Change patterns between vectors can be compared and analyzed. The swing motion analysis server 120 may precisely analyze the continuous swing motion of the golfer based on the change pattern between the feature vectors according to the frame change.

For example, the swing motion analysis server 120 may compare and analyze the previous motion and the current motion of the golfer through the change pattern between the feature vectors according to the frame change. Through this, it is possible to determine how much the golfer's golf skill has improved.

As another example, the swing motion analysis server 120 compares and analyzes a specific swing motion and a continuous swing motion with the motion of an expert to identify and improve deficiencies.

A network refers to a connection structure that allows information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound communication, Visible Light Communication (VLC), LiFi, and the like, but are not limited thereto.

2 is a configuration diagram of a swing motion analysis server according to an embodiment of the present invention. Referring to FIG. 2 , the swing motion analysis server 120 includes a joint extraction unit 210 , a transformation unit 220 , a motion extraction unit 230 , a motion feature value calculation unit 240 , and a similarity calculation unit 250 . can do. In addition, the transform unit 220 may include a feature vector calculating unit 221 and a skeleton unit 222 , and the operation feature value calculating unit 240 includes a pattern calculating unit 241 and an arrangement relation unit 242 . can do. However, the above components 210 to 250 are merely illustrative of components that can be controlled by the swing motion analysis server 120 .

Each component of the swing motion analysis server 120 of FIG. 2 is generally connected through a network. For example, as shown in FIG. 2 , the joint extraction unit 210 , the transformation unit 220 , the motion extraction unit 230 , the motion feature value calculation unit 240 , and the similarity calculation unit 250 simultaneously or time They can be connected at intervals.

The joint extractor 210 according to an embodiment of the present invention may extract n joints of a human body from a captured image using a deep learning network, and extract n joint coordinate values for each frame. For example, the joint extractor 210 may extract 17 joints of the human body from the captured image using a deep learning network, and extract the 17 joints as two-dimensional coordinate values for each frame.

3 is an exemplary view for explaining a joint according to an embodiment of the present invention. Referring to FIG. 3 , the joint 310 extracted from the joint extraction unit 210 has 16 points in the left and right pairs of eyes, ears, shoulders, elbows, wrists, pelvis, knees and ankles, and a nose is added for a total of 17 can be a dog

The motion extraction unit 230 according to an embodiment of the present invention may extract a specific motion section from a preset golf swing motion from the captured image. For example, the motion extractor 230 may extract a section in which a golf swing motion (eg, address, back swing top, impact, and finish) is performed from the captured image.

4 is an exemplary view for explaining a main posture of a golf swing operation according to an embodiment of the present invention. Referring to FIG. 4 , in general, a golf swing operation may be divided into four postures. For example, a golf swing motion may be divided into an address (a), a back swing (backswing top, b), an impact (c), and a finish (d). The address (a) posture is a posture in which the feet are positioned to swing and the club face is aimed at the ball, and the back swing (b) posture is a posture in which the club is lifted backwards, and the greater the shoulder rotation, the greater the flying distance. Impact (c) posture is the posture at the moment the club head hits the ball, and finish (d) is the swing closing posture.

For example, the motion extractor 230 extracts the entire swing section including the address (a) to the finish (d) from the captured image, or a specific posture, for example, only the back swing (b) of the swing section from the captured image. It is possible to extract the included section.

5 is an exemplary diagram for explaining a flowchart of a swing motion analysis server according to an embodiment of the present invention. Referring to FIG. 5 , the swing motion analysis server 120 may receive an image of a golf swing motion from the photographing device 110 , and extract the joint coordinate values from the joint extraction unit 210 , and a conversion unit ( 220), the joint coordinate value may be converted into a joint feature vector, and the motion feature value calculating unit 240 may calculate a motion feature value for a specific motion based on the joint feature vector, and the similarity calculator 250 may calculate the joint coordinate value. A similarity score may be calculated based on the motion feature value and compared with the motion of a reference image.

The joint extractor 210 may classify the n joints into a preset number of groups based on the human body structure. For example, the joint extractor 210 may classify 17 joints extracted from the captured image into 5 groups. Specifically, the joint extractor 210 may classify each of the 17 joints into a left arm, a right arm, a left leg, a right leg, and other groups based on the human body structure.

The transform unit 220 may convert the n joint coordinate values into a joint feature vector. For example, the transform unit 220 extracts 17 joint coordinate values from the captured image using the learned deep learning network, calculates a position change vector according to frame change from the extracted joint coordinate value, and converts it into a joint feature vector. can be converted

The motion feature value calculator 240 may calculate the motion feature value based on the joint feature vector in a specific motion section. For example, the motion feature value calculating unit 240 may calculate the motion feature value based on a change pattern of the joint feature vector according to the frame change.

The similarity calculator 250 may analyze a similarity with a specific motion among preset golf swing motions in the reference image based on the motion feature value and calculate a motion similarity score. For example, the similarity calculator 250 may calculate a similarity score of the upper body motion by comparing the calculated upper body motion feature value with the upper body motion feature value in a reference image.

6 is an exemplary diagram for explaining a flowchart of a conversion unit according to an embodiment of the present invention. The transform unit 220 may include a feature vector calculating unit 221 and a skeleton unit 222 .

First, in step S610, the joint extractor 210 may classify n joints into groups. For example, the joint extractor 210 may classify 17 joints extracted from the captured image into five groups based on the human body structure. The transform unit 220 may calculate a feature vector for each group according to a method to be described later based on the groups classified by the joint extractor 210 .

In step S620, the transform unit 220 may calculate a position change vector. The transform unit 220 according to an embodiment of the present invention calculates a position change vector according to frame change of n joint coordinate values, and a feature vector calculator 221 calculates a joint feature vector based on the position change vector. may include

)을 연산할 수 있다.For example, the feature vector calculator 221 may calculate a position change vector by calculating a position change amount of each joint coordinate value that changes according to a frame change. Specifically, the feature vector calculating unit 221 may calculate a position change amount of each joint coordinate value according to a frame change using Equation 1 below, and store the position change direction and the change magnitude of each joint coordinate value. The feature vector calculating unit 221 is configured to calculate an angle cos between the position change vectors (V ₁ , V ₂ ).

) can be calculated.

<Equation 1>

)을 연산할 수 있다. 특징 벡터 산출부(221)는 현재 프레임의 조인트 좌표값을 기준(원점)으로 Y축과 평행하게 양수(+) 방향으로 가는 단위 벡터를 하나의 벡터로 설정하고, 이전 프레임과 현재 프레임의 위치 변화 벡터 간의 차이를 다른 하나의 벡터로 설정할 수 있다. 특징 벡터 산출부(221)는 두 위치 변화 벡터의 사이각(cos

)이 양수인 경우

을 방향값으로 연산하고, 음수인 경우

을 방향값으로 연산하여, 단위 벡터와 수평 방향으로 갈 때 가중치를 크게 반영시킬 수 있다. The feature vector calculating unit 221, referring to Equation 1, divides the dot product between the position change vectors (V ₁ , V ₂ ) by the sum of the magnitudes of the two position change vectors to divide the angle cos between the two position change vectors.

) can be calculated. The feature vector calculating unit 221 sets a unit vector going in a positive (+) direction parallel to the Y axis based on the joint coordinate value of the current frame as a single vector, and changes the position of the previous frame and the current frame. The difference between vectors can be set to one vector. The feature vector calculating unit 221 is an angle (cos) between the two position change vectors.

) is positive

is calculated as a direction value, and if negative

By calculating as a direction value, the weight can be largely reflected when going in the horizontal direction with the unit vector.

또는

의 값을 각도 값으로 저장하여 총 34개의 조인트 특징 벡터를 연산할 수 있다.The feature vector calculating unit 221 calculates a position change vector according to frame change from 17 joint coordinate values using Equation 1, and the position change vector and the unit vector form

or

A total of 34 joint feature vectors can be calculated by storing the value of .

In step S630, the conversion unit 220 may define an adjacent joint. The transformation unit 220 according to an embodiment of the present invention may further include a skeleton unit 222 defining an adjacent joint based on a distance value between the joints in the group.

Referring back to FIG. 3, for example, the skeleton 222 may define the shoulder joint 331 and the elbow joint 332 positioned adjacent to each other as adjacent joints, and the elbow joint 332 and the wrist joint ( 333) can be defined as an adjacent joint. In addition, the skeleton 222 may define the pelvic joint 341 and the knee joint 342 positioned adjacent to each other as adjacent joints, and the knee joint 342 and the ankle joint 343 may be defined as adjacent joints. have.

The skeleton unit 222 may calculate the vector difference D _ab between the joints using Equation 2 below.

<Equation 2>

The skeleton 222 may calculate the vector difference D _ab between the a-th joint J _a and the b-th joint J _b using Equation 2 . Here, the vector difference (D _ab ) may be expressed as a matrix of 17*17. Specifically, the vector difference (D _ab ) matrix consists of 17 matrices having the same number of rows and columns and having a value of 0 (in the case of a distance difference from itself), and 136 pairs of matrices with the same size and opposite signs. It consists of 272 pieces.

Also, the skeleton unit 222 may define adjacent joints between the joints based on the human body structure. For example, the skeleton 222 may define the shoulder joint 331 and the wrist joint 333 that are not located adjacent to each other as adjacent joints in the body structure, and the pelvic joint 341 and the ankle joint 343 are also It can be defined as an adjacent joint.

Also, the skeleton 222 may define a relationship between adjacent joints. The relationship between adjacent joints may be based on a group divided by the joint extraction unit 210 . For example, the skeletal part 222 may define the shoulder joint 331 , the elbow joint 332 , and the wrist joint 333 defined as adjacent joints as one group, the left arm 330 , and a bone joint joint. 341 , a knee joint 342 , and an ankle joint 343 may be defined as the right leg 340 .

For another example, the skeleton 222 may define the joints 331 , 332 , 333 of the left arm 330 group and the joints (not shown) of the right arm group that are symmetric with each other in a symmetric relationship with each other, and the right leg 340 . ) group and the symmetrical joint (not shown) of the left leg group may be defined in a symmetrical relationship with each other.

The feature vector calculator 221 according to an embodiment of the present invention calculates the center point of each group as an average of the joint coordinate values within the group, calculates a group position change vector according to the frame change of the group center point, and changes the group position A joint feature vector may be calculated based on the vector.

For example, the feature vector calculator 221 may calculate the coordinate values C(x, y) of each group center point using Equation 3 below.

<Equation 3>

In Equation 3, n represents the number of joints included in the group. For example, in the case of the left arm 330 group including the shoulder joint 331 , the elbow joint 332 , and the wrist joint 333 , n is 3. The feature vector calculating unit 221 may calculate an average value (center point) of the joint coordinates within a group (center point) and an average value (center point) of the y values by using Equation (3).

The feature vector calculator 221 may calculate the center point (average value) of each group for each frame using Equation 3 . The feature vector calculating unit 221 may calculate the group position change vector by calculating the position change amount of each of the group center points that change according to the frame change.

또는

의 값을 각도 값으로 저장하여 총 10개의 조인트 특징 벡터를 산출할 수 있다. For example, the feature vector calculating unit 221 may calculate the position change amount of the group center point that changes according to the frame change of the five groups using Equation 1 described above to calculate five group position change vectors. . The feature vector calculating unit 221 is formed by the calculated five group position change vectors and the unit vector.

or

A total of 10 joint feature vectors can be calculated by storing the value of .

In step S640, the conversion unit 220 may calculate a distance value and an angle between the skeletons. The skeleton unit 222 may calculate a distance value between bones of a person based on a defined distance value between adjacent joints.

Referring back to FIG. 3 , the skeleton unit 222 may calculate a distance value between adjacent joints 331 , 332 , 333 within a group (eg, 330 ), and may calculate a distance value between adjacent joints between groups, and , an average value of the calculated distance values between adjacent joints may be calculated. The skeleton unit 222 may form the skeleton 320 by connecting joints defined as adjacent joints based on the calculated distance value and the average value. For example, the skeleton 222 may connect 17 joints 310 to form 18 skeletons 320 .

Meanwhile, the skeleton unit 222 according to an embodiment of the present invention may define a scale factor based on the average distance value of the skeleton 320 . The scale factor is an operation to match the scale between the joints extracted from each image. Even in an image of the same person, the scale of a person in the image may be different depending on the distance between the photographing device 110 and the subject, so a scale difference between feature vectors based on joint coordinate values extracted from each image may occur.

The skeleton unit 222 may use the average distance value between adjacent joints to correct the scale difference. For example, the skeleton unit 222 may calculate an average distance value of a total of 18 skeletons 320 formed. The skeleton unit 222 may define the calculated average distance value of the skeleton 320 as a scale factor, and correct a scale difference between images based on the average distance value of the skeleton 320 . Specifically, the skeleton unit 222 sets the average distance value between adjacent joints calculated from the reference image, that is, the average distance value of the skeleton 320 to 1, and compares the average distance value between adjacent joints calculated from the comparison image. It can be defined as the scale factor of the image.

Also, the skeleton unit 222 may set any one of the n joints as a reference joint, and calculate an angle between the human skeletons 320 based on an angle between the reference joint and the other joint. That is, the skeleton 222 may calculate the degree of inclination between the joints.

Referring back to FIG. 3 , for example, the skeleton 222 selects any two joints (eg, 332 and 333 ) among the 17 joints 310 , and any two joints are inclined on a two-dimensional plane. A degree 350 can be calculated. Specifically, the skeleton unit 222 sets the joint 332 having a lower index number among the two arbitrarily selected joints 332 and 333 as the reference joint, and a unit directed in the Y-axis positive (+) direction from the set reference joint. The angle between the vector and the difference vector between the reference joint and the other joint can be calculated.

)을 연산할 수 있다. 뼈대부(222)는 연산된 사이각(cos

)을 sin

로 연산할 수 있다. 뼈대부(222)는 연산된 sin

에 기초하여 뼈대(320) 간 각도를 산출할 수 있다. 구체적으로, 뼈대부(222)는 두 조인트 간의 기울어진 정도가 Y축과 평행을 이룰수록 지면에 수직하다는 것을 파악할 수 있고, 지면과 평행하게 될수록 기울었다는 것을 파악할 수 있다. 즉, 지면과 수직에 가까울수록 연산된 sin

은 '0'에 가깝고, 지면과 수평을 이룰수록 연산된 sin

은 '1'에 가까워진다. 한편, 일 예로 전술한 벡터차(D_ab)의 경우, 136개의 특징 벡터를 획득할 수 있다. For example, the skeletal part 222 is formed between the angle cos using Equation 1 described above.

) can be calculated. Skeleton 222 is the calculated in-between angle (cos)

) to sin

can be calculated with The skeleton 222 is the calculated sin

An angle between the skeletons 320 may be calculated based on . Specifically, the skeleton portion 222 can grasp that the degree of inclination between the two joints is parallel to the Y-axis, so that it is perpendicular to the ground, and it can be grasped that the degree of inclination between the two joints becomes parallel to the ground. That is, the closer to the vertical to the ground, the more the calculated sin

is close to '0', and the more horizontal it is with the ground, the more the calculated sin

is close to '1'. Meanwhile, as an example, in the case of the above-described vector difference D _ab , 136 feature vectors may be obtained.

In the embodiment shown in FIG. 3 , the skeleton unit 222 may calculate an angle between a total of 15 skeletons 320 . For example, the skeleton unit 222 sets the center joint (eg, 332) of the connected skeleton as a reference, and calculates the angle 350 between the vectors of the two joints 331 and 333 in the direction in which the skeleton advances. can The skeleton unit 222 may calculate the angle between the skeletons 320 based on a smaller value among the angles 350 between the two vectors.

7 is an exemplary diagram for explaining a flowchart of an operation feature value calculating unit according to an embodiment of the present invention. The operation feature value calculating unit 240 may include a pattern calculating unit 241 and an arrangement relation unit 242 .

In step S710 , the motion feature value calculating unit 240 may calculate a motion pattern value according to a frame change. The motion feature value calculating unit 240 according to an embodiment of the present invention may include a pattern calculating unit 241 that calculates a motion pattern value according to a frame change of a joint and a group based on a position change vector and a group position change vector. can For example, the pattern calculator 241 may calculate the movement pattern value M _i of the joint and group according to the frame change by using Equation 4 below.

<Equation 4>

,

를 이용하여 수평 방향 움직임보다 수직 방향 움직임에 따른 변화량을 더 크게 반영시킬 수 있다. Referring to Equation 4, the pattern calculating unit 241 multiplies the joint vector difference between the previous frame (k-1) and the current frame (k) by a sin value including the directionality of the joint, and calculates the value of the i-th joint or group center point. A movement pattern value M _i may be calculated. The motion pattern value (M _i ) is based on the magnitude of the joint feature vector. In addition, the pattern calculating unit 241 is a function

,

It is possible to reflect the amount of change caused by the vertical movement larger than the horizontal movement by using .

The pattern calculator 241 may calculate a movement pattern value according to a frame change at a predetermined interval, and may set a different predetermined interval for each group by reflecting group characteristics. For example, the pattern calculating unit 241 sets relatively short intervals for both arms and legs other than the rotation axis, in which the movement pattern exhibits high-frequency characteristics, so as to include all inflection points where the directionality is changed. . For another example, the pattern calculating unit 241 sets a relatively long period of a fixed period interval for the leg, which is a rotation axis in which the movement pattern exhibits low-frequency characteristics, to exclude error values due to noise and to grasp the trend in the entire continuous operation. make it possible

The pattern calculating unit 241 may more accurately analyze the movement pattern for each group included in the entire continuous operation by setting a predetermined period interval for calculating the movement pattern value according to the frame change to be different for each group.

In step S720, the operation feature value calculating unit 240 may calculate the arrangement relationship between the joint and the group. The operation feature value calculating unit 240 may further include a disposition relation unit 242 that calculates a disposition relation value between joints in a group and a disposition relation value between groups based on the angle between the skeletons.

For example, the arrangement relation unit 242 may calculate the arrangement relation value between the joints 310 in the group by using the degree of inclination between adjacent joints calculated by the skeleton unit 222 and the angle between the skeletons 320 . Referring back to FIG. 3 , the arrangement relation part 242 includes both joints 331 based on the elbow joint 332 having two adjacent joints among the three joints 331 , 332 , 333 included in the left arm 330 . , 333) to calculate the value of the angle 350 formed by the two line segments. The arrangement relation unit 242 may calculate the arrangement relation value between the joints 310 in the group by calculating the angular velocity dφ based on a predetermined period of how the calculated angle 350 changes according to the frame change.

As another example, the arrangement relation unit 242 may calculate the arrangement relation value between the joints 310 between groups using the degree of inclination between adjacent groups calculated by the skeleton unit 222 and the angle between the skeletons 320 . . As shown in FIG. 3 , in the joint of the arm group and the leg group, there is an adjacent joint between the groups. The arrangement relation unit 242 calculates the angle between the two skeletons based on the adjacent joint between the groups, and calculates the angular velocity (dφ) according to the frame change based on a certain period to calculate the arrangement relation value between the joints between the groups. .

In addition, the arrangement relation unit 242 may analyze the arrangement pattern of the rotating shaft by calculating the amount of inclination change according to the frame change of the rotating shaft. For example, the arrangement relation unit 242 may select two joints that may be axes of rotation, and track the slope of a line segment connecting the two selected joints according to a frame change.

In general, in the case of golf, the body rotates with the left foot or the right foot as a central axis. Accordingly, the arrangement relation unit 242 may track how the inclination of a line segment connecting two joints, which may be an axis of rotation, changes according to a change in the frame. For example, the arrangement relation unit 242 may calculate the angular velocity dφ according to a frame change with respect to the inclination of the line segment connecting the right shoulder joint and the right ankle or the inclination of the line segment connecting the left shoulder and the left ankle. .

Also, the arrangement relation unit 242 may analyze the arrangement pattern between the joints. For example, the arrangement relation unit 242 may use the vector difference D _ab calculated by the skeleton unit 222 to calculate the arrangement pattern value in which the relative position difference between the joints changes. The arrangement relation unit 242 may calculate a change amount dDi according to a frame change of the vector difference D _ab .

In step S730, the operation characteristic value calculating unit 240 may calculate the operation characteristic value. For example, the motion characteristic value calculating unit 240 may utilize the motion pattern value as the motion characteristic value and the arrangement relation value as the motion characteristic value. The motion characteristic value calculated by the motion feature value calculating unit 240 may be used for the motion similarity score calculated by comparing it with a reference image by the similarity calculating unit 250 to be described later.

The similarity calculating unit 250 according to an embodiment of the present invention may assign a preset weight to each group, and calculate a motion similarity score based on the weight, the group's movement pattern value, and the arrangement relation value.

For example, the similarity calculator 250 may calculate the upper body motion similarity score SM _upper using Equation 5 below.

<Equation 5>

Referring to Equation 5, the similarity calculator 250 calculates the movement pattern value M of each wrist, elbow, and shoulder joint multiplied by the scale factor S as the reference image (A). The placement relationship value (dφ) between both shoulders and each wrist calculated by the placement relationship unit 242, which is obtained by multiplying the difference between the value extracted from and the value extracted from the comparison image (A') by the weight (a) according to the upper body group _arm ) and the placement relationship value (dφ _shoulder ) between the arms and the upper body motion similarity score (SM _upper ) can be calculated by adding the difference value between the value extracted from the reference image (A) and the value extracted from the comparison image (A') . The similarity calculator 250 may measure the movement of the shoulder and the hand during the pendulum movement during the swing operation.

The similarity calculating unit 250 may calculate a lower body motion similarity score (SM _lower ) based on the movement pattern value (M _lower ) of both legs in the same way, and the movement pattern value of the group center point calculated from the group position change vector Based on (M _body ), a center of gravity similarity score (SM _body ) may be calculated.

The similarity calculating unit 250 uses the angle between the bones calculated by the skeleton unit 222, the amount of change in the rotation axis tilt according to the frame change calculated by the arrangement relation unit 242, and the arrangement relation value for a specific posture (eg, address, back swing, impact, and finish) can be calculated.

For example, the similarity calculator 250 may calculate the address posture similarity score SS _address using Equation 6 below.

<Equation 6>

Referring to Equation 6, the similarity calculator 250 calculates a disposition relationship value based on the amount of change (dDi) according to frame changes between both feet, both shoulders, left shoulder and left pelvis, right shoulder and right pelvis, shoulder and wrist, and , an address posture similarity score (SS _address ) can be calculated using the inclination between the shoulders, the joints between the shoulders and the ankles, and the angle between the bones with the elbow as the central joint. Here, since the arrangement relation value is affected by the scale, different weights may be assigned to the scale factor (s). The transformation constants α, β, and γ are constants for correcting a feature vector having a different range for each of an arrangement relation value, a slope, and an angle between the skeletons.

The similarity calculator 250 is based on the amount of change according to the frame change between the left wrist and left shoulder, right wrist and right shoulder, left elbow and left shoulder, right elbow and right shoulder, right shoulder and left pelvis, and left shoulder and right pelvis. The backswing posture similarity score (SS _backswing ) can be calculated using the placement relationship value, the inclination between the joints between the shoulder and the ankle, and the angle between the skeletons with the elbow as the center joint, and in the same way, the placement relationship value, The impact posture similarity score (SS _impact ) and the finish posture similarity score (SS _finish ) can be calculated using the inclination between joints and the angle between the bones, respectively.

Also, the similarity calculating unit 250 according to an embodiment of the present invention may calculate a similarity score for a rotational motion operation by assigning a weight to a feature vector corresponding to a change in inclination of the rotation shaft.

The similarity calculation unit 250 uses the slope of the left rotation axis formed by the left shoulder and the left ankle and the inclination of the right rotation axis formed between the right shoulder and the right ankle among the rotation axis inclination changes calculated by the arrangement relation unit 242 to score the pattern similarity score of the center of gravity of the swing. (ST _shift ) can be calculated.

For example, the similarity calculator 250 may calculate a pattern similarity score of the swing center of gravity for each section by using Equation 7 below, and calculate a final center of gravity movement pattern similarity score ST _shift by summing them.

<Equation 7>

Referring to Equation 7, the similarity calculating unit 250 divides the golf swing into two sections from the address to the back swing and from the back swing to the finish to calculate the pattern similarity score (ST _shift ) of the center of gravity of the swing. can The similarity calculator 250 may set a different comparison weight τ according to each section.

The similarity calculator 250 includes the left and right shoulder extension lines, the left and right pelvis extension lines, the left and right shoulder and pelvis extension lines, the right shoulder and pelvis extension lines, the left shoulder and right pelvis extension lines, Using the extension line of the right shoulder and the left pelvis as axes, the similarity score of the trunk rotation pattern may be calculated as in the above-described method.

The swing motion analysis server 120 according to an embodiment of the present invention may transmit the swing motion section extracted from the captured image to the user terminal in real time. The swing motion analysis server 120 may provide the swing motion section of the comparison image and the swing motion section of the reference image to the user terminal in one frame.

The swing motion analysis server 120 may convert the calculated similarity score into a score table and provide it to the user terminal. For example, the similarity score may be a sum of the similarity scores for the pendulum motion (SM), the structure (SS), and the rotation pattern (ST). In this case, the swing motion analysis server 120 may provide a final similarity score to the user terminal by differently setting weights for each item according to the evaluation criteria.

8 is a flowchart of a swing motion analysis method according to an embodiment of the present invention. The swing motion analysis method illustrated in FIG. 8 includes steps processed in time series according to the embodiments illustrated in FIGS. 1 to 7 . Therefore, even if omitted below, it is also applied to the method of analyzing the swing motion in the swing motion analysis server according to the embodiment shown in FIGS. 1 to 7 .

In step S810, the swing motion analysis server may extract n joints of the human body from the captured image using a deep learning network, and extract n joint coordinate values for each frame.

In step S820, the swing motion analysis server may convert n joint coordinate values into joint feature vectors.

In step S830, the swing motion analysis server may extract a specific motion section from among preset golf swing motions from the captured image.

In step S840, the swing motion analysis server may calculate a motion feature value based on the joint feature vector in a specific motion section.

In step S850, the swing motion analysis server may analyze the similarity with a specific motion among preset golf swing motions based on the motion characteristic value and calculate a motion similarity score.

In the above description, steps S810 to S80 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between the steps may be switched.

The method of analyzing the swing motion in the swing motion analysis server described through FIGS. 1 to 8 is in the form of a computer program stored in a computer readable recording medium executed by a computer or a recording medium including instructions executable by the computer. can also be implemented. In addition, the method of analyzing the swing motion in the swing motion analysis server described through FIGS. 1 to 8 may be implemented in the form of a computer program stored in a computer-readable recording medium that is executed by a computer.

A computer-readable recording medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable recording medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The foregoing description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

1: Swing motion analysis system
110: photographing device
120: swing motion analysis server
210: joint extraction unit
220: conversion unit
230: motion extraction unit
240: operation feature value calculation unit
250: similarity calculator

Claims (19)

In the swing motion analysis server,
a joint extraction unit that extracts n joints of a human body from a captured image using a deep learning network, and extracts the n joint coordinate values for each frame;
a transformation unit that converts the n joint coordinate values into joint feature vectors;
a motion extraction unit for extracting a specific motion section from a preset swing motion from the captured image;
an operation characteristic value calculating unit for calculating operation characteristic values based on the joint characteristic vector in the specific operation section;
A similarity calculating unit that analyzes a degree of similarity with a specific motion among the preset swing motions in a reference image based on the motion feature value and calculates a motion similarity score
Including, swing motion analysis server.
The method of claim 1,
The joint extraction unit will classify the n joints into a predetermined number of groups based on the human body structure, swing motion analysis server.
3. The method of claim 2,
The conversion unit,
A feature vector calculating unit that calculates a position change vector according to a frame change of the n joint coordinate values, and calculates the joint feature vector based on the position change vector
That comprising a, swing motion analysis server.
4. The method of claim 3,
The feature vector calculation unit,
Calculating the center point of each group as an average of the joint coordinate values within the group, calculating a group position change vector according to a frame change of the center point of the group, and calculating the joint feature vector based on the group position change vector In, swing behavior analysis server.
5. The method of claim 4,
The conversion unit,
The skeleton unit defines an adjacent joint based on a distance value between the joints in the group, and calculates a distance value between the bones of the person based on the distance value between the adjacent joints
Which will further include, swing motion analysis server.
6. The method of claim 5,
The skeleton part,
Setting any one of the n joints as a reference joint, and calculating the angle between the bones of the person based on the angle between the reference joint and the other joint, swing motion analysis server.
7. The method of claim 6,
The operation characteristic value calculation unit,
A pattern calculation unit for calculating a movement pattern value according to a frame change of the joint and the group based on the position change vector and the group position change vector
That comprising a, swing motion analysis server.
8. The method of claim 7,
The operation characteristic value calculation unit,
A placement relationship unit for calculating a placement relationship value between joints in the group and a placement relationship value between the groups based on the angle between the skeletons
Which will further include, swing motion analysis server.
9. The method of claim 8,
The similarity calculator,
A swing motion analysis server that assigns a preset weight to each group, and calculates the motion similarity score based on the weight, the group's motion pattern value, and the placement relationship value.
In the swing motion analysis method,
extracting n joints of a human body using a deep learning network from the captured image, and extracting the n joint coordinate values for each frame;
converting the n joint coordinate values into joint feature vectors;
extracting a specific motion section from among preset swing motions from the captured image;
calculating a motion feature value based on the joint feature vector in the specific motion section;
analyzing a similarity with a specific motion among the preset swing motions in a reference image based on the motion feature value and calculating a motion similarity score
Including, swing motion analysis method.
11. The method of claim 10,
The step of extracting the n joint coordinate values for each frame is to divide the n joints into a predetermined number of groups based on the body structure of the person, swing motion analysis method.
12. The method of claim 11,
Transforming the joint feature vector into the joint feature vector comprises:
calculating a position change vector according to a frame change of the n joint coordinate values, and calculating the joint feature vector based on the position change vector
That comprising a, swing motion analysis method.
13. The method of claim 12,
Calculating the joint feature vector comprises:
Calculating the center point of each group as an average of the joint coordinate values within the group, calculating a group position change vector according to a frame change of the center point of the group, and calculating the joint feature vector based on the group position change vector In and swing motion analysis method.
14. The method of claim 13,
Transforming the joint feature vector into the joint feature vector comprises:
defining an adjacent joint based on a distance value between the joints in the group, and calculating a distance value between the bones of the person based on the distance value between the adjacent joints
Which will further include, swing motion analysis method.
15. The method of claim 14,
The step of calculating the distance value between the bones of the person,
Any one of the n joints is set as a reference joint, and the angle between the skeletons of the person is calculated based on the angle between the reference joint and the other joint.
16. The method of claim 15,
The step of calculating the operation characteristic value is,
Calculating a movement pattern value according to a frame change of the joint and the group based on the position change vector and the group position change vector
That comprising a, swing motion analysis method.
17. The method of claim 16,
The step of calculating the operation characteristic value is,
calculating a placement relationship value between joints in the group and a placement relationship value between the groups based on the angle between the skeletons
Which will further include, swing motion analysis method.
18. The method of claim 17,
Calculating the similarity score comprises:
A method for analyzing swing motion, in which a preset weight is given to each group, and the motion similarity score is calculated based on the weight, the group's movement pattern value, and the arrangement relationship value.
A computer program stored on a computer-readable medium comprising a sequence of instructions for analyzing a swing motion, the computer program comprising:
When the computer program is executed by a computing device,
Extracting n joints of the human body using a deep learning network from the captured image, and extracting the n joint coordinate values for each frame,
Transform the n joint coordinate values into joint feature vectors,
Extracting a specific motion section among preset swing motions from the captured image,
calculating a motion feature value based on the joint feature vector in the specific motion section,
and a sequence of instructions for analyzing a similarity with a specific motion among the preset swing motions in a reference image based on the motion feature value and calculating a motion similarity score.

Images