JP6944144B2 - Swing analyzer, method and program - Google Patents

Description

The present invention relates to a swing analysis device, method and program for analyzing the swing motion of a golf club by a golfer.

Conventionally, there are known devices that capture the swing motion of a golf club by a golfer with a camera and analyze the swing motion based on the obtained image (Patent Documents 1, 2, etc.). The results of the analysis will be used for various purposes such as fitting golf clubs suitable for golfers, improving the form of golfers, and developing golf equipment.

Japanese Unexamined Patent Publication No. 2013-215349 Japanese Unexamined Patent Publication No. 2004-313479

In the analysis of the swing motion as described above, the behavior of the part of interest such as the grip and head of the golf club and the joint of the golfer shown in the image is often three-dimensionally specified. Patent Documents 1 and 2 disclose that swing motions are photographed from a plurality of directions by a plurality of cameras, and three-dimensional coordinates of a region of interest are derived based on a triangulation method, a DLT method, or the like. However, having to arrange the cameras in a plurality of directions may increase the size of the equipment and make it difficult to secure a space for swing analysis. In addition, such three-dimensional measurement has room for improvement in terms of analysis accuracy, calculation load, and the like.

An object of the present invention is to provide a swing analysis device, a method and a program capable of analyzing a swing motion easily and with high accuracy.

The swing analysis device according to the first aspect of the present invention is a swing analysis device for analyzing the swing motion of a golf club by a golfer, and includes an acquisition unit and a derivation unit. The acquisition unit acquires a depth image obtained by capturing the swing motion with a distance image sensor. Based on the depth image, the derivation unit is a three-dimensional vector representing the position information of a predetermined portion of the golf club or the golfer during the swing operation, or the orientation of the shaft of the golf club during the swing operation. Is derived.

The swing analysis device according to the second aspect of the present invention is the swing analysis device according to the first aspect, and the acquisition unit further acquires a two-dimensional image from the distance image sensor. The derivation unit specifies the two-dimensional coordinates of the predetermined portion based on the two-dimensional image, and specifies the depth of the predetermined portion based on the depth image, thereby determining the depth of the predetermined portion. Derivation of three-dimensional coordinates.

The swing analysis device according to the third aspect of the present invention is the swing analysis device according to the second aspect, and the derivation unit is on the two-dimensional image at the first time before the first time. A search range is set based on the two-dimensional coordinates of the predetermined part at the second time, and the two-dimensional coordinates of the predetermined part are searched within the search range.

The swing analysis device according to the fourth aspect of the present invention is a swing analysis device according to any one of the first to third aspects, and the predetermined portion is a grip of the golf club.

The swing analysis device according to the fifth aspect of the present invention is the swing analysis device according to the second or third aspect, and the predetermined portion is the grip of the golf club. The derivation unit specifies the two-dimensional coordinates of the shaft of the golf club based on the two-dimensional image, and specifies the two-dimensional coordinates of the grip based on the two-dimensional coordinates of the shaft.

The swing analysis device according to the sixth aspect of the present invention is the swing analysis device according to the first aspect, and the predetermined portion is a grip of the golf club. The acquisition unit further acquires a two-dimensional image from the distance image sensor. The derivation unit specifies the two-dimensional coordinates of the shaft of the golf club based on the two-dimensional image, and specifies the depth of the shaft in the two-dimensional coordinates based on the depth image, thereby causing the shaft. Is derived, and the three-dimensional coordinates of the grip are specified based on the distribution tendency of the three-dimensional coordinates of the shaft.

The swing analysis device according to the seventh aspect of the present invention is the swing analysis device according to the fifth or sixth aspect, and the two-dimensional image and the depth image are the golfer who performs the swing operation by the distance image sensor. This is information taken from the front of. The derivation unit specifies the two-dimensional coordinates of the shaft based on the two-dimensional image near the address, and specifies the three-dimensional coordinates of the grip based on the two-dimensional coordinates of the shaft.

The swing analysis device according to the eighth aspect of the present invention is a swing analysis device according to any of the second, third, and fifth viewpoints, and the derivation unit automatically or sets a threshold value according to the phase of the swing operation. The two-dimensional image is binarized while switching between fixed.

The swing analysis device according to the ninth aspect of the present invention is the swing analysis device according to the third aspect, and the derivation unit uses the position of the golfer's head as a reference in the phase near the top during the swing operation. , Limit the setting position of the search range.

The swing analysis device according to the tenth aspect of the present invention is the swing analysis device according to the first aspect, and the acquisition unit further acquires a time-series two-dimensional image from the distance image sensor. When the position of the grip of the golf club cannot be extracted from the two-dimensional image at the first time, the derivation unit of the golf club from the two-dimensional image at the second time immediately after the first time. The position of the shaft is specified, and the position of the grip at the second time is specified based on the position of the shaft.

The swing analysis device according to the eleventh aspect of the present invention is the swing analysis device according to the first aspect, and the predetermined portion is the shoulder of the golfer.

The swing analysis device according to the twelfth aspect of the present invention is the swing analysis device according to the eleventh aspect, the acquisition unit further acquires skeleton data representing the skeleton of the human body, and the derivation unit is the depth image. And, based on the skeleton data, the three-dimensional coordinates of the shoulder are derived.

The swing analysis apparatus according to the thirteenth aspect of the present invention is a swing analysis apparatus according to the eleventh aspect or the twelfth aspect, and the derivation part uses a convex portion extraction filter that emphasizes the central portion and suppresses the peripheral portion. , The three-dimensional coordinates of the shoulder are derived by filtering the depth image.

The swing analysis device according to the 14th viewpoint of the present invention is a swing analysis device according to the 11th viewpoint or the 12th viewpoint, and the derivation unit performs template matching using a convex image template with respect to the depth image. By applying, the three-dimensional coordinates of the shoulder are derived.

The swing analysis device according to the fifteenth aspect of the present invention is a swing analysis device according to any one of the eleventh viewpoint to the fourteenth viewpoint, and the derivation unit emphasizes the vicinity of one corner and the vicinity of the other corner. By filtering the depth image with the suppressing shoulder silhouette filter, the three-dimensional coordinates of the golfer's shoulder are derived.

The swing analysis device according to the 16th aspect of the present invention is the swing analysis device according to the 12th aspect, and the derivation unit is based on the skeleton data in at least a part of the section from the address to the takeback arm horizontal. To derive the three-dimensional coordinates of the shoulder.

The swing analysis device according to the seventeenth aspect of the present invention is the swing analysis device according to the thirteenth aspect, and the lead-out unit is the convex in at least a part of the section from the takeback arm horizontal to the downswing arm horizontal. By filtering with the part extraction filter, the three-dimensional coordinates of the shoulder are derived.

The swing analysis device according to the eighteenth aspect of the present invention is the swing analysis device according to the fourteenth aspect, and the lead-out unit is the convex in at least a part of the section from the takeback arm horizontal to the downswing arm horizontal. The three-dimensional coordinates of the shoulder are derived by performing template matching using the shape image template.

The swing analysis device according to the 19th aspect of the present invention is the swing analysis device according to the 15th aspect, and the derivation unit is at least a part from the address to the takeback arm horizontal and from the downswing arm horizontal to the finish. In the section, the three-dimensional coordinates of the shoulder are derived by filtering with the shoulder silhouette filter.

The swing analysis device according to the twentieth aspect of the present invention is a swing analysis device according to any one of the eleventh viewpoints to the nineteenth viewpoint, and the derivation unit is the first on the depth image at the first time. A search range is set with reference to the two-dimensional coordinates of the shoulder at one time or a second time before the first time, and the filtering is performed within the search range.

The swing analysis device according to the 21st aspect of the present invention is a swing analysis device according to any one of the 1st to 20th viewpoints, and the derivation unit has a predetermined multiple regression equation having a predetermined coefficient. Calibration is performed by substituting the three-dimensional coordinates of the predetermined portion.

The swing analysis device according to the 22nd aspect of the present invention is a swing analysis device according to any one of the 1st to 21st viewpoints, and the acquisition unit performs the swing operation at least by the plurality of distance image sensors. The depth images taken from the first direction and the second direction are acquired.

The swing analysis device according to the 23rd aspect of the present invention is a swing analysis device according to the 22nd aspect, and the acquisition unit obtains skeleton data representing a skeleton of a human body based on at least the depth image from the first direction. Get more. The derivation unit converts the skeleton data from the first direction into the skeleton data from the second direction, and derives the position information of the predetermined portion based on the skeleton data from the second direction. do.

The swing analysis device according to the 24th viewpoint of the present invention is a swing analysis device according to the 22nd or 23rd viewpoint, the first direction is a direction toward the front of the golfer, and the second direction is a direction toward the front of the golfer. , The direction toward the side surface of the golfer.

The swing analysis device according to the 25th aspect of the present invention is a swing analysis device according to any one of the 1st to 24th viewpoints, and the acquisition unit obtains skeleton data representing the skeleton of the human body based on the depth image. Get more. The derivation unit derives the position information of the first part included in the predetermined part based on the skeleton data.

The swing analysis device according to the 26th aspect of the present invention is a swing analysis device according to the 25th aspect, and the acquisition unit further acquires silhouette data representing a silhouette of a human body. Based on the skeleton data and the silhouette data, the derivation unit derives position information of a second portion included in the predetermined portion and different from the first portion.

The swing analysis device according to the 27th viewpoint of the present invention is a swing analysis device according to any one of the first to 26th viewpoints, and the derivation unit is a region in which depth information is not obtained in the depth image. Based on the information, the area of the silhouette specified in the silhouette data is expanded.

The swing analysis device according to the 28th aspect of the present invention is a swing analysis device according to any one of the 1st to 27th viewpoints, and the derivation unit is the above-mentioned from the right side surface side of the golfer at the top timing. In the depth image, the portion protruding toward the front side is derived as the left elbow of the golfer.

The swing analysis device according to the 29th aspect of the present invention is a swing analysis device according to any one of the 1st to 28th viewpoints, and further includes a determination unit. The determination unit compares the index value calculated based on the position information of the predetermined portion at a predetermined timing and at least one of the three-dimensional vectors with the predetermined correct answer data, thereby performing the swing operation. Judge the quality of.

The swing analysis method according to the thirtieth aspect of the present invention is a swing analysis method for analyzing the swing motion of a golf club by a golfer, and includes a step of acquiring a depth image obtained by capturing the swing motion with a distance image sensor. Based on the depth image, a step of deriving the position information of the golf club or a predetermined portion of the golfer during the swing operation, or a three-dimensional vector representing the direction of the shaft of the golf club during the swing operation. To be equipped.

The swing analysis program according to the 31st aspect of the present invention is a swing analysis program for analyzing the swing motion of a golf club by a golfer, and includes a step of acquiring a depth image obtained by capturing the swing motion with a distance image sensor. Based on the depth image, a step of deriving the position information of the golf club or a predetermined portion of the golfer during the swing operation, or a three-dimensional vector representing the direction of the shaft of the golf club during the swing operation. To the computer.

According to the present invention, a depth image obtained by a distance image sensor of a golf club swinging motion by a golfer is acquired, and based on this, position information of a predetermined portion of the golf club or the golfer during the swinging motion, or a swing motion. A three-dimensional vector representing the orientation of the shaft of the golf club inside is derived. As a result, shooting from different directions using a plurality of cameras is not always required. Therefore, the swing motion can be analyzed easily and with high accuracy.

The figure which shows the whole structure of the swing analysis system including the swing analysis apparatus which concerns on 1st Embodiment of this invention. The functional block diagram of the swing analysis system which concerns on 1st Embodiment. A flowchart showing the overall flow of the grip behavior derivation process. (A) The figure which shows the address state. (B) The figure which shows the top state. (C) The figure which shows the impact state. (D) The figure which shows the finish state. The flowchart which shows the flow of the process which directly extracts the 2D coordinates of a grip end from an IR frame. The figure of the IR frame which shows the image of a grip end and a shaft. The figure explaining the complementation of the image of the grip end. A flowchart showing the flow of processing for directly specifying the depth of the grip end from the depth frame. Diagram of a depth frame showing the position of the grip edge. Diagram of the depth frame near the grip edge. A graph showing the effect of correcting the depth of the grip edge in consideration of the image of the human body. A flowchart showing the flow of exception handling. Diagram of a depth frame showing the position of the inertial spindle of the shaft. A graph plotting the relationship between the distance and depth of a point on the inertial spindle from point b. The figure of the depth frame which shows the position of the point a and point c on the inertial spindle. A flowchart showing the flow of processing for estimating the three-dimensional coordinates of the grip end based on the depth of the lower part of the grip. The figure which shows the example of the test subject. A flowchart showing the flow of correction processing. The figure explaining the method of specifying the 3D coordinates of a grip end by principal component analysis. The figure which shows the result of the correction processing. A flowchart showing the overall flow of the left shoulder behavior derivation process. A graph showing the three-dimensional coordinates of the left shoulder derived by the left shoulder behavior derivation process. The figure which shows the example of the convex part extraction filter. The figure which shows the depth frame at the top timing. The figure which shows the depth frame after applying the convex part extraction filter. The figure which shows the binarized image of a depth frame. The figure which shows the binarized image after shrinkage. The figure which shows the depth frame after masking which applied the convex part extraction filter. The figure which shows the example of the application range of the convex part extraction filter. The figure which shows another example of the application range of a convex part extraction filter. The figure which shows the example of the left shoulder silhouette filter. The figure which shows the binarized image of a depth frame. The figure which shows the binarized image after applying the left shoulder silhouette filter. The figure which shows the example of the application range of the left shoulder silhouette filter. The figure which shows another example of the application range of the left shoulder silhouette filter. The figure which shows still another example of the application range of the left shoulder silhouette filter. The figure which shows the rectangular area used for the detection of the arm horizontal frame. The flowchart which shows the whole flow of the shaft direction derivation process. The figure which shows the graph display example of each axis component of the three-dimensional vector of a shaft. The figure which shows the 3D graph display example of the 3D vector of a shaft. The figure which shows the IR frame when the actual grip end is out of the search range. The figure which shows the limit line for setting the search range on an IR frame. The figure which shows the apex of the head on the person area image when the takeback arm is horizontal. The flowchart which shows the process of deriving the three-dimensional coordinates of the grip end which concerns on embodiment of FIG. The flowchart which shows the process of deriving the three-dimensional coordinates of the grip end which concerns on a modification. The figure which shows the example of the convex shape template. The figure which shows the position of the left shoulder extracted by the template matching which concerns on the modification which used Kinect (registered trademark) v2. A conceptual diagram illustrating switching of algorithms for left shoulder extraction. The plan view which shows the whole structure of the swing analysis system including the swing analysis apparatus which concerns on 2nd Embodiment of this invention. The functional block diagram of the swing analysis system which concerns on 2nd Embodiment. A flowchart showing the flow of the position derivation process. Front view of the golfer at various checkpoints. Right side view of the golfer at various checkpoints. The figure which shows the image which superposed the front skeleton data on the front silhouette data at the time of address. The figure which shows the image which superposed the front skeleton data on the front silhouette data at the timing slightly advanced from the address. The figure which shows the image of the right side silhouette data at the time of address. FIG. 54A shows an image in which the silhouettes of the left foot and the right foot are complemented. The figure which shows the image of the right side silhouette data at the time of the top. The figure which shows the image of the front silhouette data when the takeback arm is horizontal.

Hereinafter, swing analysis devices, methods, and programs according to some embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Overview of swing analysis system>
1 and 2 show an overall configuration diagram of the swing analysis system 100 including the swing analysis device 1 according to the first embodiment of the present invention. The swing analysis system 100 is a system for photographing the swing motion of the golf club 5 by the golfer 7 as a moving image and analyzing the golf swing based on the moving image. The above shooting is performed by the distance image sensor 2. The swing analysis device 1 constitutes a swing analysis system 100 together with the distance image sensor 2, and analyzes the golf swing by performing image processing on the image data obtained by the distance image sensor 2. The result of the analysis by the swing analysis device 1 is used for various purposes such as fitting of a golf club 5 suitable for the golfer 7, improvement of the form of the golfer 7, development of golf equipment, and the like.

In the present embodiment, the behavior of the grip 51 of the golf club 5 and the behavior of the left shoulder 71 of the golfer 7 during the swing operation are analyzed. More specifically, the three-dimensional coordinates of the grip 51 and the left shoulder 71 during the swing operation are derived. Further, in the present embodiment, the orientation of the shaft 52 of the golf club 5 during the swing operation is analyzed.

<1-2. Details of each part>
Hereinafter, details of each part of the swing analysis system 100 will be described.

<1-2-1. Distance image sensor>
The distance image sensor 2 is a camera having a distance measuring function of taking a two-dimensional image of a golfer 7 trying to hit a golf club 5 and measuring the distance to a subject. Therefore, the distance image sensor 2 can output a depth image together with the two-dimensional image. The two-dimensional image referred to here is an image obtained by projecting an image of the shooting space into a plane orthogonal to the optical axis of the camera. The depth image is an image in which the depth data of the subject in the optical axis direction of the camera is assigned to pixels within the same imaging range as the two-dimensional image.

The distance image sensor 2 used in the present embodiment captures a two-dimensional image as an infrared image (hereinafter referred to as an IR image). Further, the depth image can be obtained by a method such as a time-of-flight method using infrared rays or a dot pattern projection method. Therefore, as shown in FIG. 1, the distance image sensor 2 has an IR light emitting unit 21 that emits infrared rays forward and an IR that receives infrared rays that are emitted from the IR light emitting unit 21 and reflected on the subject and returned. It has a light receiving unit 22. The IR light receiving unit 22 is a camera having an optical system, an image sensor, and the like. In the dot pattern projection method, the infrared dot pattern emitted from the IR light emitting unit 21 is read by the IR light receiving unit 22, the dot pattern is detected by image processing inside the distance image sensor 2, and the depth is calculated based on this. NS. In the present embodiment, the IR light emitting unit 21 and the IR light receiving unit 22 are housed in the same housing 20 and are arranged in front of the housing 20. Further, in the distance image sensor 2 according to the present embodiment, an effective distance capable of measuring the depth is determined, and the depth image may include pixels for which the value of the depth cannot be obtained.

In addition to the CPU 23 that controls the entire operation of the distance image sensor 2, the distance image sensor 2 has a built-in memory 24 that stores captured image data at least temporarily. The control program that controls the operation of the distance image sensor 2 is stored in the memory 24. Further, the distance image sensor 2 also has a built-in communication unit 25, and the communication unit 25 transmits the captured image data to an external device such as the swing analysis device 1 via a wired or wireless communication line 17. It can be output to the device. In the present embodiment, the CPU 23 and the memory 24 are also housed in the housing 20 together with the IR light emitting unit 21 and the IR light receiving unit 22. It should be noted that the transfer of image data to the swing analysis device 1 does not necessarily have to be performed via the communication unit 25. For example, if the memory 24 is removable, it can be removed from the housing 20 and inserted into the reader of the swing analysis device 1 (corresponding to the communication unit 15 described later) to obtain image data in the swing analysis device 1. It can be read.

In the present embodiment, as described above, infrared imaging is performed by the distance image sensor 2, and the trajectory of the grip 51 is tracked based on the captured IR image. Therefore, a reflective sheet that efficiently reflects infrared rays is attached to the grip 51 (more accurately, the grip end 51a) as a marker so that this tracking can be facilitated. Further, as will be described later, in the present embodiment, when the grip end 51a cannot be seen due to the shooting angle, the position of the grip end 51a is specified based on the position of the shaft 52 of the golf club 5, and the position of the shaft 52 is specified. The orientation is also specified. Therefore, a similar infrared reflective sheet is attached to the shaft 52 as a marker. In the present embodiment, the distance image sensor 2 is installed in front of the golfer 7 in order to photograph the golfer 7 from the front side.

<1-2-2. Swing analyzer ＞
As shown in FIG. 2, the swing analysis device 1 is manufactured by installing a swing analysis program 3 stored in a computer-readable recording medium 30 such as a CD-ROM from the recording medium 30 on a general-purpose personal computer. Will be done. The swing analysis program 3 is software for analyzing a golf swing based on image data sent from the distance image sensor 2, and causes the swing analysis device 1 to execute an operation described later.

The swing analysis device 1 includes a display unit 11, an input unit 12, a storage unit 13, a control unit 14, and a communication unit 15. These units 11 to 15 are connected to each other via a bus line 16 and can communicate with each other. The display unit 11 can be configured by a liquid crystal display or the like, and displays the result of golf swing analysis or the like to the user. The term "user" as used herein is a general term for golfers 7 themselves, their instructors, developers of golf equipment, and other persons who require the results of golf swing analysis. The input unit 12 can be composed of a mouse, a keyboard, a touch panel, and the like, and receives an operation from the user on the swing analysis device 1.

The storage unit 13 can be configured by a hard disk or the like. In addition to storing the swing analysis program 3 in the storage unit 13, image data sent from the distance image sensor 2 is stored. The control unit 14 can be composed of a CPU, a ROM, a RAM, and the like. The control unit 14 virtually operates as the acquisition unit 14a, the position derivation unit 14b, and the orientation derivation unit 14c by reading and executing the swing analysis program 3 in the storage unit 13. Details of the operation of each part 14a to 14c will be described later. The communication unit 15 functions as a communication interface for receiving data from an external device such as the distance image sensor 2 via the communication line 17.

<1-3. Swing analysis method>
Hereinafter, a golf swing analysis method executed by the swing analysis system 100 will be described. In this analysis method, first, the golfer 7 is made to test-hit the golf club 5, and the state is photographed by the distance image sensor 2. The image data of the IR image and the depth image taken by the distance image sensor 2 is sent from the distance image sensor 2 to the swing analysis device 1. As described above, in the present embodiment, the behavior of the grip 51 of the golf club 5 during the swing operation, the behavior of the left shoulder 71 of the golfer 7, and the orientation of the shaft 52 of the golf club 5 during the swing operation are performed by the swing analysis device 1. Is analyzed. In the following, first, the process of deriving the behavior of the grip 51 (hereinafter, the grip behavior deriving process) will be described, and then the process of deriving the behavior of the left shoulder 71 of the golfer 7 (hereinafter, the left shoulder behavior deriving process) will be described. After that, a process of deriving the direction of the shaft 52 of the golf club 5 (hereinafter, shaft direction deriving process) will be described.

<1-3-1. Grip behavior derivation process>
In the grip behavior derivation process, three-dimensional coordinates representing the trajectory of the grip end 51a during the swing operation are derived based on the image data of the IR image and the depth image taken by the distance image sensor 2. The image data to be analyzed is a moving image. Therefore, hereinafter, the IR image and the depth image to be analyzed may be referred to as an IR frame and a depth frame, respectively, and may be simply referred to as a frame. The three-dimensional coordinates at each timing during the swing operation are derived mainly based on the IR frame and the depth frame at that timing.

FIG. 3 is a flowchart showing the entire flow of the grip behavior derivation process. In the grip behavior derivation process, first, the acquisition unit 14a reads the image data acquired from the distance image sensor 2 and stored in the storage unit 13 and expands the image data into the memory. Then, the position derivation unit 14b determines a frame to be analyzed among these image data (step S1). In the present embodiment, the period from the address during the swing operation to a little after the impact is the target of analysis. Therefore, in step S1, the address and the timing of the impact are determined, and the frame during this period and several frames after the impact are determined as the analysis targets. Also, here, the top timing is determined in preparation for later processing. The timing of the address, the top, and the impact can be determined based on the output data from the inertial sensor attached to the grip 51, for example, or can be determined from the image data itself by image processing. Since various methods for determining the timing of such an address, top, and impact are known, detailed description thereof will be omitted here.

The swing motion of a golf club generally proceeds in the order of address, top, impact, and finish. The address means an initial state in which the head of the golf club 5 is arranged near the ball as shown in FIG. 4 (A), and the top means the golf club 5 from the address as shown in FIG. 4 (B). It means the state where the head is swung up most after taking back. The impact means the state at the moment when the golf club 5 is swung down from the top and the head collides with the ball as shown in FIG. 4 (C), and the finish means the state at the moment when the head collides with the ball, and the finish means the impact as shown in FIG. 4 (D). After that, it means a state in which the golf club 5 is swung forward.

When the period to be analyzed is determined in step S1, the position derivation unit 14b derives the three-dimensional coordinates of the grip end 51a near the address (step S2). In the present embodiment, since the image data is captured from the front of the golfer 7, the grip end 51a is hidden in the golfer 7's hand near the address and is not reflected in the frame. Therefore, in step S2, the position of the grip end 51a is estimated from the position of the shaft 52 reflected in the IR frame.

Specifically, the target of the analysis in step S2 is the frame of the initial period, for example, three frames near the address. As described above, since the shaft 52 is attached with a marker that efficiently reflects infrared rays, the image of the marker clearly appears on the IR frame. The position derivation unit 14b divides each IR frame into a marker image of the shaft 52 and other regions by binarizing each IR frame using an appropriate threshold value, and specifies the two-dimensional coordinates of the shaft 52. do. The threshold value may be a preset value or a value calculated from an IR frame by a discriminant analysis method or the like. Subsequently, the position derivation unit 14b specifies the depth value corresponding to the two-dimensional coordinates specified here from the depth frame at the same timing. As a result, the three-dimensional coordinates of the shaft 52 are derived. By the way, the length of the marker attached to the shaft 52 and the distance from the center of the marker (for example, the geometric center of gravity) to the grip end 51a are known, and these values are stored in the storage unit 13 in advance. There is. Therefore, the position deriving unit 14b is based on these lengths and distances, the length of the marker of the shaft 52 reflected in the IR frame, and the three-dimensional coordinates of the center of the marker, and the three-dimensional position of the grip end 51a near the address. Derived the coordinates.

When step S2 is completed, the three-dimensional coordinates of the grip end 51a after the initial period are specified by steps S3 to S7. Steps S3 to S7 are repeatedly executed in order along the time axis for each frame included in the analysis period determined in step S1, excluding the frames in the initial period of step S2.

Specifically, first, in step S3, the position deriving unit 14b identifies the two-dimensional coordinates of the grip end 51a directly from the IR frame by extracting the image of the grip end 51a reflected in the IR frame. Then, when the two-dimensional coordinates are successfully specified (“YES” in step S4), the position derivation unit 14b directly specifies the depth value corresponding to the two-dimensional coordinates from the depth frame (step). S5). As a result, the three-dimensional coordinates of the grip end 51a are derived. On the other hand, if the identification of the two-dimensional coordinates fails in step S3 (“NO” in step S4), the position deriving unit 14b is an exception for deriving the three-dimensional coordinates of the grip end 51a by another method. The process is executed (step S6). Similarly, even if the identification of the depth value fails in step S5 (“NO” in step S7), the position deriving unit 14b executes the same exception handling. Details of the processes of steps S3, S5 and S6 will be described later.

When the above processing for the period to be analyzed determined in step S1 is completed, the three-dimensional coordinates of the grip end 51a during the swing operation are derived. The position deriving unit 14b executes calibration in step S8, which will be described later, with respect to the three-dimensional coordinates of these grip ends 51a. Also, these calibrated values may still contain some degree of error. In particular, when the image of the grip end 51a is not reflected on the IR frame and the three-dimensional coordinates of the grip end 51a are estimated by various methods described later, an error may appear remarkably. Therefore, the position derivation unit 14b corrects these calibrated three-dimensional coordinates by a correction process (step S9). The details of this correction process will be described later.

By the above processing, the three-dimensional coordinates of the grip end 51a during the swing operation are derived with high accuracy. The three-dimensional coordinates of these grip ends 51a are stored in the storage unit 13 and are appropriately displayed on the display unit 11 in response to a user command. For example, it is also possible to graphically display the trajectory of the grip end 51a during the swing operation. Further, the three-dimensional coordinates of these grip ends 51a stored in the storage unit 13 can be read out by another program and used for further analysis of the golf swing.

<1-3-1-1. Details of step S3>
Hereinafter, the details of step S3 will be described with reference to FIG. Step S3 is a process of specifying the two-dimensional coordinates of the grip end 51a directly from the IR frame by extracting the image of the grip end 51a reflected in the IR frame. As described above, step S3 is repeatedly executed for each frame after the initial period to be analyzed in step S2. Therefore, in the following description of step S3, the flow of processing for a specific frame (hereinafter referred to as a target frame) at a specific time (hereinafter referred to as a target time) after the initial period will be described. The target frame includes an IR frame and a depth frame, which are referred to as a target IR frame and a target depth frame, respectively.

First, the position derivation unit 14b sets a search range R1 on the target frame, which is smaller than the image size of the target frame, in order to reduce the subsequent calculation load (step S21). Specifically, the two-dimensional coordinates of the grip end 51a at the target time are estimated from the two-dimensional coordinates of the grip end 51a in a predetermined number of frames (for example, three) immediately before the target time. This estimation can be performed, for example, based on the orientation of the vector between the frames of the grip ends 51a of the predetermined number of frames immediately before. Then, for example, a search range R1 having a predetermined size (for example, 50 pixels × 50 pixels) centered on the two-dimensional coordinates is set around the estimated two-dimensional coordinates.

In the following steps S22 to S24, the position derivation unit 14b binarizes the target IR frame in the search range R1 to separate the region representing the marker attached to the grip end 51a and the shaft 52 from the other regions. conduct. Therefore, first, in step S22, a threshold value for binarization is set. The threshold value can be calculated by various known methods, but in the present embodiment, it is calculated by a discriminant analysis method. Alternatively, the threshold value may be a preset value. Subsequently, in step S23, the suitability of the threshold value is determined, and if it is determined to be appropriate, the process proceeds to step S24. In step S24, the target IR frame is binarized within the search range R1 using the threshold value. At this time, it is preferable to perform expansion treatment and reduction treatment in order to prevent holes in the “1” region (region corresponding to the marker). FIG. 6 shows an example of the target IR frame for reference. FIG. 6 shows an image of a marker attached to the shaft 52 in addition to the grip end 51a. On the other hand, if it is determined in step S23 that the threshold value is inappropriate, step S3 ends, and then the exception handling in step S6 proceeds through the above-mentioned step S4. In step S23 according to the present embodiment, when the threshold value is greater than 0.1 in the search range R1 in the target IR frame when the range in which the brightness can be taken is normalized to 0 to 1, ". If it is "appropriate" and not, it is judged to be "inappropriate". In addition to this, if the maximum frequency of the brightness exceeding the above threshold value is a predetermined value (for example, 2 pixels) or more, it can be determined as "appropriate", and if not, it can be determined as "inappropriate". ..

Subsequently, in step S25, in step S25, the grip end 51a does not appear in the search range R1 in the target IR frame, or protrudes from the search range R1 (overlaps the outer frame of the search range R1). ) Whether or not. Then, when it is determined that the grip end 51a is reflected in the search range R1 and does not protrude (hereinafter, condition 1), it is considered that the search range R1 is appropriate, and the process proceeds to step S26. On the other hand, if it is determined that the condition 1 is not satisfied, it is determined that the search range R1 is inappropriate, and the process proceeds to step S20, and the search range R1 is reset. In step S20, a search range R1 larger than the previous search range R1 by a predetermined amount is set, and then steps S22 and subsequent steps are repeated. If it is determined that the pixel is protruding, the image can be enlarged by a predetermined amount (for example, 10 pixels) only in the direction of the protrusion. Although it is not clear from FIG. 5, in the present embodiment, an upper limit is provided for the number of repetitions of step S20, and when the grip end 51a does not appear within the search range R1 even after the upper limit is repeated. , Step S3 is forcibly ended. Further, if it protrudes even after repeating the upper limit number of times, the process proceeds to step S26 forcibly.

In step S26, the position deriving unit 14b removes the image of the marker of the shaft 52 when it is included in the search range R1. Specifically, the predetermined three sides of the outer frame of the search range R1 are examined, and if the "1" region exists, it is replaced with "0". As a result, the "1" region in the search range R1 becomes an region corresponding only to the grip end 51a. In the section from the address to a little after the impact, it is unlikely that the shaft will come to the left side of the grip 51, so the upper and lower sides and the right side of the grip 51 are set as the predetermined three sides. ..

In the following step S27, the position derivation unit 14b performs a labeling process to determine whether or not the image of the grip end 51a in the search range R1 is separated into a plurality of regions. As shown in FIG. 7 (A), in the target IR frame, the image of the grip end 51a may be elongated due to motion blur, especially during a downswing. Then, when this is binarized by the above method, as shown in FIG. 7B, the “1” region representing the grip end 51a can be separated into a plurality of regions. Therefore, when it is determined that the image of the grip end 51a in the search range R1 is separated into a plurality of regions, the position deriving unit 14b combines them as shown in FIG. 7 (C). Step S28). Specifically, the closer endpoints of the plurality of "1" regions are connected by a "1" line, and the inner hole is complemented by a "1".

In step S29, the position derivation unit 14b calculates the center of the “1” region in the search range R1. The center of the "1" region can be calculated by various methods, for example, as a geometric center of gravity. Then, the two-dimensional coordinates of the center are specified as the two-dimensional coordinates of the grip end 51a.

<1-3-1-2. Details of step S5>
Next, the details of step S5 will be described with reference to FIG. Step S5 is a process of directly specifying the depth value of the grip end 51a that was successfully extracted in step S3 from the depth frame. As described above, step S5 is repeatedly executed for each frame after the initial period to be analyzed in step S2, similarly to step S3. Therefore, in the description of step S5 below, the flow of processing for the target frame will be described as in the case of step S3.

First, in step S31, the position deriving unit 14b extracts the depth value at the position corresponding to the two-dimensional coordinates of the grip end 51a specified in step S3 from the target depth frame. FIG. 9 is a diagram in which the positions of the two-dimensional coordinates of the grip end 51a identified in step S3 are plotted on the target depth frame.

In the following step S32, the position derivation unit 14b determines the suitability of the depth value extracted in step S31. Then, if it is determined to be appropriate, it is determined that the value of the depth of the grip end 51a has been successfully specified, and the process proceeds to step S33 and subsequent steps. On the other hand, if it is determined in step S32 that the depth value extracted in step S31 is inappropriate, it is determined that the identification of the depth value of the grip end 51a has failed, and step S5 is terminated. In step S32 according to the present embodiment, if the depth value extracted in step S31 is within a predetermined range, for example, in the range of 2000 mm to 3500 mm, it is "appropriate", and if not, it is "inappropriate". Is judged. Further, even if the data is missing in the first place and the depth value is not extracted in step S31, it is determined to be "inappropriate".

In step S33, the position derivation unit 14b specifies the two-dimensional coordinates of the center of the shaft 52 from the target IR frame, and extracts the depth value at the position corresponding to the two-dimensional coordinates from the target depth frame. As a result, the three-dimensional coordinates of the center of the shaft 52 are derived. Specifically, the target IR frame is binarized by the same method as in step S24, and the image of the grip end 51a already extracted from the target IR frame is removed to obtain an image of the marker of the shaft 52 on the target IR frame. The "1" region to be represented is extracted. Then, the two-dimensional coordinates of the center of the "1" region (for example, the geometric center of gravity) are specified as the two-dimensional coordinates of the center of the shaft 52. Although omitted from FIG. 8 and the like for the sake of simplicity, if the derivation of the three-dimensional coordinates of the center of the shaft 52 fails, step S5 is forcibly terminated, and step S7 is followed to step S6. Exception handling shall proceed.

In the following step S34, the position deriving unit 14b calculates the angle of the shaft 52 on the target IR frame based on the two-dimensional coordinates of the grip end 51a and the two-dimensional coordinates of the center of the shaft 52. This value is used in step S35, which will be described later.

In the following steps S35 to S37, the position deriving unit 14b determines whether or not the value of the depth of the grip end 51a should be regained. The condition for regaining the depth value of the grip end 51a is
There are the following three.
(1) The angle of the shaft 52 is within a predetermined range (for example, before 11 o'clock when viewed from the front of the golfer) (step S35).
(2) The difference in the depth value of the grip end 51a between the previous frame and the target frame is a predetermined value (for example, 50 mm) or more (step S36).
(3) Regardless of the exception processing in step S6, the number of frames (for example, three) in which the depth value of the grip end 51a can be directly extracted from the depth frame continues (step S37).

If any of the conditions of steps S35 to S37 is not satisfied, the depth value of the grip end 51a specified in step S31 is determined, and step S5 ends. On the other hand, when all the conditions of steps S35 to S37 are satisfied, the position deriving unit 14b determines that the depth value of the grip end 51a specified in step S31 is not the grip end 51a but the depth value of the human body. Then, the value of the depth of the grip end 51a is retaken (step S38).

Specifically, in step S38, in the target depth frame, a search range R2 having a predetermined size (for example, 10 pixels × 10 pixels) is set around the two-dimensional coordinates of the grip end 51a (see FIG. 10). Further, the position deriving unit 14b subtracts a predetermined value (for example, 50 mm) from the depth value of the grip end 51a specified in step S31 (moves to the front side), and is within the search range R2 based on the target depth frame. It is judged that everything behind the subtracted value is the human body. Then, the depth values corresponding to all the points in the search range R2 excluding the image of the human body (which is considered to be the grip end 51a and the image of the finger) are specified, and the maximum value (deepest value) is used as the grip. Let it be the value of the depth of the end 51a. FIG. 11 is a graph showing before and after the correction of the depth value in step S38. The graph on the left is a graph in which the values in step S31 are plotted, and in the graph on the right, the positions of the two points circled in the thick frame are corrected.

<1-3-1-3. Details of step S6 (exception handling)>
Hereinafter, the exception handling in step S6 will be described with reference to FIG. The exception processing is a process for deriving the three-dimensional coordinates of the grip end 51a by another method when the derivation of the three-dimensional coordinates of the grip end 51a fails in steps S3 and S5.

First, in step S41, the position deriving unit 14b extracts an image of the marker of the shaft 52 from the target IR frame. Specifically, the target IR frame is binarized by the same method as in step S24, and the image of the grip end 51a is appropriately removed from the target IR frame to extract the image of the marker of the shaft 52 on the target IR frame. If the image of the shaft 52 is extracted here, the process proceeds to step S42, and if the image cannot be extracted, the process proceeds to step S48.

In step S42, the position deriving unit 14b identifies the two-dimensional coordinates of all the points included in the area of the marker image of the shaft 52 extracted in step S41. Further, the position derivation unit 14b specifies a depth value corresponding to the two-dimensional coordinates specified here from the target depth frame. As a result, the three-dimensional coordinates of the marker on the shaft 52 are derived.

Subsequently, the position deriving unit 14b estimates the two-dimensional coordinates of the grip end 51a based on the two-dimensional coordinates of the marker of the shaft 52 (step S43). Specifically, the two-dimensional coordinates of the grip end 51a are targeted by referring to the length of the marker on the shaft 52 and the distance from the center of the marker to the grip end 51a stored in the storage unit 13. It is specified based on the length of the marker on the shaft 52 reflected in the IR frame and the two-dimensional coordinates of the center of the marker.

In the following step S44, the position derivation unit 14b determines the suitability of the depth value of the shaft 52 extracted in step S42, and if it is determined to be appropriate, proceeds to step S45. On the other hand, if it is determined in step S44 that the depth value extracted in step S42 is inappropriate, the process proceeds to step S46. In step S44 according to the present embodiment, it is determined to be "appropriate" when the following two conditions are satisfied, and "inappropriate" when at least one of them is not satisfied. The condition (2) is a condition for checking whether the image of the marker of the shaft 52 is correctly extracted in step S41, that is, whether the data of the legs or the like is extracted.
(1) The value of the depth of the center of the shaft 52 is within a predetermined range (for example, 1000 mm to 3500 mm).
(2) Percentage of all points in the area corresponding to the marker image of the shaft 52 on the target depth frame, where the depth value is "0" (pixel value when the depth value cannot be obtained). Is less than or equal to a predetermined value (for example, 40%).

In step S45, the position deriving unit 14b estimates the three-dimensional coordinates of the grip end 51a from the three-dimensional coordinates of the marker on the shaft 52. In the present embodiment, first, on the target IR frame, the inertial spindle A1 of the shaft 52 is extracted from the image of the marker of the shaft 52, and the depth value of each point on the inertial spindle A1 is acquired from the target depth frame (FIG. See 13). In addition, we exclude outliers from these depth values. The outlier is a value outside the predetermined range (for example, 1000 mm or less, or 3500 mm or more), or a value that greatly deviates from the reference value (for example, a value that differs by 300 mm or more). The reference value is, for example, the minimum value of the depth of the point on the ab line when the angle of the shaft 52 is before 9 o'clock, and the maximum value of the depth of the point on the ab line after the angle of the shaft 52 is 9 o'clock. Can be set. Point a is the point on the inertial spindle A1 closest to the grip end 51a in the target IR frame (excluding outliers), and point b is the inertia farthest from the grip end 51a in the target IR frame. It is a point on the spindle A1 (excluding outliers) (see FIG. 13). Subsequently, in the target IR frame, the distance from the point b to each point (excluding outliers) on the inertial spindle A1 is calculated. Then, the distance and depth from point b for each point (excluding outliers) on the inertial spindle A1 are plotted in the distance-depth plane (see FIG. 14), and the primary representing the relationship between the distance and the depth. Derive an approximate expression. Subsequently, the distance from the point b in the target IR frame to the grip end 51a is calculated, and the depth of the grip end 51a is calculated by substituting this distance into the above-mentioned first-order approximation formula. As a result, the three-dimensional coordinates of the grip end 51a are derived. When step S45 ends, exception handling ends.

The inertial spindle A1 is specified by the method shown below. That is, first, an ellipse having the same moment of inertia as the image of the marker on the shaft 52 is defined. Then, let L be the length of the major axis of the ellipse, and specify the angle θ formed by the major axis of the ellipse and the x-axis (horizontal axis of the target frame). _{Further, the coordinates (x 1} , y ₁ ) of the center of gravity of the marker of the shaft 52 are derived. At this time, the coordinates of the start point (x _s , y _s ) and the end point (x _e , y _e ) of the inertial spindle A1 are as follows.
x _s = x ₁ −L · cos θ / 2
y _s = y ₁ −L · sin θ / 2
x _e = x ₁ + L · cos θ / 2
y _e = x ₁ + L · cos θ / 2

In the following S47, the three-dimensional coordinates of the grip end 51a are estimated based on the depth value of the lower point of the grip 51 instead of the grip end 51a. Step S46, which is executed before step S47, is a step for selecting a target frame in which the estimation algorithm of step S47 works effectively. Specifically, if it is determined in step S46 that the following two conditions are satisfied, the process proceeds to step S47, and if it is determined that at least one of the conditions is not satisfied, the process proceeds to step S48.
(1) The target frame is a frame near the address or the top (for example, five frames before and after).
(2) On the a-c line, there is one or more points having a depth other than the outliers.

The point c is a point in the target IR frame that is separated from the point a by a predetermined distance (for example, 20 pixels) in the direction of the grip end 51a on the line connecting the point a and the grip end 51a. .. The outlier of (2) is a value outside the predetermined range (for example, 1000 mm or less, or 3500 mm or more), or a value that greatly deviates from the reference value (for example, a value different by 150 mm or more). The reference value can also be set in the same manner as described in step S45.

Details of step S47 are shown in FIG. That is, first, in step S471, the position deriving unit 14b derives a first-order approximate expression representing the distribution of points on the a-c line. Specifically, in the target IR frame, the point on the ac line farthest from the grip end 51a (excluding outliers) is set as the reference point, and each point on the ac line from the reference point (excluding outliers). ) Is calculated. Then, a first-order approximate expression expressing the relationship between the distance from the reference point and the depth of each point (excluding outliers) on the ac line is derived.

Subsequently, in step S472, the position derivation unit 14b determines whether or not the slope of the linear approximation formula calculated in step S471 is 0, and if it is not 0, the process proceeds to step S473. In step S473, the position derivation unit 14b calculates the distance from the reference point to the grip end 51a in the target IR frame, and substitutes this distance into the first-order approximation formula to calculate the depth of the grip end 51a. .. As a result, the three-dimensional coordinates of the grip end 51a are derived. When step S473 ends, exception handling ends.

On the other hand, when it is determined in step S472 that the slope of the first-order approximation formula is 0, the position derivation unit 14b determines the depth at the target time from the sequence of movements of the grip end 51a in the immediately preceding three frames. Is estimated (step S474). For example, the depth D at the target time can be determined according to the following equation. However, D _{1 to} D ₃ are the depths D of the frames one to three before, respectively. As a result, the three-dimensional coordinates of the grip end 51a are derived. When step 474 ends, exception handling ends.
D = 2 (D ₁ − D ₂ ) − (D ₂ − D ₃ ) + D ₁

Returning to FIG. 12, in step S48, the position deriving unit 14b determines whether or not the extraction of the grip end 51a has failed for five consecutive frames. If it is determined that the failure has not occurred for 5 consecutive frames, the process proceeds to step S49, and the position of the grip end 51a at the target time is estimated from the sequence of movements of the grip end 51a in the immediately preceding 3 frames. More specifically, when the two-dimensional coordinates of the grip end 51a are specified, only the depth is estimated as in step S474, and when the two-dimensional coordinates of the grip end 51a are not specified, the grip is gripped. The three-dimensional coordinates of the end 51a are estimated. On the other hand, if it is determined in step S48 that the number of consecutive failures is 5 frames or more, the position derivation unit 14b ends the exception handling without determining the position of the grip end 51a of the target frame. In step S49, the two-dimensional coordinates of the grip end 51a are specified, and the distance between the two-dimensional coordinates of the grip end 51a in the target frame and the two-dimensional coordinates of the grip end 51a in the immediately preceding frame. When is small (for example, 5 pixels or less), the depth value of the immediately preceding frame can be used as it is as the depth value of the target frame.

<1-3-1-4. Calibration>
Hereinafter, the calibration of the three-dimensional coordinates of the grip end 51a derived by the above processing will be described. This calibration is performed by substituting the three-dimensional coordinates of the grip end 51a derived by the above process into the following multiple regression equation.

In the above multiple regression equation, x, y, Z are the three-dimensional coordinates of the grip end 51a before calibration, and X, Y, Z'are the three-dimensional coordinates of the grip end 51a after calibration. In addition, Z, Z'is the depth. The coefficients a _{1 to} c ₁ , a _{2 to} f ₂ , and a _{3 to} f ₃ are values predetermined by experiments. Hereinafter, a _{method for determining the coefficients a 1 to} c ₁ , a _{2 to} f ₂ , and a _{3 to} f ₃ will be described.

First, the XYZ'coordinate system, which is a three-dimensional coordinate system for defining the three-dimensional coordinates after calibration, is defined. Then, various test subjects are arranged in front of the distance image sensor 2 and photographed. Next, the position derivation unit 14b calculates the three-dimensional coordinates (x, y, Z) of each test subject by the same processing as in steps S3 and S5 (S29, S31) described above. Further, the actual position of each test subject is measured, expressed as three-dimensional coordinates (X, Y, Z), and these values are input to the swing analysis device 1. As a result, the position derivation unit 14b has acquired the three-dimensional coordinates (x, y, Z) and the three-dimensional coordinates (X, Y, Z) of various test subjects. Therefore, the position derivation unit 14b subsequently derives a multiple regression equation expressing the relationship between (x, y, Z) and (X, Y, Z') using these data as sample data. Although the multiple regression equation of Equation 1 is a non-linear model, it can also be expressed by a linear model, and the explanatory variables of X, Y, and Z'are arbitrarily set by appropriately combining x, y, and Z. Can be done.

In the XYZ'coordinate system, for example, the ball is set as the origin, the optical axis direction of the distance image sensor 2 representing the depth is set as the Z'axis, the vertical vertical direction is set as the Y axis, and the horizontal direction is set as the X axis. be able to. In addition, although various test subjects can be used, a plate with a marker that efficiently reflects infrared rays is prepared in a grid pattern as shown in FIG. 17 (a quadrangle surrounded by grid intersections and grids). The center of the region can function as a test subject), and by arranging this at various positions in the Z'axis direction and taking a picture, multiple regression analysis can be efficiently performed.

<1-3-1-1-5. Correction process>
Hereinafter, the correction process of step S9 will be described with reference to FIG. This correction process is a process for correcting the three-dimensional coordinates of the grip end 51a derived by the above process. As already described, an error may appear remarkably in a frame in which the image of the grip end 51a is not reflected on the IR frame and the three-dimensional coordinates of the grip end 51a are estimated based on the position of the shaft 52 or the like. .. Since such an error is particularly likely to occur in the vicinity of the address, in the present embodiment, the following steps S53 to S57 related to this correction process are executed for the frame in the vicinity of the address. However, it is also possible to execute steps S53 to S57 not only in the vicinity of the address but also in the frame of an arbitrary period. In the present embodiment, the three-dimensional coordinates after the calibration in step S8 are the targets of correction, but the three-dimensional coordinates before the calibration may be corrected.

First, as step S51, the position derivation unit 14b identifies the frame from which the grip end 51a can be first extracted on the IR frame (the frame in which the grip end 51a was first extracted in step S7; hereinafter, referred to as the last frame). do.

In the following step S52, the position derivation unit 14b is a frame up to the last frame in which the grip end 51a could not be extracted on the IR frame (a frame that became “NO” in steps S4 or S7). And all the frames from which the shaft 52 is extracted (hereinafter, the frames to be corrected) are specified. The correction target frame includes an IR frame and a depth frame, which are referred to as a correction target IR frame and a correction target depth frame, respectively.

Subsequent steps S53 to S57 are repeatedly executed for each correction target frame. Specifically, in step S53, the position deriving unit 14b extracts an image of the marker of the shaft 52 from the correction target IR frame. The specific method is the same as in step S41, and the result of step S41 can be appropriately diverted.

In the following step S54, the position deriving unit 14b identifies the two-dimensional coordinates of all the points included in the area of the marker image of the shaft 52 extracted in step S53. Further, the position derivation unit 14b specifies a depth value corresponding to the two-dimensional coordinates specified here from the correction target depth frame. As a result, the three-dimensional coordinates of the marker on the shaft 52 are derived. The results of step S42 can also be used for these processes.

In the following step S55, the position derivation unit 14b identifies all the points having an appropriate depth value (hereinafter referred to as target points) among all the points extracted in step S53. Whether or not the depth value is appropriate is determined, for example, depending on whether or not the depth value is within a predetermined range (for example, 1000 mm to 3500 mm).

In the following step S56, the position derivation unit 14b performs principal component analysis on the three-dimensional coordinates (X, Y, Z') of all the target points, and derives the first principal component. Specifically, the covariance matrix Σ of the three-dimensional coordinates (X, Y, Z') of all the target points is calculated. Then, the eigenvalues of the covariance matrix Σ are solved to calculate the _{three eigenvalues λ 1} , λ ₂ , λ ₃ and the corresponding eigenvectors ω ₁ , ω ₂ , ω _{3 (λ 1} ≧ λ ₂ ≧ λ _3). ). Then, the eigenvector ω ₁ (first principal component) corresponding to the maximum eigenvalue λ _{1 is specified.}

In the following step S57, the three-dimensional coordinates of the grip end 51a are specified on the assumption that the shaft 52 extends along _{the first principal component ω 1 (see FIG. 19).} Specifically, the position deriving unit 14b calculates the three-dimensional distance L from the center of the marker on the shaft 52 to the grip end 51a in the XYZ'coordinate system. This distance L is calculated by referring to the length of the marker on the shaft 52 and the value of the distance from the center of the marker to the grip end 51a stored in the storage unit 13. Then, on the straight line along the first principal component ω ₁ (that is, the straight line representing the shaft 52), the three-dimensional coordinates (X, Y, Z') of the points separated by the distance L from the center of the marker of the shaft 52 are set. It is specified and used as the three-dimensional coordinates of the grip end 51a.

When the above steps S53 to S57 for all the frames to be corrected are completed, the correction process is completed. FIG. 20 is a diagram showing the result of the actual correction processing. The “before correction” graph in FIG. 20 is the Z'value (depth) of the grip end 51a derived in steps S1 to S8, and the “after correction” graph is the grip end 51a after this correction processing. Z'value (depth). In addition, "True Value" is a 3D motion analysis system (actually manufactured by Interliha) that can read the 3D coordinates of markers with high accuracy based on images taken from various angles by a large number of infrared cameras. VICON was used, but of course other 3D motion analysis systems are also available) for the position of the grip end 51a. From FIG. 20, it can be seen that the coordinates of the depth of the grip end 51a can be brought closer to the true value by this correction process.

<1-3-2. Left shoulder behavior derivation process>
The left shoulder behavior derivation process will be described below. In the left shoulder behavior derivation process, three-dimensional coordinates representing the trajectory of the left shoulder 71 of the golfer 7 during the swing operation are derived based on the depth image captured by the distance image sensor 2. As in the case of the grip behavior derivation process, the image data to be analyzed is a moving image. Therefore, here as well, the depth image to be analyzed is called a depth frame. The three-dimensional coordinates at each timing during the swing operation are derived mainly based on the depth frame at that timing.

FIG. 21 is a flowchart showing the entire flow of the left shoulder behavior derivation process. In the left shoulder behavior derivation process, first, the acquisition unit 14a reads the depth frame at each timing during the swing operation, which is acquired from the distance image sensor 2 and stored in the storage unit 13, and expands it into the memory. Then, the acquisition unit 14a acquires the skeleton data at each timing from these depth frames (step S61). Skeleton data is data that represents the positions (three-dimensional coordinates) of major joints in the human body. Since various methods for deriving skeleton data from the depth frame are known, detailed description thereof will be omitted here. Further, the distance image sensor Kinect (registered trademark) (which may be v1 or v2; the same shall apply hereinafter) is provided with a library for reading skeleton data from a depth image. In the embodiment, the skeleton data is acquired using the library. The CPU 23 of the distance image sensor 2 may derive the skeleton data from the depth frame. In this case, the acquisition unit 14a may simply read the skeleton data sent from the distance image sensor 2.

Since the skeleton data includes the three-dimensional coordinates of the left shoulder 71, it can be said that step S61 is a step of deriving the behavior of the left shoulder 71. However, the position of the left shoulder 71 specified by the Kinect-derived skeleton data does not always maintain sufficient accuracy in the entire section during the snuig operation, although it depends on the purpose of use. Therefore, in the present embodiment, the three-dimensional coordinates of the left shoulder 71 are also derived by the convex portion extraction filter and the left shoulder silhouette filter, which will be described later, and these are appropriately switched. This makes it possible to derive the three-dimensional coordinates of the left shoulder 71 with high accuracy in various sections. Depending on the purpose of use, the position of the left shoulder 71 can be specified only from the skeleton data, or the position of the left shoulder 71 can be specified only from the convex extraction filter or the left shoulder silhouette filter. That is, these three methods can be used alone or in combination as appropriate, depending on the purpose of use.

Subsequently, in step S62, the position derivation unit 14b determines a frame to be analyzed among the depth frames acquired by the acquisition unit 14a or more (step S62). In the present embodiment, the period from the address during the swing operation to the finish is the target of analysis. Therefore, in step S62, the address and the finish timing are determined, and the depth frame during this period is determined as the target of the analysis according to the subsequent steps S63 to 69. The address and finish timing can be determined based on, for example, the output data from the inertial sensor attached to the grip 51, or can be determined from the image data itself by image processing. Since various methods for determining such an address and finish timing are known, detailed description thereof will be omitted here.

In the following step S63, the position deriving unit 14b reduces the image size of each depth frame. Various compression algorithms can be used, for example, bicubic or nearest neighbor can be used. By this compression processing, the following calculation load can be reduced. Further, if compression is performed so that the reduced image size becomes a constant value, it is not necessary to generate various filters described later according to the image size, and the processing becomes efficient.

In the following step S64, the position derivation unit 14b binarizes each depth frame. This binarization is performed by setting the pixel value in which the depth data exists in the area where the person is captured in the depth frame (for example, the depth range of 2200 mm to 2800 mm) to be "1" and the other pixel values to be "0". Will be done. This binarized image is an image obtained by extracting the silhouettes of subjects (mainly golfers 7 and golf clubs 5) existing within the effective distance range of the distance image sensor 2. The use of this binarized image will be described later.

In the following step S65, the position derivation unit 14b derives the three-dimensional coordinates of the left shoulder 71 from each depth frame by using the convex portion extraction filter. Further, in step S66, the position deriving unit 14b derives the three-dimensional coordinates of the left shoulder 71 from each depth frame by using the left shoulder silhouette filter. Details of the extraction process of the left shoulder 71 using these filters will be described later.

As described above, when the three-dimensional coordinates of the left shoulder 71 are derived by the three methods (the method using the skeleton data, the convex extraction filter and the left shoulder silhouette filter), the position derivation unit 14b derives the timing for switching the three methods (step). S67). As a result of diligent studies, the present inventors have found that the accuracy of the method based on the skeleton data is particularly high in the section from the address to the takeback arm horizontal during the swing motion, and is convex in the section from the takeback arm horizontal to the downswing arm horizontal. It was found that the accuracy of the method using the partial extraction filter is particularly high, and the accuracy of the method using the left shoulder silhouette filter is particularly high in the section from the downswing arm horizontal to the finish. This is because the skeleton data derived from Kinect (registered trademark) does not fully consider the twisting motion of the shoulder, and it is expected that the accuracy may decrease when the left shoulder 71 starts to rotate after the takeback arm is horizontal. That is, in the Kinect (registered trademark) skeleton model, since both shoulders and hips (both hip joints) are modeled as rigid bodies, rotation of only shoulders or hips is not evaluated. Therefore, since the reliability of the skeleton data tends to decrease after the takeback arm horizontal in which only the shoulder rotates, the switching according to the present embodiment is effective. Further, when the golfer 7 is photographed from the front, the left shoulder 71 is present inside the entire silhouette of the golfer 7 in the section from the takeback arm horizontal to the downswing arm horizontal. Then, as will be described in detail later, since the convex portion extraction filter is a filter that emphasizes the convex portion in the depth frame, the left shoulder 71 can be accurately detected by the method using the convex portion extraction filter. On the other hand, when the golfer 7 is photographed from the front, the left shoulder 71 is present near the outline of the entire silhouette of the golfer 7 in the section from the downswing arm horizontal to the finish. Then, as will be described in detail later, the left shoulder 71 can be accurately detected by the method using the left shoulder silhouette filter for emphasizing the contour of the left shoulder 71. For the same reason, the left shoulder silhouette filter can be suitably used even in the section from the address to the takeback arm horizontal.

From the above, in the present embodiment, the takeback arm horizontal and the downswing arm horizontal timings are derived as the switching timings. Details of the method for determining the timing of these arm horizontals will be described later. The takeback arm horizontal refers to the timing at which the left arm becomes horizontal during takeback, and the downswing arm horizontal refers to the timing at which the left arm becomes horizontal during the downswing.

When the arm horizontal timing is derived, the position deriving unit 14b switches the three-dimensional coordinates of the left shoulder 71 by three methods along the time axis, and shifts the trajectory of the left shoulder 71 during the swing operation in the three-dimensional space. Derivation (step S68). Specifically, the three-dimensional coordinates of the left shoulder 71 derived from the skeleton data in the section from the address to the takeback arm horizontal, and the left shoulder 71 derived from the convex extraction filter in the section from the takeback arm horizontal to the downswing arm horizontal. And the three-dimensional coordinates of the left shoulder 71 derived from the left shoulder silhouette filter in the section from the downswing arm horizontal to the finish are connected, and the left shoulder 71 in the xyZ coordinate system from the address to the finish. Orbit is derived. After that, the trajectory of the left shoulder 71 is calibrated in the same manner as in step S8 (step S69). FIG. 22 is an example of the trajectory of the left shoulder 71 after calibration.

By the above processing, the three-dimensional coordinates of the left shoulder 71 during the swing operation are derived with high accuracy. The three-dimensional coordinates of the left shoulder 71 are stored in the storage unit 13, and are appropriately displayed on the display unit 11 according to a user's command. For example, it is possible to graphically display the trajectory of the left shoulder 71 during the swing operation. Further, the three-dimensional coordinates of these left shoulders 71 stored in the storage unit 13 can be read out by another program and used for further analysis of the golf swing.

<1-3-2-1. Left shoulder extraction with convex extraction filter>
Hereinafter, a method of extracting the left shoulder 71 by the convex portion extraction filter will be described. The convex extraction filter is a filter that emphasizes the central portion and suppresses the peripheral portion in the region to which the filter is applied. That is, the convex portion extraction filter is a filter that emphasizes the convex portion whose center protrudes in the depth frame, and is, for example, a filter as shown in FIG. 23. Each quadrangle in FIG. 23 represents a pixel, and the numerical value in each quadrangle is a coefficient multiplied by the pixel value of the pixel corresponding to the quadrangle. Then, for each pixel in the filter, the pixel value of the pixel is multiplied by the corresponding coefficient, and the sum of these multiplication values is the pixel value of the central pixel after the filter is applied.

FIG. 24 is a depth frame at the top timing. When the convex portion extraction filter of FIG. 23 is applied to the entire depth frame, the image shown in FIG. 25 is obtained. In the depth frame of FIG. 25, since the gap between the contour of the golfer 7 and the background is large, not only the convex left shoulder 71 but also the contour of the golfer 7 is emphasized. As a result, the left shoulder 71 is buried, and there is a risk that the position of the left shoulder 71 cannot be extracted correctly. Therefore, in the present embodiment, in order to deal with such a problem, the binarized image derived in step S64 is used. In addition, in the depth frame shown in the following figures including FIGS. 24 and 25, the image of the golfer is reflected on the mirror surface.

FIG. 26 is a binarized image that has undergone step S64, and the region “1” mainly shows the silhouette of the golfer 7, and the background region is the region “0”. In the present embodiment, the position deriving unit 14b shrinks the binarized image by a predetermined amount in order to remove the contour of the golfer 7. The binarized image after shrinkage is shown in FIG. 27. Then, the position derivation unit 14b masks the depth frame (see FIG. 25) after applying the convex portion extraction filter by using the binarized image after the contraction as a mask image. The frame after this masking is shown in FIG. As shown in FIG. 28, by this masking, the contour of the golfer 7 is removed from the depth frame after applying the convex extraction filter, and the left shoulder 71 can be easily extracted. In FIG. 28, the position of the left shoulder 71 is highlighted for reference. The position derivation unit 14b specifies the two-dimensional coordinates of the pixel having the maximum pixel value in the frame after masking, and sets the two-dimensional coordinates of the left shoulder 71. Further, the position derivation unit 14b derives the three-dimensional coordinates of the left shoulder 71 by extracting the depth data of the left shoulder 71 in the two-dimensional coordinates from the initial depth frame.

In order to improve the accuracy and / or reduce the calculation load, the applicable range of the convex extraction filter may be a part of the depth frame instead of the entire depth frame. For example, a region of an appropriate size including the two-dimensional coordinates of the left shoulder 71 specified on the same or immediately preceding frame can be the applicable range of the convex extraction filter. In the section from the address to the takeback arm horizontal and the section from the downswing arm horizontal to the finish, a region of a predetermined size centered on the two-dimensional coordinates of the left shoulder 71 included in the skeleton data of the same frame is extracted as a convex portion. It is preferable to set it as the application range R3 of the filter (see FIG. 29). Further, at the timing of the takeback arm horizontal, the area of a predetermined size centered on the two-dimensional coordinates of the left shoulder 71 included in the skeleton data of the previous frame is covered by the convex portion extraction filter application range R3 (see FIG. 29). ) Is preferable. Further, in the section from immediately after the takeback arm horizontal to the downswing arm horizontal, the two-dimensional coordinates of the left shoulder 71 specified by using the convex extraction filter in the previous frame are included in the position to the right of a predetermined size. The region is preferably set to the application range R4 (see FIG. 30) of the convex extraction filter. The reason why the rectangular area extending to the left with respect to the left shoulder 71 is set as the application range R4 is to avoid a part where the response value of the filter becomes very large on the right side (hand side). ..

<1-3-2-2. Left shoulder extraction with left shoulder silhouette filter>
Hereinafter, a method of extracting the left shoulder 71 by the left shoulder silhouette filter will be described. The left shoulder silhouette filter is a filter that emphasizes the lower right portion and suppresses the lower left, upper left, and upper right portions in the area to which the filter is applied. That is, the left shoulder silhouette filter is a filter that emphasizes the convex portion in which the lower right part protrudes in the depth frame, and is, for example, a filter as shown in FIG. 31. Similar to the convex extraction filter, each quadrangle in FIG. 31 represents a pixel, and the numerical value in each quadrangle is a coefficient multiplied by the pixel value of the pixel corresponding to the quadrangle. Then, for each pixel in the filter, the pixel value of the pixel is multiplied by the corresponding coefficient, and the sum of these multiplication values is the pixel value of the central pixel after the filter is applied.

The position derivation unit 14b applies the left shoulder silhouette filter to the binarized image derived in step S64. FIG. 32 is a binarized image derived in step S64, and FIG. 33 shows an image in which the left shoulder silhouette filter of FIG. 31 is applied to the binarized image. Then, the position derivation unit 14b specifies the two-dimensional coordinates of the pixel having the maximum pixel value in the frame after the application of the left shoulder silhouette filter, and sets the two-dimensional coordinates of the left shoulder 71. Further, the position derivation unit 14b derives the three-dimensional coordinates of the left shoulder 71 by extracting the depth data of the left shoulder 71 in the two-dimensional coordinates from the initial depth frame. In FIG. 33, the position of the left shoulder 71 is highlighted for reference.

In addition, in order to improve the accuracy and / or reduce the calculation load, the applicable range of the left shoulder silhouette filter may be a part of the depth frame instead of the entire depth frame. For example, a region of an appropriate size including the two-dimensional coordinates of the left shoulder 71 specified on the same or immediately preceding frame can be the applicable range of the left shoulder silhouette filter. In the section from the address to the takeback arm horizontal, the area of a predetermined size having the two-dimensional coordinates of the left shoulder 71 included in the skeleton data of the same frame as the lower right end point is covered by the left shoulder silhouette filter application range R5 (FIG. 34). It is preferable to set as (see). Further, at the timing of the downswing arm horizontal, the area of a predetermined size having the upper left end point of the above-mentioned R4 area set in the previous frame as the lower right end point is covered by the left shoulder silhouette filter application range R6 ( It is preferable to set as (see FIG. 35). Further, in the section from immediately after the downswing arm horizontal to the finish, the area of a predetermined size centered on the two-dimensional coordinates of the left shoulder 71 specified by using the left shoulder silhouette filter in the previous frame is filtered by the left shoulder silhouette filter. It is preferable that the applicable range is R7 (see FIG. 36).

<1-3-2-3. How to determine the timing of arm horizontal>
Hereinafter, a method for determining the timing of the takeback arm horizontal and the downswing arm horizontal according to the present embodiment will be described.

First, as shown in FIG. 37, the position derivation unit 14b acquires the two-dimensional coordinates of the right shoulder from the skeleton data at the time of addressing. Then, with reference to the two-dimensional coordinates of the right shoulder, the two-dimensional coordinates of the point A that advances a predetermined distance (for example, 30 pixels) in the positive direction of the x-axis and advances a predetermined distance (for example, 15 pixels) in the positive direction of the y-axis. To identify. The x-axis positive direction is a direction in which the golfer 7 travels from the left-hand side to the right-hand side along the horizontal direction. Further, the y-axis positive direction is a direction from top to bottom along the vertical direction. Subsequently, the position deriving unit 14b occupies a range of a predetermined distance (for example, 115 pixels) in the positive direction of the x-axis from the point A, and occupies a range of a predetermined distance (for example, 30 pixels) in the positive direction of the y-axis. The region R8 is formed.

After that, the position derivation unit 14b counts the number of pixels in which the depth data exists in the above-mentioned rectangular region R8 on the depth frame for each depth frame included in the section from the address to the finish. Then, all the depth frames in which the number of pixels is equal to or greater than a predetermined number (for example, 400 pixels) (hereinafter, arm horizontal condition) are specified. At this time, normally, in the section from the address to the finish, depth frames satisfying the arm horizontal condition appear in two groups along the time axis. Therefore, after identifying these two groups, the position deriving unit 14b specifies the takeback arm horizontal frame from the first half group and the downswing arm horizontal frame from the second half group. When a plurality of depth frames satisfying the arm horizontal condition appear in one group, for example, the first frame can be an arm horizontal frame. In the section from the address to the finish, if the depth frame satisfying the arm horizontal condition does not appear in two groups, appropriate error processing is performed.

<1-3-3. Shaft direction derivation process>
Hereinafter, the shaft direction derivation process will be described. In the shaft direction derivation process, a three-dimensional vector representing the direction of the shaft 52 during the swing operation is derived based on the depth image captured by the distance image sensor 2. As in the case of the grip behavior derivation process, the image data to be analyzed is a moving image. Therefore, the IR image and the depth image to be analyzed are also referred to as an IR frame and a depth frame.

FIG. 38 is a flowchart showing the entire flow of the shaft direction derivation process. First, in step S70, the acquisition unit 14a reads the IR frame and the depth frame acquired from the distance image sensor 2 and stored in the storage unit 13 and expands them into the memory. Then, the orientation derivation unit 14c determines a frame to be analyzed among these image data. In the present embodiment, the period from the address during the swing operation to a little after the impact is the target of analysis. The details are the same as in step S1.

Subsequent steps S71 to S77 are repeatedly executed for each frame included in the period to be analyzed. Therefore, in the following description, the IR frame and the depth frame at the time when the direction of the shaft 52 is to be derived (hereinafter referred to as the target time) are referred to as the target IR frame and the target depth, respectively, as in the case of the grip behavior derivation process. Called a frame. In addition, each frame or these may be collectively referred to as a target frame.

In the following step S71, the orientation deriving unit 14c extracts an image of the marker of the shaft 52 from the target IR frame. The specific process is the same as in step S41. If the image of the shaft 52 is extracted here, the process proceeds to step S72, and if the image cannot be extracted, the process proceeds to the next frame process.

In step S72, the orientation derivation unit 14c identifies the two-dimensional coordinates of all the points included in the area of the marker image of the shaft 52 extracted in step S71. Further, the orientation derivation unit 14c specifies the depth values of all the points whose two-dimensional coordinates are specified here from the target depth frame. However, the point that the depth value specified here is judged to be inappropriate can be removed from the following analysis. Whether or not the depth value is appropriate is determined, for example, depending on whether or not the depth value is within a predetermined range (for example, 1000 mm to 3500 mm). From the above, the three-dimensional coordinates of the points included in the shaft 52 are derived. Then, when the three-dimensional coordinates of the shaft 52 are derived in a predetermined number or more (one or more points in the present embodiment), the process proceeds to step S73, and when the three-dimensional coordinates are not derived, the process proceeds to the next frame process.

In step S73, the orientation deriving unit 14c calculates the three-dimensional coordinates of the point representing the shaft 52 from the three-dimensional coordinates of all the points included in the shaft 52 derived in step S72, and in the present embodiment, the three-dimensional coordinates of the center of gravity. .. The position of the center of gravity in each axial direction can be calculated as the average value of the coordinate values in the axial direction of all the points derived in step S72.

In the following step S74, the orientation deriving unit 14c determines whether or not the three-dimensional coordinates of the grip end 51a at the target time can be derived in the above steps S3 to S7, and if it can be extracted, the orientation deriving unit 14c proceeds to step S75. If the process has not been performed, the process proceeds to step S76. If the above steps S3 to S7 have not been executed for the target frame at the time of executing this step, it may be executed in this step to try to derive the three-dimensional coordinates of the grip end 51a.

In step S75, the orientation deriving unit 14c derives a three-dimensional vector representing the orientation of the shaft 52 based on the three-dimensional coordinates of the grip end 51a and the three-dimensional coordinates of the center of gravity of the shaft 52. The direction of the shaft 52 can be derived from the grip end 51a toward the center of gravity of the shaft 52, or vice versa.

On the other hand, in step S76, the orientation derivation unit 14c determines whether or not the three-dimensional coordinates of the shaft 52 are derived by a predetermined number or more (for example, two points or more) in step S72, and if they are derived, the step. Proceed to S77, and if it has not been derived, the process proceeds to the next frame. In step S77, principal component analysis is performed on the three-dimensional coordinates of all the points included in the shaft 52 derived in step S72. Then, the eigenvector (first principal component) corresponding to the maximum eigenvalue is a three-dimensional vector representing the direction of 52 of the shaft.

When the above steps S71 to S77 for all the times to be analyzed are completed, the process proceeds to step S78. In step S78, a three-dimensional vector representing the orientation of the shaft 52 at a time when the orientation of the shaft 52 has not yet been derived is calculated from the orientation of the shaft 52 at the time before and after. For example, it can be the average of three-dimensional vectors representing the orientation of the shaft 52 at the time before and after.

From the above, the orientation of the shaft 52 with respect to all the times to be analyzed is three-dimensionally derived. The orientation of such a shaft 52 can be displayed as a graph for each axis as shown in FIG. 39, or can be displayed as a three-dimensional graph as shown in FIG. 40. In the example of FIG. 40, the position of the grip end 51a and the position of the center of gravity of the shaft 52 are indicated by symbols ◯ and × so as to be superimposed on the straight line representing the shaft 52.

<1-4. Modification example>
Although some embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention. For example, the following changes can be made.

<1-4-1>
Although the IR image was used in the grip behavior derivation process according to the first embodiment, a color image can be used instead of the IR image. In this case, the distance image sensor may be equipped with a visible light receiving unit (for example, an RGB camera) that receives visible light.

<1-4-2>
The part to be analyzed using the depth image and / or the skeleton data is not limited to the grip 51 and the left shoulder 71. For example, the behavior (three-dimensional coordinates) of the shaft, head, right shoulder, left arm, right arm, etc. can be specified. Further, at this time, a marker can be appropriately attached to the site to be analyzed.

<1-4-3>
The method for determining the arm horizontal timing is not limited to the above. For example, a linear infrared reflective tape may be attached to the arm, an image of the reflective tape may be extracted from the IR image, and the timing at which the image becomes horizontal may be detected.

<1-4-4>
Instead of the process using the principal component analysis in steps S56, S57, and S77, the process using the least squares method can be executed. Specifically, a regression line is derived from the distribution of the three-dimensional coordinates of the shaft 52 by the method of least squares, and this is determined to be the axis on which the shaft 52 extends. Then, the position of the grip end 51a can be searched for on the extension line of this axis.

<1-4-5>
In step S22, the threshold value for binarization for extracting the image of the grip end 51a was automatically calculated by the discriminant analysis method. However, the threshold value can be, for example, a fixed value obtained by multiplying the maximum luminance value (limit value) by a predetermined coefficient of 0 to 1, for example, a value 0.1 times the maximum luminance value. In this case, step S23 can be omitted.

When the distance between the mark of the grip end 51a and the mark of the shaft 52 becomes narrow on the target IR frame, the mark of the shaft 52 is reflected in the search range R1 of the grip end 51a. In this case, if the brightness value of the mark of the shaft 52 is higher than that of the mark of the grip end 51a, the image of the grip end 51a may disappear when binarization is performed using the discriminant analysis method. Therefore, even if the brightness value of the mark at the grip end 51a is low, it is preferable to set the threshold value for binarization to a value of a predetermined ratio of the maximum brightness value so that the mark can be extracted. The threshold value, which is a fixed value according to this modification, is preferably used only in a predetermined phase in which the mark of the shaft 52 is easily reflected in the search range R1. That is, it is preferable that the threshold value according to the present modification and the threshold value automatically set by the discriminant analysis method or the like are appropriately switched according to the phase during the swing operation. Examples of such a phase include a section from the shaft 8 o'clock (during takeback, the time when the shaft 52 points to the clock 8 o'clock in front view) to the takeback arm horizontal, or a part thereof. .. It should be noted that this section is a section in which the grip end 51a is easily moved away from the camera arranged in front of the golfer or is easily placed behind the golfer's hand.

<1-4-6>
In step S21 according to the first embodiment, the search range R1 of the grip end 51a is set based on the moving direction (vector direction) of the grip end 51a in the predetermined number of frames immediately before. However, in a phase in which the grip end 51a is hidden by the golfer's body and is difficult to be reflected in the target IR frame, it may be difficult to accurately identify the position of the grip end 51a. That is, in the above phase, the position of the grip end 51a tends to be estimated from the image of the shaft 52, and especially when such a situation continues, the search range R1 ignores the actual movement of the grip end 51a. May go on. In such a case, as shown in FIG. 41A, if the image of the shaft 52 enters the search range R1, the accuracy of detecting the position of the grip end 51a may decrease. Therefore, in such a predetermined phase, it is preferable to set the search range R1 so as not to exceed the predetermined limit line.

An example of the above phases is near the top. In the vicinity of the top, there is a high possibility that the grip end 51a is not reflected in the target IR frame, and the movement of the grip end 51a is stopped or almost stopped. Then, in the vicinity of the top, it is preferable to set a limit line that limits the setting position of the search range R1 at the position of the golfer's head (more specifically, for example, the apex of the head) (see FIG. 41B). This is because, in the vicinity of the top, the grip end 51a usually does not advance to the right side of the golfer's head in front view. In this case, the estimated value of the position of the grip end 51a does not advance to the right of the limit line in front view.

The position of the golfer's head may be extracted by image processing the IR frame and / or the depth frame, or may be determined based on the skeleton data. The position of the golfer's head can be extracted from the person area image, and for example, the highest point of the person area image can be set as the head position. The person area image is an image that mainly shows only the area in which a person appears, and can be derived from a depth frame or an IR frame. Since various methods for deriving such a person area image are known, detailed description thereof will be omitted here. Further, Kinect (registered trademark), which is a distance image sensor, is provided with a library for reading a person area image from a depth image, and the acquisition unit 14a can acquire a person area image using the library. .. The CPU 23 of the distance image sensor 2 may derive a person region image from the depth frame. In this case, the acquisition unit 14a may simply read the person area image sent from the distance image sensor 2.

Further, the reference image for extracting the position of the golfer's head may be an image at the timing when the position of the grip end 51a is to be extracted, or may be an image at a predetermined timing. Preferably, it can be an image when the takeback arm is horizontal. FIG. 41C shows a person area image when the takeback arm is horizontal. In this way, the accuracy of extracting the position of the head can be improved by using the image at the timing when it is unlikely that other parts such as the hand will come near the head. The method for determining the horizontal takeback arm timing is as shown in 3-2-3.

<1-4-7>
In the first embodiment, as shown in FIG. 3, an attempt is first made to directly derive the three-dimensional coordinates of the grip end 51a from the target IR frame and the target depth frame (steps S3 and S5), and if it fails, the target IR frame is used. The shaft 52 is extracted from the shaft 52, and the three-dimensional coordinates of the grip end 51a are derived based on the position of the extracted shaft 52 (exception processing S6). In other words, the process of deriving the three-dimensional coordinates of the grip end 51a included in the process of the first embodiment can be expressed as a process including steps S101 to S104 as shown in FIG. 42.

However, the process of deriving the three-dimensional coordinates of the grip end 51a can also be configured as shown in FIG. 43. That is, first, it is determined whether or not the grip end 51a could be extracted from the IR frame at the previous time (step S201). Then, when it is determined that the extraction can be performed in step S201, the three-dimensional coordinates of the grip end 51a are tried to be derived from the target IR frame and the target depth frame (step S101). Subsequent steps are the same as in FIG. 42. On the other hand, if it is determined that the extraction could not be performed in S201, the target IR frame is not executed, that is, without attempting to derive the three-dimensional coordinates of the grip end 51a from the target IR frame and the target depth frame. The shaft 52 is extracted from the above (step S103). Then, the three-dimensional coordinates of the grip end 51a are derived based on the extracted position of the shaft 52 (step S104).

The process according to FIG. 43 is superior to the process according to FIG. 42 in the following points. That is, when the grip end 51a has not been extracted from the IR frame at the previous time and the position of the grip end 51a is estimated from the position of the shaft 52, the position may include an error of a certain value or more. Then, if the search range R1 is set in step S101 for the next frame using the information on the position of the grip end 51a including such an error, the above error may accumulate. Therefore, when the grip end 51a cannot be extracted from the IR frame in the previous frame, the position of the shaft 52 is specified first instead of searching for the grip end 51a within the search range R1. Then, if the position of the grip end 51a is specified based on the position of the shaft 52, it is possible to prevent the accumulation of errors.

The processing of steps S103 and S104 according to this modification can be processed in the same manner as the exception processing S6 described above, or can be changed. For example, after deriving the two-dimensional coordinates of the grip end 51a based on the two-dimensional coordinates of the shaft 52 by the same method as the exception processing S6 described above, the target IR frame is searched based on the two-dimensional coordinates of the grip end 51a. The range R1 may be set. Then, the grip end 51a can be further searched within the search range R1 in the same manner as in step S3, and then the three-dimensional coordinates of the grip end 51a can be derived in the same manner as in steps S3 to S7.

<1-4-8>
In step S65 according to the first embodiment, left shoulder extraction was performed using a convex extraction filter. However, instead of this, the position of the left shoulder can be specified by template matching using a convex template. The method by template matching using this convex template can improve the accuracy of left shoulder extraction especially in the section from the takeback arm horizontal to the downswing arm horizontal.

FIG. 44 is an example of a convex template. As shown in the figure, the convex template is an image template having a pixel value that is the smallest in the center and gradually increases from the center to the periphery (the largest in the center and from the center to the periphery). The image may have a pixel value that gradually decreases). That is, if template matching is performed using this template, it is possible to easily extract an image that captures a subject whose center protrudes and which protrudes to the opposite side toward the periphery. Various known methods for processing template matching are known, but a brief description is as follows. That is, within the applicable range of the above-mentioned filter in the depth frame, while shifting the convex template up and down little by little (for example, one pixel at a time), the similarity between the convex template and the partial image of the depth frame overlapping the convex template is adjusted. I will calculate. The similarity is, for example, SAD (the sum of the absolute values of the differences between the pixel values). Then, the region having the highest degree of similarity (minimum in the case of SAD) in the depth frame is specified as the position of the left shoulder.

By the way, as described above, Kinect (registered trademark) v1 and v2 are general-purpose distance image sensors. The template matching according to this modification is particularly useful when extracting the left shoulder from a relatively high-quality image such as a depth frame output from Kinect (registered trademark) v2. This is because the template matching according to this modification has a property of being less likely to respond to unevenness other than the left shoulder, for example, fine unevenness such as unevenness of clothes, as compared with the case of using the convex portion extraction filter according to the first embodiment. Is. Therefore, the error caused by responding to fine irregularities other than the left shoulder is more reliably prevented. Note that FIG. 45 is a diagram showing the position of the left shoulder extracted by template matching according to this modification with respect to the depth frame output from Kinect (registered trademark) v2. From the figure, it can be seen that the position of the left shoulder is correctly detected.

<1-4-9>
In the first embodiment, left shoulder extraction from the skeleton data, left shoulder extraction by the convex extraction filter, and left shoulder extraction by the left shoulder silhouette filter were performed for the entire period of the analysis. However, in order to shorten the calculation time, it is preferable to perform these left shoulder extractions only for the section in which the position of each left shoulder is finally used. FIG. 46 is an example of such processing.

In FIG. 46, first, the position derivation unit 14b specifies the address, the shaft at 8 o'clock, the top, and the impact time. Then, the position of the left shoulder 71 is derived in the order from the address to the impact. The position of the left shoulder 71 is derived from the skeleton data until the takeback arm is horizontal from the address. Specifically, the position of the left shoulder 71 is derived from the skeleton data until the shaft reaches 8 o'clock from the address. The timing at 8 o'clock on the shaft can be determined from the orientation of the shaft 52 derived by the shaft direction derivation process. After 8 o'clock on the shaft, the number of pixels in which depth data exists in the rectangular area R8 of the depth frame at the current time is counted, and if it is less than a predetermined number (for example, 400 pixels), the left shoulder 71 is based on the skeleton data. Determine the position of. On the other hand, if it is more than a predetermined number, the current time is determined to be the time horizontal to the takeback arm.

From the take-back arm horizontal to the downswing arm horizontal, the left shoulder 71 is derived based on the convex extraction filter. Specifically, the left shoulder 71 is derived based on the convex extraction filter from the takeback arm horizontal to the top. After the top, the number of pixels in which depth data exists in the rectangular area R8 of the depth frame at the current time is counted, and if it is less than a predetermined number (for example, 400 pixels), the left shoulder 71 is based on the convex extraction filter. Determine the position of. On the other hand, if it is more than a predetermined number, the current time is determined to be the time horizontal to the downswing arm. After that, from the downswing arm horizontal to the impact, the left shoulder 71 is derived based on the left shoulder silhouette filter. Further, instead of the processing using the convex portion extraction filter, the template matching of the modified example 1-4-8 may be executed.

<2. Second Embodiment>
<2-1. Overview of swing analysis system>
Hereinafter, the swing analysis system 200 including the swing analysis device 201 according to the second embodiment will be described. FIG. 47 is a plan view of the swing analysis system 200. The swing analysis system 200 is also a system for analyzing a golf swing based on a moving image of the swing motion of the golf club 5 by the golfer 7, and also in the second embodiment, the moving image is captured by the distance image sensor 2. Will be done. However, unlike the first embodiment, in the second embodiment, a plurality of distance image sensors 2 are used, and the swing motion is photographed from a plurality of directions. Here, as shown in FIG. 47, an example in which two distance image sensors 2F and 2S are used will be described, but of course, three or more distance image sensors can also be used. As a result, the swing motion can be accurately captured from multiple directions.

Further, in the second embodiment, in addition to the depth image and the two-dimensional image output from the distance image sensor 2, and the skeleton data based on the depth image, the silhouette data representing the silhouette of the human body, which will be described later, is analyzed. Then, by appropriately combining such various types of information while taking advantage of their respective advantages, the accuracy of swing analysis is improved.

In the present embodiment, the grip behavior derivation process, the left shoulder behavior derivation process, and the shaft direction derivation process are executed as in the first embodiment, whereby the three-dimensional coordinates of the grip end 51a and the left shoulder 71 and the orientation of the shaft 52 are executed. A three-dimensional vector representing is derived. Then, in the present embodiment, in addition to the same information as in the first embodiment, the quality of the swing motion is determined based on the position information of various parts of the body of the golfer 7 other than the left shoulder 71.

Hereinafter, for the sake of simplicity, the features peculiar to the second embodiment will be mainly described while omitting the description of the common points with the first embodiment. The same reference numerals are given to the elements common to those of the first embodiment.

<2-2. Details of each part>
Hereinafter, details of each part of the swing analysis system 200 will be described with reference to FIG. 48.

<2-2-1. Distance image sensor>
As shown in FIG. 48, the distance image sensors 2F and 2S have the same configuration as the distance image sensor 2 described above. The distance image sensor 2F is installed in front of the golfer 7 in order to photograph the golfer 7 from the front side, similarly to the distance image sensor 2 according to the first embodiment. On the other hand, the distance image sensor 2S is installed on the right side of the golfer 7 in order to photograph the golfer 7 from the right side.

Further, also in the second embodiment, since the IR image is taken by the distance image sensors 2F and 2S, an infrared reflective sheet is attached to the grip 51 and the shaft 52 as markers.

<2-2-2. Swing analyzer ＞
Next, the swing analysis device 201 will be described. As shown in FIG. 48, the swing analysis device 201 has the same hardware configuration as the swing analysis device 1 according to the first embodiment. However, since processing different from that of the first embodiment is executed, the swing analysis program 203 is installed in the storage unit 13 instead of the swing analysis program 3. As a result, the control unit 14 operates as the acquisition unit 14a, the first position derivation unit 14b, and the orientation derivation unit 14c, and also operates as the second position derivation unit 14d and the determination unit 14e. Details of the operation of each part 14d and 14e will be described later. In the present embodiment, the position out-licensing unit 14b is referred to as the first position out-licensing unit 14b in order to distinguish it from the second position out-licensing unit 14d.

The swing analysis device 201 may be composed of one computer or a plurality of computers. In particular, a computer may be connected to each of the distance image sensors 2F and 2S.

<2-3. Swing analysis method>
Hereinafter, a golf swing analysis method executed by the swing analysis system 200 will be described. Also in this analysis method, first, the golfer 7 is made to test-hit the golf club 5, and the state is photographed by the distance image sensors 2F and 2S. The image data of the IR frame and the depth frame captured by the distance image sensors 2F and 2S are sent from the distance image sensors 2F and 2S to the swing analysis device 201. After that, the grip behavior derivation process, the left shoulder behavior derivation process, and the shaft direction derivation process are executed. Further, two series of image data from the distance image sensors 2F and 2S in the storage unit 13 (actually, two series of image data of the depth frame and the IR frame are derived from each sensor, so it is called 4 series. Is also possible) is synchronized by the acquisition unit 14a. Synchronization can be realized by various methods, for example, it can be realized by detecting the timing at which the same feature appears in the image data of each system by image processing, and shooting with the distance image sensors 2F and 2S can be performed. It is also possible to use a separate device that triggers at the same time. After that, the position derivation process is executed. In the position derivation process, the position information of a predetermined part of the body of the golfer 7 other than the left shoulder 71 at a predetermined timing (hereinafter, checkpoint) is derived. After that, the pass / fail determination process is executed. In the quality determination process, the quality of the swing motion is determined based on the position information of the grip end 51a, the position information of a predetermined part of the body of the golfer 7 including the left shoulder 71, and the three-dimensional vector representing the direction of the shaft 52. .. Further, in the position derivation process, the calibration of step S202, which will be described later, is performed. In the following, after explaining the position derivation process, the process of setting the parameters required for this calibration (calibration parameter setting process) will be described, and finally, the pass / fail determination process will be described.

<2-3-1. Position derivation process>
As described above, the position derivation process is a process for deriving the position information of a predetermined part of the body of the golfer 7 other than the left shoulder 71 at the checkpoint. FIG. 49 is a flowchart showing the flow of the position derivation process.

First, in step S201, the acquisition unit 14a reads the image data stored in the storage unit 13 and expands it into the memory. Then, the second position derivation unit 14d determines the frame of the checkpoint to be analyzed among these image data. The checkpoints according to the present embodiment are eight timings of address, takeback shaft 8 o'clock, takeback shaft 9 o'clock, takeback arm horizontal, top, downswing arm horizontal, downswing shaft 9 o'clock, and impact (8 o'clock). (See FIGS. 50 and 51). These checkpoints appear in this order during the swing motion. 50 and 51 are a front view and a right side view of the golfer 7 at these checkpoints, respectively. The takeback shaft 8 o'clock and the takeback shaft 9 o'clock are timings at which the shaft 52 generally points in the directions of 8 o'clock and 9 o'clock of the clock during takeback when the golfer 7 is viewed from the front, respectively. The downswing shaft at 9 o'clock is the timing when the golfer 7 is viewed from the front and the shaft 52 generally points in the direction of 9 o'clock on the clock during the downswing.

The frame of these checkpoints can be determined in various ways. The address, takeback arm horizontal, top, downswing arm horizontal, and impact frame can be determined, for example, in the manner already described. The frame at 8 o'clock on the takeback shaft is, for example, the section from the address to the horizontal takeback arm, and the grip end 51a is on the right side of the right foot on the IR frame (hereinafter referred to as the front IR frame) taken from the front of the golfer 7. The frame can be specified by image processing, and can be derived as a frame obtained by returning 1 to 3 frames from this. In the following, similarly to the front IR frame, the depth frame taken from the front of the golfer 7 is referred to as a front depth frame, and the IR frame taken from the right side of the golfer 7 is referred to as a right side IR frame. A depth frame taken from the right side of the golfer 7 is called a right side depth frame. Further, a frame photographed from the front of the golfer 7 is referred to as a front frame, and a frame photographed from the right side of the golfer 7 is referred to as a right side frame.

The frame at 9 o'clock on the takeback shaft is derived as, for example, a frame in which the angle of the shaft 52 is 90 ° (horizontal) when viewed from the front of the golfer 7 in the section where the takeback shaft is horizontal from 8 o'clock. Can be done. The frame at 9 o'clock of the downswing shaft can be derived as, for example, a frame in which the angle of the shaft 52 is 90 ° (horizontal) when viewed from the front of the golfer 7 in the section from the downswing arm horizontal to the impact. At this time, the angle of the shaft 52 can be specified by using a three-dimensional vector representing the direction of the shaft 52 derived by the shaft direction derivation process or a two-dimensional vector (derived from the IR frame).

In the following steps S202 to S205, the position information of various parts of the body of the golfer 7 at the checkpoint is derived. Here, the position information is mainly derived as the position information in the first plane and / or the second plane. The first plane is a plane orthogonal to the optical axis of the distance image sensor 2F, that is, a plane orthogonal to the direction from the antinode to the back of the golfer 7. The second plane is a plane orthogonal to the optical axis of the distance image sensor 2S, that is, a plane orthogonal to the flying ball line direction. Further, here, the position information is appropriately required from the already calculated three-dimensional coordinates of the grip end 51a, the three-dimensional coordinates of the left shoulder 71, the three-dimensional vector representing the direction of the shaft 52, the skeleton data, and the silhouette data. It is calculated using the above. The skeleton data has already been described, but the silhouette data will be described later.

First, in step S202, the acquisition unit 14a reads out the front depth frame at each checkpoint during the swing operation stored in the storage unit 13 and expands it into the memory. Then, the acquisition unit 14a acquires the skeleton data at each checkpoint from these front depth frames. The mode of acquiring the skeleton data is the same as that of the first embodiment. Then, the second position derivation unit 14d converts the skeleton data at each checkpoint viewed from the front side based on the front depth frame into the skeleton data viewed from the right side. This is to address the following issues: That is, the skeleton data based on the depth frame taken from the front (more accurately, the position information of the points of interest such as joints included in the skeleton data) is highly reliable, but the skeleton data based on the depth frame taken from the side is It may be unreliable. Therefore, in the following, instead of the skeleton data based on the right side depth frame, the skeleton data obtained by converting the skeleton data based on the front depth frame (hereinafter referred to as the front skeleton data) is used. Hereinafter, unless otherwise specified, the right side skeleton data means the skeleton data after conversion from the front skeleton data. Further, in step S202, the second position derivation unit 14d converts the three-dimensional coordinates of the points of interest such as joints included in the right side surface skeleton data into the two-dimensional coordinates in the second plane.

The conversion of the skeleton data is performed based on the calibration parameters stored in the storage unit 13 in advance. The conversion of skeleton data means converting the three-dimensional coordinates in one three-dimensional space into the three-dimensional coordinates in the three-dimensional space set by another definition. Therefore, if the relative relationship of the three-dimensional space before and after the conversion is known, such conversion is possible. The parameter showing this relative relationship is the calibration parameter used here, and is set in the calibration parameter setting process described later.

In the following step S203, the second position derivation unit 14d is derived by the grip behavior derivation process, the left shoulder derivation process, and the shaft direction derivation process, and the three-dimensional coordinates of the grip 51 as seen from the front side, the three-dimensional coordinates of the left shoulder 71, and the shaft. The three-dimensional vectors representing the directions of 52 are converted into data viewed from the right side. This is because, in the present embodiment, as will be described later, the swing analysis is executed based on the image data viewed from the right side. In the following, unless otherwise specified, in the parts where the analysis of the image data viewed from the right side is described, the position information of the grip end 51a after conversion by step S203, the position information of the left shoulder 71, and the orientation of the shaft 52 are shown. , Is the subject of analysis.

By the way, in the present embodiment, the position information of the grip end 51a derived by the grip behavior derivation process and the shaft direction derivation process and the orientation of the shaft 52 are calculated based on the image data from the front, so that in step S203. The error may be large in the position information after conversion and the orientation of the shaft 52. In such a case, the position information of the grip end 51a and the orientation of the shaft 52 can be redistributed by image processing based on the image from the right side surface (including the silhouette data, the depth frame and the IR frame). .. Further, the error may also be large for the data obtained by converting the position information of the left shoulder 71 derived by the left shoulder behavior derivation process in step S203. In such a case, the position information of the left shoulder 71 can be redistributed by image processing based on the image from the right side surface (including the silhouette data, the depth frame and the IR frame).

In the following steps S204 and S205, the position information of a predetermined portion at each checkpoint determined in step S201 is derived. The position information derived here is used to determine the quality of the swing operation in the quality determination process.

As described above, steps S204 and S205 are steps for deriving the position information of a predetermined portion at each checkpoint. Of these, in step S204, the position information of the predetermined portion in the first plane is derived, and in step S205, the position information of the predetermined portion in the second plane is derived. In the present embodiment, the part from which the position information is derived differs for each checkpoint. The parts from which the position information for each checkpoint according to the present embodiment is derived will be described below. When step S205 is completed, the position derivation process is completed.

<2-3-1-1. Address>
The second position deriving unit 14d derives the position information of the golfer 7's right shoulder, shoulder center, umbilical cord, right ankle and left ankle in the first plane at the time of addressing from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the front skeleton data at the time of addressing are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

In addition, the second position derivation unit 14d derives the position information of the left foot width reference position, the right foot width reference position, the navel left position, the navel right position, and the left heel in the first plane at the time of addressing from the silhouette data. From the left foot width reference position and the right foot width reference position here, the width of the stance at the time of addressing can be specified. From the navel left position and the navel right position here, the width of the waist at the time of addressing can be specified.

Here, the silhouette data is data indicating the silhouette of the golfer 7 on each frame, and is acquired by the acquisition unit 14a. More specifically, the acquisition unit 14a reads the depth frame stored in the storage unit 13 and expands it into the memory. Then, the acquisition unit 14a acquires silhouette data from these depth frames. That is, since the golfer 7 exists on the relatively front side in the imaging range of the distance image sensors 2F and 2S, the silhouette of the human body is derived by extracting the region having the pixel value representing the shallow depth from the depth frame. Can be done. Therefore, more accurately, the silhouette referred to here includes a silhouette of a golf club 5 or the like. Kinect (registered trademark), which is a distance image sensor, is provided with a library for reading silhouette data from a depth image, and in the present embodiment, silhouette data is acquired using the library. Further, the silhouette data may be derived from the depth frame by the CPU 23 of the distance image sensors 2F and 2S. In this case, the acquisition unit 14a may simply read the silhouette data sent from the distance image sensor 2. Further, the silhouette data can be derived by image processing the IR frame instead of the depth frame.

The silhouette data directly extracted using the Kinect (registered trademark) library is usually not binarized. In that case, the acquisition unit 14a can binarize the silhouette data so that only the silhouette of the golfer 7 (more accurately, the golf club 5 is also included) is represented by 1 and the others are represented by 0. Further, the acquisition unit 14a can erase a small area classified as a silhouette by repeating expansion and contraction of the silhouette data, and can acquire a more accurate silhouette. In addition, the acquisition unit 14a can also erase a small area classified as a silhouette by performing a labeling process. Hereinafter, the silhouette data processed in this way is also referred to as silhouette data. The above silhouette data is derived as front silhouette data and right side silhouette data by the same method from each of the front depth frame and the right side depth frame.

Return to the description of the method of deriving the position information of the left foot width reference position, the right foot width reference position, the navel left position, the navel right position, and the left heel. These position information are derived by image processing the front silhouette data by the second position derivation unit 14d. FIG. 52 is an image in which the front skeleton data is superimposed on the front silhouette data at the time of addressing. In the present embodiment, as shown in FIG. 52, a rectangular frame L1 surrounding the silhouette portion is set on the image of the front silhouette data. The left and right boundaries of the frame L1 are set so as to have a predetermined distance from the silhouette. Further, the upper boundary of the frame L1 coincides with the uppermost point included in the silhouette, and the lower boundary coincides with the lower end of the silhouette excluding the silhouette of the linear shaft 52. Subsequently, on the image of the front silhouette data, a frame L2 extending from the left ankle to the right end of the frame L1 in the left-right direction and extending from the left ankle to the lower end of the frame L1 in the vertical direction is set. Further, on the image of the front silhouette data, a frame L3 extending from the right ankle to the left end of the frame L1 in the left-right direction and extending from the right ankle to the lower end of the frame L1 in the vertical direction is set. Then, the horizontal position of the point in the silhouette farthest from the left ankle in the frame L2 is specified as the left foot width reference position, and the horizontal position of the point in the silhouette farthest from the right ankle in the frame L3 is determined. Specified as the right foot width reference position.

Further, as shown in FIG. 52, on the image of the front silhouette data at the time of addressing, the left end point in the silhouette is specified as the right navel position on a horizontal straight line passing through the navel. In addition, the right end point in the silhouette on the same straight line is specified as the left navel position.

The position information of the left heel is derived from the image of the front silhouette data at the timing slightly advanced from the address (for example, advanced by one frame). FIG. 53 is an image in which the front skeleton data is superimposed on the front silhouette data at the relevant timing. In the present embodiment, as shown in FIG. 53, a rectangular frame L1 similar to the above is set on the image of the front silhouette data. Subsequently, on the image of the front silhouette data, a frame L4 extending from the left ankle to the center of the frame L1 in the left-right direction and extending from the left ankle to the lower end of the frame L1 in the vertical direction is set. Then, the point in the silhouette farthest from the left ankle in the frame L4 is specified as the position of the left heel.

Next, the second position derivation unit 14d derives the position information of the golfer 7's right shoulder, neck, waist, left knee, right knee, right ankle and right elbow in the second plane at the time of addressing from the skeleton data. do. At this time, the two-dimensional coordinates of these parts included in the right side surface skeleton data at the time of addressing are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

In addition, the second position derivation unit 14d derives the position information of the left foot tip, the right foot tip, the left kneecap, and the right kneecap in the second plane at the time of addressing from the silhouette data. Here, for the left foot and the right foot, the reflected wave of infrared data from the ground becomes noise, and the depth information may not be acquired in the depth frame. In this case, the silhouettes of the left foot and the right foot do not appear in the silhouette data based on the depth frame. In the present embodiment, in order to deal with such a case, the area of the silhouette specified in the above-mentioned silhouette data is expanded, in other words, the area of the silhouette is complemented with the area corresponding to the left foot and the right foot. It is said. Specifically, the second position derivation unit 14d derives the regions corresponding to the left foot and the right foot based on the information of the region where the depth information is missing on the right surface depth frame at the time of addressing. FIG. 54A is an image of the right side silhouette data based on the right side depth frame at the time of addressing. In the present embodiment, the second position derivation unit 14d sets a frame L5 of a predetermined size around the right ankle with the right ankle as a reference on the image of the silhouette data of the right side surface. Then, in this frame L5, the region where the depth information is 0 is extracted as the region corresponding to the left foot and the right foot. FIG. 54B is an image in which the silhouettes of the left foot and the right foot are complemented. The second position deriving unit 14d sets a frame L6 having a predetermined size around the right ankle with reference to the right ankle, and specifies the right end point of the silhouette of the left foot in the frame L6 as the position of the left foot tip. Similarly, the second position derivation unit 14d sets a frame L7 of a predetermined size around the right ankle with reference to the right ankle, and specifies the right end point of the silhouette of the right foot in the frame L7 as the position of the right foot tip.

The position information of the left kneecap and the right kneecap determines the positions of the left kneecap and the right kneecap by offsetting the positions of the left knee and the right knee in the depth direction on the front depth frame, and these are determined on the right side in the same manner as in step S202. It is converted to the position viewed from the surface side, and the positions of the left and right kneecaps after conversion are specified as position information of points offset in the contour direction of the silhouette.

<2-3-1-2. Takeback shaft 8 o'clock>
The second position derivation unit 14d derives the position information of the right shoulder and the navel of the golfer 7 in the first plane at 8 o'clock of the takeback shaft from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the front skeleton data at 8 o'clock of the takeback shaft are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

<2-3-1-3. Takeback shaft 9 o'clock>
In the present embodiment, the position information of the golfer 7 portion is not specified at the timing of the takeback shaft at 9 o'clock. However, as will be described later, the quality of the swing operation at the timing of the takeback shaft 9 o'clock is determined based on the position information of the grip end 51a at the takeback shaft 9 o'clock.

<2-3-1-4. Takeback arm horizontal ＞
The second position derivation unit 14d derives the position information of the neck and waist of the golfer 7 in the second plane when the takeback arm is horizontal from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the right side surface skeleton data when the takeback arm is horizontal are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

<2-3-1-1-5. Top ＞
The second position deriving unit 14d derives the position information of the golfer 7's head, navel, left knee, and right knee in the first plane at the top from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the front skeleton data at the time of top are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

Further, the second position deriving unit 14d derives the position information of the right navel position in the first plane at the top from the silhouette data. The method for deriving the right umbilicus position here is the same as that at the time of addressing, except that the image of the frontal silhouette data at the time of impact is used instead of the image of the frontal silhouette data at the time of addressing.

Next, the second position derivation unit 14d derives the position information of the left knee, the right knee, the neck, and the waist of the golfer 7 in the second plane at the time of the top from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the skeleton data on the right side surface at the time of top are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

Further, the second position deriving unit 14d derives the position information of the left shoulder 71, the left kneecap, the right kneecap, the left elbow and the right elbow of the golfer 7 in the second plane at the time of the top from the silhouette data. The two-dimensional coordinates of the left shoulder 71 that have already been derived are used for deriving the position information of the left shoulder 71. FIG. 55 is an image of the silhouette data of the right side surface at the time of top. Specifically, in the present embodiment, as shown in FIG. 55, the position of the left shoulder 71 that has already been extracted on the image of the silhouette data on the right side at the top is overlapped with the contour of the silhouette toward the left. And the position after the movement is set as the new position of the left shoulder.

Further, the position information of the right kneecap is derived by the same method as at the time of addressing. On the other hand, the position information of the left kneecap is derived by the following method. First, on the image of the silhouette data of the right side surface, a frame L10 (see FIG. 55) having a predetermined size is set around the position of the left knee that has already been extracted. Subsequently, the curvature of the contour of the silhouette included in the frame L10 is calculated, and the point where the curvature is maximum is specified as the position of the new left kneecap.

The position information of the right elbow is also derived based on the curvature of the silhouette. More specifically, on the image of the silhouette data of the right side surface, a frame L11 (see FIG. 55) having a predetermined size is set around the position of the grip end 51a as a reference. Subsequently, the curvature of the contour of the silhouette included in the frame L11 is calculated, and the point where the curvature is maximum is specified as the position of the new right elbow.

The position information of the left elbow is derived by the following method. First, the midpoint between the position of the re-extracted left shoulder 71 and the grip end 51a is set as the temporary left elbow. Next, a frame L12 of a predetermined size (see FIG. 55) is set around the temporary left elbow position as a reference, and a point shallower than the left shoulder 71 in the frame L12 based on the right side depth frame, in other words. And, the point protruding toward the front side with respect to the distance image sensor 2F is specified as the position of the left elbow.

<2-3-1-1-6. Downswing arm horizontal>
The second position deriving unit 14d derives the position information of the left knee of the golfer 7 in the first plane when the downswing arm is horizontal from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the front skeleton data when the downswing arm is horizontal are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

Next, the second position deriving unit 14d derives the position information of the neck and the waist of the golfer 7 in the second plane when the downswing arm is horizontal from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the right side surface skeleton data when the downswing arm is horizontal are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

<2-3-1-1-7. Downswing shaft 9 o'clock>
The second position deriving unit 14d derives the position information of the hand in the first plane at 9 o'clock of the downswing shaft from the silhouette data. In the present embodiment, the position of the hand is specified as a position moved to the head side by a predetermined length along the extending direction of the shaft 52 from the grip end 51a on the image of the front silhouette data at 9 o'clock of the downswing shaft. NS.

<2-3-1-8. Impact ＞
The second position derivation unit 14d derives the position information of the golfer 7's head, shoulder center, right shoulder, left elbow, and navel in the first plane at the time of impact from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the front skeleton data at the time of impact are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

Further, the second position deriving unit 14d derives the position information of the umbilicus left position and the umbilicus right position in the first plane at the time of impact from the silhouette data. From the navel left position and the navel right position here, the width of the waist at the time of impact can be specified. Further, the method of deriving the umbilicus left position and the umbilicus right position here is the same as that at the time of addressing, except that the image of the frontal silhouette data at the time of impact is used instead of the image of the frontal silhouette data at the time of addressing. Since the arm tends to overlap the left navel position on the image of the front silhouette data at the time of impact, when specifying the left navel position, a predetermined time is taken from the impact (for example, two frames before). ) It is preferable to perform the same processing on the image.

Next, the second position derivation unit 14d derives the position information of the golfer 7's neck, waist, right knee, and right ankle in the second plane at the time of impact from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the right side surface skeleton data at the time of impact are extracted, and the two-dimensional coordinates are used as they are as the position information of these parts.

<2-3-2. Calibration parameter setting process>
Next, the calibration parameter setting process will be described. First, one or a plurality of marks that reflect infrared rays via a stand or the like are arranged in the vicinity of the position where the golfer 7 stands. This mark is arranged at a position where it can be photographed from both the distance image sensors 2F and 2S. Then, this mark is photographed by the distance image sensors 2F and 2S, and the three-dimensional coordinates of this mark are acquired. Here, the three-dimensional coordinates of the mark taken by the distance image sensor 2F are represented as (FXi, FYi, FZi), and the three-dimensional coordinates of the mark at the same position taken by the distance image sensor 2S are (SXi, SYi, SZi). ). However, i = 1, 2, ..., N (n ≧ 2). Here, n sets of data sets (FXi, FYi, FZi, SXi, SYi, SZi) are prepared by moving the marks and the like.

At this time, the conversion coefficients a to l for converting the three-dimensional coordinates of the distance image sensor 2F into the three-dimensional coordinates of the distance image sensor 2S satisfy the following equations.

By substituting n sets of data sets (FXi, FYi, FZi, SXi, SYi, SZi) into the above equation, the conversion coefficients a to l are calculated. The conversion coefficients a to l are calibration parameters. The calibration parameters can be calculated by other methods as a matter of course.

<2-3-3. Pass / fail judgment process>
Next, the quality determination process for determining the quality of the swing operation will be described. The pass / fail determination process is executed by the determination unit 14e.

The pass / fail determination process is executed based on the position information of various parts (including the left shoulder 71) of the golfer 7, the position information of the grip end 51a, and the orientation information of the shaft 52. Specifically, the determination unit 14e calculates an index value for determining the quality of the swing operation based on this information. There are various index values, which are set for each checkpoint. In this embodiment, for example, the following index values are calculated. The following is a list of index values for frontal analysis and index values for right side analysis.

(Indicator value at address-front)
The position of the left heel with respect to the ball position (origin), the width of the stance with respect to the shoulder width, the balance of the upper body, the balance of the lower body. From the position information, the balance of the upper body is the left shoulder, the right shoulder, the left foot width reference position and the right foot width reference position. It can be derived from the position information of.

(Index value at address-right side)
The position of the right shoulder with respect to the balance point line, the position of the right knee with respect to the balance point line, and the spine angle. From the position information, the spine angle can be derived from the positions of the neck and waist. The balance point line is a line drawn in the vertical direction from the position of the right foot on the right side frame.

(Indicator value at 8 o'clock on takeback shaft-front)
The orientation of the grip end 51a (whether toward the umbilical cord), the angle difference from the ideal shaft line, the amount of movement of the right shoulder from the address The orientation of the grip end 51a is the position information of the grip end 51a, the orientation and address of the shaft 52a. From the position information of the umbilical cord at the time, the angle difference from the ideal shaft line is from the angle of the shaft 52 and the predetermined ideal angle, and the amount of movement of the right shoulder from the address is the position information of the right shoulder and the position of the right shoulder at the time of addressing. It can be derived from the information.

(Indicator value at 9 o'clock on takeback shaft-front)
Position of grip end 51a with respect to reference axis at address This index value can be derived from the position information of grip end 51a and the right shoulder and shoulder center at address.

(Takeback arm horizontal index value-front)
Wristcock angle, difference in height between both elbows, left shoulder movement from address Note that the wristcock angle is from the direction of the shaft 52, and the left shoulder movement from address is the takeback arm horizontal and left shoulder at address. It can be derived from the position information of.

The difference in height between the elbows can be derived, for example, as follows. FIG. 56 shows an image of the front silhouette data when the takeback arm is horizontal. Here, first, in addition to the above-mentioned frame L1, frames L8 and L9 as shown in FIG. 56 are set with reference to the positions of the grip end 51a and the left shoulder 71. Next, the two reference points shown in FIG. 56 are specified on the image of the front silhouette data when the takeback arm is horizontal, and the width between these reference points is derived as an index value. Then, when the index value is larger than a predetermined value, a line segment connecting the reference points is specified on the image of the front silhouette data when the takeback arm is horizontal. Subsequently, the depth information on the line segment is derived from the front depth frame when the takeback arm is horizontal. Then, the difference between the depth based on the depth information of the points on the line segment included in the frame L8 and the depth based on the depth information of the points on the line segment included in the frame L9 is derived as an index value. Which of the left elbow and the right elbow is higher is determined according to the magnitude of the index value.

(Takeback arm horizontal index value-right side)
Spine angle based on the address The spine angle can be derived from the position information of the neck and waist and the spine angle at the time of addressing.

(Top index value-front)
Overswing, warp of the upper body, amount of left and right movement from the address, position of the right knee based on the address, position of the head based on the address, position of the left shoulder based on the address Note that the overswing is the position of the shaft 52. From the angle, the warp of the upper body is from the position information of the head and umbilical cord, the amount of lateral movement is from the position information of the right umbilical cord and the umbilical cord position, and the position information of the right foot width and the umbilical cord at the time of addressing. From the position information of the right knee and the position information of the right foot width reference position at the time of addressing, the position of the head is from the position of the head and the position of the right foot width reference position at the time of addressing, and the position of the left shoulder is the position information of the left shoulder and It can be derived from the position information of the left shoulder and the center of the shoulder at the time of addressing.

(Top index value-right side)
The amount of movement of the right knee from the address, the amount of movement of the left knee from the address, the position of the wrist based on the address, the height of both elbows, the spine angle based on the address The amount of movement of the right knee is the right From the position information of the right kneecap at the time of address and the kneecap, the amount of movement of the left knee is from the position information of the left kneecap and the left kneecap at the time of address, and the position of the wrist is the position information of the neck and grip end 51a and the right at the time of address. From the position information of the elbow and the ball (origin), the height of both elbows can be derived from the position information of the left elbow and the right elbow, and the spine angle can be derived from the position information of the neck and waist and the spine angle at the time of addressing.

(Downswing arm horizontal index value-front)
Wristcock angle, left knee position with respect to left foot width reference position at address Note that the wristcock angle is from the direction of the shaft 52, and the left knee position is the left knee position information and left foot width reference position at address. It can be derived from the position information of.

(Downswing arm horizontal index value-right side)
Spine angle based on the address The spine angle can be derived from the position information of the neck and waist and the spine angle at the time of addressing.

(Index value at 9 o'clock on the downswing shaft-front)
Changes in the angle of the wrist cock, the position of the grip end 51a with respect to the reference axis at the time of address The change in the angle of the wrist cock includes the orientation of the shaft 52, the position information of the left shoulder and the position information of the grip end 51a, and the downswing arm horizontal. From the orientation of the shaft 52 at the time, the position of the grip end 51a can be derived from the position information of the grip end 51a and the position information of the right shoulder and the center of the shoulder at the time of addressing.

(Impact index value-front)
Left / right movement amount, up / down movement amount, head position, shoulder rotation amount, spine inclination, left elbow closing, position of grip end 51a with respect to reference axis at address. From the position information of the left shoulder, right shoulder and shoulder center at the time, the amount of vertical movement is from the position information of the umbilical cord at the time of address and impact, and the position of the head is the position information of the head and the position of the left shoulder and the center of the shoulder at the time of address. From the information, the amount of rotation of the shoulder is from the position information of the left and right shoulders and the position information of the left and right shoulders at the time of addressing, the inclination of the spine is from the position information of the umbilical cord and the center of the shoulder, and the closing of the left elbow is the left. The position of the grip end 51a can be derived from the position information of the elbow, the left shoulder and the grip end 51a from the position information of the grip end 51a and the position information of the left shoulder and the center of the shoulder at the time of addressing.

(Impact index value-right side)
Address-based spine angle, address-based knee angle Note that the spine angle is based on the position information of the neck and waist, and the spine angle at the time of addressing, and the knee angle is the position of the waist, right knee, and right ankle. It can be derived from the information and the position information of the waist, right knee and right ankle at the time of addressing.

In the present embodiment, the correct answer data of the index value as described above (the range of the index value when the swing operation is determined to be good) is set in advance in the storage unit 13. The determination unit 12e determines the quality of the swing operation for each index value by comparing the correct answer data with the calculated index value. It is also possible to comprehensively evaluate the quality of the swing operation by giving points to the determination results for each index value and adding them. The determination result is appropriately displayed on the display unit 11. For example, in the form of a report, a list of judgment results and a comprehensive evaluation for each index value can be displayed.

1 Swing analysis device 2 Distance image sensor 3 Swing analysis program 5 Golf club 7 Golfer 14a Acquisition unit 14b Position derivation unit (derivation unit)
14c direction derivation part (derivation part)
51 Grip 52 Shaft 71 Left shoulder 100 Swing analysis system

Claims (38)

A swing analysis device for analyzing the swing motion of a golf club by a golfer.
An acquisition unit that acquires a depth image and a two-dimensional image obtained by capturing the swing motion with a distance image sensor, and
By image processing the two-dimensional image, the two-dimensional coordinates of the tracking target of the predetermined part of the golf club during the swing operation are specified, and the tracking target of the predetermined part is determined based on the depth image . A derivation unit for deriving the three-dimensional coordinates of the tracking target of the predetermined part by specifying the depth is provided.
Swing analyzer. The derivation unit sets a search range on the two-dimensional image at the first time based on the two-dimensional coordinates of the predetermined part at the second time before the first time, and sets the search range. Search for the two-dimensional coordinates of the predetermined part within.
The swing analysis device according to claim 1. In the phase near the top during the swing operation, the derivation unit limits the set position of the search range with reference to the position of the golfer's head.
The swing analysis device according to claim 2. The predetermined portion is a grip of the golf club.
The swing analysis apparatus according to any one of claims 1 to 3. The predetermined portion is a grip of the golf club.
The derivation unit specifies the two-dimensional coordinates of the shaft of the golf club based on the two-dimensional image, and specifies the two-dimensional coordinates of the grip based on the two-dimensional coordinates of the shaft.
The swing analysis apparatus according to any one of claims 1 to 3. A swing analysis device for analyzing the swing motion of a golf club by a golfer.
An acquisition unit that acquires a depth image and a two-dimensional image obtained by capturing the swing motion with a distance image sensor, and
By image processing the two-dimensional image, the identified two-dimensional coordinates of the tracking target of the golf club shaft, based on the depth image, identifies the depth of the two-dimensional coordinates of the tracking target of the shaft To derive the three-dimensional coordinates of the tracking target of the shaft, and to specify the three-dimensional coordinates of a predetermined part based on the distribution tendency of the three-dimensional coordinates of the tracking target of the shaft.
The predetermined portion is a grip of the golf club.
Swing analyzer. The two-dimensional image and the depth image are information obtained by capturing the swing motion from the front of the golfer by the distance image sensor.
The derivation unit specifies the two-dimensional coordinates of the shaft based on the two-dimensional image near the address, and specifies the three-dimensional coordinates of the grip based on the two-dimensional coordinates of the shaft.
The swing analysis device according to claim 5 or 6. The derivation unit binarizes the two-dimensional image while switching the threshold value between automatic and fixed according to the phase of the swing operation.
The swing analysis device according to any one of claims 1 to 7. The acquisition unit further acquires a time-series two-dimensional image from the distance image sensor.
When the position of the grip of the golf club cannot be extracted from the two-dimensional image at the first time, the derivation unit of the golf club from the two-dimensional image at the second time immediately after the first time. The position of the shaft is specified, and the position of the grip at the second time is specified based on the position of the shaft.
The swing analysis device according to claim 1. A swing analysis device for analyzing the swing motion of a golf club by a golfer.
An acquisition unit that acquires a depth image of the swing motion taken from the front side of the golfer by a distance image sensor.
A binarized image obtained by binarizing the depth image with a depth value is generated, and the depth image is masked with the binarized image to emphasize the vicinity of one corner and suppress the vicinity of the other corners. The masked depth image is filtered by the silhouette filter, the two-dimensional coordinates of the predetermined portion during the swing operation are derived from the filtered depth image, and the predetermined portion is based on the depth image before filtering. A derivation unit that derives the three-dimensional coordinates of the predetermined part during the swing operation by specifying the depth of the
With
The predetermined part is the shoulder of the golfer.
Swing analyzer. The acquisition unit further acquires skeleton data representing the skeleton of the human body.
The derivation unit derives the three-dimensional coordinates of the shoulder based on the depth image and the skeleton data.
The swing analysis device according to claim 10. The derivation unit derives the three-dimensional coordinates of the shoulder by filtering the depth image with a convex extraction filter that emphasizes the central portion and suppresses the peripheral portion.
The swing analysis device according to claim 10 or 11. The derivation unit derives the three-dimensional coordinates of the shoulder by performing template matching on the depth image using a convex image template.
The swing analysis device according to claim 10 or 11. The derivation unit derives the three-dimensional coordinates of the shoulder based on the skeleton data in at least a part of the section from the address to the takeback arm horizontal.
The swing analysis device according to claim 11. The derivation unit derives the three-dimensional coordinates of the shoulder by filtering with the convex portion extraction filter in at least a part of the section from the takeback arm horizontal to the downswing arm horizontal.
The swing analysis device according to claim 12. The derivation unit derives the three-dimensional coordinates of the shoulder by performing template matching using the convex image template in at least a part of the section from the takeback arm horizontal to the downswing arm horizontal.
The swing analysis device according to claim 13. A swing analysis device for analyzing the swing motion of a golf club by a golfer.
An acquisition unit that acquires a depth image of the swing motion captured by a distance image sensor, and
A derivation unit that derives the three-dimensional coordinates of a predetermined part during the swing operation by filtering the depth image with a shoulder silhouette filter that emphasizes the vicinity of one corner and suppresses the vicinity of the other corner.
With
The predetermined part is the shoulder of the golfer.
The derivation unit derives the three-dimensional coordinates of the shoulder by filtering with the shoulder silhouette filter in at least a part of the section from the address to the takeback arm horizontal and from the downswing arm horizontal to the finish.
Swing analyzer. The derivation unit sets a search range on the depth image at the first time with reference to the two-dimensional coordinates of the shoulder at the first time or the second time before the first time. Perform the filtering within the search range.
The swing analysis device according to any one of claims 10 to 17. A swing analysis device for analyzing the swing motion of a golf club by a golfer.
An acquisition unit that acquires a depth image of the swing motion captured by a distance image sensor, and
Based on the depth image, the three-dimensional coordinates of the predetermined part of the golf club or the golfer during the swing operation are derived, and the three-dimensional of the predetermined part is expressed in a predetermined multiple regression equation having a predetermined coefficient. It is equipped with a derivation unit that performs calibration by substituting coordinates.
The multiple regression equation is a multiple regression equation in which each of the coordinates in the three directions after the calibration uses all the coordinates in the three directions before the calibration as explanatory variables.
Swing analyzer. The acquisition unit acquires the depth image obtained by capturing the swing motion from at least the first direction and the second direction by the plurality of distance image sensors.
The swing analysis apparatus according to any one of claims 1 to 19. The acquisition unit further acquires skeleton data representing the skeleton of the human body based on the depth image from at least the first direction.
The derivation unit converts the skeleton data from the first direction into the skeleton data from the second direction, and derives the position information of the predetermined portion based on the skeleton data from the second direction. do,
The swing analysis device according to claim 20. The first direction is a direction toward the front of the golfer, and the second direction is a direction toward the side surface of the golfer.
The swing analysis device according to claim 20 or 21. A swing analysis device for analyzing the swing motion of a golf club by a golfer.
An acquisition unit that acquires a depth image of the swing motion captured by a distance image sensor, and
A derivation unit for deriving position information of a predetermined portion of the golf club or the golfer during the swing operation based on the depth image is provided.
The acquisition unit further acquires skeleton data representing the skeleton of the human body based on the depth image.
The derivation unit derives the position information of the first part included in the predetermined part based on the skeleton data, and derives the position information.
The acquisition unit further acquires silhouette data representing the silhouette of the human body, and obtains the silhouette data.
Based on the skeleton data and the silhouette data, the derivation unit derives the position information of the second part included in the predetermined part and different from the first part.
Swing analyzer. The derivation unit expands the area for copying the silhouette specified in the silhouette data based on the information of the area for which the depth information is not obtained in the depth image.
The swing analysis device according to claim 23. In the depth image from the right side of the golfer at the top timing, the lead-out unit derives a portion protruding toward the front side as the left elbow of the golfer.
The swing analysis apparatus according to any one of claims 1 to 24. A determination unit for determining the quality of the swing operation is further provided by comparing an index value calculated based on the position information of the predetermined portion at a predetermined timing with predetermined correct answer data.
The swing analysis apparatus according to any one of claims 1 to 25. A swing analysis method for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image and a two-dimensional image of the swing motion taken by a distance image sensor, and
By performing image processing on the two-dimensional image, a step of specifying the two-dimensional coordinates of a tracking target of a predetermined portion of the golf club during the swing operation, and a step of specifying the two-dimensional coordinates.
Based on the depth image, by specifying the depth of the tracking target of the predetermined portion, and a step of deriving three-dimensional coordinates of the tracking target of the predetermined portion,
Swing analysis method. A swing analysis method for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image and a two-dimensional image of the swing motion taken by a distance image sensor, and
By image processing the two-dimensional image, the identified two-dimensional coordinates of the tracking target of the golf club shaft, based on the depth image, identifies the depth of the two-dimensional coordinates of the tracking target of the shaft To derive the three-dimensional coordinates of the tracking target of the shaft,
A step of specifying the three-dimensional coordinates of a predetermined part based on the distribution tendency of the three-dimensional coordinates of the tracking target of the shaft is provided.
The predetermined portion is a grip of the golf club.
Swing analysis method. A swing analysis method for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image of the swing motion taken from the front side of the golfer by a distance image sensor, and
A step of generating a binarized image obtained by binarizing the depth image with a depth value and masking the depth image with the binarized image.
The masked depth image is filtered by a shoulder silhouette filter that emphasizes the vicinity of one corner and suppresses the vicinity of the other corner, and the two-dimensional coordinates of the predetermined portion during the swing operation are obtained from the filtered depth image. A step of deriving and deriving the three-dimensional coordinates of the predetermined part during the swing operation by specifying the depth of the predetermined part based on the depth image before filtering is provided.
The predetermined part is the shoulder of the golfer.
Swing analysis method. A swing analysis method for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image of the swing motion taken by a distance image sensor, and
Based on the depth image, a step of deriving the three-dimensional coordinates of a predetermined portion of the golf club or the golfer during the swing operation, and
A step of performing calibration by substituting the three-dimensional coordinates of the predetermined part into a predetermined multiple regression equation having a predetermined coefficient is provided.
The multiple regression equation is a multiple regression equation in which each of the coordinates in the three directions after the calibration uses all the coordinates in the three directions before the calibration as explanatory variables.
Swing analysis method. A swing analysis method for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image of the swing motion taken by a distance image sensor, and
The step of acquiring skeleton data representing the skeleton of the human body based on the depth image, and
Steps to acquire silhouette data representing the silhouette of the human body,
A step of deriving the position information of a predetermined portion of the golf club or the golfer during the swing operation based on the depth image is provided.
The step of deriving the position information of the predetermined part is
A step of deriving the position information of the first part included in the predetermined part based on the skeleton data, and
A step of deriving position information of a second part different from the first part included in the predetermined part based on the skeleton data and the silhouette data is included.
Swing analysis method. A swing analysis program for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image and a two-dimensional image of the swing motion taken by a distance image sensor, and
By performing image processing on the two-dimensional image, a step of specifying the two-dimensional coordinates of a tracking target of a predetermined portion of the golf club during the swing operation, and a step of specifying the two-dimensional coordinates.
A computer is made to perform a step of deriving the three-dimensional coordinates of the tracking target of the predetermined part by specifying the depth of the tracking target of the predetermined part based on the depth image.
Swing analysis program. A swing analysis program for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image and a two-dimensional image of the swing motion taken by a distance image sensor, and
By image processing the two-dimensional image, the identified two-dimensional coordinates of the tracking target of the golf club shaft, based on the depth image, identifies the depth of the two-dimensional coordinates of the tracking target of the shaft To derive the three-dimensional coordinates of the tracking target of the shaft,
A computer is made to perform a step of specifying the three-dimensional coordinates of a predetermined part based on the distribution tendency of the three-dimensional coordinates of the tracking target of the shaft.
The predetermined portion is a grip of the golf club.
Swing analysis program. A swing analysis program for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image of the swing motion taken from the front side of the golfer by a distance image sensor, and
A step of generating a binarized image obtained by binarizing the depth image with a depth value and masking the depth image with the binarized image.
The masked depth image is filtered by a shoulder silhouette filter that emphasizes the vicinity of one corner and suppresses the vicinity of the other corner, and the two-dimensional coordinates of the predetermined portion during the swing operation are obtained from the filtered depth image. A computer is made to perform a step of deriving and deriving the three-dimensional coordinates of the predetermined part during the swing operation by specifying the depth of the predetermined part based on the depth image before filtering.
The predetermined part is the shoulder of the golfer.
Swing analysis program. A swing analysis program for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image of the swing motion taken by a distance image sensor, and
Based on the depth image, a step of deriving the three-dimensional coordinates of a predetermined portion of the golf club or the golfer during the swing operation, and
A computer is made to perform a step of performing calibration by substituting the three-dimensional coordinates of the predetermined part into a predetermined multiple regression equation having a predetermined coefficient.
The multiple regression equation is a multiple regression equation in which each of the coordinates in the three directions after the calibration uses all the coordinates in the three directions before the calibration as explanatory variables.
Swing analysis program. A swing analysis program for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image of the swing motion taken by a distance image sensor, and
The step of acquiring skeleton data representing the skeleton of the human body based on the depth image, and
Steps to acquire silhouette data representing the silhouette of the human body,
Based on the depth image, the computer is made to perform a step of deriving the position information of the golf club or the golfer at a predetermined portion during the swing operation.
The step of deriving the position information of the predetermined part is
A step of deriving the position information of the first part included in the predetermined part based on the skeleton data, and
A step of deriving position information of a second part different from the first part included in the predetermined part based on the skeleton data and the silhouette data is included.
Swing analysis program. A swing analysis method for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image of the swing motion taken by a distance image sensor, and
It is provided with a step of deriving the three-dimensional coordinates of a predetermined part during the swing operation by filtering the depth image with a shoulder silhouette filter that emphasizes the vicinity of one corner and suppresses the vicinity of the other corner.
The predetermined part is the shoulder of the golfer.
The derivation step is a step of deriving the three-dimensional coordinates of the shoulder by filtering with the shoulder silhouette filter in at least a part of the section from the address to the takeback arm horizontal and from the downswing arm horizontal to the finish. including,
Swing analysis method. A swing analysis program for analyzing the swing motion of a golf club by a golfer.
A step of acquiring a depth image of the swing motion taken by a distance image sensor, and
The computer executes a step of deriving the three-dimensional coordinates of a predetermined part during the swing operation by filtering the depth image with a shoulder silhouette filter that emphasizes the vicinity of one corner and suppresses the vicinity of the other corner. Let me
The predetermined part is the shoulder of the golfer.
The derivation step is a step of deriving the three-dimensional coordinates of the shoulder by filtering with the shoulder silhouette filter in at least a part of the section from the address to the takeback arm horizontal and from the downswing arm horizontal to the finish. including,
Swing analysis program.

Images