JP2017074350A - Swing analysis apparatus, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a swing analysis apparatus capable of analyzing a swing motion in a simple and highly accurate manner.SOLUTION: Provided is a swing analysis apparatus 1 for analyzing a swing motion of a golf club performed by a golfer 7. The swing analysis apparatus comprises an acquisition unit and a derivation unit. The acquisition unit acquires a depth image of the swing motion captured by a distance image sensor 2. The derivation unit derives, based on the depth image, positional information of a predetermined position of the golf club or of the golfer during the swing motion, or a three-dimensional vector indicative of the orientation of the shaft of the golf club during the swing motion.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a swing analysis apparatus, method, and program for analyzing a golf club swing motion by a golfer.

  2. Description of the Related Art Conventionally, an apparatus that captures a golf club swing motion by a golfer with a camera and analyzes the swing motion based on the obtained image is known (Patent Documents 1, 2, etc.). The results of the analysis are used in various applications such as fitting a golf club suitable for a golfer, improving a golfer's foam, and developing golf equipment.

JP 2013-215349 A JP 2004-313479 A

  In the analysis of the swing motion as described above, it is often performed to three-dimensionally identify the behavior of a site of interest, such as a golf club grip or head, a golfer's joint, or the like that appears in the image. Patent Documents 1 and 2 disclose that a swing motion is photographed from a plurality of directions by a plurality of cameras, and the three-dimensional coordinates of the region of interest are derived based on a triangulation method, a DLT method, or the like. However, having to arrange the cameras in a plurality of directions can increase the size of the equipment and make it difficult to secure a space for performing the swing analysis. In addition, such three-dimensional measurement has room for improvement in terms of analysis accuracy and calculation load.

  An object of the present invention is to provide a swing analysis device, method, and program that can analyze swing motion easily and with high accuracy.

  A swing analysis device according to a first aspect of the present invention is a swing analysis device for analyzing a golf club swing motion by a golfer, and includes an acquisition unit and a derivation unit. The acquisition unit acquires a depth image obtained by capturing the swing motion with a distance image sensor. The derivation unit is based on the depth image, and is a three-dimensional vector representing position information of a predetermined part of the golf club or the golfer during the swing operation or a shaft direction of the golf club during the swing operation. Is derived.

  The swing analysis apparatus according to a second aspect of the present invention is the swing analysis apparatus according to the first aspect, wherein the acquisition unit further acquires a two-dimensional image from the distance image sensor. The derivation unit specifies the two-dimensional coordinates of the predetermined part based on the two-dimensional image, and specifies the depth of the predetermined part based on the depth image, thereby Derive three-dimensional coordinates.

  The swing analysis apparatus according to a third aspect of the present invention is the swing analysis apparatus according to the second aspect, wherein the derivation unit is on the two-dimensional image at the first time and is before the first time. A search range is set based on the two-dimensional coordinates of the predetermined part at the second time, and the two-dimensional coordinates of the predetermined part are searched within the search range.

  A swing analyzing apparatus according to a fourth aspect of the present invention is the swing analyzing apparatus according to any one of the first to third aspects, wherein the predetermined part is a grip of the golf club.

  A swing analysis device according to a fifth aspect of the present invention is the swing analysis device according to the second or third aspect, wherein the predetermined part is a grip of the golf club. The derivation unit specifies the two-dimensional coordinates of the shaft of the golf club based on the two-dimensional image, and specifies the two-dimensional coordinates of the grip based on the two-dimensional coordinates of the shaft.

  A swing analysis apparatus according to a sixth aspect of the present invention is the swing analysis apparatus according to the first aspect, wherein the predetermined part is a grip of the golf club. The acquisition unit further acquires a two-dimensional image from the distance image sensor. The derivation unit specifies the two-dimensional coordinates of the shaft of the golf club based on the two-dimensional image, and specifies the depth in the two-dimensional coordinates of the shaft based on the depth image. The three-dimensional coordinates of the grip are derived, and the three-dimensional coordinates of the grip are specified based on the distribution tendency of the three-dimensional coordinates of the shaft.

  A swing analysis device according to a seventh aspect of the present invention is the swing analysis device according to the fifth or sixth aspect, wherein the two-dimensional image and the depth image are obtained by performing the swing operation using the distance image sensor. This is information taken from the front. The deriving unit identifies the two-dimensional coordinates of the shaft based on the two-dimensional image near the address, and identifies the three-dimensional coordinates of the grip based on the two-dimensional coordinates of the shaft.

  A swing analysis device according to an eighth aspect of the present invention is the swing analysis device according to any of the second, third, fifth, and seventh aspects, wherein the derivation unit automatically sets a threshold value according to the phase of the swing operation or The two-dimensional image is binarized while switching between fixed.

  A swing analysis device according to a ninth aspect of the present invention is the swing analysis device according to the third aspect, wherein the derivation unit is based on the position of the golfer's head in the phase near the top during the swing operation. The setting position of the search range is limited.

  A swing analysis apparatus according to a tenth aspect of the present invention is the swing analysis apparatus according to the first aspect, wherein the acquisition unit further acquires a time-series two-dimensional image from the distance image sensor. If the position of the grip of the golf club cannot be extracted from the two-dimensional image at the first time, the derivation unit can determine the golf club from the two-dimensional image at the second time immediately after the first time. The position of the shaft is specified, and the position of the grip at the second time is specified based on the position of the shaft.

  A swing analyzing apparatus according to an eleventh aspect of the present invention is the swing analyzing apparatus according to the first aspect, wherein the predetermined part is a shoulder of the golfer.

  A swing analysis device according to a twelfth aspect of the present invention is the swing analysis device according to the eleventh aspect, wherein the acquisition unit further acquires skeleton data representing a skeleton of the human body, and the derivation unit includes the depth image. And the three-dimensional coordinates of the shoulder are derived based on the skeleton data.

  A swing analysis apparatus according to a thirteenth aspect of the present invention is the swing analysis apparatus according to the eleventh aspect or the twelfth aspect, wherein the derivation unit uses a convex extraction filter that emphasizes the central part and suppresses the peripheral part. Then, the shoulder image is derived by filtering the depth image.

  A swing analysis apparatus according to a fourteenth aspect of the present invention is the swing analysis apparatus according to the eleventh aspect or the twelfth aspect, in which the derivation unit performs template matching on the depth image using a convex image template. As a result, the three-dimensional coordinates of the shoulder are derived.

  A swing analysis device according to a fifteenth aspect of the present invention is the swing analysis device according to any of the eleventh to fourteenth aspects, wherein the derivation unit emphasizes the vicinity of one corner and the vicinity of the other corner. The depth image is filtered by the suppressing shoulder silhouette filter to derive the three-dimensional coordinates of the golfer's shoulder.

  A swing analysis apparatus according to a sixteenth aspect of the present invention is the swing analysis apparatus according to the twelfth aspect, wherein the derivation unit is based on the skeleton data in at least a part of the section from the address to the takeback arm horizontal. Then, the three-dimensional coordinates of the shoulder are derived.

  A swing analysis apparatus according to a seventeenth aspect of the present invention is the swing analysis apparatus according to the thirteenth aspect, wherein the derivation unit is configured to project the convex in at least a part of a section from the takeback arm level to the downswing arm level. By performing filtering using a partial extraction filter, the three-dimensional coordinates of the shoulder are derived.

  A swing analysis apparatus according to an eighteenth aspect of the present invention is the swing analysis apparatus according to the fourteenth aspect, wherein the derivation unit is configured to project the convex in at least a part of a section from the takeback arm level to the downswing arm level. The three-dimensional coordinates of the shoulder are derived by performing template matching using a shape image template.

  A swing analysis apparatus according to a nineteenth aspect of the present invention is the swing analysis apparatus according to the fifteenth aspect, wherein the derivation unit includes at least a part from the address to the takeback arm level and from the downswing arm level to the finish. In the section, the three-dimensional coordinates of the shoulder are derived by performing filtering using the shoulder silhouette filter.

  A swing analysis apparatus according to a twentieth aspect of the present invention is the swing analysis apparatus according to any of the eleventh to nineteenth aspects, wherein the derivation unit is configured to perform the first analysis on the depth image at a first time. A search range is set based on the two-dimensional coordinates of the shoulder at one time or at a second time before the first time, and the filtering is performed within the search range.

  A swing analysis apparatus according to a twenty-first aspect of the present invention is the swing analysis apparatus according to any one of the first to twentieth aspects, wherein the derivation unit uses a predetermined multiple regression equation having a predetermined coefficient. Calibration is performed by substituting the three-dimensional coordinates of the predetermined part.

  A swing analysis apparatus according to a twenty-second aspect of the present invention is the swing analysis apparatus according to any one of the first to twenty-first aspects, wherein the acquisition unit performs at least the swing operation by a plurality of the distance image sensors. The depth images taken from the first direction and the second direction are acquired.

  A swing analysis apparatus according to a twenty-third aspect of the present invention is the swing analysis apparatus according to the twenty-second aspect, wherein the acquisition unit generates skeleton data representing a human skeleton based on at least the depth image from the first direction. Get more. The derivation unit converts the skeleton data from the first direction into the skeleton data from the second direction, and derives position information of the predetermined part based on the skeleton data from the second direction. To do.

  A swing analysis device according to a twenty-fourth aspect of the present invention is the swing analysis device according to the twenty-second or twenty-third aspect, wherein the first direction is a direction toward the front of the golfer, and the second direction is The direction toward the side surface of the golfer.

  A swing analysis device according to a twenty-fifth aspect of the present invention is the swing analysis device according to any one of the first to twenty-fourth aspects, wherein the acquisition unit generates skeleton data representing a human skeleton based on the depth image. Get more. The derivation unit derives position information of a first part included in the predetermined part based on the skeleton data.

  A swing analysis apparatus according to a twenty-sixth aspect of the present invention is the swing analysis apparatus according to the twenty-fifth aspect, in which the acquisition unit further acquires silhouette data representing a silhouette of a human body. The derivation unit derives position information of a second part different from the first part included in the predetermined part based on the skeleton data and the silhouette data.

  A swing analysis apparatus according to a twenty-seventh aspect of the present invention is the swing analysis apparatus according to any one of the first to twenty-sixth aspects, wherein the derivation unit is a region for which depth information is not obtained in the depth image. Based on the information, the silhouette area specified in the silhouette data is expanded.

  A swing analysis device according to a twenty-eighth aspect of the present invention is the swing analysis device according to any one of the first to the twenty-seventh aspects, wherein the derivation unit is the right side surface of the golfer at a top timing. In the depth image, a portion protruding to the near side is derived as the left elbow of the golfer.

  A swing analysis device according to a twenty-ninth aspect of the present invention is the swing analysis device according to any one of the first to twenty-eighth aspects, further comprising a determination unit. The determination unit compares the index value calculated based on at least one of the position information of the predetermined part and the three-dimensional vector at a predetermined timing with predetermined correct answer data, thereby performing the swing motion Judge the quality of the.

  A swing analysis method according to a thirtieth aspect of the present invention is a swing analysis method for analyzing a golf club swing motion by a golfer, and obtaining a depth image obtained by photographing the swing motion with a distance image sensor; Deriving, based on the depth image, a position information of a predetermined part of the golf club or the golfer during the swing operation, or a three-dimensional vector representing a direction of a shaft of the golf club during the swing operation; Is provided.

  A swing analysis program according to a thirty-first aspect of the present invention is a swing analysis program for analyzing a golf club swing motion by a golfer, and obtaining a depth image obtained by photographing the swing motion with a distance image sensor; Deriving, based on the depth image, a position information of a predetermined part of the golf club or the golfer during the swing operation, or a three-dimensional vector representing a direction of a shaft of the golf club during the swing operation; Is executed on the computer.

  According to the present invention, a depth image obtained by photographing a golf club swing motion by a golfer with a distance image sensor is acquired, and based on this, position information of a predetermined part of the golf club or the golfer during the swing motion or the swing motion A three-dimensional vector representing the orientation of the shaft of the golf club inside is derived. As a result, shooting from different directions using a plurality of cameras is not necessarily required. Therefore, the swing motion can be analyzed easily and with high accuracy.

The figure which shows the whole structure of the swing analysis system containing the swing analyzer which concerns on 1st Embodiment of this invention. The functional block diagram of the swing analysis system which concerns on 1st Embodiment. The flowchart which shows the whole flow of a grip behavior derivation process. (A) The figure which shows an address state. (B) The figure which shows a top state. (C) The figure which shows an impact state. (D) The figure which shows a finish state. The flowchart which shows the flow of the process which extracts directly the two-dimensional coordinate of a grip end from IR frame. The figure of IR frame which shows the image of a grip end and a shaft. The figure explaining the complement of the image of a grip end. The flowchart which shows the flow of the process which pinpoints the depth of a grip end directly from a depth frame. The figure of the depth frame which shows the position of a grip end. The figure of the depth frame in the vicinity of the grip end. The graph which shows the effect of the correction | amendment of the depth of the grip end which considered the image of the human body. The flowchart which shows the flow of exception processing. The figure of the depth frame which shows the position of the inertia principal axis of a shaft. The graph which plotted the relationship between the distance from b point of the point on an inertial principal axis, and depth. The figure of the depth frame which shows the position of a point and c point on an inertial principal axis. The flowchart which shows the flow of the process which estimates the three-dimensional coordinate of a grip end based on the depth of the lower part of a grip. The figure which shows the example of a test subject. The flowchart which shows the flow of a correction process. The figure explaining the specific method of the three-dimensional coordinate of the grip end by principal component analysis. The figure which shows the result of a correction process. The flowchart which shows the whole flow of left shoulder behavior derivation processing. The graph which shows the three-dimensional coordinate of the left shoulder derived | led-out by the left shoulder behavior deriving process. The figure which shows the example of a convex part extraction filter. The figure which shows the depth frame in the top timing. The figure which shows the depth frame after convex part extraction filter application. The figure which shows the binarized image of a depth frame. The figure which shows the binarized image after shrinkage | contraction. The figure which shows the depth frame after masking to which the convex part extraction filter was applied. The figure which shows the example of the application range of a convex part extraction filter. The figure which shows another example of the application range of a convex part extraction filter. The figure which shows the example of a left shoulder silhouette filter. The figure which shows the binarized image of a depth frame. The figure which shows the binarized image after a left shoulder silhouette filter application. The figure which shows the example of the application range of a left shoulder silhouette filter. The figure which shows another example of the application range of a left shoulder silhouette filter. The figure which shows another example of the application range of a left shoulder silhouette filter. The figure which shows the rectangular area | region used for the detection of an arm horizontal frame. The flowchart which shows the whole flow of a shaft direction derivation | leading-out process. The figure which shows the example of a graph display of each axis | shaft component of the three-dimensional vector of a shaft. The figure which shows the three-dimensional graph display example of the three-dimensional vector of a shaft. The figure which shows IR frame when an actual grip end is outside a search range. The figure which shows the limit line for setting the search range on IR frame. The figure which shows the vertex of the head on the person area image at the time of a takeback arm level. The flowchart which shows the process which derives | leads-out the three-dimensional coordinate of the grip end which concerns on embodiment of FIG. The flowchart which shows the process which derives | leads-out the three-dimensional coordinate of the grip end which concerns on a modification. The figure which shows the example of a convex shape template. The figure which shows the position of the left shoulder extracted by the template matching which concerns on the modification using Kinect (trademark) v2. The conceptual diagram explaining the switching of the algorithm for left shoulder extraction. The top view which shows the whole structure of the swing analysis system containing the swing analyzer which concerns on 2nd Embodiment of this invention. The functional block diagram of the swing analysis system which concerns on 2nd Embodiment. The flowchart which shows the flow of a position derivation process. Front view of golfer at various checkpoints. A right side view of the golfer at various checkpoints. The figure which shows the image which superimposed the front skeleton data on the front silhouette data at the time of an address. The figure which shows the image which superimposed the front skeleton data on the front silhouette data in the timing slightly advanced from the address. The figure which shows the image of the right side silhouette data at the time of an address. FIG. 54B is a diagram showing an image in which silhouettes of the left foot and the right foot are complemented in the image of FIG. 54A. The figure which shows the image of the right side silhouette data at the time of a top. The figure which shows the image of the front silhouette data at the time of a takeback arm level.

  Hereinafter, swing analysis devices, methods, and programs according to some embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Overview of swing analysis system>
1 and 2 show an overall configuration diagram of a swing analysis system 100 including a swing analysis apparatus 1 according to the first embodiment of the present invention. The swing analysis system 100 is a system for taking a swing motion of the golf club 5 by the golfer 7 as a moving image and analyzing the golf swing based on the moving image. The above photographing is performed by the distance image sensor 2. The swing analysis apparatus 1 constitutes a swing analysis system 100 together with the distance image sensor 2 and analyzes the golf swing by performing image processing on the image data obtained by the distance image sensor 2. The result of the analysis by the swing analysis device 1 is used for various purposes such as fitting of the golf club 5 suitable for the golfer 7, improvement of the form of the golfer 7, development of golf equipment, and the like.

  In the present embodiment, the behavior of the grip 51 of the golf club 5 during the swing motion and the behavior of the left shoulder 71 of the golfer 7 are analyzed. More specifically, the three-dimensional coordinates of the grip 51 and the left shoulder 71 during the swing motion are derived. In the present embodiment, the orientation of the shaft 52 of the golf club 5 during the swing operation is analyzed.

<1-2. Details of each part>
Hereinafter, details of each part of the swing analysis system 100 will be described.

<1-2-1. Distance image sensor>
The distance image sensor 2 is a camera having a ranging function that measures a distance to a subject while shooting a golfer 7 trial hitting the golf club 5 as a two-dimensional image. Therefore, the distance image sensor 2 can output a depth image together with the two-dimensional image. Note that the two-dimensional image here is an image obtained by projecting an image of the photographing space onto a plane orthogonal to the optical axis of the camera. The depth image is an image in which data on the depth of the subject in the optical axis direction of the camera is assigned to pixels within substantially the same imaging range as the two-dimensional image.

  The distance image sensor 2 used in the present embodiment captures a two-dimensional image as an infrared image (hereinafter referred to as an IR image). The depth image is obtained by a method such as a time-of-flight method using infrared rays or a dot pattern projection method. Accordingly, as shown in FIG. 1, the distance image sensor 2 has an IR light emitting unit 21 that emits infrared rays toward the front, and an IR that receives the infrared rays that are emitted from the IR light emitting unit 21 and reflected back to the subject. A light receiving unit 22. The IR light receiving unit 22 is a camera having an optical system, an image sensor, and the like. In the dot pattern projection method, the IR dot pattern emitted from the IR light emitting unit 21 is read by the IR light receiving unit 22, the dot pattern is detected by image processing inside the distance image sensor 2, and the depth is calculated based on this. The In the present embodiment, the IR light emitting unit 21 and the IR light receiving unit 22 are accommodated in the same housing 20 and disposed in front of the housing 20. In the distance image sensor 2 according to the present embodiment, an effective distance capable of measuring the depth is determined, and the depth image may include pixels for which a depth value cannot be obtained.

  In addition to the CPU 23 that controls the entire operation of the distance image sensor 2, the distance image sensor 2 includes a memory 24 that stores captured image data at least temporarily. A control program for controlling the operation of the distance image sensor 2 is stored in the memory 24. The distance image sensor 2 also includes a communication unit 25, and the communication unit 25 transmits captured image data to an external device such as the swing analysis device 1 via a wired or wireless communication line 17. Can be output to the device. In the present embodiment, the CPU 23 and the memory 24 are also housed in the housing 20 together with the IR light emitting unit 21 and the IR light receiving unit 22. It is not always necessary to transfer the image data to the swing analysis device 1 via the communication unit 25. For example, if the memory 24 is detachable, it is removed from the housing 20 and inserted into a reader (corresponding to a communication unit 15 described later) of the swing analysis device 1 so that the image data is captured by the swing analysis device 1. Can be read.

  In the present embodiment, as described above, infrared imaging is performed by the distance image sensor 2, and the trajectory of the grip 51 is tracked based on the captured IR image. Therefore, a reflection sheet that efficiently reflects infrared rays is attached to the grip 51 (more precisely, the grip end 51a) as a marker so as to facilitate this tracking. Further, as will be described later, in this embodiment, when the grip end 51a is not visible depending on the shooting angle, the position of the grip end 51a is specified based on the position of the shaft 52 of the golf club 5, and the shaft 52 The direction is also specified. Accordingly, a similar infrared reflecting sheet is also affixed to the shaft 52 as a marker. In the present embodiment, the distance image sensor 2 is installed in front of the golfer 7 so as to photograph the golfer 7 from the front side.

<1-2-2. Swing analyzer>
As shown in FIG. 2, the swing analysis device 1 is manufactured by installing a swing analysis program 3 stored in a computer-readable recording medium 30 such as a CD-ROM from the recording medium 30 to a general-purpose personal computer. Is done. The swing analysis program 3 is software for analyzing a golf swing based on image data sent from the distance image sensor 2, and causes the swing analysis device 1 to perform an operation described later.

  The swing analysis apparatus 1 includes a display unit 11, an input unit 12, a storage unit 13, a control unit 14, and a communication unit 15. These units 11 to 15 are connected to each other via the bus line 16 and can communicate with each other. The display unit 11 can be composed of a liquid crystal display or the like, and displays the result of golf swing analysis or the like to the user. In addition, a user here is a general term for those who require the result of golf swing analysis, such as golfer 7 itself, its instructor, and a developer of golf equipment. The input unit 12 can be configured with a mouse, a keyboard, a touch panel, and the like, and accepts an operation from the user for the swing analysis apparatus 1.

  The storage unit 13 can be configured with a hard disk or the like. In the storage unit 13, the swing analysis program 3 is stored, and image data sent from the distance image sensor 2 is stored. The control part 14 can be comprised from CPU, ROM, RAM, etc. The control unit 14 virtually operates as an acquisition unit 14a, a position deriving unit 14b, and a direction deriving unit 14c by reading and executing the swing analysis program 3 in the storage unit 13. Details of the operations of the units 14a to 14c will be described later. The communication unit 15 functions as a communication interface that receives data from an external device such as the distance image sensor 2 via the communication line 17.

<1-3. Swing analysis method>
Hereinafter, a golf swing analysis method executed by the swing analysis system 100 will be described. In this analysis method, first, the golfer 7 makes a trial hit with the golf club 5, and the state is photographed by the distance image sensor 2. The image data of the IR image and the depth image captured by the distance image sensor 2 is sent from the distance image sensor 2 to the swing analysis apparatus 1. As already described, in this embodiment, the swing analysis device 1 causes the behavior of the grip 51 of the golf club 5 during the swing operation and the behavior of the left shoulder 71 of the golfer 7 and the orientation of the shaft 52 of the golf club 5 during the swing operation. Are analyzed. In the following, a process for deriving the behavior of the grip 51 (hereinafter referred to as grip behavior deriving process) will be described first, and then a process for deriving the behavior of the left shoulder 71 of the golfer 7 (hereinafter referred to as left shoulder behavior deriving process) will be described. Thereafter, processing for deriving the direction of the shaft 52 of the golf club 5 (hereinafter, shaft direction deriving processing) will be described.

<1-3-1. Grip behavior derivation process>
In the grip behavior deriving process, three-dimensional coordinates representing the trajectory of the grip end 51a during the swing operation are derived based on the IR image and depth image data captured by the distance image sensor 2. The image data to be analyzed is a moving image. Therefore, hereinafter, the IR image and the depth image to be analyzed may be referred to as an IR frame and a depth frame, respectively, and may be simply referred to as a frame. The three-dimensional coordinates at each timing during the swing motion are derived mainly based on the IR frame and the depth frame at that timing.

  FIG. 3 is a flowchart showing the overall flow of the grip behavior derivation process. In the grip behavior derivation process, first, the acquisition unit 14a reads the image data acquired from the distance image sensor 2 and stored in the storage unit 13, and develops it in the memory. Then, the position deriving unit 14b determines a frame to be analyzed among these image data (step S1). In the present embodiment, a period from the address during the swing operation to a little later than the impact is an analysis target. Therefore, in step S1, the address and impact timing are determined, and the frames in the meantime and several frames after the impact are determined as objects to be analyzed. Here, the top timing is also determined in preparation for later processing. The timing of the address, top and impact can be determined based on output data from an inertial sensor attached to the grip 51, for example, or can be determined by image processing from the image data itself. Various methods for determining the address, top, and impact timing are known, and detailed description thereof is omitted here.

  Note that the golf club swing operation generally proceeds in the order of address, top, impact, and finish. The address means an initial state in which the head of the golf club 5 is arranged near the ball, as shown in FIG. 4A, and the top means the golf club 5 from the address as shown in FIG. It means a state where the head is swung up and the head is swung up most. As shown in FIG. 4C, the impact means a state at the moment when the golf club 5 is swung down from the top and the head collides with the ball. The finish means the impact as shown in FIG. After that, the golf club 5 is swung forward.

  When the period to be analyzed is determined in step S1, the position deriving unit 14b derives the three-dimensional coordinates of the grip end 51a near the address (step S2). In this embodiment, since image data is taken from the front of the golfer 7, it is hidden near the golfer 7 near the address, and the grip end 51a is not shown in the frame. Therefore, in step S2, the position of the grip end 51a is estimated from the position of the shaft 52 shown in the IR frame.

  Specifically, the analysis target in step S2 is a frame in the initial period, for example, three frames near the address. As described above, since the marker that efficiently reflects the infrared rays is attached to the shaft 52, an image of the marker clearly appears on the IR frame. The position deriving unit 14b binarizes each IR frame using an appropriate threshold value, thereby dividing each IR frame into an image of the marker of the shaft 52 and other regions, and specifying the two-dimensional coordinates of the shaft 52. To do. The threshold value may be a preset value or a value calculated from an IR frame by a discriminant analysis method or the like. Subsequently, the position deriving unit 14b specifies the depth value corresponding to the two-dimensional coordinate specified here from the depth frame at the same timing. Thereby, the three-dimensional coordinates of the shaft 52 are derived. By the way, the length of the marker attached to the shaft 52 and the distance from the center of the marker (for example, the geometric center of gravity) to the grip end 51a are known, and these values are stored in the storage unit 13 in advance. Yes. Therefore, the position deriving unit 14b determines the three-dimensional shape of the grip end 51a near the address based on these lengths and distances, the marker length of the shaft 52 reflected in the IR frame, and the three-dimensional coordinates of the marker center. Deriving coordinates.

  When step S2 ends, the three-dimensional coordinates of the grip end 51a after the initial period are specified by steps S3 to S7. Steps S3 to S7 are repeatedly executed in order along the time axis for each frame included in the analysis period determined in step S1 except for the frame in the initial period of step S2.

  Specifically, first, in step S3, the position deriving unit 14b extracts the image of the grip end 51a that appears in the IR frame, thereby specifying the two-dimensional coordinates of the grip end 51a directly from the IR frame. When the two-dimensional coordinates are successfully identified (“YES” in step S4), the position deriving unit 14b directly identifies the depth value corresponding to the two-dimensional coordinates from the depth frame (step S4). S5). Thereby, the three-dimensional coordinate of the grip end 51a is derived. On the other hand, if the specification of the two-dimensional coordinates fails in step S3 (“NO” in step S4), the position deriving unit 14b uses an exception for deriving the three-dimensional coordinates of the grip end 51a by another method. Processing is executed (step S6). Similarly, even when the specification of the depth value fails in step S5 (“NO” in step S7), the position deriving unit 14b executes the same exception process. Details of the processes of steps S3, S5, and S6 will be described later.

  When the above processing for the period to be analyzed determined in step S1 is completed, the three-dimensional coordinates of the grip end 51a during the swing motion are derived. The position deriving unit 14b performs calibration in step S8 described later on the three-dimensional coordinates of the grip end 51a. In addition, these post-calibration values may still contain some degree of error. In particular, when an image of the grip end 51a is not captured in the IR frame and the three-dimensional coordinates of the grip end 51a are estimated by various methods to be described later, an error may appear significantly. Therefore, the position deriving unit 14b corrects these three-dimensional coordinates after calibration by a correction process (step S9). Details of this correction processing will be described later.

  Through the above processing, the three-dimensional coordinates of the grip end 51a during the swing operation are derived with high accuracy. The three-dimensional coordinates of these grip ends 51a are stored in the storage unit 13 and are appropriately displayed on the display unit 11 in accordance with a user command. For example, the trajectory of the grip end 51a during the swing operation can be displayed graphically. Further, the three-dimensional coordinates of these grip ends 51a stored in the storage unit 13 can be read out by another program and used for further golf swing analysis.

<1-3-1-1. Details of Step S3>
Hereinafter, the details of step S3 will be described with reference to FIG. Step S3 is a process of specifying the two-dimensional coordinates of the grip end 51a directly from the IR frame by extracting the image of the grip end 51a reflected in the IR frame. As described above, step S3 is repeatedly executed for each frame after the initial period to be analyzed in step S2. Therefore, in the following description of step S3, the flow of processing for a specific frame (hereinafter referred to as a target frame) at a specific time (hereinafter referred to as a target time) after the initial period will be described. The target frame includes an IR frame and a depth frame, which are referred to as a target IR frame and a target depth frame, respectively.

  First, the position deriving unit 14b sets a search range R1 smaller than the image size of the target frame on the target frame in order to reduce the subsequent calculation load (step S21). Specifically, the two-dimensional coordinates of the grip end 51a at the target time are estimated from the two-dimensional coordinates of the grip end 51a in a predetermined number of frames (for example, three) immediately before the target time. This estimation can be performed based on, for example, the direction of the vector between frames of the grip end 51a of the predetermined number of frames immediately before. Then, for example, a search range R1 having a predetermined size (for example, 50 pixels × 50 pixels) around the two-dimensional coordinates is set around the estimated two-dimensional coordinates.

  In subsequent steps S22 to S24, the position deriving unit 14b binarizes the target IR frame for separating the region representing the marker attached to the grip end 51a and the shaft 52 from other regions within the search range R1. Do. For this purpose, first, in step S22, a threshold value for binarization is set. The threshold value can be calculated by various known methods. In this embodiment, the threshold value is calculated by a discriminant analysis method. Alternatively, the threshold value may be a preset value. Subsequently, in step S23, whether or not the threshold is appropriate is determined. If it is determined that the threshold is appropriate, the process proceeds to step S24. In step S24, binarization of the target IR frame within the search range R1 is performed using the threshold value. At this time, it is preferable to perform an expansion process and a reduction process in order to prevent the “1” area (area corresponding to the marker) from being lost. FIG. 6 shows an example of a target IR frame as a reference. FIG. 6 shows an image of the marker attached to the shaft 52 in addition to the grip end 51a. On the other hand, if it is determined in step S23 that the threshold value is inappropriate, step S3 ends, and then the process proceeds to the exception process in step S6 through step S4 described above. In step S23 according to the present embodiment, in the search range R1 in the target IR frame, when the threshold value is larger than 0.1 when the range that the brightness can take is normalized to 0 to 1, “Appropriate”, otherwise, it is determined as “inappropriate”. In addition to this, it is possible to determine “appropriate” when the maximum frequency of luminance that is equal to or higher than the threshold is equal to or higher than a predetermined value (for example, 2 pixels), and “inappropriate” otherwise. .

  Subsequently, in step S25, the position deriving unit 14b does not include the grip end 51a in the search range R1 in the target IR frame or protrudes from the search range R1 (overlaps the outer frame of the search range R1). ) Or not. If it is determined that the grip end 51a is in the search range R1 and does not protrude (hereinafter, condition 1), it is determined that the search range R1 is appropriate, and the process proceeds to step S26. On the other hand, when it is determined that the condition 1 is not satisfied, it is determined that the search range R1 is inappropriate, the process proceeds to step S20, and the search range R1 is reset. In step S20, a search range R1 that is larger than the previous search range R1 by a predetermined amount is set, and then step S22 and subsequent steps are repeated. If it is determined that the image is protruding, the image can be enlarged by a predetermined amount (for example, 10 pixels) only in the protruding direction. Although not clear from FIG. 5, in the present embodiment, there is an upper limit for the number of repetitions of step S20, and when the grip end 51a does not appear within the search range R1 even if the upper limit is repeated. Step S3 is forcibly ended. If the upper limit is repeated, the process proceeds to step S26 forcibly.

  In step S26, when the image of the marker of the shaft 52 is included in the search range R1, the position deriving unit 14b removes this. Specifically, predetermined three sides of the outer frame of the search range R1 are examined, and if a “1” region exists, it is replaced with “0”. Thereby, the “1” region in the search range R1 is a region corresponding only to the grip end 51a. Since it is unlikely that the shaft will come to the left side of the grip 51 in the section from the address to a little after the impact, the upper and lower sides and the right side of the grip 51 are set as the predetermined three sides. .

  In subsequent step S27, the position deriving unit 14b performs a labeling process, and determines whether or not the image of the grip end 51a in the search range R1 is separated into a plurality of regions. As shown in FIG. 7A, in the target IR frame, particularly during the downswing, the image of the grip end 51a may appear long due to motion blur. When this is binarized by the above method, as shown in FIG. 7B, the “1” region representing the grip end 51a can be separated into a plurality of regions. Therefore, when it is determined that the image of the grip end 51a in the search range R1 is separated into a plurality of regions, the position deriving unit 14b combines them as shown in FIG. Step S28). Specifically, closer end points of a plurality of “1” regions are connected by a line “1”, and the inner hole is complemented by “1”.

  In step S29, the position deriving unit 14b calculates the center of the “1” region within the search range R1. The center of the “1” region can be calculated by various methods. For example, it can be calculated as a geometric center of gravity. Then, the two-dimensional coordinates of the center are specified as the two-dimensional coordinates of the grip end 51a.

<1-3-1-2. Details of Step S5>
Next, the details of step S5 will be described with reference to FIG. Step S5 is a process for directly specifying the depth value of the grip end 51a that has been successfully extracted in step S3 from the depth frame. As described above, step S5 is repeatedly executed for each frame after the initial period to be analyzed in step S2, as in step S3. Therefore, in the description of step S5 below, the flow of processing for the target frame will be described as in step S3.

  First, in step S31, the position deriving unit 14b extracts a depth value at a position corresponding to the two-dimensional coordinates of the grip end 51a specified in step S3 from the target depth frame. FIG. 9 is a diagram in which the two-dimensional coordinate position of the grip end 51a specified in step S3 is plotted on the target depth frame.

  In subsequent step S32, the position deriving unit 14b determines whether the depth value extracted in step S31 is appropriate. If it is determined to be appropriate, it is determined that the depth value of the grip end 51a has been successfully specified, and the process proceeds to step S33 and subsequent steps. On the other hand, if it is determined in step S32 that the depth value extracted in step S31 is inappropriate, it is determined that the specification of the depth value of the grip end 51a has failed, and step S5 is ended. In step S32 according to the present embodiment, “appropriate” is obtained when the depth value extracted in step S31 is within a predetermined range, for example, within a range of 2000 mm to 3500 mm, and “inappropriate” otherwise. It is judged. Further, if data is missing in the first place and no depth value is extracted in step S31, it is determined as “inappropriate”.

  In step S33, the position deriving unit 14b specifies the two-dimensional coordinates of the center of the shaft 52 from the target IR frame, and extracts the value of the depth at the position corresponding to the two-dimensional coordinates from the target depth frame. Thereby, the three-dimensional coordinates of the center of the shaft 52 are derived. Specifically, the target IR frame is binarized in the same manner as in step S24, and the image of the grip end 51a already extracted from the target IR frame is removed, whereby the image of the marker of the shaft 52 on the target IR frame. The “1” area to be extracted is extracted. Then, the two-dimensional coordinates of the center of the “1” region (for example, the geometric center of gravity) are specified as the two-dimensional coordinates of the center of the shaft 52. Although omitted from FIG. 8 and the like for the sake of simplicity, if the derivation of the three-dimensional coordinates of the center of the shaft 52 fails here, step S5 is forcibly terminated, and step S6 is followed by step S6. Let's proceed to exception handling.

  In subsequent step S34, the position deriving unit 14b calculates the angle of the shaft 52 on the target IR frame based on the two-dimensional coordinates of the grip end 51a and the two-dimensional coordinates of the center of the shaft 52. This value is used in step S35 described later.

In subsequent steps S35 to S37, the position deriving unit 14b determines whether or not the depth value of the grip end 51a should be taken again. The conditions for re-taking the depth value of the grip end 51a are as follows:
The following are three.
(1) The angle of the shaft 52 is within a predetermined range (for example, before 11 o'clock when viewed from the front of the golfer) (step S35).
(2) The difference in the depth value of the grip end 51a between the previous frame and the target frame is equal to or greater than a predetermined value (for example, 50 mm) (step S36).
(3) A predetermined number (for example, three) or more of frames in which the depth value of the grip end 51a can be directly extracted from the depth frame irrespective of the exception processing in step S6 continues (step S37).

  If any of the conditions in steps S35 to S37 is not satisfied, the value of the depth of the grip end 51a specified in step S31 is determined, and step S5 is ended. On the other hand, when all the conditions in steps S35 to S37 are satisfied, the position deriving unit 14b determines that the depth value of the grip end 51a specified in step S31 is not the grip end 51a but the human body depth value. Then, the value of the depth of the grip end 51a is read again (step S38).

  Specifically, in step S38, a search range R2 having a predetermined size (for example, 10 pixels × 10 pixels) is set around the two-dimensional coordinates of the grip end 51a in the target depth frame (see FIG. 10). Further, the position deriving unit 14b subtracts a predetermined value (for example, 50 mm) from the depth value of the grip end 51a specified in step S31 (moves to the near side), and within the search range R2 based on the target depth frame. It is determined that everything behind the subtracted value is a human body. Then, the depth value corresponding to all the points in the region excluding the human image in the search range R2 (considered as the grip end 51a and the finger image) is specified, and the maximum value (the deepest value) is gripped. It is set as the depth value of the end 51a. FIG. 11 is a graph showing before and after the correction of the depth value in step S38. The graph on the left is a graph in which the values in step S31 are plotted. In the graph on the right, the positions of two points surrounded by a thick circle are corrected.

<1-3-1-3. Details of Step S6 (Exception Processing)>
Hereinafter, the exception processing in step S6 will be described with reference to FIG. The exception process is a process for deriving the three-dimensional coordinates of the grip end 51a by another method when the derivation of the three-dimensional coordinates of the grip end 51a fails in steps S3 and S5.

  First, in step S41, the position deriving unit 14b extracts a marker image of the shaft 52 from the target IR frame. Specifically, the target IR frame is binarized by the same method as in step S24, and the image of the grip end 51a is appropriately removed therefrom, whereby the image of the marker of the shaft 52 on the target IR frame is extracted. If the image of the shaft 52 has been extracted, the process proceeds to step S42. If the image has not been extracted, the process proceeds to step S48.

  In step S42, the position deriving unit 14b specifies the two-dimensional coordinates of all points included in the region of the marker image of the shaft 52 extracted in step S41. In addition, the position deriving unit 14b specifies the depth value corresponding to the two-dimensional coordinates specified here from the target depth frame. Thereby, the three-dimensional coordinates of the marker of the shaft 52 are derived.

  Subsequently, the position deriving unit 14b estimates the two-dimensional coordinates of the grip end 51a based on the two-dimensional coordinates of the marker of the shaft 52 (step S43). Specifically, the two-dimensional coordinates of the grip end 51a can be obtained by referring to the marker length of the shaft 52 and the distance from the center of the marker to the grip end 51a stored in the storage unit 13. It is specified based on the length of the marker of the shaft 52 reflected in the IR frame and the two-dimensional coordinates of the center of the marker.

In subsequent step S44, the position deriving unit 14b determines whether or not the depth value of the shaft 52 extracted in step S42 is appropriate, and if it is determined to be appropriate, the process proceeds to step S45. On the other hand, if it is determined in step S44 that the depth value extracted in step S42 is inappropriate, the process proceeds to step S46. Note that in step S44 according to the present embodiment, it is determined as “appropriate” when the following two conditions are satisfied, and “inappropriate” when at least one of the following conditions is not satisfied. The condition (2) is a condition for checking whether or not the marker image of the shaft 52 has been correctly extracted in step S41, that is, whether or not data such as a leg has been extracted.
(1) The value of the depth of the center of the shaft 52 is within a predetermined range (for example, 1000 mm to 3500 mm).
(2) Ratio of depth value “0” (pixel value when depth value is not obtained) among all points in the region corresponding to the marker image of shaft 52 on the target depth frame Is below a predetermined value (for example, 40%).

  In step S45, the position deriving unit 14b estimates the three-dimensional coordinates of the grip end 51a from the three-dimensional coordinates of the marker of the shaft 52. In the present embodiment, first, the inertia principal axis A1 of the shaft 52 is extracted from the marker image of the shaft 52 on the target IR frame, and the depth value of each point on the inertia principal axis A1 is acquired from the target depth frame (FIG. 13). Further, outliers are excluded from these depth values. The outlier is a value outside a predetermined range (for example, 1000 mm or less, or 3500 mm or more), or a value greatly deviating from the reference value (for example, a value different by 300 mm or more). The reference value is, for example, the minimum value of the depth of the point on the line ab before the angle of the shaft 52 before 9 o'clock, and the maximum value of the depth of the point on the line ab after the angle of the shaft 52 is 9 o'clock. Can be set. Note that point a is the point on the inertial principal axis A1 closest to the grip end 51a in the target IR frame (excluding outliers), and point b is the inertia farthest from the grip end 51a in the target IR frame. It is a point on the main axis A1 (excluding outliers) (see FIG. 13). Subsequently, in the target IR frame, the distance from the point b to each point on the inertial main axis A1 (excluding outliers) is calculated. Then, the distance and depth from the point b for each point (excluding outliers) on the inertial principal axis A1 are plotted in a distance-depth plane (see FIG. 14), and a primary representing the relationship between the distance and the depth. An approximate expression is derived. Subsequently, the distance from the point b in the target IR frame to the grip end 51a is calculated, and the depth of the grip end 51a is calculated by substituting this distance into the primary approximation formula. Thereby, the three-dimensional coordinate of the grip end 51a is derived. When step S45 ends, the exception processing ends.

The inertia main axis A1 is specified by the following method. That is, first, an ellipse having the same second moment as the marker image of the shaft 52 is defined. Then, the length of the major axis of the ellipse is L, and the angle θ formed by the major axis of the ellipse and the x axis (the horizontal axis of the target frame) is specified. Further, the coordinates (x ₁ , y ₁ ) of the center of gravity of the marker on the shaft 52 are derived. At this time, the coordinates of the start point (x _s , y _s ) and the end point (x _e , y _e ) of the inertial spindle A1 are as follows.
x _s = x ₁ −L · cos θ / 2
y _s = y ₁ −L · sin θ / 2
x _e = x ₁ + L · cos θ / 2
y _e = x ₁ + L · cos θ / 2

In the following S47, the three-dimensional coordinates of the grip end 51a are estimated based on the depth value of the lower point of the grip 51 instead of the grip end 51a. Step S46, which is executed before step S47, is a step for selecting a target frame in which the estimation algorithm of step S47 functions effectively. Specifically, if it is determined in step S46 that the following two conditions are satisfied, the process proceeds to step S47, and if it is determined that at least one of the conditions is not satisfied, the process proceeds to step S48.
(1) The target frame is an address or a frame near the top (for example, five frames before and after).
(2) There are one or more points having a depth other than an outlier on the a-c line.

  Note that the point c is a point separated from the point a in the direction of the grip end 51a by a predetermined distance (for example, 20 pixels) on the line connecting the point a and the grip end 51a in the target IR frame. . The outlier in (2) is a value outside a predetermined range (for example, 1000 mm or less, or 3500 mm or more), or a value greatly deviating from the reference value (for example, a value different by 150 mm or more). The reference value can also be set in the same manner as described in step S45.

  Details of step S47 are shown in FIG. That is, first, as step S471, the position deriving unit 14b derives a first-order approximation expression representing the distribution of points on the a-c line. Specifically, in the target IR frame, a point on the a-c line (excluding outliers) farthest from the grip end 51a is used as a reference point, and each point on the ac line from the reference point (excluding outliers is excluded). ) Is calculated. Then, a linear approximate expression representing the relationship between the distance from the reference point and the depth of each point on the a-c line (excluding outliers) is derived.

  Subsequently, in step S472, the position deriving unit 14b determines whether or not the slope of the primary approximation calculated in step S471 is 0, and if not, the process proceeds to step S473. In step S473, the position deriving unit 14b calculates the distance from the reference point to the grip end 51a in the target IR frame, and calculates the depth of the grip end 51a by substituting this distance into the primary approximation formula. . Thereby, the three-dimensional coordinate of the grip end 51a is derived. When step S473 ends, the exception processing ends.

On the other hand, if it is determined in step S472 that the slope of the first-order approximation equation is 0, the position deriving unit 14b determines the depth at the target time from the sequence of movement of the grip end 51a in the immediately preceding three frames. Is estimated (step S474). For example, the depth D at the target time can be determined according to the following equation. However, D _{1 to} D ₃ are the depths D of the previous frame to the previous three frames, respectively. Thereby, the three-dimensional coordinate of the grip end 51a is derived. When step 474 ends, the exception processing ends.
_{D = 2 (D 1 -D 2} ) - (D 2 -D 3) + D 1

  Returning to FIG. 12, in step S48, the position deriving unit 14b determines whether or not the extraction of the grip end 51a has failed for five consecutive frames. If it is determined that five frames have not failed, the process proceeds to step S49, and the position of the grip end 51a at the target time is estimated from the movement sequence of the grip end 51a in the immediately preceding three frames. More specifically, when the two-dimensional coordinates of the grip end 51a are specified, only the depth is estimated as in step S474, and when the two-dimensional coordinates of the grip end 51a are not specified, the grip The three-dimensional coordinates of the end 51a are estimated. On the other hand, if it is determined in step S48 that the number of consecutive failures is 5 frames or more, the position deriving unit 14b ends the exception process without determining the position of the grip end 51a of the target frame. In step S49, the two-dimensional coordinates of the grip end 51a are specified, and the distance between the two-dimensional coordinates of the grip end 51a in the target frame and the two-dimensional coordinates of the grip end 51a in the immediately preceding frame. Is small (for example, 5 pixels or less), the depth value of the immediately preceding frame can be used as it is as the depth value of the target frame.

<1-3-1-4. Calibration>
Hereinafter, the calibration of the three-dimensional coordinates of the grip end 51a derived by the above processing will be described. This calibration is performed by substituting the three-dimensional coordinates of the grip end 51a derived by the above processing into the following multiple regression equation.

In the above multiple regression equations, x, y, and Z are the three-dimensional coordinates of the grip end 51a before calibration, and X, Y, and Z ′ are the three-dimensional coordinates of the grip end 51a after calibration. Z and Z ′ are depths. The coefficients a _{1 to} c ₁ , a _{2 to} f ₂ , and a _{3 to} f ₃ are values determined in advance by experiments. Hereinafter, a method for determining the coefficients a _{1 to} c ₁ , a _{2 to} f ₂ , and a _{3 to} f ₃ will be described.

  First, an XYZ ′ coordinate system that is a three-dimensional coordinate system for defining the three-dimensional coordinates after calibration is defined. Then, various test subjects are arranged in front of the distance image sensor 2 and photographed. Next, the position deriving unit 14b calculates the three-dimensional coordinates (x, y, Z) of each test subject by the same processing as in the above-described steps S3 and S5 (S29, S31). Further, the actual position of each test subject is measured and expressed as three-dimensional coordinates (X, Y, Z), and these values are input to the swing analysis apparatus 1. As a result, the position deriving unit 14b acquires the three-dimensional coordinates (x, y, Z) and the three-dimensional coordinates (X, Y, Z) of various test subjects. Therefore, subsequently, the position deriving unit 14b derives a multiple regression equation representing the relationship between (x, y, Z) and (X, Y, Z ′) using these data as sample data. The multiple regression equation of Equation 1 is a non-linear model, but it can also be expressed by a linear model, and the explanatory variables of X, Y, and Z ′ can be arbitrarily set by appropriately combining x, y, and Z. Can do.

  The XYZ ′ coordinate system sets, for example, the ball as the origin, the optical axis direction of the distance image sensor 2 representing the depth as the Z ′ axis, the vertical vertical direction as the Y axis, and the horizontal direction as the X axis. be able to. A variety of test subjects can be used, but a plate with markers that efficiently reflect infrared rays in a lattice shape as shown in FIG. 17 is prepared (a rectangular intersection surrounded by lattice intersections and lattices). The center of this area can function as a test subject), and this can be arranged at various positions in the Z′-axis direction and photographed to efficiently perform multiple regression analysis.

<1-3-1-5. Correction processing>
Hereinafter, the correction process in step S9 will be described with reference to FIG. This correction process is a process for correcting the three-dimensional coordinates of the grip end 51a derived by the above process. As described above, an error may appear remarkably in a frame in which the image of the grip end 51a is not shown in the IR frame and the three-dimensional coordinates of the grip end 51a are estimated based on the position of the shaft 52 or the like. . Since such an error is likely to occur particularly in the vicinity of the address, in the present embodiment, the following steps S53 to S57 relating to the present correction process are executed for a frame near the address. However, steps S53 to S57 can be executed not only in the vicinity of the address but also in a frame of an arbitrary period. In the present embodiment, the three-dimensional coordinates after calibration in step S8 are to be corrected, but the three-dimensional coordinates before calibration may be corrected.

  First, in step S51, the position deriving unit 14b first identifies a frame in which the grip end 51a can be extracted on the IR frame (a frame that first becomes “YES” in step S7, hereinafter referred to as the last frame). To do.

  In the subsequent step S52, the position deriving unit 14b, among the frames up to the last frame, the frame in which the grip end 51a could not be extracted on the IR frame (the frame in which “NO” was determined in step S4 or S7). And all the frames from which the shaft 52 is extracted (hereinafter referred to as correction target frames) are specified. The correction target frame includes an IR frame and a depth frame, which are referred to as a correction target IR frame and a correction target depth frame, respectively.

  The subsequent steps S53 to S57 are repeatedly executed for each correction target frame. Specifically, in step S53, the position deriving unit 14b extracts a marker image of the shaft 52 from the correction target IR frame. The specific method is the same as step S41, and the result of step S41 can be appropriately used.

  In subsequent step S54, the position deriving unit 14b specifies the two-dimensional coordinates of all the points included in the marker image area of the shaft 52 extracted in step S53. Further, the position deriving unit 14b specifies a depth value corresponding to the two-dimensional coordinate specified here from the correction target depth frame. Thereby, the three-dimensional coordinates of the marker of the shaft 52 are derived. Also in these processes, the result of step S42 can be used.

  In subsequent step S55, the position deriving unit 14b specifies all the points having appropriate depth values (hereinafter referred to as target points) among all the points extracted in step S53. Whether or not the depth value is appropriate is determined depending on, for example, whether or not the depth value is within a predetermined range (for example, 1000 mm to 3500 mm).

In subsequent step S56, the position deriving unit 14b performs principal component analysis on the three-dimensional coordinates (X, Y, Z ′) of all the target points to derive the first principal component. Specifically, the covariance matrix Σ of the three-dimensional coordinates (X, Y, Z ′) of all target points is calculated. Subsequently, the eigen equation of the covariance matrix Σ is solved to calculate _three eigenvalues λ ₁ , λ ₂ , λ ₃ and corresponding eigenvectors ω ₁ , ω ₂ , ω ₃ (λ ₁ ≧ λ ₂ ≧ λ _3). And). Then, the eigenvector ω ₁ (first principal component) corresponding to the maximum eigenvalue λ ₁ is specified.

In the subsequent step S57, the along the first principal component omega ₁ on the assumption that the shaft 52 is extended, three-dimensional coordinates of the grip end 51a is specified (see FIG. 19). Specifically, the position deriving unit 14b calculates a three-dimensional distance L from the center of the marker of the shaft 52 to the grip end 51a in the XYZ ′ coordinate system. The distance L is calculated by referring to the length of the marker of the shaft 52 and the value of the distance from the center of the marker to the grip end 51a, which are stored in the storage unit 13. Then, on a straight line (that is, a straight line representing the shaft 52) along the first principal component ω ₁ , the three-dimensional coordinates (X, Y, Z ′) of a point separated from the center of the marker of the shaft 52 by the distance L are obtained. The three-dimensional coordinates of the grip end 51a are specified.

  When the above steps S53 to S57 are completed for all the correction target frames, the correction process ends. FIG. 20 is a diagram illustrating a result of actual correction processing. The “before correction” graph in FIG. 20 is the Z ′ value (depth) of the grip end 51a derived in steps S1 to S8, and the “after correction” graph is the grip end 51a after this correction processing. This is the Z ′ value (depth). The “true value” is a 3D motion analysis system (actually manufactured by Interliha) that can read the 3D coordinates of the marker with high accuracy based on images taken from various angles by a large number of infrared cameras. This is the position of the grip end 51a measured by the other VICON (of course, other three-dimensional motion analysis systems are also available). FIG. 20 shows that the depth coordinate of the grip end 51a can be brought closer to the true value by this correction processing.

<1-3-2. Left shoulder behavior derivation process>
Hereinafter, the left shoulder behavior derivation process will be described. In the left shoulder behavior deriving process, three-dimensional coordinates representing the trajectory of the left shoulder 71 of the golfer 7 during the swing motion are derived based on the depth image captured by the distance image sensor 2. As in the case of the grip behavior derivation process, the image data to be analyzed is a moving image. Accordingly, the depth image to be analyzed is also referred to as a depth frame here. The three-dimensional coordinates at each timing during the swing motion are derived mainly based on the depth frame at that timing.

  FIG. 21 is a flowchart showing the overall flow of the left shoulder behavior derivation process. In the left shoulder behavior derivation process, first, the acquisition unit 14a reads the depth frame at each timing during the swing motion acquired from the distance image sensor 2 and stored in the storage unit 13, and expands it in the memory. Then, the acquisition unit 14a acquires skeleton data at each timing from these depth frames (step S61). Skeleton data is data representing the positions (three-dimensional coordinates) of major joints of the human body. Since various methods for deriving skeleton data from the depth frame are known, detailed description thereof is omitted here. In addition, Kinect (registered trademark) (which may be v1 or v2; the same applies hereinafter), which is a distance image sensor, provides a library for reading out skeleton data from a depth image. In the embodiment, skeleton data is acquired using the library. Note that the CPU 23 of the distance image sensor 2 may derive skeleton data from the depth frame. In this case, the acquisition unit 14a may simply read the skeleton data sent from the distance image sensor 2.

  Since the skeleton data includes the three-dimensional coordinates of the left shoulder 71, it can be said that step S61 is a step of deriving the behavior of the left shoulder 71. However, although the position of the left shoulder 71 specified by the skeleton data derived from Kinect depends on the purpose of use, it does not always maintain sufficient accuracy in the entire section during the snug operation. For this reason, in the present embodiment, the three-dimensional coordinates of the left shoulder 71 are derived also by a convex extraction filter and a left shoulder silhouette filter, which will be described later, and these are switched as appropriate. As a result, the three-dimensional coordinates of the left shoulder 71 can be derived with high accuracy in various sections. Although depending on the purpose of use, the position of the left shoulder 71 can be specified only from the skeleton data, or the position of the left shoulder 71 can be specified from only the convex portion extraction filter or only the left shoulder silhouette filter. That is, these three methods can be used singly or in combination with any one of them depending on the purpose of use.

  Subsequently, in step S62, the position deriving unit 14b determines a frame to be analyzed from the depth frames acquired by the acquiring unit 14a (step S62). In the present embodiment, the period from the address during the swing operation to the finish is the object of analysis. Accordingly, in step S62, the address and finish timing are determined, and the depth frame in between is determined as an analysis target according to subsequent steps S63-69. The address and finish timing can be determined based on output data from an inertial sensor attached to the grip 51, for example, or can be determined from image data itself by image processing. Since various methods for determining the address and finish timing are known, detailed description is omitted here.

  In subsequent step S63, the position deriving unit 14b reduces the image size of each depth frame. Various compression algorithms can be used. For example, bicubic or nearest neighbor can be used. This compression processing can reduce the following calculation load. Further, if compression is performed so that the reduced image size becomes a constant value, it is not necessary to generate various filters described later according to the image size, and the processing becomes efficient.

  In subsequent step S64, the position deriving unit 14b binarizes each depth frame. This binarization is performed by setting a pixel value in which depth data exists in an area in which a person is captured in a depth frame (for example, a depth of 2200 mm to 2800 mm) to “1”, and other values to “0”. Done. This binarized image is an image obtained by extracting silhouettes of subjects (mainly, the golfer 7 and the golf club 5) existing within the range of the effective distance of the distance image sensor 2. The use of the binarized image will be described later.

  In subsequent step S65, the position deriving unit 14b derives the three-dimensional coordinates of the left shoulder 71 from each depth frame using the convex extraction filter. In step S66, the position deriving unit 14b derives the three-dimensional coordinates of the left shoulder 71 from each depth frame using the left shoulder silhouette filter. Details of the extraction process of the left shoulder 71 using these filters will be described later.

  As described above, when the three-dimensional coordinates of the left shoulder 71 are derived by the three methods (the method using the skeleton data, the convex portion extraction filter, and the left shoulder silhouette filter), the position deriving unit 14b derives the timing for switching the three methods (step S67). As a result of intensive studies, the present inventors have found that the accuracy of the method based on the skeleton data is particularly high in the section from the address to the takeback arm level during the swing operation, and in the section from the takeback arm level to the downswing arm level. It was found that the accuracy of the method using the partial extraction filter is particularly high, and the accuracy of the method using the left shoulder silhouette filter is particularly high in the section from the horizontal to the finish of the downswing arm. This is presumably because in the skeleton data derived from Kinect (registered trademark), the torsional motion of the shoulder is not sufficiently taken into account, and therefore the accuracy can be lowered when the left shoulder 71 starts to rotate after the takeback arm is level. That is, in the Kinect (registered trademark) skeleton model, both shoulders and waist (both hip joints) are modeled as rigid bodies, and therefore, rotation of only the shoulder or only the waist is not evaluated. Therefore, after the takeback arm level where only the shoulder rotates, the reliability of the skeleton data is likely to be lowered, so that the switching according to the present embodiment is effective. Further, when the golfer 7 is photographed from the front, the left shoulder 71 exists in the entire silhouette of the golfer 7 in the section from the takeback arm level to the downswing arm level. Then, although details will be described later, since the convex portion extraction filter is a filter that emphasizes the convex portion in the depth frame, the left shoulder 71 can be accurately detected by the method using the convex portion extraction filter. On the other hand, when the golfer 7 is photographed from the front, the left shoulder 71 exists in the vicinity of the contour of the entire silhouette of the golfer 7 in the section from the level of the downswing arm to the finish. Then, although details will be described later, the left shoulder 71 can be accurately detected by the method using the left shoulder silhouette filter for enhancing the contour of the left shoulder 71. For the same reason, the left shoulder silhouette filter can be preferably used even in the horizontal section from the address to the takeback arm.

  As described above, in this embodiment, the takeback arm horizontal and downswing arm horizontal timings are derived as the switching timing. Details of the determination method of the arm horizontal timing will be described later. The takeback arm level refers to the timing when the left arm becomes horizontal during takeback, and the downswing arm level refers to the timing when the left arm becomes horizontal during the downswing.

  When the arm horizontal timing is derived, the position deriving unit 14b switches the three-dimensional coordinates of the left shoulder 71 according to the three methods along the time axis, and changes the trajectory of the left shoulder 71 during the swing motion in the three-dimensional space. Derived (step S68). Specifically, the three-dimensional coordinates of the left shoulder 71 derived from the skeleton data in the section from the address to the takeback arm level, and the left shoulder 71 derived from the convex extraction filter in the section from the takeback arm level to the downswing arm level. Are connected to the three-dimensional coordinates of the left shoulder 71 derived from the left shoulder silhouette filter in the section from the horizontal to the finish of the downswing arm, and the left shoulder 71 in the xyZ coordinate system from the address to the finish is connected. Is derived. Thereafter, the trajectory of the left shoulder 71 is calibrated by the same method as in step S8 (step S69). FIG. 22 is an example of the trajectory of the left shoulder 71 after calibration.

  Through the above processing, the three-dimensional coordinates of the left shoulder 71 during the swing motion are derived with high accuracy. The three-dimensional coordinates of the left shoulder 71 are stored in the storage unit 13 and appropriately displayed on the display unit 11 according to a user command. For example, the trajectory of the left shoulder 71 during the swing operation can be displayed graphically. Further, the three-dimensional coordinates of these left shoulders 71 stored in the storage unit 13 can be read out by another program and used for further golf swing analysis.

<1-3-2-1. Left shoulder extraction with convex extraction filter>
Hereinafter, a method for extracting the left shoulder 71 using the convex portion extraction filter will be described. The convex portion extraction filter is a filter that emphasizes the central portion and suppresses the peripheral portion in the region to which the filter is applied. That is, the convex portion extraction filter is a filter that emphasizes a convex portion that protrudes from the center and appears in the depth frame, for example, a filter as shown in FIG. Each square in FIG. 23 represents a pixel, and the numerical value in each square is a coefficient multiplied by the pixel value of the pixel corresponding to the square. Then, for each pixel in the filter, the pixel value of the pixel is multiplied by a corresponding coefficient, and the sum of these multiplied values is the pixel value of the center pixel after the filter is applied.

  FIG. 24 is a depth frame at the top timing. When the convex portion extraction filter of FIG. 23 is applied to the entire depth frame, the image shown in FIG. 25 is obtained. In the depth frame of FIG. 25, since the gap between the contour of the golfer 7 and the background is large, not only the convex left shoulder 71 but also the contour of the golfer 7 is emphasized. As a result, the left shoulder 71 is buried and there is a possibility that the position of the left shoulder 71 cannot be extracted correctly. Therefore, in this embodiment, the binarized image derived in step S64 is used to deal with such a problem. In addition, in the depth frame shown in the subsequent figures including FIG. 24 and FIG. 25, the golfer's image is reflected on the mirror surface.

  FIG. 26 is a binarized image that has undergone step S64. The region “1” mainly indicates the silhouette of the golfer 7, and the background region is the region “0”. In the present embodiment, the position deriving unit 14b contracts the binarized image by a predetermined amount in order to remove the contour of the golfer 7. The binarized image after contraction is shown in FIG. Then, the position deriving unit 14b masks the depth frame (see FIG. 25) after the convex portion extraction filter is applied, using the binary image after contraction as a mask image. FIG. 28 shows the frame after the masking. As shown in FIG. 28, by this masking, the contour of the golfer 7 is removed from the depth frame after the convex portion extraction filter is applied, and the left shoulder 71 can be easily extracted. In FIG. 28, the position of the left shoulder 71 is highlighted for reference. The position deriving unit 14b specifies the two-dimensional coordinate of the pixel having the maximum pixel value in the frame after masking, and sets it as the two-dimensional coordinate of the left shoulder 71. Further, the position deriving unit 14b derives the three-dimensional coordinates of the left shoulder 71 by extracting the depth data at the two-dimensional coordinates of the left shoulder 71 from the initial depth frame.

  In order to improve the accuracy and / or reduce the calculation load, the application range of the convex portion extraction filter may be set to a partial range instead of the entire depth frame. For example, an area of an appropriate size including the two-dimensional coordinates of the left shoulder 71 specified on the same or the previous frame can be set as the application range of the convex portion extraction filter. In addition, in the section from the address to the takeback arm horizontal section and the downswing arm horizontal section to the finish, a region having a predetermined size centered on the two-dimensional coordinates of the left shoulder 71 included in the skeleton data of the same frame is extracted as a convex portion. It is preferably set as the filter application range R3 (see FIG. 29). In addition, at the horizontal timing of the takeback arm, a region having a predetermined size centered on the two-dimensional coordinates of the left shoulder 71 included in the skeleton data of the previous frame is applied to the application range R3 of the convex portion extraction filter (see FIG. 29). ) Is preferably set. Further, in the section from immediately after the takeback arm level to the downswing arm level, the two-dimensional coordinates of the left shoulder 71 specified by using the convex portion extraction filter in the previous frame are included at a right-side position. It is preferable that the region be an application range R4 (see FIG. 30) of the convex portion extraction filter. The reason why the rectangular area extending to the left with reference to the left shoulder 71 is set as the application range R4 is to avoid a portion where the response value of the filter becomes very large on the right side (hand side). .

<1-3-2-2. Left shoulder extraction with left shoulder silhouette filter>
Hereinafter, a method for extracting the left shoulder 71 using the left shoulder silhouette filter will be described. The left shoulder silhouette filter is a filter that emphasizes the lower right portion and suppresses the lower left, upper left, and upper right portions in the region to which the filter is applied. In other words, the left shoulder silhouette filter is a filter that emphasizes a convex portion that projects in the depth frame and protrudes in the lower right, and is, for example, a filter as shown in FIG. As with the convex portion extraction filter, each square in FIG. 31 represents a pixel, and the numerical value in each square is a coefficient to be multiplied by the pixel value of the pixel corresponding to the square. Then, for each pixel in the filter, the pixel value of the pixel is multiplied by a corresponding coefficient, and the sum of these multiplied values is the pixel value of the center pixel after the filter is applied.

  The position deriving unit 14b applies the left shoulder silhouette filter to the binarized image derived in step S64. 32 is a binarized image derived in step S64, and FIG. 33 shows an image obtained by applying the left shoulder silhouette filter of FIG. 31 to this binarized image. Then, the position deriving unit 14b specifies the two-dimensional coordinate of the pixel having the maximum pixel value in the frame after the application of the left shoulder silhouette filter, and sets it as the two-dimensional coordinate of the left shoulder 71. Further, the position deriving unit 14b derives the three-dimensional coordinates of the left shoulder 71 by extracting the depth data at the two-dimensional coordinates of the left shoulder 71 from the initial depth frame. In FIG. 33, the position of the left shoulder 71 is highlighted for reference.

  In order to improve the accuracy and / or reduce the calculation load, the application range of the left shoulder silhouette filter may not be the entire depth frame but may be within a part of the range. For example, an area of an appropriate size including the two-dimensional coordinates of the left shoulder 71 specified on the same or the previous frame can be set as the application range of the left shoulder silhouette filter. Note that in the horizontal section from the address to the takeback arm, a region of a predetermined size with the two-dimensional coordinates of the left shoulder 71 included in the skeleton data of the same frame as the lower right end point is applied to the application range R5 of the left shoulder silhouette filter (FIG. 34). Reference) is preferably set. At the horizontal timing of the downswing arm, a region of a predetermined size having the upper left end point of the above R4 region set in the previous frame as the lower right end point is applied to the application range R6 ( It is preferable to set as shown in FIG. Further, in the section from immediately after the downswing arm level to the finish, an area of a predetermined size centered on the two-dimensional coordinates of the left shoulder 71 specified by using the left shoulder silhouette filter in the previous frame is represented by the left shoulder silhouette filter. The application range R7 (see FIG. 36) is preferable.

<1-3-2-2-3. How to determine horizontal arm timing>
Hereinafter, a method for determining the timing of the takeback arm level and the downswing arm level according to the present embodiment will be described.

First, as shown in FIG. 37, the position deriving unit 14b acquires the two-dimensional coordinates of the right shoulder from the skeleton data at the time of addressing. Then, with the two-dimensional coordinates of the right shoulder as a reference, the two-dimensional coordinates of the point A advanced by a predetermined distance (for example, 30 pixels) in the positive x-axis direction and advanced by the predetermined distance (for example, 15 pixels) in the positive y-axis direction Is identified. The x-axis positive direction is a direction that proceeds from the left hand side of the golfer 7 to the right hand side along the horizontal direction. The positive y-axis direction is a direction from top to bottom along the vertical direction. Subsequently, the position deriving unit 14b occupies a range of a predetermined distance (for example, 115 pixels) from the point A in the x-axis positive direction, and occupies a range of a predetermined distance (for example, 30 pixels) in the y-axis positive direction. Region R8 is formed.

  Thereafter, the position deriving unit 14b counts, for each depth frame included in the section from the address to the finish, the number of pixels in which the depth data exists in the above-described rectangular region R8 on the depth frame. Then, all depth frames in which the number of pixels is equal to or greater than a predetermined number (for example, 400 pixels) (hereinafter referred to as arm horizontal conditions) are specified. At this time, normally, in the section from the address to the finish, the depth frames that satisfy the arm horizontal condition appear in two groups along the time axis. Therefore, after specifying these two groups, the position deriving unit 14b specifies a take-back arm horizontal frame from the first group and a down-swing arm horizontal frame from the second group. When a plurality of depth frames satisfying the arm horizontal condition appear in one group, for example, the top frame can be set as the arm horizontal frame. It should be noted that in the section from the address to the finish, if the depth frame that satisfies the arm horizontal condition is large and does not appear in two groups, appropriate error processing is performed.

<1-3-3. Shaft direction derivation process>
Hereinafter, the shaft direction derivation process will be described. In the shaft direction deriving process, a three-dimensional vector representing the direction of the shaft 52 during the swing operation is derived based on the depth image captured by the distance image sensor 2. As in the case of the grip behavior derivation process, the image data to be analyzed is a moving image. Accordingly, the IR image and depth image to be analyzed are also referred to as IR frame and depth frame here.

FIG. 38 is a flowchart showing the overall flow of the shaft direction derivation process. First, in step S70, the acquisition unit 14a reads the IR frame and the depth frame acquired from the distance image sensor 2 and stored in the storage unit 13, and develops them in the memory. Then, the direction deriving unit 14c determines a frame to be analyzed among these image data. In the present embodiment, a period from the address during the swing operation to a little later than the impact is an analysis target. Details are the same as in step S1.

  The subsequent steps S71 to S77 are repeatedly executed for each frame included in the period to be analyzed. Therefore, in the following description, as in the grip behavior derivation process, the IR frame and the depth frame at the time (hereinafter, the target time) at which the direction of the shaft 52 is to be derived are respectively represented as the target IR frame and the target depth. Called a frame. Each frame or these may be collectively referred to as a target frame.

  In subsequent step S71, the direction deriving unit 14c extracts an image of the marker of the shaft 52 from the target IR frame. The specific process is the same as that in step S41. If the image of the shaft 52 has been extracted, the process proceeds to step S72. If the image has not been extracted, the process proceeds to the next frame.

In step S72, the direction deriving unit 14c specifies the two-dimensional coordinates of all points included in the region of the marker image of the shaft 52 extracted in step S71. In addition, the direction deriving unit 14c specifies the depth values of all the points for which the two-dimensional coordinates are specified here from the target depth frame. However, the point determined that the depth value specified here is not appropriate can be removed from the following analysis. Whether or not the depth value is appropriate is determined depending on, for example, whether or not the depth value is within a predetermined range (for example, 1000 mm to 3500 mm). From the above, the three-dimensional coordinates of the points included in the shaft 52 are derived. If the predetermined number or more (one or more points in this embodiment) of the three-dimensional coordinates of the shaft 52 are derived, the process proceeds to step S73, and if not derived, the process proceeds to the next frame.

  In step S73, the direction deriving unit 14c calculates a point representing the shaft 52, in this embodiment, the three-dimensional coordinate of the center of gravity, from the three-dimensional coordinates of all the points included in the shaft 52 derived in step S72. . The position of the center of gravity in each axial direction can be calculated as the average value of the coordinate values in the axial direction of all points derived in step S72.

  In subsequent step S74, the direction deriving unit 14c determines whether or not the three-dimensional coordinates of the grip end 51a at the target time can be derived in the above-described steps S3 to S7. If it has not been derived, the process proceeds to step S76. If the above steps S3 to S7 are not executed for the target frame at the time of execution of this step, it may be executed in this step to try to derive the three-dimensional coordinates of the grip end 51a.

  In step S75, the direction deriving unit 14c derives a three-dimensional vector representing the direction of the shaft 52 based on the three-dimensional coordinates of the grip end 51a and the three-dimensional coordinates of the center of gravity of the shaft 52. The direction of the shaft 52 can be derived as the direction from the grip end 51a toward the center of gravity of the shaft 52, or vice versa.

  On the other hand, in step S76, the direction deriving unit 14c determines whether or not a predetermined number or more (for example, two or more) of the three-dimensional coordinates of the shaft 52 have been derived in step S72. The process proceeds to S77, and if it has not been derived, the process proceeds to the next frame. In step S77, principal component analysis is performed on the three-dimensional coordinates of all the points included in the shaft 52 derived in step S72. Then, the eigenvector (first principal component) corresponding to the maximum eigenvalue is set as a three-dimensional vector representing the direction of the shaft 52.

  When the above steps S71 to S77 for all times to be analyzed are completed, the process proceeds to step S78. In step S78, a three-dimensional vector representing the direction of the shaft 52 at a time when the direction of the shaft 52 has not yet been derived is calculated from the direction of the shaft 52 at the previous and subsequent times. For example, the average of three-dimensional vectors representing the direction of the shaft 52 at the previous and subsequent times can be used.

  From the above, the orientation of the shaft 52 with respect to all the times to be analyzed is derived three-dimensionally. Such a direction of the shaft 52 can be displayed in a graph for each axis as shown in FIG. 39, for example, and can also be displayed in a three-dimensional graph as shown in FIG. In the example of FIG. 40, the position of the grip end 51 a and the position of the center of gravity of the shaft 52 are indicated by symbols “◯” and “X” so as to overlap with a straight line representing the shaft 52.

<1-4. Modification>
As mentioned above, although several embodiment of this invention was described, this invention is not limited to the said embodiment, A various change is possible unless it deviates from the meaning. For example, the following changes can be made.

<1-4-1>
In the grip behavior deriving process according to the first embodiment, an IR image is used. However, a color image can be used instead of the IR image. In this case, the distance image sensor may be equipped with a visible light receiving unit (for example, an RGB camera) that receives visible light.

<1-4-2>
The part to be analyzed using the depth image and / or skeleton data is not limited to the grip 51 or the left shoulder 71. For example, the behavior (three-dimensional coordinates) of the shaft, head, right shoulder, left arm, right arm, etc. can be specified. At this time, a marker can be appropriately attached to a site to be analyzed.

<1-4-3>
The method for determining the arm horizontal timing is not limited to the above. For example, a linear infrared reflective tape may be attached to the arm, an image of the reflective tape may be extracted from the IR image, and the timing at which the image becomes horizontal may be detected.

<1-4-4>
Instead of the process using principal component analysis in steps S56, S57, and S77, a process using the least square method can be executed. Specifically, a regression line is derived from the distribution of the three-dimensional coordinates of the shaft 52 by the least square method, and this is determined as the axis on which the shaft 52 extends. Then, the position of the grip end 51a can be searched for on the extension line of this axis.

<1-4-5>
In step S22, the threshold value for binarization for extracting the image of the grip end 51a was automatically calculated by the discriminant analysis method. However, the threshold value can be a fixed value obtained by multiplying the maximum luminance value (limit value) by a predetermined coefficient of 0 to 1, for example, a value that is 0.1 times the maximum luminance value. In this case, step S23 can be omitted.

  When the gap between the mark on the grip end 51a and the mark on the shaft 52 becomes narrow on the target IR frame, the mark on the shaft 52 appears in the search range R1 of the grip end 51a. In this case, if the mark of the shaft 52 has a higher luminance value than the mark of the grip end 51a, the binarization using the discriminant analysis method may cause the image of the grip end 51a to disappear. Therefore, it is preferable to set the threshold for binarization to a value of a predetermined ratio of the maximum luminance value so that even if the luminance value of the mark at the grip end 51a is low, it can be extracted. Note that the threshold value, which is a fixed value according to this modification, is preferably used only in a predetermined phase in which the mark of the shaft 52 is easily reflected in the search range R1. That is, it is preferable that the threshold value according to this modification and the threshold value automatically set by the discriminant analysis method or the like are appropriately switched according to the phase during the swing operation. Examples of such a phase include a section from the shaft 8 o'clock (during takeback and the time when the shaft 52 points to 8 o'clock in the front view) to the takeback arm level, or a part thereof. . Note that this section is a section in which the grip end 51a is likely to move away or enter into the shadow of the golfer's hand when viewed from the camera arranged in front.

<1-4-6>
In step S21 according to the first embodiment, the search range R1 of the grip end 51a is set based on the moving direction (vector direction) of the grip end 51a in the predetermined number of frames immediately before. However, in the phase in which the grip end 51a is not easily captured in the target IR frame because it is hidden by the golfer's body, it may be difficult to specify the exact position of the grip end 51a. That is, in the above-described phase, the position of the grip end 51a tends to be estimated from the image of the shaft 52. In particular, when such a situation continues, the search range R1 ignores the actual movement of the grip end 51a. May go on. In such a case, as shown in FIG. 41A in particular, if the image of the shaft 52 enters the search range R1, the accuracy in detecting the position of the grip end 51a can be reduced. Therefore, in such a predetermined phase, it is preferable to set the search range R1 so as not to exceed a predetermined limit line.

  An example of such a phase is near the top. In the vicinity of the top, there is a high possibility that the grip end 51a does not appear in the target IR frame, and the movement of the grip end 51a is stopped or almost stopped. In the vicinity of the top, it is preferable to set the limit line for limiting the setting position of the search range R1 to the position of the golfer's head (more specifically, for example, the vertex of the head) (see FIG. 41B). This is because, in the vicinity of the top, the grip end 51a usually does not move rightward in front view than the golfer's head. In this case, the estimated value of the position of the grip end 51a does not advance to the right of the limit line in front view.

  The position of the golfer's head may be extracted by image processing of the IR frame and / or the depth frame, or may be determined based on skeleton data. The golfer's head position can be extracted from the person area image. For example, the top point of the person area image can be set as the head position. The person area image is an image mainly showing only an area in which a person is captured, and can be derived from a depth frame or can be derived from an IR frame. Since various methods for deriving such a person area image are known, detailed description thereof is omitted here. Also, Kinect (registered trademark), which is a distance image sensor, is provided with a library for reading a person area image from a depth image, and the acquisition unit 14a can acquire a person area image using the library. . Note that the CPU 23 of the distance image sensor 2 may derive a person region image from the depth frame. In this case, the acquisition unit 14a may simply read the person area image sent from the distance image sensor 2.

  The image serving as a reference for extracting the position of the golfer's head may be an image at a timing when the position of the grip end 51a is to be extracted, or may be an image at a predetermined timing. Preferably, the image can be an image when the takeback arm is horizontal. FIG. 41C shows a person area image when the takeback arm is horizontal. In this way, the accuracy of extraction of the position of the head can be improved by using an image at a timing at which it is unlikely that another part such as a hand will come near the head. Note that the method for determining the horizontal timing of the takeback arm is as shown in 3-2-3.

<1-4-7>
In the first embodiment, as shown in FIG. 3, first, an attempt is made to directly derive the three-dimensional coordinates of the grip end 51a from the target IR frame and the target depth frame (steps S3 and S5). Is extracted, and the three-dimensional coordinates of the grip end 51a are derived based on the extracted position of the shaft 52 (exception processing S6). In other words, the process of deriving the three-dimensional coordinates of the grip end 51a included in the process of the first embodiment can be expressed as a process including steps S101 to S104 as shown in FIG.

  However, the process for deriving the three-dimensional coordinates of the grip end 51a can be configured as shown in FIG. That is, first, it is determined whether or not the grip end 51a has been extracted from the IR frame at the previous time (step S201). If it is determined in step S201 that extraction has been performed, an attempt is made to derive the three-dimensional coordinates of the grip end 51a from the target IR frame and the target depth frame (step S101). The subsequent steps are the same as in FIG. On the other hand, if it is determined in S201 that the target IR frame could not be extracted, step S101 is not executed, that is, without trying to derive the three-dimensional coordinates of the grip end 51a from the target IR frame and the target depth frame. The shaft 52 is extracted from (step S103). Then, the three-dimensional coordinates of the grip end 51a are derived based on the extracted position of the shaft 52 (step S104).

  The process according to FIG. 43 is superior to the process according to FIG. 42 in the following points. That is, when the grip end 51a has not been extracted from the IR frame at the previous time and the position of the grip end 51a is estimated from the position of the shaft 52, the position may include an error of a certain level or more. Then, if the search range R1 is set in step S101 for the next frame using the information on the position of the grip end 51a including such an error, the above error may be accumulated. Therefore, when the grip end 51a cannot be extracted from the IR frame in the previous frame, the position of the shaft 52 is specified first instead of searching for the grip end 51a within the search range R1. If the position of the grip end 51a is specified based on the position of the shaft 52, accumulation of errors can be prevented.

  The processes in steps S103 and S104 according to this modification can be performed in the same manner as the exception process S6 described above, or can be changed. For example, after the two-dimensional coordinates of the grip end 51a are derived based on the two-dimensional coordinates of the shaft 52 by the same method as the exception process S6 described above, the target IR frame is searched using the two-dimensional coordinates of the grip end 51a as a reference. A range R1 may be set. Then, similarly to step S3, the grip end 51a is further searched within the search range R1, and thereafter, the three-dimensional coordinates of the grip end 51a can be derived similarly to steps S3 to S7.

<1-4-8>
In step S65 according to the first embodiment, left shoulder extraction is performed using a convex portion extraction filter. However, instead of this, the position of the left shoulder can also be specified by template matching using a convex template. The template matching method using this convex template can improve the accuracy of the left shoulder extraction particularly in the section from the takeback arm level to the downswing arm level.

FIG. 44 is an example of a convex template. As shown in the figure, the convex template is an image template having a pixel value that is smallest at the center and gradually increases from the center toward the periphery (the center is the largest, and from the center toward the periphery). It can also be an image having pixel values that gradually decrease). That is, if template matching is performed using this template, an image capturing a subject having a center projecting and projecting to the opposite side toward the periphery can be easily extracted. Various template matching processing methods are known, but a brief description is as follows. That is, within the application range of the above-described filter in the depth frame, the degree of similarity between the convex template and the partial image of the depth frame that overlaps the convex template is slightly shifted up and down (for example, one pixel at a time). I will calculate. The similarity is, for example, SAD (sum of absolute values of differences between pixel values). Then, the region having the highest similarity in the depth frame (minimum in the case of SAD) is specified as the position of the left shoulder.

  As described above, Kinect (registered trademark) v1 and v2 are general-purpose distance image sensors. Template matching according to this modification is particularly useful when the left shoulder is extracted from a relatively high-quality image such as a depth frame output from Kinect (registered trademark) v2. The template matching according to this modification has a property that it is less likely to respond to unevenness other than the left shoulder, for example, unevenness such as clothing unevenness, as compared with the case where the convex portion extraction filter according to the first embodiment is used. It is. Therefore, an error caused by responding to fine irregularities other than the left shoulder is more reliably prevented. FIG. 45 is a diagram showing the position of the left shoulder extracted by template matching according to this modification with respect to the depth frame output from Kinect (registered trademark) v2. From the figure, it can be seen that the position of the left shoulder is correctly detected.

<1-4-9>
In the first embodiment, the left shoulder extraction from the skeleton data, the left shoulder extraction by the convex portion extraction filter, and the left shoulder extraction by the left shoulder silhouette filter were performed for the entire analysis period. However, in order to shorten the calculation time, it is preferable to perform the left shoulder extraction only for the section where the position of the left shoulder by each is finally used. FIG. 46 shows an example of such processing.

  In FIG. 46, first, the position deriving unit 14b specifies the address, shaft 8 o'clock, top, and impact time. Then, the position of the left shoulder 71 is derived in order from the address toward the impact. Until the takeback arm level is reached from the address, the position of the left shoulder 71 is derived from the skeleton data. Specifically, the position of the left shoulder 71 is derived from the skeleton data until the shaft reaches 8:00 from the address. The timing at 8 o'clock of the shaft can be determined from the orientation of the shaft 52 derived by the shaft direction deriving process. After the shaft 8 o'clock, the number of pixels in which the depth data exists in the rectangular area R8 of the depth frame at the current time is counted. If the number is less than a predetermined number (for example, 400 pixels), the left shoulder 71 is based on the skeleton data. Determine the position. On the other hand, if the number is greater than the predetermined number, the current time is determined as the horizontal time of the takeback arm.

  From the takeback arm level to the downswing arm level, the left shoulder 71 is derived based on the convex extraction filter. Specifically, the left shoulder 71 is derived based on the convex extraction filter until the takeback arm level reaches the top from the horizontal. After the top, the number of pixels in which the depth data exists in the rectangular region R8 of the depth frame at the current time is counted. If the number is less than a predetermined number (for example, 400 pixels), the left shoulder 71 is based on the convex portion extraction filter. Determine the position. On the other hand, if it is greater than the predetermined number, the current time is determined as the time of the downswing arm horizontal. Thereafter, from the downswing arm level to the impact, the left shoulder 71 is derived based on the left shoulder silhouette filter. Moreover, it may replace with the process using a convex part extraction filter, and may perform the template matching of modification 1-4-8.

<2. Second Embodiment>
<2-1. Overview of swing analysis system>
Hereinafter, the swing analysis system 200 including the swing analysis apparatus 201 according to the second embodiment will be described. FIG. 47 is a plan view of the swing analysis system 200. The swing analysis system 200 is also a system for analyzing a golf swing based on a movie that captures the swing motion of the golf club 5 by the golfer 7. In the second embodiment, the movie is also captured by the distance image sensor 2. Is done. However, unlike the first embodiment, in the second embodiment, a plurality of distance image sensors 2 are used, and a swing motion is photographed from a plurality of directions. Here, as shown in FIG. 47, an example in which two distance image sensors 2F and 2S are used will be described. Of course, it is also possible to use three or more distance image sensors. As a result, the swing motion can be accurately captured from multiple directions.

  In the second embodiment, in addition to the depth image and two-dimensional image output from the distance image sensor 2 and skeleton data based on the depth image, silhouette data representing the silhouette of the human body described later is analyzed. Then, the accuracy of swing analysis is improved by appropriately combining such various types of information while taking advantage of the respective advantages.

  In the present embodiment, the grip behavior derivation process, the left shoulder behavior derivation process, and the shaft direction derivation process are executed as in the first embodiment, whereby the three-dimensional coordinates of the grip end 51a and the left shoulder 71 and the orientation of the shaft 52 are performed. A three-dimensional vector representing is derived. In the present embodiment, whether or not the swing motion is good is determined based on position information of various parts of the body of the golfer 7 other than the left shoulder 71 in addition to the same information as in the first embodiment.

  Hereinafter, for the sake of simplicity, descriptions of features common to the first embodiment will be omitted, and features specific to the second embodiment will be mainly described. In addition, the same referential mark is attached | subjected about the element which is common in 1st Embodiment.

<2-2. Details of each part>
Hereinafter, the details of each part of the swing analysis system 200 will be described with reference to FIG.

<2-2-1. Distance image sensor>
The distance image sensors 2F and 2S have the same configuration as the distance image sensor 2 already described, as shown in FIG. Similar to the distance image sensor 2 according to the first embodiment, the distance image sensor 2F is installed in front of the golfer 7 so as to photograph the golfer 7 from the front side. On the other hand, the distance image sensor 2S is installed on the right side of the golfer 7 so as to photograph the golfer 7 from the right side.

  Also in the second embodiment, since IR images are taken by the distance image sensors 2F and 2S, an infrared reflection sheet is attached to the grip 51 and the shaft 52 as a marker.

<2-2-2. Swing analyzer>
Next, the swing analysis apparatus 201 will be described. As shown in FIG. 48, the swing analysis device 201 has the same hardware configuration as the swing analysis device 1 according to the first embodiment. However, since processing different from that of the first embodiment is executed, a swing analysis program 203 is installed in the storage unit 13 instead of the swing analysis program 3. Accordingly, the control unit 14 operates as the acquisition unit 14a, the first position deriving unit 14b, and the direction deriving unit 14c, and also operates as the second position deriving unit 14d and the determination unit 14e. Details of the operation of each of the units 14d and 14e will be described later. In the present embodiment, the position deriving unit 14b is referred to as a first position deriving unit 14b for distinction from the second position deriving unit 14d.

  Note that the swing analysis apparatus 201 can be composed of one computer or a plurality of computers. In particular, a computer may be connected to each of the distance image sensors 2F and 2S.

<2-3. Swing analysis method>
Hereinafter, a golf swing analysis method executed by the swing analysis system 200 will be described. Also in this analysis method, first, the golfer 7 makes a trial hit of the golf club 5, and the state is photographed by the distance image sensors 2F and 2S. The IR frame and depth frame image data captured by the distance image sensors 2F and 2S are sent from the distance image sensors 2F and 2S to the swing analysis apparatus 201. Thereafter, grip behavior derivation processing, left shoulder behavior derivation processing, and shaft direction derivation processing are executed. Also, two series of image data from the distance image sensors 2F and 2S in the storage unit 13 (in reality, two series of image data of a depth frame and an IR frame are derived from each sensor. Can also be synchronized by the acquisition unit 14a. The synchronization can be realized by various methods. For example, the synchronization can be realized by detecting the timing at which the same feature appears in the image data of each system by image processing, or photographing by the distance image sensors 2F and 2S. A separate device that triggers simultaneously can also be used. Thereafter, a position derivation process is executed. In the position deriving process, position information of a predetermined part of the body of the golfer 7 other than the left shoulder 71 at a predetermined timing (hereinafter referred to as a check point) is derived. Thereafter, pass / fail judgment processing is executed. In the quality determination process, the quality of the swing motion is determined based on the position information of the grip end 51a, the position information of a predetermined part of the body of the golfer 7 including the left shoulder 71, and the three-dimensional vector representing the direction of the shaft 52. . In the position derivation process, calibration in step S202 described later is performed. Hereinafter, after describing the position derivation process, a process for setting parameters necessary for the calibration (calibration parameter setting process) will be described, and finally, the quality determination process will be described.

<2-3-1. Position derivation process>
As described above, the position deriving process is a process of deriving position information of a predetermined part of the body of the golfer 7 other than the left shoulder 71 at the check point. FIG. 49 is a flowchart showing the flow of the position derivation process.

  First, in step S201, the acquisition unit 14a reads the image data stored in the storage unit 13 and develops it in the memory. Then, the second position deriving unit 14d determines a checkpoint frame to be analyzed among these image data. Check points according to the present embodiment are eight timings of address, takeback shaft 8 o'clock, takeback shaft 9 o'clock, takeback arm horizontal, top, down swing arm horizontal, down swing shaft 9 o'clock, and impact ( 50 and 51). These checkpoints appear in this order during the swing movement. 50 and 51 are a front view and a right side view of the golfer 7 at these check points, respectively. The takeback shaft 8 o'clock and the takeback shaft 9 o'clock are timings when the golf player 7 is viewed from the front and the shaft 52 generally points in the direction of 8 o'clock and 9 o'clock of the timepiece during the takeback. 9 o'clock of the down swing shaft is a timing when the golf player 7 is viewed from the front, and the shaft 52 generally points in the 9 o'clock direction of the timepiece during the down swing.

  These checkpoint frames can be determined in various ways. The address, takeback arm level, top, downswing arm level, and impact frames can be determined, for example, by the method described above. The frame of the takeback shaft 8 o'clock is, for example, the section from the address to the horizontal of the takeback arm, and the grip end 51a on the right side of the right foot tip on the IR frame (hereinafter referred to as the front IR frame) taken from the front of the golfer 7. The frame that has become can be identified by image processing, and can be derived as a frame back by 1 to 3 frames. Hereinafter, similarly to the front IR frame, the depth frame shot from the front of the golfer 7 is referred to as a front depth frame, and the IR frame shot from the right side of the golfer 7 is referred to as a right side IR frame. The depth frame shot from the right side surface of the golfer 7 is referred to as a right side depth frame. A frame taken from the front of the golfer 7 is called a front frame, and a frame taken from the right side of the golfer 7 is called a right side frame.

  The frame at the time of the takeback shaft 9 is derived, for example, as a frame in which the angle of the shaft 52 is 90 ° (horizontal) when viewed from the front of the golfer 7 in the horizontal section of the takeback arm from the time of the takeback shaft 8 o'clock. Can do. The frame at the time of the downswing shaft 9 can be derived, for example, as a frame in which the angle of the shaft 52 is 90 ° (horizontal) when viewed from the front of the golfer 7 in the section from the horizontal of the downswing arm to the impact. At this time, the angle of the shaft 52 can be specified by using a three-dimensional vector representing the direction of the shaft 52 derived by the shaft direction deriving process or a two-dimensional vector (derived from the IR frame).

  In subsequent steps S202 to S205, position information of various parts of the body of the golfer 7 at the checkpoint is derived. Here, the position information is mainly derived as position information in the first plane and / or the second plane. The first plane is a plane orthogonal to the optical axis of the distance image sensor 2F, that is, a plane orthogonal to the direction from the belly of the golfer 7 toward the back. The second plane is a plane orthogonal to the optical axis of the distance image sensor 2S, that is, a plane orthogonal to the flying ball direction. Further, here, the position information is necessary as appropriate from the already calculated three-dimensional coordinates of the grip end 51a, the three-dimensional coordinates of the left shoulder 71, the three-dimensional vector indicating the orientation of the shaft 52, skeleton data, and silhouette data. It is calculated using anything. The skeleton data has already been described, but the silhouette data will be described later.

  First, in step S202, the acquisition unit 14a reads the front depth frame at each check point during the swing operation stored in the storage unit 13 and develops it in the memory. And the acquisition part 14a acquires the skeleton data in each check point from these front depth frames. The manner of acquiring the skeleton data is the same as in the first embodiment. Then, the second position deriving unit 14d converts the skeleton data at each check point viewed from the front side based on the front depth frame into skeleton data viewed from the right side. This is to deal with the following problems. That is, the skeleton data based on the depth frame photographed from the front (more precisely, the position information of the attention point such as the joint included in the skeleton data) is highly reliable, but the skeleton data based on the depth frame photographed from the side is Reliability may be low. Therefore, in the following, skeleton data obtained by converting skeleton data based on the front depth frame (hereinafter referred to as front skeleton data) is used instead of the skeleton data based on the right side depth frame. Hereinafter, unless otherwise specified, the right side skeleton data means skeleton data after conversion from the front side skeleton data. In step S202, the second position deriving unit 14d converts the three-dimensional coordinates of the point of interest such as a joint included in the right-side skeleton data into two-dimensional coordinates in the second plane.

  The conversion of the skeleton data is performed based on calibration parameters stored in the storage unit 13 in advance. The conversion of skeleton data means that three-dimensional coordinates in a certain three-dimensional space are converted into three-dimensional coordinates in a three-dimensional space set by another definition. Therefore, if the relative relationship between the three-dimensional space before and after the conversion is known, such conversion is possible. A parameter indicating this relative relationship is a calibration parameter used here, and is set in a calibration parameter setting process described later.

  In subsequent step S203, the second position deriving unit 14d determines the three-dimensional coordinates of the grip 51, the three-dimensional coordinates of the left shoulder 71, and the shaft derived from the front side derived by the grip behavior deriving process, the left shoulder deriving process, and the shaft direction deriving process. The three-dimensional vector representing the orientation of 52 is converted into data viewed from the right side. This is because, in the present embodiment, as will be described later, swing analysis is executed based on image data viewed from the right side. In the following, unless otherwise specified, in the portion describing the analysis of the image data viewed from the right side, the position information of the grip end 51a, the position information of the left shoulder 71 and the orientation of the shaft 52 after the conversion in step S203 are the same. , Subject to analysis.

  By the way, in the present embodiment, the position information of the grip end 51a and the orientation of the shaft 52 derived by the grip behavior deriving process and the shaft direction deriving process are calculated based on the image data from the front. The positional information after the conversion and the orientation of the shaft 52 may increase errors. In such a case, the position information of the grip end 51a and the direction of the shaft 52 can be re-derived by image processing based on images from the right side surface (including silhouette data, depth frames, and IR frames). . In addition, the error may also increase for the data obtained by converting the position information of the left shoulder 71 derived by the left shoulder behavior deriving process in step S203. In such a case, the position information of the left shoulder 71 can be re-derived by image processing based on an image (including silhouette data, a depth frame, and an IR frame) from the right side surface.

  In subsequent steps S204 and S205, position information of a predetermined part at each check point determined in step S201 is derived. The position information derived here is used to determine whether the swing motion is good or bad in the quality determination process.

  As described above, steps S204 and S205 are steps for deriving position information of a predetermined part at each check point. Among these, in step S204, position information of a predetermined part in the first plane is derived, and in step S205, position information of a predetermined part in the second plane is derived. In the present embodiment, the part from which the position information is derived differs for each check point. The parts from which the position information for each check point according to the present embodiment is derived will be described together below. When step S205 ends, the position derivation process ends.

<2-3-1-1. Address>
The second position deriving unit 14d derives position information of the right shoulder, shoulder center, navel, right ankle, and left ankle of the golfer 7 in the first plane at the time of addressing from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the front skeleton data at the time of addressing are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

  Further, the second position deriving unit 14d derives position information of the left foot width reference position, the right foot width reference position, the umbilical left position, the umbilical right position, and the left footpad in the first plane at the time of address from the silhouette data. From the left foot width reference position and the right foot width reference position, the stance width at the time of addressing can be specified. The waist width at the time of addressing can be specified from the left navel position and right navel position here.

  Here, the silhouette data is data indicating the silhouette of the golfer 7 on each frame, and is acquired by the acquisition unit 14a. More specifically, the acquisition unit 14a reads the depth frame stored in the storage unit 13 and develops it in the memory. And the acquisition part 14a acquires silhouette data from these depth frames. That is, since the golfer 7 exists relatively near in the imaging range of the distance image sensors 2F and 2S, the golfer 7 derives a silhouette of the human body by extracting a region having a pixel value representing a shallow depth from the depth frame. Can do. Therefore, more precisely, the silhouette here includes a silhouette of the golf club 5 or the like. It should be noted that Kinect (registered trademark), which is a distance image sensor, is provided with a library for reading silhouette data from a depth image. In this embodiment, silhouette data is acquired using the library. The silhouette data may be derived from the depth frame by the CPU 23 of the distance image sensors 2F and 2S. In this case, the acquisition unit 14a may simply read out the silhouette data sent from the distance image sensor 2. Silhouette data can also be derived by image processing of IR frames instead of depth frames.

  Note that silhouette data directly extracted using a Kinect (registered trademark) library is not usually binarized. In that case, the acquisition unit 14a can binarize the silhouette data so that only the silhouette of the golfer 7 (more precisely, the golf club 5 is included) is 1 and the others are 0. Moreover, the acquisition part 14a can erase | acquire the small area classified into the silhouette by repeating expansion | swelling and shrinkage | contraction with respect to silhouette data, and can acquire a more exact silhouette. The acquisition unit 14a can also erase a small area classified as a silhouette by performing a labeling process. Hereinafter, the silhouette data processed in this way is also referred to as silhouette data. The above silhouette data is derived as front silhouette data and right side silhouette data from the front depth frame and the right side depth frame in the same manner.

  Returning to the description of the method for deriving the position information of the left foot width reference position, right foot width reference position, left navel position, right navel position and left footpad. These pieces of position information are derived by subjecting the front silhouette data to image processing by the second position deriving unit 14d. FIG. 52 is an image in which front skeleton data is superimposed on front silhouette data at the time of addressing. In the present embodiment, as shown in FIG. 52, a rectangular frame L1 surrounding the silhouette portion is set on the front silhouette data image. The left and right boundaries of the frame L1 are set so as to leave a predetermined interval from the silhouette. Further, the upper boundary of the frame L1 coincides with the uppermost point included in the silhouette, and the lower boundary coincides with the lower end of the silhouette excluding the silhouette of the linear shaft 52. Subsequently, on the image of the front silhouette data, a frame L2 is set that extends from the left ankle to the right end of the frame L1 in the left-right direction and extends from the left ankle to the lower end of the frame L1 in the vertical direction. Further, on the front silhouette data image, a frame L3 is set that extends from the right ankle to the left end of the frame L1 in the left-right direction and extends from the right ankle to the lower end of the frame L1 in the vertical direction. Then, the horizontal position of the point in the silhouette farthest from the left ankle in the frame L2 is specified as the left foot width reference position, and the horizontal position of the point in the silhouette farthest from the right ankle in the frame L3 is It is specified as the right foot width reference position.

  As shown in FIG. 52, the left end point in the silhouette is specified as the right navel position on the horizontal straight line passing through the navel on the image of the front silhouette data at the time of addressing. Further, the right end point in the silhouette on the same line is specified as the left navel position.

  The position information of the left footpad is derived from the image of the front silhouette data at a timing slightly advanced from the address (for example, one frame advanced). FIG. 53 is an image obtained by superimposing the front skeleton data on the front silhouette data at the timing. In the present embodiment, as shown in FIG. 53, a rectangular frame L1 similar to the above is set on the front silhouette data image. Subsequently, on the front silhouette data image, a frame L4 is set that extends from the left ankle to the center of the frame L1 in the left-right direction and extends from the left ankle to the lower end of the frame L1 in the vertical direction. Then, the point in the silhouette farthest from the left ankle in the frame L4 is specified as the position of the left footpad.

  Next, the second position deriving unit 14d derives the position information of the right shoulder, neck, waist, left knee, right knee, right ankle and right elbow of the golfer 7 in the second plane at the time of addressing from the skeleton data. To do. At this time, the two-dimensional coordinates of these parts included in the right-side skeleton data at the time of addressing are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

  Further, the second position deriving unit 14d derives position information of the left foot tip, the right foot tip, the left kneecap, and the right kneecap in the second plane at the time of addressing from the silhouette data. Here, for the left foot and the right foot, the reflected wave of infrared data from the ground becomes noise, and depth information may not be acquired in the depth frame. In this case, the silhouette of the left foot and the right foot does not appear in the silhouette data based on the depth frame. In the present embodiment, in order to deal with such a case, the silhouette region specified in the above-described silhouette data is expanded, in other words, processing for complementing the region corresponding to the left foot and the right foot is performed on the silhouette region. Is called. Specifically, the second position deriving unit 14d derives regions corresponding to the left foot and the right foot based on information on a region where depth information is missing on the right-side depth frame at the time of addressing. FIG. 54A is an image of right side silhouette data based on the right side depth frame at the time of addressing. In the present embodiment, the second position deriving unit 14d sets a frame L5 of a predetermined size around the right ankle on the right ankle silhouette image on the right side silhouette data image. And the area | region where depth information is 0 in this frame L5 is extracted as an area | region corresponding to a left foot and a right foot. FIG. 54B is an image in which the silhouettes of the left foot and the right foot are complemented. The second position deriving unit 14d sets a frame L6 having a predetermined size around the right ankle with reference to the right ankle, and specifies the right end point of the silhouette of the left foot in the frame L6 as the position of the left foot. Similarly, the second position deriving unit 14d sets a frame L7 of a predetermined size around the right ankle with reference to the right ankle, and specifies the right end point of the silhouette of the right foot in the frame L7 as the position of the right foot tip.

  The position information of the left and right knee heads is determined by offsetting the positions of the left and right knees in the depth direction on the front depth frame, and determining the positions of the left and right knee heads in the same manner as in step S202. The position is converted into a position viewed from the surface side, and is specified as position information of a point obtained by offsetting the converted positions of the left and right knee caps in the contour direction of the silhouette.

<2-3-1-2. Take back shaft 8 o'clock>
The second position deriving unit 14d derives position information of the right shoulder and navel of the golfer 7 in the first plane at the time of the takeback shaft 8 from the skeleton data. At this time, two-dimensional coordinates of these parts included in the front skeleton data at the time of the takeback shaft 8 are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

<2-3-1-3. Take back shaft 9 o'clock>
In the present embodiment, the position information of the part of the golfer 7 is not specified at the timing of the takeback shaft 9. However, as described later, the quality of the swing operation at the timing of the takeback shaft 9 is determined based on the position information of the grip end 51a at the time of the takeback shaft 9.

<2-3-1-4. Take back arm horizontal>
The second position deriving unit 14d derives position information of the neck and waist of the golfer 7 in the second plane when the takeback arm is horizontal from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the right-side skeleton data when the takeback arm is horizontal are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

<2-3-1-5. Top>
The second position deriving unit 14d derives position information of the head, navel, left knee, and right knee of the golfer 7 in the first plane at the time of the top from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the front skeleton data at the top time are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

  The second position deriving unit 14d derives position information of the right umbilicus position in the first plane at the time of top from the silhouette data. The method for deriving the right umbilicus position is the same as that at the time of addressing except that the image of the front silhouette data at the time of impact is used instead of the image of the front silhouette data at the time of addressing.

  Next, the second position deriving unit 14d derives position information of the left knee, right knee, neck, and waist of the golfer 7 in the second plane at the time of top from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the right side skeleton data at the time of the top are extracted, and the two-dimensional coordinates are directly used as the position information of these parts.

  Further, the second position deriving unit 14d derives position information of the left shoulder 71, the left knee head, the right knee head, the left elbow, and the right elbow of the golfer 7 in the second plane at the time of the top from the silhouette data. For deriving the position information of the left shoulder 71, the already derived two-dimensional coordinates of the left shoulder 71 are used. FIG. 55 is an image of right side silhouette data at the time of top. Specifically, in the present embodiment, as shown in FIG. 55, the position of the left shoulder 71 that has already been extracted on the image of the right side silhouette data at the top is overlapped with the outline of the silhouette in the left direction. The position after the movement is set as a new left shoulder position.

  The position information of the right kneecap is derived by the same method as that at the time of addressing. On the other hand, the position information of the left kneecap is derived by the following method. First, a frame L10 of a predetermined size (see FIG. 55) is set around the position of the left knee that has already been extracted on the right side silhouette data image. Subsequently, the curvature of the outline of the silhouette included in the frame L10 is calculated, and the point having the maximum curvature is specified as the position of the new left kneecap.

  The position information of the right elbow is also derived based on the curvature of the silhouette. More specifically, a frame L11 of a predetermined size (see FIG. 55) is set around the grip end 51a on the right side silhouette data image. Subsequently, the curvature of the outline of the silhouette included in the frame L11 is calculated, and the point having the maximum curvature is specified as the position of the new right elbow.

  The position information of the left elbow is derived by the following method. First, the midpoint between the re-extracted position of the left shoulder 71 and the grip end 51a is taken as a temporary left elbow. Next, a frame L12 (see FIG. 55) of a predetermined size is set around the position of the provisional left elbow, and based on the right-side depth frame, a point that is shallower than the left shoulder 71 in the frame L12, in other words, And the point which protrudes to the near side on the basis of the distance image sensor 2F is specified as the position of the left elbow.

<2-3-1-6. Down swing arm horizontal>
The second position deriving unit 14d derives position information of the left knee of the golfer 7 in the first plane when the downswing arm is horizontal from the skeleton data. At this time, two-dimensional coordinates of these parts included in the front skeleton data when the downswing arm is horizontal are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

  Next, the second position deriving unit 14d derives position information of the neck and waist of the golfer 7 in the second plane when the downswing arm is horizontal from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the right-side skeleton data when the downswing arm is horizontal are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

<2-3-1-7. Down swing shaft 9 o'clock>
The second position deriving unit 14d derives position information of the hand in the first plane at the time of the downswing shaft 9 from the silhouette data. In the present embodiment, the position of the hand is specified as a position moved by a predetermined length from the grip end 51a to the head side along the extending direction of the shaft 52 on the image of the front silhouette data at the time of the down swing shaft 9. The

<2-3-1-8. Impact>
The second position deriving unit 14d derives position information of the head, shoulder center, right shoulder, left elbow, and navel of the golfer 7 in the first plane at the time of impact from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the front skeleton data at the time of impact are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

  In addition, the second position deriving unit 14d derives position information of the left navel position and the right navel position in the first plane at the time of impact from the silhouette data. The waist width at the time of impact can be specified from the left navel position and right navel position here. The method for deriving the left umbilicus position and the right umbilicus position here is the same as that at the time of addressing except that the image of the front silhouette data at the time of impact is used instead of the image of the front silhouette data at the time of addressing. Since the arm tends to overlap the left umbilicus position on the front silhouette data image at the time of impact, when the left umbilicus position is specified, a predetermined time has passed from the impact (for example, 2 frames before ) It is preferable to perform the same processing on the image.

  Next, the second position deriving unit 14d derives the position information of the neck, waist, right knee, and right ankle of the golfer 7 in the second plane at the time of impact from the skeleton data. At this time, the two-dimensional coordinates of these parts included in the right side skeleton data at the time of impact are extracted, and the two-dimensional coordinates are directly used as position information of these parts.

<2-3-2. Calibration parameter setting process>
Next, calibration parameter setting processing will be described. First, in the vicinity of the position where the golfer 7 stands, one or more marks that reflect infrared rays through a stand or the like are arranged. This mark is arranged at a position where photographing can be performed from both the distance image sensors 2F and 2S. The mark is photographed by the distance image sensors 2F and 2S, and the three-dimensional coordinates of the mark are acquired. Here, the three-dimensional coordinates of the mark photographed by the distance image sensor 2F are represented as (FXi, FYi, FZi), and the three-dimensional coordinates of the mark at the same position photographed by the distance image sensor 2S are represented by (SXi, SYi, SZi). ). However, i = 1, 2,..., N (n ≧ 2). Here, n data sets (FXi, FYi, FZi, SXi, SYi, SZi) are prepared by moving the mark.

At this time, conversion coefficients a to l for converting the three-dimensional coordinates of the distance image sensor 2F into the three-dimensional coordinates of the distance image sensor 2S satisfy the following expressions.

  By substituting n data sets (FXi, FYi, FZi, SXi, SYi, SZi) into the above equation, conversion coefficients a to l are calculated. These conversion coefficients a to l are calibration parameters. The calibration parameter can be naturally calculated by other methods.

<2-3-3. Pass / Fail Judgment Processing>
Next, a quality determination process for determining quality of the swing motion will be described. The quality determination process is executed by the determination unit 14e.

  The pass / fail determination process is executed based on position information of various parts of the golfer 7 (including the left shoulder 71), position information of the grip end 51a, and information on the orientation of the shaft 52. Specifically, the determination unit 14e calculates an index value for determining whether or not the swing motion is good based on these pieces of information. There are various index values, which are set for each checkpoint. In the present embodiment, for example, the following index values are calculated. The following list is divided into index values for analyzing the front and index values for analyzing the right side.

(Index value at address-front)
The position of the left footpad with respect to the ball position (origin), the width of the stance with respect to the shoulder width, the balance of the upper body, the balance of the lower body The width of the stance with respect to the shoulder width is that of the right shoulder, left shoulder, left foot width reference position and right foot width reference position. From the position information, the upper body balance is the left shoulder, right shoulder, left foot width reference position and right foot width reference position, and the lower body balance is the left navel position, the right navel position, the left foot width reference position, and the right foot width reference position. It can be derived from the position information.

(Index value at address-right side)
The position of the right shoulder relative to the balance point line, the position of the right knee relative to the balance point line, and the spine angle The position of the right shoulder is based on the position information of the right shoulder and the right foot, and the position of the right knee is the position of the right knee head and the right foot. From the position information, the spine angle can be derived from the neck and waist positions. The balance point line is a line drawn in the vertical direction from the position of the right foot on the right side frame.

(Indicator value at 8 o'clock of takeback shaft-front)
The direction of the grip end 51a (toward the navel), the angle difference from the ideal shaft line, the amount of right shoulder movement from the address. The direction of the grip end 51a is the position information of the grip end 51a, the direction of the shaft 52a, and the address. From the position information of the navel at the time, the angle difference from the ideal shaft line is from the angle of the shaft 52 and the predetermined ideal angle, the amount of right shoulder movement from the address is the position information of the right shoulder and the position of the right shoulder at the address It can be derived from information.

(Index value of takeback shaft 9 o'clock-front)
The position of the grip end 51a with respect to the reference axis at the time of addressing The index value can be derived from the position information of the grip end 51a, and the right shoulder and the shoulder center at the time of addressing.

(Takeback arm horizontal index value-front)
Wrist cock angle, difference in height between both elbows, left shoulder movement from the address Note that the wrist cock angle is from the direction of the shaft 52, and the left shoulder movement from the address is the takeback arm level and the left shoulder at the address. It can be derived from the position information.

  The difference in height between both elbows can be derived, for example, as follows. FIG. 56 shows an image of front silhouette data when the takeback arm is horizontal. Here, first, frames L8 and L9 as shown in FIG. 56 are set based on the positions of the grip end 51a and the left shoulder 71 in addition to the frame L1 described above. Next, two reference points shown in FIG. 56 are identified on the image of the front silhouette data when the takeback arm is horizontal, and the width between these reference points is derived as an index value. When the index value is larger than the predetermined value, a line segment connecting the reference points is specified on the image of the front silhouette data when the takeback arm is horizontal. Subsequently, depth information on the line segment is derived from the front depth frame when the takeback arm is horizontal. And the difference between the depth based on the depth information of the point on the line segment included in the frame L8 and the depth based on the depth information of the point on the line segment included in the frame L9 is derived as an index value. Whether the left elbow or the right elbow is higher is determined according to the magnitude of the index value.

(Takeback arm horizontal index value-right side)
The spine angle based on the address The spine angle can be derived from the position information of the neck and waist and the spine angle at the address.

(Top index value-front)
Overswing, upper body warp, left / right movement from address, right knee position based on address, head position based on address, left shoulder position based on address From the angle, the upper body warp is from the head and umbilical position information, the left and right movement is the right umbilical position and umbilical position position information, the right foot width reference position at the address and the umbilical position information, the right knee position is From the position information of the right knee and the reference position of the right foot width at the time of address, the position of the head from the position of the head and the reference position of the right foot width at the time of address, the position of the left shoulder is the position information of the left shoulder and It can be derived from the position information of the left shoulder and the shoulder center at the time of addressing.

(Top index value-right side)
The amount of movement of the right knee from the address, the amount of movement of the left knee from the address, the wrist position based on the address, the height of both elbows, and the spine angle based on the address. From the position information of the right knee head at the time of knee head and address, the amount of movement of the left knee is from the position information of the left knee head and the left knee head at the time of address, and the position of the wrist is the position information of the neck and grip end 51a and the right at the time of addressing From the position information of the elbow and the ball (origin), the height of both elbows can be derived from the position information of the left and right elbows, and the spine angle can be derived from the position information of the neck and waist and the spine angle at the time of addressing.

(Down swing arm horizontal index value-front)
Wrist cock angle, left knee position relative to left foot width reference position at address Note that the wrist cock angle is based on the orientation of the shaft 52, the left knee position is the left knee position information and the left foot width reference position at the address It can be derived from the position information.

(Down swing arm horizontal index value-right side)
The spine angle based on the address The spine angle can be derived from the position information of the neck and waist and the spine angle at the address.

(Indicator value for down swing shaft 9 o'clock-front)
Wrist cock angle change, position of grip end 51a with respect to reference axis at address. Wrist cock angle change depends on shaft 52 orientation, left shoulder position information and grip end 51a position information, and downswing arm level. From the orientation of the shaft 52 at the time, the position of the grip end 51a can be derived from the position information of the grip end 51a and the position information of the right shoulder and the center of the shoulder at the time of addressing.

(Impact index value-front)
Left / right movement amount, vertical movement amount, head position, shoulder rotation amount, spine tilt, left elbow closure, position of grip end 51a with respect to the reference axis at the time of address. From the position information of the left shoulder, right shoulder and center of the shoulder at the time, the vertical movement amount is from the position information of the navel at the time of addressing and impact, the position of the head is the position information of the head and the position of the left shoulder and the center of the shoulder at the time of addressing From the information, the amount of rotation of the shoulder is the position information of the left shoulder and the right shoulder and the position information of the left shoulder and the right shoulder at the time of addressing, the inclination of the spine is the position information of the umbilicus and the center of the shoulder, the left elbow is From the position information of the elbow, the left shoulder and the grip end 51a, the position of the grip end 51a can be derived from the position information of the grip end 51a and the position information of the left shoulder and the shoulder center at the time of addressing.

(Impact index value-right side)
The spine angle based on the address and the knee angle based on the address Note that the spine angle is the position of the waist, right knee, and right ankle based on the information on the position of the neck and waist and the spine angle at the time of the address. It can be derived from the information and positional information of the waist, right knee and right ankle at the time of addressing.

  In the present embodiment, the correct data of the index value as described above (the range of the index value when the swing motion is determined to be good) is preset in the storage unit 13. The determination unit 12e determines the quality of the swing motion for each index value by comparing the correct answer data with the calculated index value. Also, it is possible to comprehensively evaluate the quality of the swing motion by giving points to the determination result for each index value and adding the points. The determination result is appropriately displayed on the display unit 11. For example, a list of determination results for each index value and a comprehensive evaluation can be displayed in the form of a report.

DESCRIPTION OF SYMBOLS 1 Swing analysis apparatus 2 Distance image sensor 3 Swing analysis program 5 Golf club 7 Golfer 14a Acquisition part 14b Position derivation | leading-out part (derivation | leading-out part)
14c Direction derivation unit (derivation unit)
51 Grip 52 Shaft 71 Left shoulder 100 Swing analysis system

Claims (31)

A swing analysis device for analyzing a swing motion of a golf club by a golfer,
An acquisition unit for acquiring a depth image obtained by capturing the swing motion by a distance image sensor;
Based on the depth image, a derivation unit that derives position information of a predetermined part of the golf club or the golfer during the swing operation or a direction of a shaft of the golf club during the swing operation With
Swing analysis device. The acquisition unit further acquires a two-dimensional image from the distance image sensor,
The derivation unit specifies the two-dimensional coordinates of the predetermined part based on the two-dimensional image, and specifies the depth of the predetermined part based on the depth image, thereby Derive 3D coordinates,
The swing analysis apparatus according to claim 1. The derivation unit sets a search range on the basis of the two-dimensional coordinates of the predetermined part at a second time before the first time on the two-dimensional image at the first time, and the search range Search for the two-dimensional coordinates of the predetermined part,
The swing analysis apparatus according to claim 2. The predetermined part is a grip of the golf club;
The swing analysis apparatus according to any one of claims 1 to 3. The predetermined part is a grip of the golf club;
The derivation unit specifies the two-dimensional coordinates of the shaft of the golf club based on the two-dimensional image, and specifies the two-dimensional coordinates of the grip based on the two-dimensional coordinates of the shaft.
The swing analysis apparatus according to claim 2 or 3. The predetermined part is a grip of the golf club;
The acquisition unit further acquires a two-dimensional image from the distance image sensor,
The derivation unit specifies the two-dimensional coordinates of the shaft of the golf club based on the two-dimensional image, and specifies the depth in the two-dimensional coordinates of the shaft based on the depth image. And the three-dimensional coordinates of the grip are specified based on the distribution tendency of the three-dimensional coordinates of the shaft.
The swing analysis apparatus according to claim 1. The two-dimensional image and the depth image are information obtained by photographing the swing motion from the front of the golfer with the distance image sensor,
The derivation unit specifies the two-dimensional coordinates of the shaft based on the two-dimensional image near the address, and specifies the three-dimensional coordinates of the grip based on the two-dimensional coordinates of the shaft.
The analysis device according to claim 5 or 6. The derivation unit binarizes the two-dimensional image while switching a threshold value between automatic and fixed according to the phase of the swing operation.
The analysis device according to claim 2. The derivation unit restricts the set position of the search range in the phase near the top during the swing operation with reference to the position of the head of the golfer.
The analysis device according to claim 3. The acquisition unit further acquires a time-series two-dimensional image from the distance image sensor,
If the position of the grip of the golf club cannot be extracted from the two-dimensional image at the first time, the derivation unit can determine the golf club from the two-dimensional image at the second time immediately after the first time. Identifying the position of the shaft, and identifying the position of the grip at the second time based on the position of the shaft;
The analysis device according to claim 1. The predetermined part is a shoulder of the golfer;
The swing analysis apparatus according to claim 1. The acquisition unit further acquires skeleton data representing a skeleton of a human body,
The derivation unit derives the three-dimensional coordinates of the shoulder based on the depth image and the skeleton data.
The swing analysis apparatus according to claim 11. The derivation unit derives the three-dimensional coordinates of the shoulder by filtering the depth image with a convex extraction filter that emphasizes a central part and suppresses a peripheral part.
The swing analysis apparatus according to claim 11 or 12. The derivation unit derives the three-dimensional coordinates of the shoulder by applying template matching to the depth image using a convex image template.
The swing analysis apparatus according to claim 11 or 12. The derivation unit derives three-dimensional coordinates of the golfer's shoulder by filtering the depth image by a shoulder silhouette filter that emphasizes the vicinity of one corner and suppresses the vicinity of the other corner.
The swing analysis apparatus according to any one of claims 11 to 14. The derivation unit derives the three-dimensional coordinates of the shoulder based on the skeleton data in at least a part of the section from the address to the takeback arm horizontal.
The swing analysis apparatus according to claim 12. The derivation unit derives the three-dimensional coordinates of the shoulder by performing filtering by the convex portion extraction filter in at least a part of the section from the takeback arm horizontal to the downswing arm horizontal.
The swing analysis apparatus according to claim 13. The derivation unit derives the three-dimensional coordinates of the shoulder by performing template matching with the convex image template in at least a partial section from the takeback arm horizontal to the downswing arm horizontal.
The swing analysis apparatus according to claim 14. The derivation unit derives the three-dimensional coordinates of the shoulder by performing filtering by the shoulder silhouette filter in at least a part of the section from the address to the takeback arm level and the downswing arm level to the finish.
The swing analysis apparatus according to claim 15. The derivation unit sets a search range on the basis of the two-dimensional coordinates of the shoulder at the first time or the second time before the first time on the depth image at the first time, Performing the filtering within the search range;
The swing analysis apparatus according to any one of claims 11 to 19. The derivation unit performs calibration by substituting the three-dimensional coordinates of the predetermined part into a predetermined multiple regression equation having a predetermined coefficient.
The swing analysis apparatus according to any one of claims 1 to 20. The acquisition unit acquires the depth image obtained by capturing the swing motion from at least a first direction and a second direction by a plurality of the distance image sensors.
The swing analysis apparatus according to any one of claims 1 to 21. The acquisition unit further acquires skeleton data representing a skeleton of a human body based on at least the depth image from the first direction,
The derivation unit converts the skeleton data from the first direction into the skeleton data from the second direction, and derives position information of the predetermined part based on the skeleton data from the second direction. To
The swing analysis apparatus according to claim 22. The first direction is a direction toward the front of the golfer, and the second direction is a direction toward the side of the golfer.
The swing analysis apparatus according to claim 22 or 23. The acquisition unit further acquires skeleton data representing a human body skeleton based on the depth image,
The deriving unit derives position information of a first part included in the predetermined part based on the skeleton data;
The swing analysis apparatus according to any one of claims 1 to 24. The acquisition unit further acquires silhouette data representing a silhouette of a human body,
The derivation unit derives position information of a second part different from the first part included in the predetermined part based on the skeleton data and the silhouette data.
The swing analysis apparatus according to claim 25. The derivation unit expands the region of the silhouette specified in the silhouette data based on information of a region where depth information is not obtained in the depth image.
The swing analysis apparatus according to any one of claims 1 to 26. In the depth image from the right side of the golfer at the timing of the top, the deriving unit derives a position protruding to the near side as the left elbow of the golfer.
The swing analysis apparatus according to any one of claims 1 to 27. The quality of the swing motion is determined by comparing an index value calculated based on at least one of the position information of the predetermined part and the three-dimensional vector at a predetermined timing with predetermined correct answer data. A determination unit;
The swing analysis apparatus according to any one of claims 1 to 28. A swing analysis method for analyzing a swing motion of a golf club by a golfer,
Obtaining a depth image obtained by photographing the swing motion by a distance image sensor;
Deriving, based on the depth image, a position information of a predetermined part of the golf club or the golfer during the swing operation, or a three-dimensional vector representing a direction of a shaft of the golf club during the swing operation; Comprising
Swing analysis method. A swing analysis program for analyzing the swing motion of a golf club by a golfer,
Obtaining a depth image obtained by photographing the swing motion by a distance image sensor;
Deriving, based on the depth image, a position information of a predetermined part of the golf club or the golfer during the swing operation, or a three-dimensional vector representing a direction of a shaft of the golf club during the swing operation; To run on a computer,
Swing analysis program.