JP2017119102A - Motion analysis device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a motion analysis device capable of accurately analyzing motion of a mobile object.SOLUTION: The motion analysis device for analyzing motion of a mobile object is provided. The motion analysis device comprises a first acquisition part, a second acquisition part and a derivation part. The first acquisition part acquires time-series sensor data, including at least one of time-series angular velocity data, acceleration data and terrestrial magnetism data output from a sensor unit. The sensor unit is attached to the mobile object, and includes at least one of an angular velocity sensor, an acceleration sensor and a terrestrial magnetism sensor. The second acquisition part acquires time-series image data including time-series depth images obtained by photographing motion of the mobile object using a distance image sensor and a time-series two-dimensional image. The derivation part derives a three-dimensional behavior of the mobile object based on the sensor data and the image data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a motion analysis apparatus, method, and program for analyzing motion of a moving body.

  Conventionally, a technique for attaching an inertial sensor including an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor to a moving body and estimating the behavior of the moving body based on an output value of the inertial sensor is known (Patent Document 1, etc.). In addition, the movement of a moving body is photographed with a camera, and the behavior of the moving body is estimated based on the obtained image data (Patent Documents 2, 3, etc.).

JP 2011-227017 A JP 2013-215349 A JP 2004-313479 A

  However, when the analysis is performed based on the output value of the inertial sensor as in Patent Document 1, since the output value includes an error due to a drift component of the inertial sensor, the accuracy of the analysis may be reduced. For example, when integrating the acceleration data, which is the output value of the inertial sensor, to obtain the speed and position, errors can be accumulated by this integration. Further, when the moving body moves at a high speed, the error of the angular velocity sensor becomes particularly large, and it may be difficult to estimate the posture of the moving body. Further, particularly when the rotational motion is large, it can be difficult to correctly evaluate the influence of gravitational acceleration included in the output value of the acceleration sensor.

  On the other hand, when performing analysis based on image data as in Patent Documents 2 and 3, there is often a problem in terms of accuracy. For example, when the position of a moving object is estimated by image processing and the speed and acceleration are obtained by differentiating the data of the position, even if the error in position estimation is small, the error may increase due to differentiation. .

  An object of the present invention is to provide a motion analysis apparatus, method, and program capable of analyzing the motion of a moving object with high accuracy.

  A motion analysis device according to a first aspect of the present invention is a motion analysis device for analyzing motion of a moving body, and includes a first acquisition unit, a second acquisition unit, and a derivation unit. The first acquisition unit acquires time-series sensor data including at least one of time-series angular velocity data, acceleration data, and geomagnetic data output from the sensor unit. The sensor unit is attached to the moving body and includes at least one of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor. The second acquisition unit acquires time-series image data including a time-series depth image and a two-dimensional image obtained by photographing the movement of the moving body by the first distance image sensor. The deriving unit derives a three-dimensional behavior of the moving body based on the sensor data and the image data.

  A motion analysis apparatus according to a second aspect of the present invention is the motion analysis apparatus according to the first aspect, wherein the derivation unit minimizes an objective function including a function defined using the image data and the sensor data. The posture of the moving body is derived as an optimal solution to be maximized or maximized.

  A motion analysis apparatus according to a third aspect of the present invention is the motion analysis apparatus according to the second aspect, wherein the function is defined using a direction of the moving body derived from the image data.

  A motion analysis apparatus according to a fourth aspect of the present invention is the motion analysis apparatus according to the second or third aspect, wherein the function is defined using at least one of the geomagnetic data and acceleration data.

  A motion analysis apparatus according to a fifth aspect of the present invention is the motion analysis apparatus according to any one of the second to fourth aspects, wherein the derivation unit includes an initial posture of the moving body and time-series angular velocity data. Is used to calculate a temporary posture of the moving body. The function is defined using the temporary posture.

  A motion analysis apparatus according to a sixth aspect of the present invention is the motion analysis apparatus according to the fifth aspect, wherein the derivation unit is an initial direction of the moving body and initial acceleration data derived from the image data. Alternatively, using the angular velocity data, a stationary posture of the moving body is derived as an initial posture of the moving body.

  A motion analysis apparatus according to a seventh aspect of the present invention is the motion analysis apparatus according to any one of the first to sixth aspects, wherein the derivation unit uses a differential filter for the posture of the moving body and the posture of the moving body. The angular velocity of the moving body is derived based on the differential data filtered by the above.

  A motion analysis apparatus according to an eighth aspect of the present invention is the motion analysis apparatus according to any of the first to seventh aspects, wherein the derivation unit is derived from the image data using the acceleration data. By smoothing at least one of the temporary position of the moving body and the temporary speed obtained by differentiating the temporary position, at least one of the position and speed of the moving body is derived.

  The motion analysis apparatus according to a ninth aspect of the present invention is the motion analysis apparatus according to the eighth aspect, wherein the derivation unit filters the acceleration of the mobile object by filtering the speed of the mobile object with a differential filter. To derive.

  A motion analysis apparatus according to a tenth aspect of the present invention is the motion analysis apparatus according to any one of the first to ninth aspects, wherein the derivation unit has a direction of the moving body derived from the image data. Time alignment of the time-series image data and the time-series sensor data is performed so that the degree of coincidence between the series data and the time-series data of the moving body direction derived from the sensor data is the highest. .

  A motion analysis apparatus according to an eleventh aspect of the present invention is the motion analysis apparatus according to any one of the first to tenth aspects, further including a third acquisition unit. The third acquisition unit acquires time-series image data including a time-series depth image and a two-dimensional image obtained by capturing the motion of the moving body from a direction different from the first distance image sensor by the second distance image sensor. To do. The derivation unit derives a three-dimensional behavior of the moving body based on the sensor data and the image data acquired by the second acquisition unit and the third acquisition unit.

  A motion analysis apparatus according to a twelfth aspect of the present invention is the motion analysis apparatus according to any of the first to eleventh aspects, wherein the derivation unit is configured to move the moving body at a first time based on the image data. The movement between the second time different from the first time and the first time so that the posture of the moving body is derived and the posture of the moving body matches the instantaneous posture at the first time Interpolate body posture.

A motion analysis method according to a thirteenth aspect of the present invention is a motion analysis method for analyzing the motion of a moving body, and includes the following steps (1) to (3).
(1) Using the sensor unit, obtaining time-series sensor data including at least one of time-series angular velocity data, acceleration data, and geomagnetic data. The sensor unit is attached to the moving body and includes at least one of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor.
(2) A step of acquiring time-series image data including a time-series depth image and a two-dimensional image obtained by photographing the motion of the moving body using a distance image sensor.
(3) Deriving a three-dimensional behavior of the moving body based on the sensor data and the image data.

The motion analysis program according to the fourteenth aspect of the present invention is a motion analysis program for analyzing the motion of a moving body, and causes a computer to execute the following steps (1) to (3).
(1) A step of acquiring time-series sensor data including at least one of time-series angular velocity data, acceleration data, and geomagnetic data output from the sensor unit. The sensor unit is attached to the moving body and includes at least one of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor.
(2) A step of acquiring time-series image data including a time-series depth image and a two-dimensional image obtained by photographing the movement of the moving body by a distance image sensor.
(3) Deriving a three-dimensional behavior of the moving body based on the sensor data and the image data.

  According to the present invention, the motion of the moving body includes (1) time-series sensor data including at least one of time-series angular velocity data, acceleration data, and geomagnetic data, and (2) a time-series depth image and two Derived based on time-series image data including a dimensional image. As a result, the error included in the sensor data is reduced based on the image data. Alternatively, the error included in the image data is reduced based on the sensor data. Therefore, the movement of the moving body can be analyzed with high accuracy.

The figure which shows the whole structure of the motion analysis system containing the motion analysis apparatus which concerns on 1st Embodiment of this invention. The functional block diagram of the motion analysis system of FIG. The figure explaining xyz local coordinate system on the basis of the grip vicinity of a golf club. (A) The figure which shows an address state. (B) The figure which shows a top state. (C) The figure which shows an impact state. (D) The figure which shows a finish state. The flowchart which shows the flow of the motion analysis method which concerns on 1st Embodiment. The flowchart which shows the flow of the time adjustment process which concerns on 1st Embodiment. The figure explaining the direction of a two-dimensional shaft. The conceptual diagram explaining the principle of a time adjustment process. The flowchart which shows the flow of the derivation | leading-out process of the attitude | position of an address period. The flowchart which shows the flow of the derivation | leading-out process of the attitude | position from the address to the impact vicinity. The flowchart which shows the flow of the derivation | leading-out process of the position and speed of a grip. The flowchart which shows the flow of the derivation | leading-out process of the angular velocity and angular acceleration of a grip. The flowchart which shows the flow of the derivation | leading-out process of the acceleration of a grip. The graph which shows Example 1 and Reference Example 1. FIG. The graph which shows Example 2 and Reference Example 2. FIG. The graph which shows Example 3 and Reference Example 3 (position of the grip in the X-axis direction). The graph which shows Example 3 and Reference Example 3 (position of the Y-axis direction of a grip). The graph which shows Example 3 and Reference Example 3 (position of a grip in the Z-axis direction). The graph which shows Example 4 and Reference Example 4 (position of the X-axis direction of a grip). The graph which shows Example 4 and Reference Example 4 (position of the Y-axis direction of a grip). The graph which shows Example 4 and Reference Example 4 (position of a grip in the Z-axis direction). The figure which shows the whole structure of the motion analysis system containing the motion analysis apparatus which concerns on 2nd Embodiment of this invention. The functional block diagram of the exercise | movement analysis system of FIG. The flowchart which shows the flow of the motion analysis method which concerns on 2nd Embodiment. The flowchart which shows the flow of the time adjustment process which concerns on 2nd Embodiment. The flowchart which shows the flow of the correction process of the attitude | position in a down swing period. 10 is a graph showing Example 5. The flowchart which shows the flow of the derivation | leading-out process of the attitude | position of the initial time which concerns on a modification.

  Hereinafter, some embodiments in the case of using a motion analysis device, method, and program according to the present invention for golf swing analysis will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Outline of Motion Analysis System>
1 and 2 show an overall configuration diagram of a motion analysis system 100 including a motion analysis device 1 according to the first embodiment of the present invention. The motion analysis system 100 is configured to be suitable for analyzing the swing motion of the golf club 5 by the golfer 7. In addition to time-series angular velocity data, acceleration data, and geomagnetic data (hereinafter sometimes referred to as sensor data) output from the inertial sensor unit 4 attached to the golf club 5, the motion analysis apparatus 1 captures a swing motion. The swing motion is analyzed based on the time-series image data (that is, moving image data). The above photographing is performed by the distance image sensor 2. The motion analysis apparatus 1 constitutes a motion analysis system 100 together with the inertial sensor unit 4 and the distance image sensor 2. The result of the analysis by the motion analysis apparatus 1 is used for various purposes such as fitting of the golf club 5 suitable for the golfer 7, improvement of the form of the golfer 7, development of golf equipment, and the like.

  In the present embodiment, the three-dimensional behavior of the grip 51 and the shaft 52 of the golf club 5 that moves during the swing operation is mainly analyzed. More specifically, the posture of the shaft 52 is derived together with the position (three-dimensional coordinates), speed, acceleration, angular velocity, and angular acceleration of the grip 51 during the swing operation.

<1-2. Details of each part>
Hereinafter, details of each part of the motion analysis system 100 will be described.

<1-2-1. Distance image sensor>
The distance image sensor 2 is a camera having a ranging function that measures a distance to a subject while shooting a golfer 7 trial hitting the golf club 5 as a two-dimensional image. Therefore, the distance image sensor 2 can output a time-series depth image together with a time-series two-dimensional image. Note that the two-dimensional image here is an image obtained by projecting an image of the photographing space onto a plane orthogonal to the optical axis of the camera. The depth image is an image in which data on the depth of the subject in the optical axis direction of the camera is assigned to pixels within substantially the same imaging range as the two-dimensional image.

  The distance image sensor 2 used in the present embodiment captures a two-dimensional image as an infrared image (hereinafter referred to as an IR image). The depth image is obtained by a method such as a time-of-flight method using infrared rays or a dot pattern projection method. Accordingly, as shown in FIG. 1, the distance image sensor 2 has an IR light emitting unit 21 that emits infrared rays toward the front, and an IR that receives the infrared rays that are emitted from the IR light emitting unit 21 and reflected back to the subject. A light receiving unit 22. The IR light receiving unit 22 is a camera having an optical system, an image sensor, and the like. In the dot pattern projection method, the IR dot pattern emitted from the IR light emitting unit 21 is read by the IR light receiving unit 22, the dot pattern is detected by image processing inside the distance image sensor 2, and the depth is calculated based on this. The In the present embodiment, the IR light emitting unit 21 and the IR light receiving unit 22 are accommodated in the same housing 20 and disposed in front of the housing 20. In the distance image sensor 2 according to the present embodiment, an effective distance capable of measuring the depth is determined, and the depth image may include pixels for which a depth value cannot be obtained.

  In addition to the CPU 23 that controls the entire operation of the distance image sensor 2, the distance image sensor 2 includes a memory 24 that stores captured image data at least temporarily. A control program for controlling the operation of the distance image sensor 2 is stored in the memory 24. The distance image sensor 2 also includes a communication unit 25, and the communication unit 25 transmits captured image data to an external device such as the motion analysis apparatus 1 via a wired or wireless communication line 17. Can be output to the device. In the present embodiment, the CPU 23 and the memory 24 are also housed in the housing 20 together with the IR light emitting unit 21 and the IR light receiving unit 22. It is not always necessary to transfer the image data to the motion analysis apparatus 1 via the communication unit 25. For example, if the memory 24 is detachable, it is removed from the housing 20 and inserted into a reader (corresponding to a communication unit 15 described later) of the motion analysis device 1, so that the motion analysis device 1 captures image data. Can be read.

  In the present embodiment, as described above, infrared imaging is performed by the distance image sensor 2, and the trajectory of the grip 51 is tracked based on the captured IR image. Therefore, a reflection sheet that efficiently reflects infrared rays is attached to the grip 51 (more precisely, the grip end 51a) as a marker so as to facilitate this tracking. A similar infrared reflective sheet is also attached to the shaft 52 as a marker. In the present embodiment, the distance image sensor 2 is installed in front of the golfer 7 so as to photograph the golfer 7 from the front side.

<1-2-2. Inertial sensor unit>
As shown in FIGS. 1 and 3, the inertial sensor unit 4 is attached to the grip end 51a of the golf club 5, and measures the behavior of the grip end 51a. The inertial sensor unit 4 according to the present embodiment is configured to be detachable and can be attached to an arbitrary golf club 5. The golf club 5 is a general golf club, and includes a shaft 52, a head 53 provided at one end of the shaft 52, and a grip 51 provided at the other end of the shaft 52. The inertial sensor unit 4 is configured to be small and light so as not to hinder the swing operation. As shown in FIG. 2, an acceleration sensor 41, an angular velocity sensor 42, and a geomagnetic sensor 43 are mounted on the inertial sensor unit 4 according to the present embodiment. The inertial sensor unit 4 is also equipped with a communication device 40 for transmitting sensor data output from these sensors 41 to 43 to an external device such as the motion analysis device 1. In the present embodiment, the communication device 40 is wireless so as not to hinder the swing operation, but may be connected to the motion analysis device 1 in a wired manner via a cable.

The acceleration sensor 41, the angular velocity sensor 42, and the geomagnetic sensor 43 respectively measure acceleration, angular velocity, and geomagnetism in the xyz local coordinate system. More specifically, the acceleration sensor 41 measures accelerations a _x , a _y , and a _z in the x-axis, y-axis, and z-axis directions. The angular velocity sensor 42 measures angular velocities ω _x , ω _y , and ω _z around the x axis, the y axis, and the z axis. Geomagnetic sensor 43 measures the x-axis, y-axis and z-axis direction terrestrial magnetism m _x, m _y, a m _z. These sensor data are acquired as time-series data of a predetermined sampling period Δt. The xyz local coordinate system is a three-axis orthogonal coordinate system defined as shown in FIG. That is, the z axis coincides with the direction in which the shaft 52 extends, and the direction from the head 53 toward the grip 51 is the positive z axis direction. The y-axis is oriented so as to be as much as possible along the flying direction of the golf club 5 at the time of addressing, that is, generally along the face-back direction. The x-axis is oriented so as to be orthogonal to the y-axis and the z-axis, that is, substantially along the toe-heel direction, and the direction from the heel side to the toe side is the x-axis positive direction.

  In the present embodiment, sensor data from the acceleration sensor 41, the angular velocity sensor 42, and the geomagnetic sensor 43 are transmitted to the motion analysis apparatus 1 in real time via the communication device 40. However, for example, sensor data may be stored in a storage device in the inertial sensor unit 4, and the sensor data may be taken out from the storage device after the swing operation is finished and transferred to the motion analysis device 1.

<1-2-3. Motion analysis device>
The motion analysis apparatus 1 is a general-purpose personal computer as hardware, and is realized as, for example, a tablet computer, a smartphone, a notebook computer, or a desktop computer. As shown in FIG. 2, the motion analysis apparatus 1 is manufactured by installing a motion analysis program 3 from a computer-readable recording medium 30 such as a CD-ROM in a general-purpose computer. The motion analysis program 3 is software for analyzing a golf swing based on the sensor data sent from the inertial sensor unit 4 and the image data sent from the distance image sensor 2. Execute the action to be performed.

  The motion analysis apparatus 1 includes a display unit 11, an input unit 12, a storage unit 13, a control unit 14, and a communication unit 15. These units 11 to 15 are connected to each other via the bus line 16 and can communicate with each other. The display unit 11 can be composed of a liquid crystal display or the like, and displays the result of golf swing analysis or the like to the user. In addition, a user here is a general term for those who require the result of golf swing analysis, such as golfer 7 itself, its instructor, and a developer of golf equipment. The input unit 12 can be configured with a mouse, a keyboard, a touch panel, and the like, and accepts an operation from the user for the motion analysis apparatus 1.

  The storage unit 13 can be configured with a hard disk or the like. In the storage unit 13, the motion analysis program 3 is stored, and sensor data sent from the inertial sensor unit 4 is stored. Further, the image data sent from the distance image sensor 2 is also stored in the storage unit 13. The control part 14 can be comprised from CPU, ROM, RAM, etc. The control unit 14 virtually operates as the first acquisition unit 14a, the second acquisition unit 14b, the derivation unit 14c, and the display control unit 14d by reading and executing the motion analysis program 3 in the storage unit 13. Details of the operations of the units 14a to 14d will be described later. The communication unit 15 functions as a communication interface that receives data from an external device such as the inertial sensor unit 4 and the distance image sensor 2 via the communication line 17.

<1-3. Motion analysis method>
Hereinafter, a motion analysis method for a golf swing will be described with reference to FIG. This motion analysis method is realized by the first data collection process (S1), the second data collection process (S2), and the motion analysis process (S3 to S9). The first data collection process is a process for collecting sensor data output from the inertial sensor unit 4, and the second data collection process is a process for collecting image data output from the distance image sensor 2. The motion analysis processing is processing for analyzing the three-dimensional behavior of the golf club 5 by the motion analysis device 1 based on the sensor data and the image data. Hereinafter, these processes will be described in order.

<1-3-1. First data collection process>
In the first data collection process, the golf club 5 with the inertial sensor unit 4 is swung by the golfer 7. In this case, the inertial sensor unit 4, the acceleration a _x in the golf swing, a _y, a _z, the angular velocity ω _x, ω _y, ω _z and geomagnetism m _x, m _y, m _z of the sensor data is detected. Further, these sensor data are transmitted to the motion analysis device 1 via the communication device 40 of the inertial sensor unit 4. On the other hand, on the side of the motion analysis device 1, the first acquisition unit 14 a receives this via the communication unit 15 and stores it in the storage unit 13. In the present embodiment, time-series sensor data from at least a little before the address (hereinafter referred to as initial time t ₁ ) to the finish is collected.

  Note that the golf club swing operation generally proceeds in the order of address, top, impact, and finish. The address means a stationary state in which the head 53 of the golf club 5 is arranged near the ball as shown in FIG. 4A, and the top means that the golf club 5 is moved from the address as shown in FIG. 4B. It means a state in which the head 53 is swung up by taking back. As shown in FIG. 4 (C), the impact means a state at the moment when the golf club 5 is swung down from the top and the head 53 collides with the ball, and the finish is as shown in FIG. 4 (D). It means a state in which the golf club 5 is swung forward after impact.

<1-3-2. Second data collection process>
In the second data collection process, the distance image sensor 2 captures the swing motion of the golf club 5. That is, the distance image sensor 2 detects image data including an IR image and a depth image capturing a golf swing. The second data collection process is performed on the same swing operation as the first data collection process in parallel with the first data collection process. The detected image data is transmitted to the motion analysis apparatus 1 via the communication unit 25 of the distance image sensor 2. On the other hand, on the side of the motion analysis device 1, the second acquisition unit 14 b receives this via the communication unit 15 and stores it in the storage unit 13. In this embodiment, at least time-series image data from the initial time t ₁ slightly before the address to the finish is collected.

<1-3-3. Motion analysis processing>
The general flow of the motion analysis process is as follows. That is, first, time adjustment is performed on the time-series sensor data acquired by the first data collection process and the time-series image data acquired by the second data collection process (S3). In other words, the derivation unit 14c synchronizes the sensor data and the image data.

In step S4, deriving unit 14c is, the period from the initial time t ₁ to the address (hereinafter, referred to as an address period) to derive the posture of the shaft 52. At this time, in this embodiment, optimization using Lagrange's undetermined multiplier method is executed. Many golfers 7 perform a waggle operation that swings the golf club 5 back and forth lightly during the address period. However, compared with the swing operation after the address, the golf club 5, especially the grip 51 hardly moves during the address period. For this reason, in step S4, an algorithm suitable for the motion analysis of the stationary moving body and the substantially stationary moving body is used.

  In subsequent step S5, the deriving unit 14c derives the attitude of the shaft 52 from the address to the vicinity of the impact. Therefore, in step S5, an algorithm suitable for not only the motion of the stationary moving body and the substantially stationary moving body but also the analysis of the moving body moving at high speed is used. In this embodiment, again, optimization using the Lagrange multiplier method is performed.

  In subsequent step S6, the deriving unit 14c derives the position and speed of the grip 51 from the address to the vicinity of the impact. Here, in this embodiment, smoothing using a Kalman filter or the like is executed.

  In subsequent step S7, the deriving unit 14c derives the angular velocity and the angular acceleration of the grip 51 from the address to the vicinity of the impact. Here, in this embodiment, filtering using a pseudo differential filter is executed.

  In subsequent step S8, the deriving unit 14c derives the acceleration of the grip 51 from the address to the vicinity of the impact. In the present embodiment, filtering using a pseudo differential filter is also executed here.

  In subsequent step S9, the display control unit 14d displays information indicating the three-dimensional behavior derived in steps S4 to S8 on the display unit 11. At this time, for example, it can be displayed as numerical information or character information, or can be displayed graphically using a graph, computer graphics, or the like.

<1-3-3-1. Time adjustment (S3)>
Hereinafter, the time adjustment processing of the time-series sensor data and the time-series image data in step S3 will be described with reference to FIG.

  First, in step S11, the deriving unit 14c refers to the sensor data in the storage unit 13 and derives the address, top, and impact time. Since various methods for deriving these times are known, detailed description is omitted here.

In subsequent step S12, the deriving unit 14c derives the posture matrix N at each time from the address to the vicinity of the impact based on the sensor data in the storage unit 13. Now, the posture matrix N is expressed by the following equation. The posture matrix N is a matrix for converting values in the xyz local coordinate system into values in the XYZ inertial coordinate system.

  The XYZ inertial coordinate system is a three-axis orthogonal coordinate system defined as shown in FIG. That is, the Z-axis is a direction from the vertically lower side to the upper side, the X-axis is a direction from the back to the stomach of the golfer 7, and the Y-axis is parallel to the ground plane and goes from the ball hitting point to the target point. Direction.

The meanings of the nine components of the posture matrix N are as follows.
Component a: Cosine of the angle formed by the X axis of the inertial coordinate system and the x axis of the local coordinate system Component b: Cosine of the angle formed by the Y axis of the inertial coordinate system and the x axis of the local coordinate system Component c: Inertia Cosine of angle formed by Z axis of coordinate system and x axis of local coordinate system Component d: Cosine of angle formed by X axis of inertial coordinate system and y axis of local coordinate system Component e: Y of inertial coordinate system Cosine of angle between axis and y-axis of local coordinate system Component f: cosine of angle between Z-axis of inertial coordinate system and y-axis of local coordinate system Component g: X-axis of local coordinate system and local Cosine of angle between z axis of coordinate system Component h: Cosine of angle between Y axis of inertial coordinate system and z axis of local coordinate system Component i: Z axis of inertial coordinate system and z of local coordinate system Here, the vector (a, b, c) represents a unit vector in the x-axis direction, and the vector (d, e, f) represents a single vector in the y-axis direction. Represents a vector, the vector (g, h, i) represents a unit vector in the z-axis direction.

Since the golf club 5 is stationary at the time of addressing, the acceleration sensor 42 detects only the gravitational acceleration in the Z-axis direction. From this fact, c, f, and i at the time of address can be calculated according to the following equation.

  The remaining a, b, d, e, g, and h are predetermined weights with c, f, and i as explanatory variables and a, b, d, e, g, and h as objective variables, respectively. It can be calculated from a regression equation. This multiple regression equation can be determined by multiple regression analysis of a large number of true value data of a to i. Alternatively, the attitude matrix can be expressed as an Euler transformation matrix as shown in the following equation. Therefore, if three values of nine components of the posture matrix are known, φ, θ, and ψ can be specified, and the remaining components can be calculated. Various methods can be used for calculating the attitude matrix, and any method may be used. However, the method described here is also disclosed in, for example, Japanese Patent Laid-Open No. 2013-56074.

After deriving the posture matrix N at the time of addressing, the deriving unit 14c updates the posture matrix N from time to time at intervals of the sampling period Δt, with the posture matrix N at the time of addressing as an initial value. More specifically, first, the attitude matrix N is expressed by the following equation using Euler parameters q = (q ₀ , q ₁ , q ₂ , q ₃ ).

Therefore, when a to i at a certain time are known, the Euler parameters q = (q ₀ , q ₁ , q ₂ , q ₃ ) at the time can be calculated according to the following equation.

  Now, the values of a to i that define the attitude matrix N at the time of address are known. Therefore, according to the above formula, first, the Euler parameter q at the time of addressing is calculated.

Euler parameter q _{t + 1} after a lapse of a minute time from time t is expressed by the following equation using Euler parameter q _t at time t.

Further, the first-order differential equation representing the time change of the Euler parameter q is represented by the following equation.

  By using the equations (6) and (7), the Euler parameter q at time t can be sequentially updated to the Euler parameter q at the next time t + Δt based on the angular velocity data. Here, the Euler parameter q from the address to the impact is calculated. The posture matrix N from the address to the impact is calculated by sequentially substituting the Euler parameter q from the address to the impact into the equation (4).

  Note that, as a method of calculating the Euler parameter q and the attitude matrix N based on the sensor data output from the inertial sensor unit, not only the method described here but also various methods are known. For example, “Yasumi Neda, Kouta Fujihashi, Kuniyuki Omagari, Ken Fujiwara, Saburo Matsunaga“ Attitude Control System for the Small Satellite Cute-1.7 Series ”, 16th Space Engineering Conference, Tokyo, January 25, 2007 It is also possible to use the technology disclosed in.

In subsequent step S13, the deriving unit 14c derives the time-series direction θ _s of the shaft 52 at each time from the address to the vicinity of the impact (see FIG. 7) based on the image data in the storage unit 13. Specifically, in this embodiment, the image of the linear shaft 52 is detected by performing image processing on the IR image. Then, the angle formed by the direction in which the image of the shaft 52 extends and the Z direction is derived as the direction θ _s of the shaft 52.

In step S14, it is deriving unit 14c, based on the posture matrix N of the time series of step S12, to derive the orientation theta _s shaft 52 of a time series at each time from the address to the vicinity of the impact. Specifically, θ _s can be calculated according to the following equation.

Next, the derivation unit 14c sets both time series so that the degree of coincidence between the direction θ _s of the time series shaft 52 derived from the sensor data and the direction θ _s of the time series shaft 52 in step S13 is the highest. Align the data. For example, as shown in FIG. 8, the deriving unit 14c determines the point at which the waveforms of the two time series data most closely match while moving the two time series data back and forth along the time axis. Then, based on the timing at that time, the time axis of the sensor data and the time axis of the image data are relatively time aligned.

In the present embodiment, the time is adjusted based on the orientation θ _s of the two-dimensional shaft 52, but the time can also be adjusted based on the orientation θ _s of the three-dimensional shaft 52. That is, it is possible to derive the orientation (g, h, i) of the three-dimensional shaft 52 from the posture matrix N, and if the depth image is used, the three-dimensional shaft 52 can be derived from the image data. It is possible to derive the orientation. Therefore, it is only necessary to derive the time-series data of the direction of the three-dimensional shaft 52 derived from the sensor data and the image data, and determine the timing at which the waveforms of these time-series data most closely match in three dimensions. .

<1-3-3-2. Derivation of attitude during address period (S4)>
The flow of the address period attitude derivation process will be described below with reference to FIG. Here, the Euler parameter q can be mutually converted with the attitude matrix N as described above. Further, as described above, the row vector included in the posture matrix N represents the directions of the three axes of the local coordinate system in the inertial coordinate system. Therefore, if at least one of the Euler parameter q and the attitude matrix N is known, the direction of the three axes of the local coordinate system in the inertial coordinate system, that is, the direction (g, h, i) of the shaft 52 parallel to the z axis can be determined. I understand. Therefore, in the following, the Euler parameter q and the attitude matrix N are derived as the attitude of the shaft 52.

  First, in step S21, the deriving unit 14c derives postures q and N at each time included in the address period. In the following step S26, the posture q is optimized by Lagrange's undetermined multiplier method. In this sense, the postures q and N derived in step S21 can be said to be temporary postures.

Specifically, deriving unit 14c calculates the Euler parameter q of the initial time t ₁ is the first time included in the address period. Specifically, deriving unit 14c derives the orientation of the shaft 52 in three dimensions from the IR image and depth image of the initial time t _1. That is, the image of the shaft 52 is detected from the IR image by image processing, and the Y coordinate and Z coordinate of each point included in the image are derived. Then, the depth information at these points is extracted from the depth image and converted to the X coordinate. Thereafter, the direction (g, h, i) of the three-dimensional shaft 52 is derived from these three-dimensional coordinates. Further, deriving unit 14c, the acceleration a _x of the initial time t _1, a _y, based on the fact that a _z represents almost gravitational only acceleration data of the initial time _{t 1 (a x, a y} , a The direction of three-dimensional gravity is derived from _z ). Then, the deriving unit 14c derives the outer product of the direction (g, h, i) of the shaft 52 and the direction of gravity, and sets this as the y-axis direction (d, e, f). Further, an outer product of the direction (g, h, i) of the three-dimensional shaft 52 and the direction (d, e, f) of the y axis is derived, and this is expressed as the direction (a, b, c) of the x axis. To do. Thereby, the posture matrix N at the initial time t ₁ is derived. In addition, the derivation unit 14c converts the attitude matrix N at the initial time t _{1 into} the Euler parameter q at the initial time t ₁ according to the equation (5).

In step S21, the derivation unit 14c updates the posture matrix N from time to time at the sampling period Δt with the posture matrix N at the initial time t ₁ as an initial value. Since the specific method is the same as that in step S12 described above, detailed description thereof is omitted here.

In subsequent step S22, the derivation unit 14c calculates the geomagnetism m ^B = (m ^B _x , m ^B _y , m ^B _z ) = (m _x , m _y , m _z ) in the local coordinate system at the initial time t ₁ , It is converted into geomagnetism m ^I = (m ^I _x , m ^I _y , m ^I _z ) in the inertial coordinate system. In addition, not only m ^I and m ^B , when B is added to the right shoulder, it indicates a vector of the local coordinate system, and when I is added to the right shoulder, it is a vector of the inertial coordinate system. Represents that. m ^I can be constant regardless of the time, and can be calculated according to the following equation from m ^{B at the} initial time t ₁ using the Euler parameter q. Note that the Euler parameter q at the initial time t ₁ derived in step S21 is substituted for q below.
However,
It is.

The geomagnetism m ^I can be derived by another method. For example, a magnetic sensor different from the magnetic sensor 43 is arranged in a state where the three axes of the magnetic sensor coincide with the three axes of the inertial coordinate system at the origin of the inertial coordinate system. In this case, the output value of the magnetic sensor represents the geomagnetism m ^I in the inertial coordinate system.

In step S23, deriving unit 14c, to any combination of time t _i, t _j included in the address period is the difference between the posture q _i at time t _i, the attitude q _j at time t _j , Q _BjBi is derived. At this time, the order of t _j and t _i does not matter. The attitude difference q _BjBi can be calculated according to the following equation. The Euler parameters q _i and q _{j that} are substituted into the following expression are the Euler parameters q calculated in step S21.

The attitude difference q _BjBi can be derived by another method. For example, the Euler parameter q at each time included in the address period to be substituted into Equation 11 is not calculated in step S21 but can be calculated by the following method. That is, in step S21, Euler parameters q each time after the initial time t ₁ is calculated in the same manner as step S12 that slide into sequentially updated Euler parameter q of the initial time t ₁ by using the angular velocity data It was. However, for the initial time t ₁ each time the orientation matrix N later than, as is the case of the initial time t _1, the image data and the acceleration data may be calculated by a method of calculating the outer product by using.

Alternatively, the attitude difference q _BjBi in the address period can be approximated as a unit quaternion (1, 0, ₀ , 0). This is because the grip 51 is stationary or hardly moving during the address period, and there is almost no difference in posture.

In subsequent step S24, the deriving unit 14c derives a matrix A _kj defined below for a combination of arbitrary times t _i and t _j included in the address period. A _kj is substituted into the objective function in the optimization process in the subsequent step S26.

Hereinafter, A _kj and each symbol on the right side in the above formula will be described. First, vectors representing the directions of the three-dimensional shaft 52 in the inertial coordinate system at times t _i and t _j are b ^I _i and b ^I _j , respectively, and the three-dimensional vectors in the local coordinate system at times t _i and t _j are used. It is assumed that vectors representing the direction of the shaft 52 are b ^B _i and b ^B _j respectively. Incidentally, b ^B _i, b B _j is a constant value regardless of the time [0 0 -1] can be a ^T (the superscript T denotes the transpose), therefore, b ^B _i, b B _j are all Can be simply expressed as b ^B. At this time, the following equation holds.

Further, the following equation is derived from Equation 11.

Here, when the equations of Equations 13 and 14 are modified, the following equation is established.
However,
It is. When the formula of Formula 15 is transformed, the following formula is obtained.

From the equation (17), it can be seen that a linear relationship is established with respect to q _i . Rearranging this formula leads to the following relational formula:

  The relational expression of Equation 18 is an identity that always holds if there is no error in the image data and sensor data. The relational expression of Expression 18 is used to define the objective function used for the optimization in step S26.

Here, returning to the equation (12), b ^I _jx , b ^I _jy , b ^I _jz are X, Y, b ^I _j representing the orientation of the three-dimensional shaft 52 in the inertial coordinate system at time t _j . Means the Z-axis component. Further, b _jix , b _jiy , and b _jiz mean X, Y, and Z axis components of b _ji defined by _equation (16). Note that the hat of the symbol means that it is normalized.

In step S24, the deriving unit 14c derives the orientation b ^I _j of the three-dimensional shaft 52 at each time in the address period based on the IR image and the depth image in the address period. Then, in order to derive A _kj , the values of b ^I _jx , b ^I _jy , and b ^I _jz that are the triaxial components are substituted into the equation (12). Further, b ^B = representing the orientation of the shaft _{52 (b jx, b jy,} b jz) = a [0 0 -1] ^T, by substituting q _BjBi in step S23 in the numerical formula 16, a b _ji Then, this is substituted into the equation (12).

In subsequent step S25, the deriving unit 14c derives a matrix A _mj defined below for any combination of times t _i and t _j included in the address period. Similarly to A _kj , A _mj is substituted into the objective function in the optimization process in the subsequent step S26.

Similar to the case described in step S24, the following equation is derived. This relational expression is used to define the objective function used for the optimization in step S26.

Here, returning to Equation 19, m ^I _jx , m ^I _jy , and m ^I _jz mean X, Y, and Z axis components of m ^I _j representing the geomagnetism in the inertial coordinate system at time t _j . . _{M jix} , m _jiy , and m _jiz mean X, Y, and Z axis components of m _ji defined by the following equations.

In step S25, the deriving unit 14c substitutes the geomagnetic data derived in step S22 as m ^I _{j on} the right side of Equation 19. Further, m _ji is derived by substituting m ^B _j = (m _jx , m _jy , m _jz ), which is the geomagnetic sensor data at time t _j , and q _BjBi in step S23 into the equation (21), Furthermore, this is substituted into the right side of Equation 19.

In the subsequent step S26, the posture q _i at each time t _i in the address period is optimized by the Lagrange multiplier method. Specifically, the following objective function J ₁ is defined at each time t _i .
However,
It is.

Here, as is clear from Equations 18 and 20, F _k and F _m are functions that should be zero if there is no error in the image data or sensor data. Further, (1-q _i ^T q _i ) is a constraint condition. Accordingly, the deriving unit 14c derives an optimal solution of the Euler parameter q _i that minimizes the objective function J ₁ using A _kj and _Amj calculated in steps S24 and S25. In addition, the deriving unit 14c converts the Euler parameters q _i after optimization at each time t _i into an attitude matrix N according to the equation (4).

Note that w _k and w _m are weighting factors set in accordance with how much image data or geomagnetic data is to be trusted.

In the present embodiment, the objective function J ₁ is minimized as follows. That is, in order to minimize the objective function J ₁ , it is a necessary condition that its differential value (partial differential value) becomes zero, in other words, the following equation holds. Here, I ₄ is a 4 × 4 unit matrix.

The above equation represents that the eigenvalue of the matrix of the following equation is λ, and q _i is an eigenvector.

Accordingly, the deriving unit 14c calculates a matrix of Formula 25 from the results of steps S24 and S25, derives an eigenvector of magnitude 1 corresponding to the minimum eigenvalue of the matrix, and sets this as q _i .

<1-3-3-3. Derivation of posture from address to near impact (S5)>
Hereinafter, the flow of the posture deriving process from the address to the vicinity of the impact will be described with reference to FIG. In this process, an algorithm similar to the attitude derivation process in the address period is used. The main difference between the two processes is that the objective function J ₂ for optimization includes a function F _n based on the orientation of the two-dimensional shaft 52 derived from image data in addition to the functions F _k and F _m. It is in. Details will be described below.

In step S31, the deriving unit 14c derives temporary postures q and N at each time from the address to the vicinity of the impact. Specifically, the derivation unit 14c uses the data of the angular velocities ω _x , ω _y , and ω _z and uses the Euler parameter q at the time of optimization as an initial value according to the equations 6 and 7, to determine the Euler parameter q Is updated at intervals of the sampling period Δt. Thereafter, the posture q at each time is converted into a posture matrix N in accordance with the equation (5).

In the subsequent step S32, the deriving unit 14c calculates the geomagnetism m ^B = (m ^B _x , m ^B _y , m ^B _z ) = (m _x , m _y , m _z ) in the local coordinate system at the time of the inertial coordinates. The geomagnetism in the system is converted to m ^I = (m ^I _x , m ^I _y , m ^I _z ). A specific conversion method is the same as that in step S22. However, the Euler parameter q used for the conversion here is the Euler parameter q calculated in step S31.

In step S33, deriving unit 14c, an arbitrary time t _i that is included in the period from the address to the vicinity of the impact, to the combination of t _j, and the posture q _i at time t _i, the time t posture in _j q _j Q _BjBi , which is a difference from the above, is derived. At this time, the order of t _j and t _i does not matter. The attitude difference q _BjBi can be calculated according to the equation (11). Note that the Euler parameters q _i and q _{j that} are substituted into the equation (11) are the Euler parameters q calculated in step S31.

In subsequent step S34, the deriving unit 14c derives a matrix A _kj defined as shown in _Expression 12 for a combination of arbitrary times t _i and t _j included in the period from the address to the vicinity of the impact. A _kj is substituted into the objective function in the optimization process in the subsequent step S37.

In step S34, the deriving unit 14c derives the orientation b ^I _j of the three-dimensional shaft 52 at each time in the period based on the IR image and the depth image in the period from the address to the vicinity of the impact. Then, in order to derive A _kj , the values of b ^I _jx , b ^I _jy , and b ^I _jz that are the triaxial components are substituted into the equation (12). Further, b ^B = representing the orientation of the shaft _{52 (b jx, b jy,} b jz) = a [0 0 -1] ^T, by substituting q _BjBi in step S33 in the numerical formula 16, a b _ji Then, this is substituted into the equation (12).

In subsequent step S35, the deriving unit 14c derives a matrix _Amj defined as shown in _Equation 19 for a combination of arbitrary times t _i and t _j included in the period from the address to the vicinity of the impact. A _mj is substituted into the objective function in the optimization process in the subsequent step S37.

In step S35, the deriving unit 14c substitutes the geomagnetic data derived in step S32 as m ^I _{j on} the right side of Equation 19. Further, m _ji is derived by substituting m ^B _j = (m _jx , m _jy , m _jz ), which is the geomagnetic sensor data at time t _j , and q _BjBi in step S33 into the equation (21), Furthermore, this is substituted into the right side of Equation 19.

In subsequent step S36, the deriving unit 14c derives a matrix A _nj defined below for any combination of times t _i and t _j included in the period from the address to the vicinity of the impact. A _nj is substituted into the objective function in the optimization process in the subsequent step S37.

Hereinafter, A _nj and each symbol on the right side of the above formula will be described. First, the direction of the two-dimensional shaft 52 in the YZ plane is considered here. Here, a projection matrix for projecting three-dimensional coordinates in the XYZ coordinate system onto the YZ plane is P _YZ . At this time, the direction b _Si two-dimensional shaft in the YZ plane at time t _i is expressed as P _YZ b ^I _i. b _Si can be expressed as follows using the Euler parameters q _i . However, u = [1 0 0] ^T.

When Equation 27 is transformed, the following is obtained.
Further, Equation 28 is transformed from Equation 14 below.

Furthermore, q _BjBi is multiplied from the right of both sides of _Equation 29.
Furthermore, from the formulas 13 and 16, it is modified as follows.

From the equation of Equation 31, it can be seen that a linear relationship is established with respect to q _i . Rearranging this formula leads to the following relational formula:

  The relational expression of Equation 32 is an identity that always holds if there is no error in the image data and sensor data. The relational expression of Expression 32 is used to define an objective function used for the optimization in step S37.

Here, returning to Equation 26, b _Sjx , b _Sjy , and b _Sjz mean X, Y, and Z axis components of b _Sj representing the orientation of the two-dimensional shaft in the YZ plane at time t _j . . Accordingly, b _Sjx = 0 as the X component.

In step S36, the derivation unit 14c derives values of the inertial coordinate system (b ^I _jx , b ^I _jy , b ^I _jz ) by using b ^B = [0 0 −1] ^T and _Expression 13 in order to derive A _nj. And this is substituted into the equation (26). Further, b ^B = representing the orientation of the shaft _{52 (b jx, b jy,} b jz) = a [0 0 -1] ^T, by substituting q _BjBi in step S33 in the numerical formula 16, a b _ji Then, this is substituted into the equation (26). Further, the deriving unit 14c substitutes b _Sj derived from the IR image into the equation (26).

In the subsequent step S37, the posture q _i at each time t _i from the address to the vicinity of the impact is optimized by Lagrange's undetermined multiplier method. Specifically, the following objective function J ₂ is defined at each time t _i .
However,
It is.

Here, as is clear from Equation 32, F _n is a function that should be zero if there is no error in the image data or sensor data, as in F _k and F _m . Further, (1-q _i ^T q _i ) is a constraint condition. Therefore, the derivation unit 14c derives an optimal solution of the Euler parameter q _i that minimizes the objective function J ₂ by using A _kj , A _mj , A _nj calculated in steps S34 to S36. In addition, the deriving unit 14c converts the Euler parameters q _i after optimization at each time t _i into an attitude matrix N according to the equation (4).

_{Incidentally, w k, w m, w} n , the data of the depth image, geomagnetic data in response to either how much trust the data of the IR image, a weighting coefficient set.

In the present embodiment, the objective function J ₂ is minimized as follows. That is, in order to minimize the objective function J ₂ is, that the differential value (the partial differential value) becomes zero, in other words, a necessary condition that the following expression is established.

The above equation represents that the eigenvalue of the matrix of the following equation is λ, and q _i is an eigenvector.

Accordingly, the deriving unit 14c calculates the matrix of Expression 36 from the results of steps S33 to S36, derives an eigenvector of size 1 corresponding to the minimum eigenvalue of the matrix, and sets this as q _i .

<1-3-3-4. Derivation of grip position and speed (S6)>
Hereinafter, the flow of the process of deriving the position and speed of the grip 51 from the address to the vicinity of the impact will be described with reference to FIG.

  The position of the grip 51 is measured by image data. However, when the sampling cycle of the distance image sensor 2 is rough, or when the grip 51 is not captured on the image, it may be difficult to measure the position. Therefore, here, the position of the grip 51 derived from the image data is smoothed by the acceleration data by using a Kalman filter or the like.

First, in step S41, the derivation unit 14c calculates the acceleration a ^B _i = (a _x , a _y , a _z ) of the local coordinate system at each time t _i from the address to the vicinity of the impact according to the following formula in the inertial coordinate system. Convert to acceleration and remove the gravitational acceleration component. Thereby, the acceleration a ^I _i of the grip 51 in the inertial coordinate system is derived. Note that the value derived in step S37 can be substituted for the Euler parameter q _i in the following equation. G is the magnitude of gravitational acceleration.

In subsequent steps S42 and S43, the derivation unit 14c uses the acceleration a ^I _i in step S41 to smooth the three-dimensional position p of the grip 51 derived from the IR image and the depth image, thereby removing the error. Derived position p and velocity p ′ (estimated value) are derived. In this sense, the three-dimensional position p of the grip 51 derived from the IR image and the depth image can be said to be a temporary position.

The system for estimating the position p and the speed p ′ of the grip 51 can be modeled as follows. Here, x is a position p and velocity p ′ to be obtained, y is a position p derived from an IR image and a depth image, u is acceleration data, w is process noise, and v is observation noise. The subscript k represents a time-series data number, and k = 1, 2,..., N (N is the number of data). It should be noted that symbols / symbols appearing in the descriptions of steps S42 and S43 are defined independently of symbols / symbols appearing in the above description and later descriptions unless otherwise specified. .
However,
And

The average value of the process noise w _k is 0, the variance is Q _k , the average value of the observation noise v _k is 0, and the variance is R _k . Q _k and R _k are factors that determine the degree of smoothing. If Q _k / R _k is large, smoothing is not performed so much, and the specific gravity of the measurement result of the distance image sensor 2 increases. On the other hand, if Q _k / R _k is small, smoothing becomes strong and the specific gravity of the measurement result of the acceleration sensor 41 becomes large.

Now, let x ⁺ _fk be the estimated value when filtering forward along the time axis, and P ⁺ _{fk the} error covariance. In step S42, the deriving unit 14c derives an initial value when k = 0 as follows. E represents an expected value.

Thereafter, the deriving unit 14c updates the initial value according to the following expression.

Note that x ⁺ _fk represents an estimated value at time k in the state of time k, and x ⁻ _fk represents an estimated value at time (k−1) in the state of time k. Similarly for error covariance, P ⁺ _fk represents an estimated value at time k in the state at time k, and P ⁻ _fk represents an estimated value at time (k−1) in the state at time k. Represent. Further, x ⁻ _fk and P ⁻ _fk can be calculated as follows.
It is.

  As described above, x = (p, p ′) from which errors are removed from the past and present acceleration data and image data is estimated. In the present embodiment, the sampling period of the image data is longer than the sampling period ΔT of the acceleration data that is the output value of the inertial sensor unit 4. Therefore, in step S42, preprocessing is performed in which the image data is appropriately subjected to linear interpolation or the like to be a data set in accordance with the acceleration data sampling period ΔT.

  In subsequent step S43, the derivation unit 14c further updates the position p and speed p ′ of the grip 51 using future acceleration data and image data by performing backward filtering along the time axis.

Specifically, x _k is derived by performing smoothing on k = N−1,... Note that the initial value x _N = x ⁺ _fN .

  As described above, the position p and the speed p ′ of the grip 51 can be derived with high accuracy using not only the past and present measurement results but also the future measurement results.

<1-3-3-5. Derivation of angular velocity and acceleration of grip (S7)>
Hereinafter, the flow of the derivation process of the angular velocity and the angular acceleration of the grip 51 from the address to the vicinity of the impact will be described with reference to FIG.

First, in step S51, the derivation unit 14c derives the differential q _i ′ of the posture at each time t _i from the address to the vicinity of the impact by applying a pseudo differential filter to the posture q _i derived in step S5. The pseudo differential filter is expressed by the following equation.

However, in the present embodiment, time differentiation is performed using the following equation obtained by discretizing the equation 44 by bilinear transformation.

In subsequent step S52, the deriving unit 14c uses the posture q _i derived in step S5 and the differential q _i ′ of the posture derived in step S51, and the angular velocity at each time t _i from the address to the vicinity of the impact. Derived ω _i . The angular velocity ω _i is derived according to the following equation.

In subsequent step S53, the deriving unit 14c calculates an angular acceleration ω _i ′ at each time t _i from the address to the vicinity of the impact by applying a pseudo differential filter to the angular velocity ω _i derived in step S52. The pseudo-differential filter is the same as Equation 45.

<1-3-3-6. Derivation of grip acceleration (S8)>
Hereinafter, the flow of the process for deriving the acceleration of the grip 51 from the address to the vicinity of the impact will be described with reference to FIG.

  First, in step S61, the deriving unit 14c applies a pseudo differential filter to the speed p ′ of the grip 51 after smoothing derived in step S6, thereby accelerating the acceleration p ′ of the grip 51 at each time from the address to the vicinity of the impact. Derives'. The pseudo-differential filter is the same as Equation 45.

In subsequent step S62, the deriving unit 14c uses the Euler parameter q _i derived in step S5, and according to the following formula, the acceleration p '' in the inertial coordinate system in step S61 at each time from the address to the vicinity of the impact. = A ^I _i is converted into acceleration a ^B _i in the local coordinate system.

<1-4. Evaluation>
Examples of the present invention will be described below, but the present invention is not limited to the following examples.

<1-4-1>
As Example 1, in order to confirm the effect of the optimization in step S4, the optimized posture matrix N obtained by executing steps S1 to S4 according to the above embodiment is used to determine the shaft of the local coordinate system. Orientation [0, 0, 1] ^T was converted to the shaft orientation in the global coordinate system. The angle between the shaft orientation vector and the vector (true value) representing the shaft orientation in the global coordinate system measured by the motion capture system was calculated as an angle error. The result is shown by a solid line in FIG. The motion capture here is, for example, a system that measures movement with high accuracy using a large number of infrared cameras, as described in JP 2011-183090 A and JP 2011-030761 A. is there. Further, as Reference Example 1, steps S1 to S3 and step S21 are executed to derive a posture matrix N, and using this posture matrix N, the shaft direction [0, 0, 1] ^T of the local coordinate system is entirely set. Converted to the orientation of the shaft in the coordinate system. The angle between the shaft orientation vector and the vector (true value) representing the shaft orientation in the global coordinate system measured by the motion capture system was calculated as an angle error. The result is shown by a dotted line in FIG. From the figure, it can be seen that the error from the true value is reduced through the optimization process of step S4.

<1-4-2>
As Example 2, in order to confirm the effect of the optimization in Step S5, the optimized posture matrix N obtained by executing Steps S1 to S5 according to the above embodiment is used to determine the shaft of the local coordinate system. Orientation [0, 0, 1] ^T was converted to the shaft orientation in the global coordinate system. The angle between the shaft orientation vector and the vector (true value) representing the shaft orientation in the global coordinate system measured by the motion capture system was calculated as an angle error. The result is shown by a solid line in FIG. Further, as Reference Example 2, steps S1 to S4 and step S31 are executed to derive a posture matrix N, and using this posture matrix N, the shaft direction [0, 0, 1] ^T of the local coordinate system is entirely set. Converted to the orientation of the shaft in the coordinate system. The angle between the shaft orientation vector and the vector (true value) representing the shaft orientation in the global coordinate system measured by the motion capture system was calculated as an angle error. The result is shown by a dotted line in FIG. From the figure, it can be seen that the error from the true value is reduced by performing the optimization process of step S5.

<1-4-3>
As Example 3, in order to confirm the effect of the smoothing in Step S6, Steps S1 to S6 according to the above embodiment were executed to calculate the grip position p. The results of calculating the difference (position error) between the position p and the true position p are shown by solid lines in FIGS. 16A to 16C. The true value was a value measured by motion capture. As Reference Example 3, acceleration data in the absolute coordinate system is calculated from the same sensor data obtained in step S1 using the attitude matrix N derived by the same method as in steps S11 and S12. The position p of the grip was calculated by integrating this twice. The results of calculating the difference (position error) between the position p and the true position p are shown by dotted lines in FIGS. 16A to 16C. From the figure, it can be seen that the error from the true value is reduced by performing the smoothing in step S6.

<1-4-4>
As Example 4, in order to confirm the effect of the smoothing in Step S43, Steps S1 to S6 according to the above embodiment were executed to calculate the grip position p. The results of calculating the difference (position error) between the position p and the true position p are shown by dotted lines in FIGS. 17A to 17C. The true value was a value measured by motion capture. Further, as Reference Example 4, steps S1 to S5, S41, and S42 were executed to calculate the grip position p. The results of calculating the difference (position error) between this position p and the true position p are shown by solid lines in FIGS. 17A to 17C. From the figure, it can be seen that the error from the true value is reduced by performing the smoothing in step S43.

<2. Second Embodiment>
18 and 19 are diagrams showing the overall configuration of a motion analysis system 200 including the motion analysis apparatus 101 according to the second embodiment of the present invention. As is clear from comparison of these figures with FIGS. 1 and 2, the motion analysis system 200 of the second embodiment includes a second distance image sensor 6 compared to the motion analysis system 100 of the first embodiment. Has been added. The second distance image sensor 6 has the same configuration as the first distance image sensor 2 as shown in FIG. The distance image sensor 6 is installed on the right side of the golfer 7 so as to photograph from a different direction from the distance image sensor 2, specifically, to photograph the golfer 7 from the right side. The distance image sensor 6 is also connected to the motion analysis apparatus 101 via the communication line 17. The motion analysis apparatus 101 may be composed of a plurality of computers. For example, the distance image sensors 2 and 6 may be connected to different computers.

  The motion analysis apparatus 101 of the second embodiment and the motion analysis apparatus 1 of the first embodiment have the same hardware configuration. However, as described later, since some software processes are different, different motion analysis programs 103 are installed. Thereby, the control unit 14 operates as the first and second acquisition units 14a and 14b, and also operates as the third acquisition unit 14e, and operates as the derivation unit 114c instead of the derivation unit 14c. In the following, elements common to the first and second embodiments are denoted by common reference numerals, and the differences between the two embodiments will be mainly described while omitting the description as appropriate.

  Note that various symbols appear in this specification, and the same symbol may be used in a different meaning, or different symbols may be used in the same meaning, but the latest definition takes precedence.

<2-1. Motion analysis method>
The motion analysis method according to the second embodiment will be described with reference to FIG. As apparent from a comparison of FIGS. 5 and 20, the motion analysis methods according to the first and second embodiments are similar, but steps S2, S3, and S6 are improved to steps S102, S103, and S106. Is different. In the second embodiment, step S105 for correcting the posture of the shaft 52 derived in step S5 is inserted after step S5. Hereinafter, steps S102, S103, S105, and S106 relating to the differences will be described in order.

<2-1-1. Second Data Collection Process (S102)>
In the second embodiment, as the second data collection process, the swing motion of the golf club 5 is photographed by the two distance image sensors 2 and 6. That is, not only the distance image sensor 2 but also the distance image sensor 6 simultaneously detects image data including a time-series IR image and a depth image capturing a golf swing. The image data detected by the distance image sensor 6 is transmitted to the motion analysis apparatus 101 via the communication unit 25 of the distance image sensor 6 in the same manner as the image data detected by the distance image sensor 2. On the other hand, on the side of the motion analysis apparatus 101, the third acquisition unit 14 e receives this via the communication unit 15 and stores it in the storage unit 13. In this embodiment, at least time-series image data from the initial time t ₁ slightly before the address to the finish is collected.

<2-1-2. Time adjustment (S103)>
In this embodiment, two series of image data derived from the two distance image sensors 2 and 6 are acquired. Therefore, in the first embodiment, only the time adjustment of the image data and the sensor data is performed, but in this embodiment, the time adjustment of these two series of image data is also performed as step S103.

  Specifically, step S103 proceeds according to the flowchart shown in FIG. As shown in the figure, in step S103, steps S11 to S14 are executed by the derivation unit 114c as in step S3. Thereby, the sensor data and the image data photographed from the front by the distance image sensor 2 (hereinafter referred to as front image data) are synchronized. Subsequently, in steps S111 and S112, the front image data and the image data photographed from the right side by the distance image sensor 6 (hereinafter referred to as right image data) are synchronized. First, in step S111, the deriving unit 114c performs image processing on the front image data, thereby deriving the three-dimensional coordinates of the grip end 51a at each time from the address to the vicinity of the impact. Similarly, the deriving unit 114c performs image processing on the right image data, thereby deriving the three-dimensional coordinates of the grip end 51a at each time from the address to the vicinity of the impact.

  In subsequent step S112, the derivation unit 114c has the highest degree of coincidence between the three-dimensional coordinates of the time-series grip end 51a derived from the front image data and the three-dimensional coordinates of the time-series grip end 51a derived from the right image data. Thus, both time series data are aligned. That is, as in step S14, the point at which the waveforms of the two time series data most closely match is determined while moving the two time series data back and forth along the time axis. Then, with the timing at that time as a reference, the time axis of the front image data and the time axis of the right image data are relatively time aligned.

  The time adjustment is thus completed. By the above processing, the right image data and sensor data are also timed through the front image data.

<2-1-3. Correction of posture during downswing period (S105)>
See FIG. The figure shows the angle error of the direction of the shaft 52 based on the attitude | position matrix N derived | led-out by step S1-S5 as mentioned above. As can be seen from the figure, this error is effectively reduced until just before the impact (corresponding to 0 seconds in the figure), but is slightly larger immediately before the impact. This is presumably because the golf club 5 moves at a high speed immediately before the impact, which roughly corresponds to the downswing period, and the bias component of the angular velocity sensor 42 (gyro sensor) increases.

Step S105 is a step of correcting the posture of the shaft 52 in the period immediately before the impact in which such an error is relatively large, in other words, the period from (impact-T ₀ seconds) to the impact (hereinafter referred to as the downswing period). It is. T ₀ can be set to 0.1 seconds to 0.5 seconds, for example, and more preferably 0.2 seconds to 0.3 seconds.

  Step S105 proceeds as shown in FIG. First, in step S121, the deriving unit 114c derives the posture q (Euler parameter) at the time of impact based on the image data. More specifically, the deriving unit 114c derives a three-dimensional vector representing the orientation of the shaft 52 by performing image processing on the depth image at the time of impact included in the right image data, and at the same time the impact included in the right image data. A two-dimensional vector representing the orientation of the shaft 52 is derived by image processing of the IR image at the time. Similarly, the deriving unit 114c derives a three-dimensional vector representing the orientation of the shaft 52 by performing image processing on the depth image at impact included in the front image data, and at the same time IR at impact included in the front image data. By processing the image, a two-dimensional vector representing the direction of the shaft 52 is derived. Further, the deriving unit 114c derives a three-dimensional vector representing the orientation of the shaft 52 by performing image processing on the paired IR images at impact included in the right image data and the front image data. At this time, using the DLT method or the like, the three-dimensional coordinates of the image of the shaft 52 are derived based on the two-dimensional coordinates of the image of the shaft 52 from two directions derived from the two IR images.

  The deriving unit 114c defines an objective function representing the magnitude of the error between the orientation of the above five shafts 52 and the unknown posture q. And the attitude | position q which minimizes the said error is derived | led-out by optimizing the said objective function. In this embodiment, the objective function is defined using all the five vectors representing the direction of the shaft 52. For example, the objective function is defined by arbitrarily combining two or more of these vectors. You can also

In subsequent step S122, the deriving unit 114c, based on the impact posture q derived in step S121 (hereinafter referred to as q _IBs ), the impact posture q derived in step S5 (hereinafter q _IBb). ). q _IBs is an Euler parameter representing an attitude with respect to an inertial system of the B _s system (corresponding to the XYZ inertial coordinate system; the same _applies hereinafter), and q _IBb is an Euler parameter representing an attitude of the B _b system with respect to the inertial system. The B _b system is a local coordinate system obtained in step S5, and the B _s system is a local coordinate system obtained in step S121.

Here, Euler parameter representing the attitude of the B _s system with respect to the B _b system is set as q _BbBs . At this time, q _BbBs is expressed as follows.

The attitude error is evaluated by an angle between the shafts when the shaft axis in the B _s system or the B _b system and the shaft axis in the B system are converted into inertial systems, respectively. The B system is a local coordinate system (true value). In order to evaluate the attitude error based on the difference in the direction of the shaft axis, q _BbBs is _considered to be _divided into a rotation that matches the shaft axis (difference in the direction of the shaft axis) and a rotation around the shaft axis as follows. q _s corresponds to the rotation that matches the shaft axis, and qψ corresponds to the rotation around the shaft axis.

s is a unit vector representing the direction of the −z axis, and s = (0, 0, −1). q _s represents an operation of rotating the s axis of the B _b system by an angle φ around a unit vector α = (α _x , α _y , 0) in the xy plane. qψ represents an operation of rotating around the s-axis by an angle ψ. Therefore, the attitude error is evaluated using the angle φ. Since α and s are orthogonal, α · s = 0. When the formula of Formula 49 is expanded, it becomes as follows.

The first term to the third term on the right side of the above formula are modified as follows.
From the above, when the vector part of the Euler parameter q is represented as V (q) and the scalar part is represented as S (q), the following is obtained.

By dividing the first formula of Formula 52 by the second formula, ψ can be expressed.

In step S122, the deriving unit 114c substitutes the impact posture q _IBs derived in step S121 and the impact posture q _IBb derived in step S5 into the equation 48, _{thereby calculating} the impact q _{Derive BbBs} . Subsequently, the deriving unit 114c derives ψ at the time of impact by substituting this q _BbBs into the formula 53. Furthermore, the deriving unit 114c derives q _s at the time of impact by substituting q _BbBs and ψ into the following equation obtained by _modifying the first equation of Formula 49.

Further, the deriving unit 114c derives q _IBb ′ _obtained by correcting the impact posture q _IBb by substituting the impact posture q _IBb derived in step S5 and the impact q _s into the following equation. To do. By the conversion according to the equation 55, the attitude error of the B _b system and the attitude error of the B _s system become equal.

In subsequent steps S123 and S124, the deriving unit 114c interpolates the posture q _IBb between the start time of the downswing period (ie, before the impact T ₀ ) and the end time (ie, the impact time). At this time, the posture q _{IBb at} the time of impact is made to coincide with the corrected posture q _IBb ′ derived in step S122. Specifically, first, the differentiation of q _IBb is considered.
Here, ω _Bm is a detection value of the angular velocity sensor 42. ω _bias is a bias component of ω _Bm . In step S123, this bias component ω _bias is derived. In _{Equation 56} , a hat symbol is attached to q _IBb in the sense that a bias component is taken into consideration.

Specifically, the deriving unit 114c substitutes q _IBb and ω _Bm for the right side of _Formula 56 at each time included in the downswing period. The q _IBb substituted at this time is the value calculated in step S5 at the start time of the downswing period. At subsequent times, the quaternion that is updated based on the differential of q _IBb calculated from the equation 56 and the equation 6 at the previous time is substituted. Then, the substitution values for all these times are added. In other words, Expression 56 is integrated using the downswing period as an integration period. This added value, that is, the integral value is an expression including ω _bias as an unknown, and represents the posture q _IBb at the time of impact. Therefore, the deriving unit 114c substitutes this integral value (formula) into the formula 48 as the posture q _IBb at impact to derive the value (formula) of q _BbBs . Further, the value (formula) of q _s is derived from the formulas 53 and 54 using the value (formula) of q _BbBs . Then, the value of the q _s (expression), to be equal to the value of q _s derived earlier in step S122, it determines the omega _bias. Here, since the z component of q _s is 0, the termination condition to be satisfied is two components. Since ω _bias has three components, the three components are calculated so as to satisfy two termination conditions. At this time, it is preferable to efficiently search for the three components of ω _bias using the Newton method or the like.

Next, in S124, the posture q _IBb in the downswing period is derived again based on the bias component ω _bias derived in step S123. Specifically, the posture q _IBb at the _start time of the downswing period derived in step S5 is updated from time to time using the equations (6 ₎ and (7). At this time, the angular velocity used for the update is an angular velocity (ω _Bm + ω _bias ) in consideration of the bias component.

In the present embodiment, at step S124, the in calculating the angular velocity (ω _Bm + ω _bias) in consideration of the bias component, the bias component omega _bias at each time is corrected by the following equation. T is the length of the downswing period, and t is the elapsed time from the beginning of the downswing period. ω _biasf is a bias component at the time of impact.
As a result, the bias component ω _bias derived in the description of the previous three paragraphs is the bias component ω _{biasf at the} impact time, and the bias component at the start time of the downswing period is zero. In addition, the bias component is set to gradually increase during the downswing period. As a result, the angular velocity can be continuously changed while avoiding an abrupt change in the angular velocity at the boundary between the downswing period and the other period (the boundary between the initial time and the immediately preceding time).

In addition, as a bias component correction method, the following method is also conceivable, and the following correction method can be used in combination with the above correction method, or can be used alone. That is, the bias component is large in the high speed region and small in the low speed region. Therefore, ω _Bm can be corrected as follows.
ω _Bm '= TH + (ω _Bm −TH) × k (when ω _Bm > TH)
ω _Bm '= −TH + (ω _Bm + TH) × k (when ω _Bm <TH)

Here, ω _Bm ′ is an angular velocity after correction (an angular velocity including a bias component), TH is a threshold value, and k is a correction coefficient. Thus, the correction process in step S105 is completed.

<2-1-4. Derivation of grip position and speed (S106)>
The grip position and speed derivation process (step S106) is the same as the process of step S6 except for the following points. That is, in step S106, the position and speed of the grip 51 are derived based not only on the front image data but also on the side image data. Specifically, the formula of Equation 39 is changed as follows. b means the bias component of the acceleration sensor.

<2-2. Evaluation>
Examples of the present invention will be described below, but the present invention is not limited to the following examples.

As Example 5, in order to confirm the effect of optimization in Step S105, the optimized posture q obtained by executing Steps S1 to S105 according to the second embodiment is used to determine the shaft of the local coordinate system. Orientation [0, 0, 1] ^T was converted to the shaft orientation in the global coordinate system. The angle between the shaft orientation vector and the vector (true value) representing the shaft orientation in the global coordinate system measured by the motion capture system was calculated as an angle error. The result is shown by a one-dot chain line in FIG. FIG. 23 also shows Example 2 and Reference Example 2 of FIG. From the figure, it can be seen that the error from the true value is reduced in the period immediately before the impact through the correction process in step S105.

<3. Modification>
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to the said embodiment, A various change is possible unless it deviates from the meaning. For example, the following changes can be made.

<3-1>
In the optimization process of step S26, the objective function J ₁ including the function F _m derived from geomagnetic data and the function F _k derived from image data is minimized. However, the objective function J ₁ can also be defined so as not to include _one of the functions F _{m and} F _k . Similarly, in the optimization process of step S37, the objective function J ₂ including F _n , F _m , and F _k is minimized, but the objective function J ₂ is at least one of the functions F _n , F _m , and F _k . It can also be defined not to include one. In addition, the function F _m derived from geomagnetic data can be defined as a function derived from acceleration data. In this case, for example, a gravity component is extracted from a _x , a _y , and a _z measured by the acceleration sensor 41. The geomagnetic m _x, m _y, the data of m _z, by replacing the data direction of gravity extracted respectively, can be calculated similarly functions F _m.

<3-2>
In the above embodiment, an IR image is taken by the distance image sensor 2, but a color image can be taken instead of the IR image and used for motion analysis. In this case, the distance image sensor 2 may be equipped with a visible light receiving unit (for example, an RGB camera) that receives visible light.

<3-3>
The motion analysis apparatus, method, and program according to the above embodiment are configured to be suitable for analyzing the motion of a golf swing, but the same algorithm can be used to analyze an arbitrary motion.

<3-4>
In the step S21, the posture of the initial time t ₁ (Euler parameters q and posture matrix N) is, the image data of the initial time t _1, and, on the basis of the acceleration data of the initial time t ₁ in the direction of gravity identified derived It was done. However, the attitude at the initial time t ₁ can also be derived from the angular velocity data. The flowchart in FIG. 24 shows an example of the process flow.

First, in step S71, the deriving unit 14c sets the Euler parameter q at the initial time t ₁ to [0, 0, 0, ₁ ], and uses the Euler parameter q as the angular velocity ω _x , Using the data of ω _y and ω _z , it is updated every moment according to the formulas 6 and 7. Thereafter, the posture q at each time is converted into a posture matrix N in accordance with the equation (5). Subsequently, in step S72, the deriving unit 14c derives a posture difference q _BjBi at times t _i and t _j in the same manner as in step S23. In step S73, a matrix A _kj is derived as in step S24. Thereafter, in step S74, the posture q _i derived in step S71 is optimized by the Lagrange's undetermined multiplier method, as in step S26. Specifically, the optimization is performed so as to minimize an evaluation function J ₃ below.

The method of this optimization is also the same as steps S26, in order to evaluate the function J ₃ is minimized relates derivative of the objective function J _3, it is necessary to number 60 is established. Since this equation represents that the eigenvalue of the matrix of Formula 61 is λ and q _i is an eigenvector, the derivation unit 14c calculates the matrix of Formula 61 and corresponds to the minimum eigenvalue of the matrix. Then, an eigenvector of magnitude 1 to be derived is derived and set as q _i . The deriving unit 14c derives this q _i for at least the initial time t ₁ and further converts it into the posture matrix N. Thereby, the posture at the initial time t ₁ is derived.

<3-5>
In the derivation process (step S5) of the posture from the address to the vicinity of the impact in the first and second embodiments, steps S32 to S37 can be omitted. In this case, in the processing after step S6, the temporary posture derived in step S31 can be used instead of the result of step S37.

1,101 motion analysis device 2 distance image sensor 6 distance image sensor 3 motion analysis program 4 inertial sensor unit 41 acceleration sensor 42 angular velocity sensor 43 geomagnetic sensor 5 golf club 7 golfer 14a first acquisition unit 14b second acquisition unit 14e third acquisition Parts 14c, 114c lead-out part 51 grip 52 shaft 100, 200 motion analysis system

Claims (14)

A motion analysis device for analyzing the motion of a moving object,
Time-series sensor data including at least one of time-series angular velocity data, acceleration data, and geomagnetic data output from a sensor unit that is attached to the moving body and includes at least one of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor. A first acquisition unit to acquire;
A second acquisition unit that acquires time-series image data including a time-series depth image and a two-dimensional image obtained by photographing the movement of the moving object by the first distance image sensor;
A derivation unit that derives a three-dimensional behavior of the moving body based on the sensor data and the image data;
Motion analysis device. The derivation unit derives the posture of the moving body as an optimal solution for minimizing or maximizing an objective function including a function defined using the image data and the sensor data.
The motion analysis apparatus according to claim 1. The function is defined using an orientation of the moving object derived from the image data.
The motion analysis apparatus according to claim 2. The function is defined using at least one of the geomagnetic data and acceleration data.
The motion analysis apparatus according to claim 2 or 3. The derivation unit calculates a temporary posture of the moving body using the initial posture of the moving body and the time-series angular velocity data.
The function is defined using the temporary posture,
The motion analysis apparatus according to claim 2. The derivation unit uses the initial orientation of the moving body and the initial acceleration data derived from the image data, or the angular velocity data as the initial posture of the moving body when the moving body is stationary. To derive the posture of
The motion analysis apparatus according to claim 5. The derivation unit derives an angular velocity of the moving body based on differential data obtained by filtering the attitude of the moving body and the attitude of the moving body using a differential filter.
The motion analysis apparatus according to claim 1. The deriving unit uses the acceleration data to smooth at least one of a temporary position of the moving body derived from the image data and a temporary speed obtained by differentiating the temporary position, thereby determining the position of the moving body. And at least one of the speed is derived,
The motion analysis apparatus according to claim 1. The derivation unit derives the acceleration of the moving body by filtering the speed of the moving body with a differential filter.
The motion analysis apparatus according to claim 8. The derivation unit has the highest degree of coincidence between the time series data of the moving body direction derived from the image data and the time series data of the moving body direction derived from the sensor data. Performing time adjustment of the time-series image data and the time-series sensor data;
The motion analysis apparatus according to claim 1. A third acquisition unit for acquiring time-series image data including a time-series depth image and a two-dimensional image obtained by capturing the movement of the moving body from a direction different from that of the first distance image sensor by the second distance image sensor; Prepared,
The derivation unit derives a three-dimensional behavior of the moving body based on the sensor data and the image data acquired by the second acquisition unit and the third acquisition unit.
The motion analysis apparatus according to claim 1. The deriving unit derives an instantaneous posture, which is the posture of the moving body at a first time, based on the image data, and the posture of the moving body matches the instantaneous posture at the first time. Interpolating the posture of the moving body between a second time different from one time and the first time;
The motion analysis apparatus according to claim 1. A motion analysis method for analyzing the motion of a moving object,
Using a sensor unit attached to the moving body and including at least one of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor, time-series sensor data including at least one of time-series angular velocity data, acceleration data, and geomagnetic data is obtained. A step to obtain,
Using a distance image sensor to acquire time-series image data including a time-series depth image and a two-dimensional image obtained by capturing the movement of the moving body;
Deriving a three-dimensional behavior of the moving body based on the sensor data and the image data;
Comprising
Motion analysis method. A motion analysis program for analyzing the motion of a moving object,
Time-series sensor data including at least one of time-series angular velocity data, acceleration data, and geomagnetic data output from a sensor unit that is attached to the moving body and includes at least one of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor. A step to obtain,
Acquiring time-series image data including a time-series depth image and a two-dimensional image obtained by photographing the movement of the moving object by a distance image sensor;
Deriving a three-dimensional behavior of the moving body based on the sensor data and the image data;
To run on a computer,
Motion analysis program.