JP2016011951A - Method and device of acquiring positional information of virtual marker, and motion measurement method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method of acquiring positional information of a virtual marker by setting the virtual marker in a simpler configuration by effectively using simultaneous measurement with an optical motion capture and a sensor module.SOLUTION: In motion measurement including an optical marker 1' for correction, and a sensor module having at least a triaxial gyro sensor 2, the sensor module is provided on a rigid body 5 to which a virtual marker Mis set, a positional relationship between the sensor module and a virtual point to which the virtual marker is set is stored, and a coordinate value of the virtual marker set to the virtual point is acquired by using the positional relationship, a coordinate value of the sensor module, and attitude information of the sensor module.

Description

The present invention relates to a method and apparatus for acquiring position information of a virtual marker, and more specifically, by performing motion measurement using an optical motion capture and motion measurement using an inertial sensor module at the same time, using the position information of the sensor module The present invention relates to a technique for setting a marker and acquiring position information of a virtual marker at the time of motion measurement. As such an inertial sensor module, a 9-axis sensor and a 6-axis sensor can be exemplified.

As a measurement method for three-dimensional motion analysis, a motion analysis system based on optical motion capture (typically an infrared reflection method is adopted) is well known. In optical motion capture, a plurality of optical markers (infrared reflective markers) are attached to a predetermined part of the subject, and the subject's motion is measured from the movement trajectory of the optical marker by photographing the subject's motion with a plurality of cameras. Optical motion capture systems are widely used in fields such as medicine, sports, robotics, and computer graphics.

However, since the three-dimensional motion measurement is originally to measure a moving person or object, a marker photographed with a camera at a certain time is hidden at another certain time t, and the position information of the marker cannot be acquired. There is a case. Although it may be considered to acquire the position information of each marker at each time by increasing the number of cameras, there is a problem that increasing the number of cameras increases the cost of the entire system. It is also conceivable to estimate and interpolate the hidden marker position information in the frame _t from the marker position information in the frame _t-1 and the frame _{t + 1} before and after the time t when the marker position information could not be acquired. Computation processing is complicated, and it is difficult to interpolate the position information of markers that are continuously hidden over a plurality of frames.

A virtual marker may be used as means for solving such a problem. The virtual marker is not a marker (actual marker) actually attached to a subject or the like, but a virtual marker whose position is specified by a relative position with respect to a plurality of actual markers. More specifically, the “virtual marker” is at a certain distance from three or more actual markers, and when there are three markers, one of the front and back surfaces of the surface created by the three markers. A point is set, a coordinate value is calculated from the coordinate value of the actual marker and the setting, and the calculated coordinate value is handled as a marker. By using the virtual marker, it is possible to avoid the phenomenon that the marker is hidden and data cannot be acquired. It is also possible to acquire time-series data of coordinate values of places that cannot be seen from the installed camera. Performs virtual point determination input. The method of virtual point setting and virtual point determination input is disclosed in Patent Document 1, and the contents thereof can be referred to.

Here, conventionally, in order to virtually set a marker, three optical markers are required (see FIG. 8). This means that as the number of somites increases, the number of markers increases, and it is desirable that the number of markers be small because labor for attaching the markers and erroneous recognition of the markers may occur.

On the other hand, in three-dimensional motion analysis, it is also known to measure motion using an inertial sensor such as an acceleration sensor or a gyroscope. Recently, a six-axis sensor (a sensor equipped with a three-axis acceleration sensor or a three-axis gyroscope) is known. Modules) and 9-axis sensors (3-axis acceleration sensors, 3-axis gyroscopes, sensor modules with 3-axis geomagnetic sensors) have begun to perform motion analysis.

Here, by combining motion analysis of optical motion capture and motion analysis using inertial sensors, it is expected to perform synergistic measurement taking advantage of each advantage. However, simply attaching an optical marker for optical motion capture and a sensor module such as a 9-axis sensor to the subject only increases the measurement data obtained, and the drawbacks of both systems cannot be solved. It is important to collaborate with optical motion capture and inertial sensor for effective simultaneous measurement.

Although the viewpoint of simultaneously performing the optical motion capture and the measurement by the inertial sensor is described in Patent Document 2, the disclosed content provides means for synchronizing information from both the optical motion capture device and the inertial sensor. Stay on what you do.
JP2013-164413A JP2013-192591A

An object of the present invention is to set a virtual marker with a simpler configuration and acquire position information of a virtual marker by effectively using simultaneous measurement by an optical motion capture and a sensor module.

The first technical means adopted by the present invention is:
With optical motion capture,
A sensor module having at least a three-axis gyro sensor;
In motion measurement with
A sensor module is provided on a rigid body where a virtual marker is set, and a positional relationship between the sensor module and a virtual point where the virtual marker is set is stored,
The coordinate value of the virtual marker set to the virtual point is acquired using the positional relationship, the coordinate value of the sensor module, and the posture information of the sensor module.
It is the acquisition method of the positional information on a virtual marker.
In one aspect, the coordinate value of the virtual marker in the world coordinate system is acquired by the coordinate value of the sensor module in the world coordinate system, the virtual point coordinate value in the sensor module coordinate system, and the gyro sensor of the sensor module. The conversion information between the sensor module coordinate system and the world coordinate system calculated using the rotation information obtained is used.
In one aspect, the virtual point coordinate value in the sensor module coordinate system includes a virtual point coordinate value in the world coordinate system when a virtual marker is set, a coordinate value of the sensor module in the world coordinate system, and a gyro sensor of the sensor module. Is obtained using the conversion information between the sensor module coordinate system and the world coordinate system, which is calculated using the rotation information obtained by the above.
In one embodiment, one or more optical markers are provided on the rigid body, the virtual point is a position of the one or more optical markers, and the virtual marker is the one Or it sets to the position of a plurality of optical markers.
By doing so, when the optical marker (real marker) on which the virtual marker is set is hidden spatially or temporally, the coordinate value of the virtual marker can be adopted as the coordinate value of the real marker.

In one aspect, the sensor module comprises a triaxial acceleration sensor,
The coordinate value of the sensor module is calculated from the measurement value of the acceleration sensor of the sensor module.
In one aspect, one or more optical markers whose positional relationship is specified with respect to the sensor module are provided on the rigid body,
The coordinate value of the sensor module is acquired using the coordinate value of the one or more optical markers acquired by the optical motion capture and the positional relationship.
More specifically, the coordinate value of the sensor module includes the coordinate value of the optical marker in the sensor module coordinate system, the coordinate value of the optical marker in the world coordinate system acquired by the optical motion capture, It is acquired using the conversion information between the sensor module coordinate system and the world coordinate system calculated using the rotation information acquired by the gyro sensor of the sensor module.
In one aspect, a sensor module and a plurality of optical markers are provided on one rigid body, and the positional relationship between each optical marker and the sensor module is stored. The coordinate value of the sensor module can be calculated from the coordinate value of the at least one visible optical marker.
A calculation method for estimating the coordinate value of the virtual marker by using a plurality of optical markers is appropriately selected by those skilled in the art. For example, the average of the estimated coordinate values of the obtained plurality of virtual markers is set as the coordinate value of the virtual marker. Alternatively, the average coordinate value of the obtained plurality of sensor modules may be used as the coordinate value of the sensor module, and the coordinate value of the virtual marker may be estimated using the coordinate value.

  The orientation information of the sensor module when the sensor module coordinate system is made to coincide with the world coordinate system is obtained, and the conversion information includes the orientation information and the rotation information acquired by the gyro sensor of the sensor module. Obtained using.

In one aspect, the virtual point coordinate value in the world coordinate system at the time of setting the virtual marker is obtained by optical motion capture using a device including an instruction unit indicating the virtual point and a marker capable of determining the coordinate value of the instruction unit. To be acquired.
In one aspect, the virtual point coordinate value in the world coordinate system at the time of setting the virtual marker is a coordinate value of the real marker acquired by optical motion capture using a real marker attached to a predetermined part.

The second technical means adopted by the present invention is:
With optical motion capture,
A sensor module having at least a three-axis gyro sensor;
In the motion measurement system with
A sensor module provided on a rigid body on which a virtual marker is set;
Storage means for storing a positional relationship between the sensor module and a virtual point where the virtual marker is set;
A calculation means for acquiring the coordinate value of the virtual marker set at the virtual point using the positional relationship, the coordinate value of the sensor module, and the posture information of the sensor module;
An apparatus for acquiring position information of a virtual marker, comprising:

The third technical means adopted by the present invention is:
With optical motion capture,
A sensor module comprising at least a 3-axis gyro sensor and a 3-axis acceleration sensor;
In motion measurement with
One sensor module is provided on a rigid body where a virtual point as a virtual marker is set,
The coordinate value of the sensor module in the world coordinate system is calculated using acceleration data acquired by the acceleration sensor of the sensor module,
The virtual point coordinate value in the sensor module coordinate system, the virtual point coordinate value in the world coordinate system when setting the virtual marker, the coordinate value of the sensor module in the world coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated using
The virtual point coordinate value in the world coordinate system at the time of measurement, the coordinate value of the sensor module in the world coordinate system, the virtual point coordinate value in the sensor module coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated using
This is a motion measurement method in which the setting of a virtual marker and the tracking of the position of the virtual marker during motion measurement are executed by one sensor module.
The third technical means can also be defined as an apparatus invention (operation measurement system).

The fourth technical means adopted by the present invention is:
With optical motion capture,
A sensor module having at least a three-axis gyro sensor;
In motion measurement with
One sensor module is provided on a rigid body where a virtual point as a virtual marker is set,
An optical marker provided on the same rigid body as the sensor module, the positional relationship of which is specified with respect to the sensor module is prepared,
The coordinate value of the sensor module in the world coordinate system is acquired using the coordinate value of the optical marker acquired by the optical motion capture and the positional relationship,
The virtual point coordinate value in the sensor module coordinate system, the virtual point coordinate value in the world coordinate system when setting the virtual marker, the coordinate value of the sensor module in the world coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated using
The virtual point coordinate value in the world coordinate system at the time of measurement, the coordinate value of the sensor module in the world coordinate system, the virtual point coordinate value in the sensor module coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated using
This is a motion measurement method in which the setting of the virtual marker and the tracking of the position of the virtual marker during motion measurement are executed by one sensor module and one optical marker.
The forty-third technical means can also be defined as an invention of the apparatus (motion measurement system).

The basic hardware of the motion measurement system according to the present invention is an optical marker, a plurality of cameras, and a computer that constitute an optical motion capture, and a 9-axis sensor that constitutes a motion measurement by a sensor module. It becomes a sensor module such as a computer. One or more computers may be used in the motion measurement system. Measurement data (camera image, acceleration data, angular velocity data, etc.) acquired in time series in each frame is stored in the storage unit of the computer, and a predetermined calculation is executed using the measurement data in the calculation unit of the computer. Is stored in the storage unit. The storage unit also stores information acquired prior to measurement.
Examples of the optical marker include an infrared reflective marker and an infrared LED.
In the embodiment to be described later, various information used for calculation is mainly shown as a matrix, but the matrix representation is only one aspect, and it is possible to define various information in other representation formats. Will be understood by those skilled in the art.

The present invention does not require three optical markers by providing a 9-axis sensor on a rigid body on which a virtual marker is set and defining the position of the virtual point when determining the virtual point using the 9-axis sensor. Provides a method for setting virtual markers.
In the conventional method, at least three optical markers are required to determine and track the virtual marker, but in the present invention, the setting of the virtual marker and the tracking of the position of the virtual marker at the time of motion measurement are performed. It can be performed by one sensor module (with acceleration sensor) or by one sensor module and one optical marker.

It is a conceptual diagram which shows the simultaneous measurement of the operation measurement by an optical motion capture, and the operation measurement by a sensor module. It is a conceptual diagram of 1st Embodiment of the setting of a virtual marker. It is a conceptual diagram of 2nd Embodiment of the setting of a virtual marker. An optical marker (correction marker) mounted on the sensor module is shown. An optical marker (correction marker) and a sensor module mounted on the same rigid link are shown. A jig used for matching the sensor coordinate system of the 9-axis sensor with the world coordinate system is shown. It is a figure which shows the method of matching the coordinate system of a body segment and the coordinate system of a sensor. It is a figure explaining the conventional setting of a virtual marker. A sensor module and two markers mounted on the same rigid link (body segment) are shown. It is a figure explaining the setting / estimation of a virtual point using a marker jig (comprising three markers and a sensor module).

[A] Motion measurement by optical motion capture and motion measurement by sensor module [A-1] Motion measurement by optical motion capture system The optical motion capture system uses a plurality of predetermined parts (head, neck, shoulder, elbow) of a subject. A plurality of optical markers 1 (for example, infrared reflection markers) mounted on a tool or a tool, a plurality of cameras 3 that simultaneously photograph a subject wearing the optical marker 1 from a plurality of angles, The two-dimensional position of the marker in the image information of the marker acquired by the camera 3 is reconstructed and the three-dimensional position of the marker is calculated, and each part of the body is calculated from the three-dimensional position of the optical marker and the three-dimensional model of the body. A processing unit that acquires a three-dimensional position (subject's posture) and a storage unit that stores measurement data and data calculated based on the measurement data A display unit for displaying such processing by the processing unit results (movement data is acquired as time-series data of the subject's posture) it consists.

The processing unit, the storage unit, and the display unit are configured by a computer 4 (for example, a general-purpose computer including an input unit, an output unit, a calculation unit, a storage unit, a display unit, and the like), and specifically acquired by the camera 3. Storage unit for storing image information and information calculated by image processing unit, display unit for displaying image information and measurement results and analysis results, image processing unit for performing image processing on image information, optical marker 1 Is provided with a calculation unit for calculating time-series data of the three-dimensional positions.

[A-2] Motion Measurement System Using Sensor Module The motion measurement system using the sensor module is a 9-axis sensor 2 attached to a predetermined part of a subject and a receiver that receives measurement data from the 9-axis sensor 2 ( (Not shown) and a computer 4 that executes calculations based on various data obtained by the receiver. The 9-axis sensor 2 is a sensor module including a 3-axis acceleration sensor, a 3-axis gyroscope, a 3-axis geomagnetic sensor, a control unit (MCU, etc.), a battery, and a wireless communication function (Bluetooth, etc.). The sensor module is attached to a predetermined part (for example, the left upper arm, the upper right arm, the left forearm, the right forearm, the left wrist, the right wrist, or the trunk) of the subject's body, a tool, or the like. The acceleration data and the angular velocity data are transmitted to the data storage / calculation computer 4 via the communication function of each sensor module, and the measurement data is analyzed to reconstruct the motion.

The computer 4 stores sensor data obtained from the receiver, reconstructs the posture of the subject, and performs motion analysis. The computer 4 can be comprised from a general purpose computer provided with an input part, an output part, a calculating part, a memory | storage part, a display part, etc. Sensor data (acceleration data, angular velocity data, geomagnetic data) input from the input unit is stored in the storage unit, and a predetermined calculation is executed by the calculation unit (for example, position information by integration of acceleration data, integration of angular velocity data) And the calculation result are stored in the storage unit and output from the output unit or / and displayed on the display unit as necessary. The measurement by the optical motion capture and the measurement by the 9-axis sensor 2 are executed synchronously, and the measurement data is acquired synchronously, so that the position information of the optical marker 1 and the 9-axis sensor are obtained at each time (frame). Each data acquired from 2 is obtained correspondingly and stored in the storage unit.

[B] Setting of virtual marker using 9-axis sensor [B-1] Outline In optical motion capture, as a method of attaching an optical marker at a position where it cannot be seen from the camera, a virtual marker is used instead of a real marker. There is a method of setting a marker. As shown in FIG. 8, conventionally, in order to set such a virtual marker _MH , three or more optical markers m ₁ , m ₂ , and m ₃ are required. This means that as the number of somites increases, the number of markers increases, and it is desirable that the number of markers be small because labor for attaching the markers and erroneous recognition of the markers may occur.

In this embodiment, one 9-axis sensor 2 is provided on a rigid body on which a virtual marker is set, and a virtual point is determined at a certain distance from the 9-axis sensor 9 in a certain posture on the same rigid body. By setting a virtual marker at a virtual point, a method for setting a virtual marker that does not require three optical markers is provided.

In the aspect shown in FIG. 2, one 9-axis sensor 2 is provided on the same rigid body 5 as the rigid body (such as a body segment) 5 for setting the virtual marker _MH, and a virtual point (not shown) on the rigid body where the virtual marker is set is provided. In the aspect, the virtual marker _MH is set using the positional relationship between the tip coordinate of the pointer P) and the 9-axis sensor 2, and the position of the virtual marker _MH is tracked using the positional relationship when measuring the motion. To do. That is, the setting of the virtual marker and the tracking of the position of the virtual marker at the time of motion measurement are executed by one 9-axis sensor.

In the aspect shown in FIG. 3, the 9-axis sensor 2 and the correction marker 1 ′ mounted on the 9-axis sensor 2 are provided on the same rigid body 5 as the rigid body (such as the body segment) 5 for setting the virtual marker _MH , and the virtual marker The virtual marker _MH is set using the positional relationship between the virtual point on the rigid body (in the illustrated embodiment, the tip coordinate of the pointer P) and the 9-axis sensor 2, and the positional relationship is measured during motion measurement. Is used to track the position of the virtual marker _MH . In other words, the setting of the virtual marker and the tracking of the position of the virtual marker at the time of motion measurement are executed by one 9-axis sensor and one optical marker (correction marker 1 ′).

[B-2] Acquisition of 9-axis sensor position information Setting the virtual marker based on the 9-axis sensor position information requires the 9-axis sensor position information. Several methods can be exemplified for obtaining the position information of the 9-axis sensor. First, by integrating the acceleration data measured by the acceleration sensor of the 9-axis sensor, the position information of the 9-axis sensor can be acquired. Second, as shown in FIGS. 4 and 5, a correction marker 1 ′ provided on the same rigid body 5 as the 9-axis sensor 2 and whose positional relationship is specified with respect to the 9-axis sensor 2 is prepared. The coordinate value of the axis sensor 2 is acquired using the coordinate value of the correction marker 1 ′ acquired by optical motion capture and the positional relationship. This second method is expected to estimate the position of the 9-axis sensor 2 with higher accuracy than the first method. A method for acquiring the position of the 9-axis sensor using the correction marker will be described in detail later.

[B-3] Determination / Input of Virtual Point The virtual point where the virtual marker _MH is set is determined as a desired part on the rigid body 5, and the coordinate value of the virtual point determined when the virtual marker is set is input to the computer 4. And stored in the storage unit. When the virtual marker _MH is set in the three-dimensional motion measurement system, a virtual point is set with a device called a pointer P. In one aspect, the pointer P includes three optical markers P1, P2, and P3, and the position of the tip can be specified by the positions of the three markers P1, P2, and P3. In a state where the subject himself or another person (such as a doctor) points to a virtual point used as a virtual marker at the tip of the pointer P, the image is taken by a plurality of cameras, and image data is transmitted to the computer. The computer calculates the coordinates of the three markers on the pointer based on the input image data, thereby acquiring the coordinates of the tip of the pointer as virtual point coordinates and inputs them to the computer 4. The method of setting virtual points and determining and inputting virtual points is disclosed in Patent Document 1, and the contents can be referred to as necessary. The input means of the virtual point coordinates is not limited to the pointer. For example, the data of one shot may be acquired with the actual marker first attached, the coordinate value of the actual marker at that time may be used instead of the coordinate value determined by the pointer, and then the actual marker may be removed.

[B-4] Setting of virtual marker using 9-axis sensor The coordinate value (such as the coordinates of the tip of the pointer) of the virtual marker in the world coordinate system when the virtual marker is determined (n = r).
And
Coordinate value (virtual point coordinate value) of the virtual point in the world coordinate system when the virtual marker is determined (n = r)
Is acquired by optical motion capture using a device including an instruction unit indicating a virtual point and a marker capable of determining the coordinate value of the instruction unit.
Alternatively, the coordinate value in the world coordinate system of the virtual point when the virtual marker is determined (n = r)
May use a real marker temporarily attached to a predetermined site and use the coordinate value of the real marker acquired by optical motion capture.

The coordinate value of the 9-axis sensor in the world coordinate system when the virtual marker is determined (n = r)
And As a method for determining the coordinate values of the 9-axis sensor, the method [B-2] can be employed. In the aspect illustrated in FIG. 2, the position information is acquired based on the acceleration data measured by the acceleration sensor of the 9-axis sensor 2. In the aspect shown in FIG. 3, the positional information of the 9-axis sensor 2 is acquired based on the coordinate value of the correction marker 1 ′.

9-axis sensor transformation matrix at the time of virtual marker determination (n = r)
And At the time of calibration, posture information of the 9-axis sensor when the sensor coordinate system of the 9-axis sensor is made to coincide with the world coordinate system is obtained, and the posture information and the 9-axis sensor at the time of virtual marker determination (n = r) A transformation matrix from the world coordinate system to the sensor coordinate system is obtained using the rotation matrix obtained by the gyro sensor.

When the virtual marker is placed in the sensor coordinate system of the 9-axis sensor and the sensor position is the origin, the coordinate value of the marker that is virtually set, that is, the coordinate value of the virtual point in the sensor coordinate system
Then, the virtual point coordinate value in the sensor coordinate system
Can be obtained as follows.

Using the virtual point coordinate value in the sensor coordinate system, the coordinate value in the world coordinate system of the virtual marker of the nth frame can be obtained as follows.
The coordinate value of the 9-axis sensor in the world coordinate system of the nth frame at the time of motion measurement
age,
The transformation matrix from the world coordinate system of the nth frame to the sensor coordinate system of the 9-axis sensor
Then,
Coordinate value in the world coordinate system of the virtual marker of the nth frame
Is
Obtained from.
As a method for determining the coordinate values of the 9-axis sensor, the method [B-2] can be employed.
In the aspect illustrated in FIG. 2, the position information is acquired based on the acceleration data measured by the acceleration sensor of the 9-axis sensor 2. The coordinate values (x _s , y _s , z _s ) of the sensor module in the world coordinate system in the nth frame are calculated from the measurement values of the acceleration sensor of the 9-axis sensor in the nth frame.
In the aspect shown in FIG. 3, the positional information of the 9-axis sensor 2 is acquired based on the coordinate value of the correction marker 1 ′. The coordinate values (x _s , y _s , z _s ) of the sensor module in the world coordinate system in the nth frame are obtained by the optical motion capture in the world coordinate system of the correction marker in the nth frame. Then, the calculation is performed using the transformation matrix from the world coordinate system to the sensor module coordinate system in the nth frame and the positional relationship between the correction marker 1 ′ and the 9-axis sensor 2 (see [C]).
The transformation matrix M _{W → S} from the world coordinate system of the nth frame to the sensor coordinate system of the 9-axis sensor is the attitude information of the 9-axis sensor when the sensor coordinate system of the 9-axis sensor is matched with the world coordinate system at the time of calibration. And a rotation matrix acquired by the gyro sensor of the 9-axis sensor of the nth frame (see [D]).

[C] Acquisition of position information of 9-axis sensor using optical motion capture [C-1] Outline Although 9-axis sensor can acquire position information using acceleration data measured by the acceleration sensor, In order to acquire the position information, it is necessary to add the values obtained by integrating the acceleration data twice. Therefore, errors are accumulated, and the position accuracy tends to be inferior to that of the optical motion capture. Therefore, using the optical motion capture, the positional accuracy of the 9-axis sensor 2 used for acquiring the position information of the hidden optical marker 1 is improved, and the virtual marker is used using the obtained position information of the 9-axis sensor 2. Set. As shown in FIGS. 4 and 5, a correction optical marker 1 ′ is prepared, and a 9-axis sensor 2 is provided on the same rigid body (such as a body segment) 5 as the correction marker 1 ′. The positional relationship of the correction marker 1 ′ is stored, and the positional information of the 9-axis sensor 2 is acquired from the positional information of the correction marker 1 ′ (obtained by optical motion capture) using the positional relationship. In the embodiment shown in FIG. 4, the correction marker 1 ′ is mounted on a 9-axis sensor 2 provided on a predetermined rigid body (such as a body segment). By attaching the 9-axis sensor 2 to the predetermined rigid body, The axis sensor 2 and the correction marker 1 ′ are provided on the same rigid body. In the embodiment shown in FIG. 5, the correction marker 1 ′ and the 9-axis sensor 2 are mounted on the same rigid body (such as a body segment) 5 apart. If the positional relationship is uniquely determined, the correction marker 1 ′ and the 9-axis sensor 2 may be in contact with each other or may be separated from each other.

[C-2] Determination of the positional relationship between the sensor position and the marker In the state where the correction marker 1 'is fixed to the same rigid body (such as a body segment) 5 as the 9-axis sensor 2, the 9-axis sensor 2 is rotated to 9 axes The rotation matrix is acquired using the gyro sensor of the sensor 2. Using this rotation information, the positional relationship between the position of the 9-axis sensor 2 and the correction marker 1 is determined before measurement and stored in the storage unit. The calculation method is shown below.

The coordinate value in the world coordinate system of the 9th axis sensor in the nth frame
(This value is unknown.) Here, for the purpose of acquiring more accurate position information of the 9-axis sensor, the acceleration data obtained from the acceleration sensor of the 9-axis sensor is not used for acquiring the position information of the 9-axis sensor.

The transformation matrix from the absolute coordinate system to the 9-axis sensor coordinate system in the nth frame
And The transformation matrix is, when to match the sensor coordinate system to the world coordinate system stores the posture information of the nine-axis sensor (calibration time), and this position information, at time t _n by 9 axis sensor of a gyro sensor And using the obtained rotation matrix.

The coordinate value in the world coordinate of the correction marker in the nth frame
And The coordinate value of the correction marker in the world coordinate system can be acquired from a camera image of optical motion capture.

The coordinate values of the correction marker when the world coordinate system and the coordinate system of the 9-axis sensor are matched and the position of the 9-axis sensor is located at the origin
And This value is synonymous with the position of the correction marker in the sensor coordinate system, and this value is also an unknown number.

At this time, the following relationship holds.
It becomes. At this time, the right side is the position of the 9-axis sensor when the correction marker is moved to the origin.

In the i-th frame, the j-th frame, and the k-th frame, since the distance from the origin to the correction marker is constant, the following relationship is established.
By solving these simultaneous equations, the coordinate value of the correction marker in the sensor coordinate system
Can be requested.

In summary, the coordinate value of the correction marker in the sensor module (9-axis sensor) coordinate system is
In three or more different postures of the sensor module (9-axis sensor), conversion information (transformation matrix) in each posture is obtained, and rotation information (rotation matrix) obtained by the gyro sensor of the sensor module (9-axis sensor) is obtained. Get using
In each posture, the distance is acquired using the fact that the distance from the sensor module (9-axis sensor) to the correction marker is the same.
The calculation is executed by the calculation unit of the computer 4, and the calculation result is stored in the storage unit of the computer 4.

[C-3] When the correction marker 1 ′ attached for improving the positional accuracy of the 9-axis sensor 2 is photographed by the camera during the acquisition measurement of the position of the 9-axis sensor using the correction marker position, as follows. Then, the coordinate value of the 9-axis sensor 2 is calculated. The calculation formula for calculating the coordinate value of the 9-axis sensor 2 from the position of the correction marker 1 'during measurement is as follows.

Find the position of the 9-axis sensor in the absolute coordinate system
And

The position of the correction marker in the sensor coordinate system
And This position information is acquired in [C-2].

The position of the correction marker in the absolute coordinate system
And The three-dimensional coordinate value of the correction marker is acquired by optical motion capture.

Conversion matrix from absolute coordinate system to sensor coordinate system of 9-axis sensor
If
Than
It becomes.
The calculation is executed by the calculation unit of the computer 4, and the calculation result is stored in the storage unit of the computer 4.

[D] Determination of conversion matrix from absolute coordinate system to 9-axis sensor coordinate system _{Mw → s} [D-1] Outline The coordinate system of 9-axis sensor and the world coordinate system (coordinate system of three-dimensional motion analysis) are one. Since this is not done, calibration work is required to match the coordinate system before measurement. Before the measurement, a transformation matrix for calculating the relationship between the coordinate system of each 9-axis sensor and the world coordinate system (camera coordinate system) is created using a jig 6 as shown in FIG. The jig 6 has a rectangular shape in plan view, and a rectangular recess 61 is formed on the upper surface 60 at intervals in the vertical and horizontal directions so that the 9-axis sensor 2 is fitted in each recess 61. It has become. The jig 6 is provided with three optical markers 1A, 1B, 1C so as to form a vertex of a right triangle. The direction of the marker 1A attached to the right-angled point of the right triangle formed by the three markers 1A, 1B, 1C and the marker 1B attached to the long side is the first axis, and the marker 1A attached to the right-angled point is short. The direction of the marker 1C attached to the side is taken as another axis. When these axes are the front and rear axes and the left and right axes, respectively, the front and rear axes and left and right axes of the 9-axis sensor 2 are placed so as to coincide with the front and rear axes and left and right axes of the jig 6. Since the front and rear axes and the left and right axes are determined, the upper and lower axes can be obtained by calculation.

[D-2]
At a certain timing, the operation measurement by the optical motion capture and the operation measurement by the 9-axis sensor 2 are simultaneously performed. The coordinate values of the three markers 1A, 1B, 1C
age,
And when
Becomes a rotation matrix of three markers 1A, 1B, and 1C. At this time, the rotation matrix obtained from the 9-axis sensor 2 is R _s (R _s is directly calculated and output in the sensor).

Through the above operation, the matrix for converting the coordinate system of the 9-axis sensor 2 to the coordinate system of the markers 1A, 1B, 1C
Is
It becomes. Here, r is a subscript meaning a frame serving as a reference for acquiring data used for creating a matrix using the jig of FIG.
The rotation matrix of the sensor acquired with the above matrix and frame number n
When
Matrix that converts the sensor coordinate system of the 9-axis sensor in frame number n to the world coordinate system
Is
The transformation matrix from the world coordinate system to the sensor coordinate system of the 9-axis sensor is
It becomes.

Thus, in the calibration before measurement, the posture information of the sensor module (9-axis sensor) when the sensor module (9-axis sensor) coordinate system is matched with the world coordinate system is obtained, and the posture information, Using the rotation information (rotation matrix) acquired by the gyro sensor of the sensor module (9-axis sensor), conversion information (conversion matrix) between the sensor module (9-axis sensor) coordinate system and the world coordinate system is acquired. The

The method of calibrating the axis of the 9-axis sensor for matching the 9-axis sensor coordinate system with the world coordinate system is not limited to the method using the jig 6 described above, and other methods can be appropriately adopted by those skilled in the art. It is. For example, draw lines on the left and right and front and back of the world coordinate system, remove the marker from the jig in Fig. 6, and then move the long side of the jig and the front and rear axes of the world coordinate system, the short side of the jig and the world coordinates. It is also possible to match the coordinate system of the sensor with the coordinate system of the marker by using the offset acquisition function of the sensor after aligning the left and right axes of the system.

[E] Method for Matching Body Segment Coordinate System and Sensor Coordinate System Another mode of simultaneous measurement by optical motion capture and sensor module will be described. When a 9-axis sensor is attached to a living body, the sensor coordinate system of the 9-axis sensor is different from the coordinate system of the body segment, which causes an error. As shown in FIG. 7, three infrared reflection markers 1 and one 9-axis sensor 2 are attached on the same rigid body (body segment) 5, and each posture matrix is calculated, so that each coordinate system is obtained. It is possible to convert. By this conversion, an error caused by not being able to attach a 9-axis sensor so as to coincide with the body segment coordinate system can be reduced. This calculation is possible even if all the markers are visible, but it is not necessary to match the coordinate system of the 9-axis sensor with the coordinate system of the markers. A case where it is closed or a case where a virtual marker is used can be exemplified. A specific calculation method is described below.

In r in the reference frame, the coordinate values of the three markers are respectively
age,
And when
Becomes a rotation matrix of three markers, which is a matrix expressing the coordinate system of the body segment.

When the coordinate value of the marker cannot be obtained because the virtual marker is used in the nth frame
The
Calculate the coordinate system of the body segment.
here,
Is a transformation matrix from the world coordinate system to the sensor coordinate system in the nth frame,
Is a transformation matrix from the world coordinate system to the sensor coordinate system in the reference frame r.

[E] The camera measurement accuracy improvement virtual marker using the 9-axis sensor may set the position of one or a plurality of real markers as a virtual point. Specifically, one or a plurality of measurement optical markers (coordinates are acquired by optical motion capture) are provided on a rigid body, and the virtual marker is the one or more measurement optical markers. When the position of the marker is set as a virtual point, and the one or more optical markers for measurement located at the virtual point are hidden, the virtual marker is used as the coordinate value of the hidden optical marker for measurement. The coordinate value of is used.

When the optical marker is hidden from the camera and cannot be seen, the position of the hidden optical marker can be estimated using information obtained from the 9-axis sensor. More specifically, the measurement by the optical motion capture and the measurement by the 9-axis sensor 2 are executed in synchronization, and the measurement data is acquired in synchronization, so that at each time (each frame), the optical marker 1 The position information and each data acquired from the 9-axis sensor 2 are obtained correspondingly. For example, when the optical marker is hidden from the camera at a certain time, the coordinates of the hidden optical marker (virtual marker) are estimated using information acquired from the 9-axis sensor at that time. Can do.

The above estimation method is based on the premise that the measurement frequency of the camera and the measurement frequency of the 9-axis sensor are the same or similar. Here, when the measurement frequency of the 9-axis sensor is higher, a point measured by the 9-axis sensor at a certain time and not measured by the camera can be handled as a hidden marker. At this time, by performing the same calculation as the estimation of the coordinate value of the hidden marker spatially from the camera, it becomes possible to acquire data at the point where the optical marker is not measured, and the position of the optical marker is more accurate. Can be interpolated. If the previously described "estimation of hidden marker using 9-axis sensor" is a state of being spatially hidden, interpolation using devices with different measurement frequencies can be considered as estimation of temporal hidden marker. . The sensor measurement frequency actually includes an acceleration sensor measurement frequency and a three-axis gyro sensor measurement frequency, both of which are higher than the camera measurement frequency. In one aspect, the measurement frequency of the acceleration sensor is the same as that of the three-axis gyro sensor, but it is not necessarily the same.

That is,
The number of measurements per unit time of the gyro sensor of the 9-axis sensor and the number of measurements per unit time of the acceleration sensor are larger than the number of times of data acquisition (frame rate) per unit time of the optical motion capture camera,
The coordinate value of the optical marker hidden in time is estimated using the positional relationship of the optical marker with respect to the sensor module, the coordinate value of the sensor module, and the attitude information of the sensor module.
Here, the coordinate value of the sensor module is acquired by the acceleration sensor, and the attitude information of the sensor module is acquired by the gyro sensor.

Thus, by interpolating the optical marker using the measurement data of the 9-axis sensor, the same effect as that obtained by using a high-speed camera can be obtained even with a low-speed (low-cost) camera. High-precision measurement can be performed, and therefore high-precision measurement can be performed at low cost. In this embodiment, a camera with an actual camera measurement frequency of 100 Hz (50 Hz and 25 Hz can be set) is normally used, and the measurement frequency of the 9-axis sensor is higher than this.

[F] Hidden marker estimation using a marker for measurement In order to estimate a hidden marker using optical motion capture, it is necessary to always show three or more optical markers. If a marker is pasted so that three markers are always reflected in one body segment, increase the number of cameras or increase the number of markers to reduce the blind spots from the camera. The marker is always shown. However, it is costly to use a large number of cameras by attaching a large number of markers.

As a method for preventing this, a method using a jig (three-point marker) with a pointer or three-point marker is known. By setting the marker virtually, it is possible to prevent the marker from being hidden, but when the three-point marker itself is hidden, the virtual marker cannot be calculated.

In this specification, it has been described that the position of the 9-axis sensor is obtained using the position information of the sensor marker (infrared reflective marker). However, if the marker disappears and is hidden for a long time, The position of the marker cannot be obtained with high accuracy, and the position information of the 9-axis sensor with high accuracy cannot be acquired.

As a method for solving such a problem, a 9-axis sensor and a plurality of optical markers are set on the same body segment. The plurality of optical markers can be sensor markers or markers to be estimated. For example, as shown in FIG. 6, a 9-axis sensor, a marker A, and a marker B are attached on the same body segment. The positional relationship between the 9-axis sensor and the marker A and the positional relationship between the 9-axis sensor and the marker B are stored. If the position information of the marker A can be acquired, the position information of the 9-axis sensor can be acquired from the positional relationship between the marker A and the 9-axis sensor and the posture information of the 9-axis sensor. The position of the marker B can be estimated from the positional relationship between the marker B and the 9-axis sensor and the posture information of the 9-axis sensor. That is, in a scene where the marker B is hidden, the marker A can be used as a sensor marker to estimate the position of the marker B. In a scene where the marker A is hidden, the marker B Can be used for estimating the position of the marker A. Here, for example, the virtual marker may be set with the position of the marker A or / and the marker B as a virtual point.

In this way, the position of any one marker on the same body segment can be estimated from the position of the other one marker, that is, if one marker on the same body segment is visible (that is, If the position of the marker can be obtained), the marker position on another same body segment can be estimated. As a result, the robustness against the blind spot of the camera can be greatly improved, so that it is possible to reduce the number of cameras and the number of markers required for measurement, and to reduce the human and monetary costs for motion analysis. Reduction can be achieved.

In FIG. 9, two markers are provided on the same body segment, but three or more markers (positional relationship with the 9-axis sensor may be stored) may be provided. In this case, the position of one or more other markers is estimated based on the position information of one or more sensor markers. For example, when a hidden optical marker is estimated using a plurality of sensor markers, an average of the estimated positions of the plurality of obtained hidden markers may be used as the estimated position of the hidden marker.

A more specific aspect will be described. As an optical marker to be mounted on the thigh, the optical marker may be attached to the outer epicondyle of the thigh and the greater trochanter to perform measurement. In walking measurement, the optical marker on the greater trochanter is the arm. It often happens that it is hidden by a swing motion. Therefore, a 9-axis sensor, a thigh outer epicondyle marker, and a greater trochanter marker are attached on the thigh (rigid body), and the positional relationship between the 9-axis sensor and the thigh outer epicondyle marker is large. The positional relationship of the trochanter marker is stored. By selecting the thigh outer epicondyle marker as a sensor marker for acquiring position information of the 9-axis sensor, and selecting the greater trochanter marker as an estimated marker whose position information is estimated, the greater trochanter marker is When hidden, the coordinate value of the greater trochanter marker is the positional relationship between the greater trochanter marker and the 9-axis sensor, the coordinate value of the thigh outer epicondyle marker, and the position of the 9-axis sensor and the outer thigh condyle marker. It can be estimated using the coordinate values of the 9-axis sensor and the posture information of the 9-axis sensor acquired based on the relationship.

Other specific embodiments will be described. A 9-axis sensor, an ulnar process marker, and a radial process marker are attached on the forearm (rigid body), and the positional relationship between the 9-axis sensor and the ulnar process marker and the positional relationship between the 9-axis sensor and the radial process marker are stored. In some scenes, the ulnar process marker is a sensor marker and the radial process marker is an estimated marker. In some scenes, the ulnar process marker is an estimated marker and the radial process marker is a sensor marker. When the pronation is performed, the radial process marker is hidden, but the coordinate value of the radial process marker is changed to the positional relationship between the radial process marker and the 9-axis sensor, the coordinate value of the ulnar process marker, and the positional relationship between the 9-axis sensor and the ulnar process marker. It can be estimated using the coordinate values of the 9-axis sensor and the attitude information of the 9-axis sensor acquired based on the information. On the other hand, the ulnar process marker is hidden during the supination operation, but the coordinate value of the ulnar process marker is the positional relationship between the ulnar process marker and the 9-axis sensor, the coordinate value of the radial process marker, the 9-axis sensor and the calcaneus. It can be estimated using the coordinate values of the 9-axis sensor and the posture information of the 9-axis sensor acquired based on the positional relationship of the protrusion markers.

In this way, in most parts of the human body, the entire surface of the body segment is not hidden in most movements, so multiple markers are provided on the body segment and the visible markers on the body segment are used as sensor markers. It is effective to estimate the position of the marker that has disappeared. Moreover, in the said aspect, although 9 axis | shaft sensor + two markers were demonstrated, you may use three or more markers. For example, in the pelvis, the four points of the left and right upper anterior iliac spines and the greater trochanter may be used as the hidden marker estimation target and the sensor marker, respectively.

As yet another specific mode, setting of a virtual marker and position estimation will be described. A 9-axis sensor is mounted on a marker jig provided with three optical markers at the time of setting and measuring a virtual point. The positional relationship between each optical marker and the 9-axis sensor is stored. Three markers on the marker jig are used as sensor markers. By doing this, if at least one marker on the marker jig is visible, the position of the other marker on the marker jig is estimated from the positional relationship with the 9-axis sensor and the posture information of the 9-axis sensor. The set virtual point can be determined. Even when a virtual point is set, if one marker on the body segment is visible, other body segment markers can be estimated and a virtual point can be set.

Referring to FIG. 10, when setting / estimating the virtual marker 1, conventionally, the three somite markers 1 to 3 had to be visible. However, if one of the three markers is seen, the other two somite markers can be estimated. For example, if only the body segment marker 1 is visible, the positions of the body segment markers 2 and 3 can be estimated even if the body segment markers 2 and 3 are hidden. Therefore, since the positions of the three body segment markers can be calculated, in order to calculate the virtual marker 1, one of the body segment markers 1 to 3 need only be visible. Similarly, it is only necessary that one of the three body segment markers 1 to 3 is visible when the virtual point is determined.

DESCRIPTION OF SYMBOLS 1 Optical marker 1 'Optical marker 2 for correction | amendment 9 Axis sensor 3 Camera 4 Computer 5 Rigid body _MH Virtual marker P Pointer

Claims (16)

With optical motion capture,
A sensor module having at least a three-axis gyro sensor;
In motion measurement with
A sensor module is provided on a rigid body where a virtual marker is set, and a positional relationship between the sensor module and a virtual point where the virtual marker is set is stored,
The coordinate value of the virtual marker set to the virtual point is acquired using the positional relationship, the coordinate value of the sensor module, and the posture information of the sensor module.
How to get virtual marker position information. Using the coordinate value of the virtual marker in the world coordinate system, the coordinate value of the sensor module in the world coordinate system, the virtual point coordinate value in the sensor module coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated by
The method for acquiring position information of a virtual marker according to claim 1. The virtual point coordinate value in the sensor module coordinate system includes the virtual point coordinate value in the world coordinate system when the virtual marker is set, the coordinate value of the sensor module in the world coordinate system, and the rotation acquired by the gyro sensor of the sensor module. Obtained using the conversion information between the sensor module coordinate system and the world coordinate system calculated using the information,
The method for acquiring position information of a virtual marker according to claim 2. The orientation information of the sensor module when the sensor module coordinate system is made to coincide with the world coordinate system is obtained, and the conversion information includes the orientation information and the rotation information acquired by the gyro sensor of the sensor module. Obtained using
The method for acquiring position information of a virtual marker according to claim 2. The sensor module includes a three-axis acceleration sensor,
The coordinate value of the sensor module is calculated from the measurement value of the acceleration sensor of the sensor module.
The acquisition method of the positional information on the virtual marker of any one of Claims 1-4. One or more optical markers whose positional relationship is specified with respect to the sensor module are provided on the rigid body,
The coordinate value of the sensor module is acquired using the coordinate value of the one or more optical markers acquired by the optical motion capture and the positional relationship.
The acquisition method of the positional information on the virtual marker of any one of Claims 1-4. With optical motion capture,
A sensor module having at least a three-axis gyro sensor;
In the motion measurement system with
A sensor module provided on a rigid body on which a virtual marker is set;
Storage means for storing a positional relationship between the sensor module and a virtual point where the virtual marker is set;
A calculation means for acquiring the coordinate value of the virtual marker set at the virtual point using the positional relationship, the coordinate value of the sensor module, and the posture information of the sensor module;
An apparatus for acquiring position information of a virtual marker. The calculation means is obtained by the coordinate value of the virtual marker in the world coordinate system, the coordinate value of the sensor module in the world coordinate system, the virtual point coordinate value in the sensor module coordinate system, and the gyro sensor of the sensor module. Obtained by using the conversion information between the sensor module coordinate system and the world coordinate system calculated using the rotation information obtained,
The virtual marker position information acquisition apparatus according to claim 7. The virtual point coordinate value in the sensor module coordinate system includes the virtual point coordinate value in the world coordinate system when the virtual marker is set, the coordinate value of the sensor module in the world coordinate system, and the rotation acquired by the gyro sensor of the sensor module. Obtained using the conversion information between the sensor module coordinate system and the world coordinate system calculated using the information,
The virtual marker position information acquisition apparatus according to claim 8. The orientation information of the sensor module when the sensor module coordinate system is made to coincide with the world coordinate system is obtained, and the conversion information includes the orientation information and the rotation information acquired by the gyro sensor of the sensor module. Obtained using
The virtual marker position information acquisition apparatus according to any one of claims 8 and 9. The sensor module includes a three-axis acceleration sensor,
The coordinate value of the sensor module is calculated from the measurement value of the acceleration sensor of the sensor module.
The acquisition apparatus of the positional information on the virtual marker of any one of Claims 7-10. One or more optical markers whose positional relationship is specified with respect to the sensor module are provided on the rigid body,
The coordinate value of the sensor module is acquired using the coordinate value of the one or more optical markers acquired by the optical motion capture and the positional relationship.
The acquisition apparatus of the positional information on the virtual marker of any one of Claims 7-10. With optical motion capture,
A sensor module comprising at least a 3-axis gyro sensor and a 3-axis acceleration sensor;
In motion measurement with
One sensor module is provided on a rigid body where a virtual point as a virtual marker is set,
The coordinate value of the sensor module in the world coordinate system is calculated using acceleration data acquired by the acceleration sensor of the sensor module,
The virtual point coordinate value in the sensor module coordinate system, the virtual point coordinate value in the world coordinate system when setting the virtual marker, the coordinate value of the sensor module in the world coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated using
The virtual point coordinate value in the world coordinate system at the time of measurement, the coordinate value of the sensor module in the world coordinate system, the virtual point coordinate value in the sensor module coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated using
The setting of the virtual marker and the tracking of the position of the virtual marker when measuring the operation are executed by one sensor module.
Motion measurement method. With optical motion capture,
A sensor module having at least a three-axis gyro sensor;
In motion measurement with
One sensor module is provided on a rigid body where a virtual point as a virtual marker is set,
An optical marker provided on the same rigid body as the sensor module, the positional relationship of which is specified with respect to the sensor module is prepared,
The coordinate value of the sensor module in the world coordinate system is acquired using the coordinate value of the optical marker acquired by the optical motion capture and the positional relationship,
The virtual point coordinate value in the sensor module coordinate system, the virtual point coordinate value in the world coordinate system when setting the virtual marker, the coordinate value of the sensor module in the world coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated using
The virtual point coordinate value in the world coordinate system at the time of measurement, the coordinate value of the sensor module in the world coordinate system, the virtual point coordinate value in the sensor module coordinate system, and the rotation information acquired by the gyro sensor of the sensor module Using the conversion information between the sensor module coordinate system and the world coordinate system calculated using
The setting of the virtual marker and the tracking of the position of the virtual marker at the time of motion measurement are executed by one sensor module and one optical marker.
Motion measurement method. On the rigid body, one or more optical markers for measurement are provided,
The virtual marker is set with the position of the one or more optical markers for measurement as a virtual point,
When the one or more optical markers for measurement located at a virtual point are hidden, the coordinate value of the virtual marker is used as the coordinate value of the hidden optical marker for measurement,
The acquisition method of the positional information on the virtual marker of any one of Claims 1-6. On the rigid body, one or more optical markers for measurement are provided,
The virtual marker is set with the position of the one or more optical markers for measurement as a virtual point,
When the one or more optical markers for measurement located at a virtual point are hidden, the coordinate value of the virtual marker is used as the coordinate value of the hidden optical marker for measurement,
The acquisition apparatus of the positional information on the virtual marker of any one of Claims 7-12.