WO2005124687A1

WO2005124687A1 - Method for marker tracking in optical motion capture system, optical motion capture method, and system

Info

Publication number: WO2005124687A1
Application number: PCT/JP2005/010644
Authority: WO
Inventors: Yoshihiko Nakamura; Katsu Yamane
Original assignee: The University Of Tokyo
Priority date: 2004-06-16
Filing date: 2005-06-10
Publication date: 2005-12-29
Also published as: JPWO2005124687A1

Abstract

In an optical motion capture system, it is possible to obtain highly-accurate high-speed data by using two types of cameras: a highly-accurate camera and a high-speed camera. The method includes: a step of imaging an object under test having a marker, by a camera group including cameras different in the frame rate and resolution and acquiring an image; a marker detection step for detecting two-dimensional position information on the marker from the image; a three-dimensional reconfiguration step for acquiring the three-dimensional position information on the marker by using the two-dimensional position information on the marker detected; and a step of unifying the three-dimensional position information on the marker acquired by the respective cameras. The unifying step has a tracking step for searching the three-dimensional position information on the marker in time series and attaching the same ID to the three-dimensional position information on the marker judged to be identical and an interpolation step for interpolating the three-dimensional position information on the marker attached by the same ID acquired by the respective cameras.

Description

Specification

Marker tracking method in optical motion capture system, optical motion capture method and system

Technical field

The present invention relates to a marker tracking method in an optical motion capture system and an optical motion capture system.

Background art

[0002] Motion capture is a method of measuring the movements of humans and animals as numerical data, and the data obtained by such measurements is wide-ranging, such as biomechanics, humanoid movement generation, and CG animation generation! / , Used in the field. There are various types of motion capture, for example, a wire type that attaches a device for data acquisition to a subject and transfers data by wire and records coordinate data, a head type, There is an optical type in which a marker that reflects light is attached to the shoulder, elbow, wrist, etc., and the coordinate data is recorded by capturing the reflected light from the marker. From the viewpoint of the subject's low restraint during exercise, the so-called passive optical motion capture is more advantageous than the wire type.

[0003] However, in the noisy optical motion capture, the amount of calculation for determining the three-dimensional position of a marker (three-dimensional reconstruction) from images acquired by a plurality of cameras is large. There is a drawback that the amount of calculation for the work (labeling) to deal with is large. In addition, since the power of the camera has a limit in the data transfer speed to the computer, a single camera cannot obtain high-precision and high-speed data.

Disclosure of the invention

Problems to be solved by the invention

[0004] It is an object of the present invention to realize real-time marker tracking in an optical motion capture system by performing marker local tracking at the stage of marker detection and three-dimensional reconstruction of a marker position. It is in.

Another object of the present invention is to provide a high-precision optical motion capture system. The purpose is to obtain high-precision and high-speed data by using two types of cameras, a camera and a high-speed camera, to compensate for the shortcomings of each other and to match each other.

Means for solving the problem

[0006] A first technical means adopted by the present invention is a marker detecting step of detecting two-dimensional position information of an image force marker acquired by each camera, and a two-dimensional position information of a marker force detected by each camera. A three-dimensional reconstruction step of obtaining the three-dimensional position information of the marker using (marker estimation vector), and a three-dimensional position of the marker determined to be the same by searching the three-dimensional position information of the marker in time series A tracking step of assigning the same ID to the information, wherein the marker detection step and / or the three-dimensional reconstruction step includes a reference marker position having an ID, and two-dimensional position information (marker marker) of the detected marker. (Estimated vector) or Z and the identity with the 3D position information are determined, and the 2D position information (marker estimation vector) or Z and And a step of assigning the same ID to the three-dimensional position information as the reference marker position.

[0007] A second technical means employed by the present invention includes a step of capturing an image of a subject marked with a heterogeneous camera group composed of cameras having different frame rates and resolutions to obtain an image, A marker detection step for detecting the two-dimensional position information of the image force marker acquired by each camera group, and acquiring three-dimensional position information of the marker using the two-dimensional position information of the marker detected in each camera group A three-dimensional reconstruction step of integrating the three-dimensional position information of the markers acquired by each force camera group. The integration step searches for the three-dimensional position information of the markers in time series. Tracking step to assign the same ID to the powerful three-dimensional position information determined to be the same, and the three-dimensional position information of the marker with the same ID acquired by each camera group Interpolating the optical motion.

[0008] A third technical means adopted by the present invention is a photographing means for photographing a subject marked with a heterogeneous camera group composed of cameras having different frame rates and resolutions to obtain an image. , For each camera group! Marker detecting means for detecting the two-dimensional position information of the marker from the acquired image, and the two-dimensional position of the marker detected in each camera group There is provided a three-dimensional reconstruction means for acquiring three-dimensional position information of the marker using the information, and an integrating means for integrating the three-dimensional position information of the marker acquired by each camera group. A tracking unit that searches the three-dimensional position information of the marker in time series and assigns the same ID to the three-dimensional position information of the marker determined to be identical, and the same ID acquired by each camera group is assigned. Means for interpolating the three-dimensional position information of the marker.

[0009] In one preferred aspect of the present invention, in the aspect, the marker detection step includes a step of determining a two-dimensional coordinate value of a reference marker position to which an ID is assigned and two-dimensional position information of the detected marker. An ID is assigned by determining the identity. Still preferably, in the three-dimensional reconstruction step, three-dimensional position information of a marker is obtained based on two-dimensional position information to which the same ID is assigned. In another preferred aspect, the three-dimensional reconstruction step assigns an ID by determining the identity between the reference marker position to which the ID is assigned and the three-dimensional position information of the reconstructed marker. is there.

[0010] The determination as to whether or not the markers are the same is made based on the distance between the position information of the detected marker or the position information of the three-dimensionally reconstructed manual force and the reference marker position. In one preferred embodiment, the reference marker position is a marker prediction position having the same ID calculated based on the measurement force position with ID obtained in the tracking step. The latest measurement marker position with ID obtained in the tracking step is used as the reference minimum force position.

[0011] In one preferred embodiment, the three-dimensional reconstruction step includes: calculating a distance between a plurality of marker estimation vectors to which the same ID has been assigned in the marker detection step; Determining whether the vectors intersect at a distance equal to or less than a threshold value. If the judgment is positive, calculate the three-dimensional position of the marker. If the judgment is negative, a search is made for a set that intersects at a distance equal to or less than the threshold with respect to the marker estimation vector used for calculating the three-dimensional position of the marker, and the newly found marker estimation vector force is also configured. A new ID is assigned to the three-dimensional position of the marker to be set.

[0012] In one preferred embodiment, a mode for capturing an image of a subject marked with a group of disparate cameras having different camera powers at different frame rates and resolutions to acquire an image. The 3D position information of the marker with the same ID obtained by each camera group is interpolated. The interpolation is for obtaining an interpolation polynomial for interpolating a marker trajectory composed of time-series three-dimensional position information of a marker to which the same ID is assigned over a plurality of frames measured by each camera group. Preferably, in calculating the coefficients of the polynomial, weighting is performed based on the average distance between marker estimation vectors at the time of three-dimensional reconstruction.

[0013] In a further preferred aspect, a marker position in a next frame of a camera of each camera group is predicted based on the interpolation. An ID is assigned to the marker prediction position, and by providing the marker prediction position information together with the ID to the marker detection step and the three-dimensional reconstruction step, the marker tracking (marker) is also performed in the marker detection step and the three-dimensional reconstruction step. ID).

The invention's effect

According to the present invention, local tracking of markers can be performed at the stage of marker detection and three-dimensional reconstruction of marker positions. By using the tracking result in marker detection, the amount of calculation required for three-dimensional reconstruction can be reduced, and real-time processing becomes possible. Furthermore, real-time labeling and joint angle calculation can be performed using the marker tracking function.

[0015] The following effects are obtained by integrating different types of cameras. One is that real-time tracking is difficult due to the large moving distance of the marker between one frame with a high-precision camera alone, but tracking is easier with the combined use of data from a high-speed camera. As a result, the accuracy of prediction is higher than when only low-speed data is used, so the accuracy of tracking (probability of correct marker correspondence)

Is improved. Second, the spatial resolution is low with high-speed cameras alone, but accuracy can be improved by using data with high-precision cameras. Correction of low-precision data is performed by using the distance between vectors as an index of accuracy and determining interpolation parameters based on the index. At this time, the distance between the vectors is generally large because the vectors of the low-resolution cameras are greatly shifted by an error of one pixel.

BEST MODE FOR CARRYING OUT THE INVENTION [A] Overall Configuration of Motion Capture System

The present invention relates to a high-precision and high-speed motion capture system by mutually complementing different types of cameras. The motion capture system includes a plurality of markers attached to a plurality of predetermined parts (a head, a neck, a shoulder, an elbow, a wrist, and the like) of a subject, imaging means for imaging the subject, and imaging means for the imaging means. It has one or more computers that are electrically connected (in a state that allows data transfer), and the image information of the markers acquired by the imaging means is transmitted to the computers, and the computer processes the image information. Is used to obtain the three-dimensional position of the marker. A computer connected to the photographing unit, a storage unit for storing the image information obtained by the photographing unit and the information calculated by the image processing unit, a display unit for displaying the image information, and an image for the image information. An image processing unit for performing the processing is configured.

[0017] The photographing means in the motion capture is composed of a plurality of video cameras, and by arranging a plurality of video cameras at different positions so as to surround the subject, a plurality of video forces can be obtained. Acquire images simultaneously. The acquired moving image is, for example, a time-series image group of still image information of several tens of frames and several hundred frames per second. In the present invention, the photographing means is composed of a group of different kinds of cameras, and the group of different kinds of cameras is one preferred! /, In an embodiment, is composed of a group of high-precision cameras and a group of high-speed cameras.

[0018] A high-precision camera and a high-speed camera refer to high-resolution cameras of different types when one has a higher resolution 'low frame rate than the other' and the other has a lower resolution 'high frame rate than the other'. A camera with a resolution of “low frame rate” is called a high precision camera, and a camera with a low resolution of “high frame rate” is called a high speed camera. In the experimental examples described below, the cameras that make up Group A have a resolution of 1004 x 1004 and a maximum frame rate of 50 fts, and the cameras that make up Group B have a resolution of 512 X 512 and a maximum frame rate of 262 ft) s, and make up Group A. The cameras that make up are high-precision cameras, and the cameras that make up Group B are high-speed cameras.

FIG. 1 is a diagram showing a configuration of a motion capture system using two types of camera groups (Gl, G2). The motion capture system according to the present invention has three block functions: marker detection processing, three-dimensional reconstruction processing, and integration processing. Each block performs the following processing. In the marker detection step (C), The camera image color marker is detected. In the three-dimensional reconstruction step (R), the three-dimensional position of the marker is calculated using the two-dimensional position information of the marker detected by the same type of camera group. In the integration step (I), data from a plurality of three-dimensional reconstruction blocks is integrated. It should be noted that FIG. 1 shows two camera groups. A force of three or more camera groups may be used.

[0020] In the marker detection process, the two-dimensional image force captured by each camera also detects a marker. The detection of power is performed by processing the images acquired by each camera, and the marker is detected as a certain coordinate in the two-dimensional image. The position of each camera constituting each group is determined in advance, and the detected marker is located on a vector (mark estimation vector) starting from the absolute position of the camera. The marker detection process is performed independently for each camera.

In the three-dimensional reconstruction processing, an intersection of a plurality of marker estimation vectors obtained from a plurality of cameras belonging to the same group at the same time is obtained, and the intersection is set as the three-dimensional position of the marker. However, in practice, marker estimation vectors do not intersect strictly due to the effects of camera pixel errors and calibration errors.Therefore, a set of vectors with a small distance is searched, and the midpoint of the common perpendicular is regarded as the intersection. . The three-dimensional reconstruction processing is performed independently for each camera group (G).

[0022] The integration process will be described. High precision data is obtained by using a high precision camera. However, high-precision cameras usually have low frame rates, so the time interval between data is large. To solve this, use a high-speed camera to obtain high frame rate data. However, the accuracy of high-speed camera data is low, so the accuracy will be reduced as it is. Therefore, high-speed and high-precision data can be obtained by integrating the two data and correcting the low-precision data using an interpolation method. The high-speed camera plays a role of providing the data to be interpolated at a high frame rate to facilitate integration and to increase the frame rate of data finally obtained as an output of the system. In order to integrate the three-dimensional position information obtained from different types of cameras, how the markers detected by camera group 1 correspond to the markers detected by camera group 2

(It is necessary to find out which and which are the data of the same marker, that is, to track the individual markers. The integration is to interpolate the marker positions measured by each camera group and to convert one continuous data. As a result, it becomes possible to predict a future marker position in which low-precision data is corrected with high-precision data. In the integration process, the three-dimensional marker position data sent from the plurality of camera groups is received, the correspondence with the previously observed marker is checked, and a unique ID is assigned (tracking). Marker position data of already measured frames is divided for each ID and stored together with the interpolation formula. At the time of integration, the current predicted position is calculated using this interpolation formula, the correspondence is checked, and the corresponding ID is assigned. For the marker that has been tracked, an interpolation polynomial is calculated in consideration of the error at the time of three-dimensional reconstruction. The error referred to here is the distance between the vectors in Eq. (7) described later.A larger value means that the reliability of the obtained three-dimensional position where the error of each vector is large is low. Give weight. As a result, an interpolation formula is calculated that approximates highly accurate data as a whole. By using this interpolation formula, a marker position is predicted when a certain camera group performs the next measurement (next frame), and the information is sent to the marker detection 3D reconstruction processing. Tracking can also be performed at the configuration stage. This enables real-time three-dimensional reconstruction and tracking as described later.

[0024] [B] Interpolation of marker locus

An interpolation process and a position prediction process for integrating data from different types of cameras will be described. The trajectory of each marker (marker with the same ID) is interpolated by a polynomial in order to predict the position of the marker and obtain data of a fixed time width after the end of the measurement. Specifically, when data measured at time t is sent, the coefficients of the polynomial interpolation function are determined using data after time t. here

Is a constant that determines the time width of data used for interpolation.

FIG. 3 shows an interpolation method. Time t (i = 0, 1,..., N; t- τ <t <t

m + i m m + 1

= t) is denoted by p, and the interval polynomial m + n i m m + 1 from time t to time t

Find f (t-t). Marker trajectory must be first differentiable at time t m m m

Then

[Number 1]

(o) = ^1} , p _m = J _m —l ( _m -t _m -i), p _m = H ― tm- \) Also, at time t, each polynomial passes through p, so

m + i m + i

△ t = t — t

i m + i m

[Number 2]

= p _{m + i} = 1, 2 _,. ,. (2) must be satisfied. Here, in order to cope with a case where the degree of the polynomial is smaller than the number of points used for interpolation, or a case where abnormal data is measured, it is not necessary to always strictly satisfy Expression (2).

If f (t) is a polynomial of degree r,

[Number 3] fm (t) two _Q f + a _x ¹ ten _{... + α, .- Λ ί +} a r (3) and write.

Clearly from equation (1),

Since the image a _r = Pm fc ^ ra _r — \-p _m , equation (2) can be summarized as follows.

[Number 5]

No "one t„ p _m -p _m )

(4) This is more

[Number 6]

Ta = u And solve this,

[Equation 7] a = ≠ p (6) and the coefficients of the polynomial can be calculated. Here, Τ ^# (vector) is a weighted pseudo-inverse of T (vector). The weight w corresponding to the data at each time is calculated using the average distance d between the marker estimation vectors during 3D reconstruction.

m + i

[Equation 8]

C

= ^a "m ~ + ~ i 7 + ^^ ( ⁷ ) where a is a coefficient for correcting the difference in camera frame rate, and C is an arbitrary constant. The weight of the marker, that is, the marker with a large error is reduced, and in general, for cameras with a high frame rate, a is reduced to reduce the influence of individual data.The interpolation method is described here. The method for determining the weight is not limited to the one described here, and any other method may be used as long as the function monotonically decreases with respect to the distance.

[0027] [C] Marker tracking

The marker tracking will be described. In this system, the above-mentioned interpolation is enabled by tracking the marker that can be determined to be the same across multiple frames and assigning the same ID. Since marker tracking is performed in real time, local tracking is performed not only in integration processing but also in marker detection and reconstruction processing. Ability to Explain Marker Tracking Based on Using Different Kinds of Cameras The marker tracking method according to the present invention is effective even in a system using only one kind of camera. In addition, here, as one preferred aspect, a force that describes a case where a marker prediction position is adopted as a reference marker position having an ID is not limited to the marker prediction position. For example, the latest measurement marker position with ID obtained in the tracking step may be adopted as the reference marker position.

FIG. 2 shows a conceptual diagram of tracking using the marker estimated position (marker predicted position). Figure 2 As shown in (2), the marker estimation positions (white circles 1, 2, and 3 with IDs) obtained by the integration processing are projected on a camera image to obtain two-dimensional coordinate values. Then, local tracking of the detected marker is performed by associating with the actually detected marker. When the correspondence between the estimated marker estimated position and the actually detected marker is confirmed, an ID is assigned to the detected marker. The detected marker (with ID) is reconstructed in three dimensions to obtain the marker position. The correspondence between the estimated marker position and the three-dimensionally reconstructed marker position is taken. These operations consist of a set of detected markers D,

Given a set of estimated markers E, the optimal assignment,

[Number 9]

{(Di, E), ( D 2, E h),..., A (¾, Ej), Ό, Find 6 _{{e {φ} 6} problems. Here, Εφ indicates that there is no corresponding estimated marker, that is, a new marker, and a new ID is added when such a marker appears.

In this embodiment, the score of the pair (D, E) is given as follows.

[Number 10]

Ρ (^, Ε = ------- (8)

Here, C is an arbitrary constant, and d (D, Ε) is the distance between the detected marker D and the estimated marker Ε. Note that Ε = φ, that is, when the corresponding estimated marker is strong, d (D, E) is set as the maximum distance d that may correspond. Then, as an evaluation function for optimization, equation (8)

max

Is used. Thereby, even if any of the markers are not detected, it is possible to search for a globally optimal assignment.

[0030] In tracking in the integration processing, IDs are added to detected markers by three-dimensional reconstruction processing. Therefore, when D and E have the same ID, a certain score is added to the pair (D, E) so that the pair is searched preferentially.

[D] 3D reconstruction

The three-dimensional reconstruction will be described. Three-dimensional reconstruction in normal motion capture In the configuration, since a search is performed for a short distance from a set of a large number of marker estimation vectors, the calculation amount is large. On the other hand, in the present invention, as described above, the IDs are already assigned to the markers detected by each camera, too! /, So that all IDs are correct! / Known, no need to search. In practice, there is a possibility that the wrong ID is assigned, so the distance estimation is used together to determine the set of marker estimation vectors, and the three-dimensional position of the force is calculated.

[0032] In a system having n cameras, three-dimensional reconstruction is performed when a set D of markers detected by camera k and a set E of estimated markers are given.

[Equation 11] Estimated marker e {} detected by camera

The combination of marker ^{€ {^ D 11} (,} ,..., A plurality find the problem. For each combination one single marker corresponding

. The search is performed as follows.

Step 1: Check the distance between marker estimation vectors having the same ID for each estimated marker. If n or more vectors intersect at a distance less than or equal to threshold d, it is assumed that a marker exists and mm max

To calculate the marker position.

Step 2: Search for a set that intersects with the marker estimation vector not used in Step 1 at a distance equal to or smaller than the threshold value in the same manner as in a normal motion capture. The marker found as a result is given a new ID.

[E] Experiment

An experiment to evaluate the accuracy and speed of this system and the results are described. The experimental equipment and conditions will be described. Using the two types of cameras shown in Table 1, a motion capture system with a total of 19 cameras was constructed. The cameras belonging to Group A are high-precision cameras with a resolution of 1004 x 1004 pixels and a maximum frame rate of 50 fps. The high-precision cameras consist of 10 high-precision cameras. The cameras belonging to Group B are high-speed cameras, with a resolution of 532 x 512 pixels and a maximum frame speed of S262 fps. The high-precision cameras consist of nine high-speed cameras. Marker detection One PC for each camera is assigned to each camera, and one of the PCs in each camera group performs 3D reconstruction. Separately, a Pentium IV 2GHz machine was used for integrated processing. The squat, walking, and kick movements were measured for each of the subjects with 10, 20, 30, and 40 markers attached, and the speed and tracking performance were evaluated. In the following evaluation experiments, τ = 0.1 s and r = 4 were used as interpolation parameters. The minimum number of vectors in the 3D reconstruction is n = 3, and the thresholds are d = 0.005m and d = 0.008m for groups A and B, respectively.

It was.

[table 1]

Table 1: Specifications of the cameras used for the experi

The speed evaluation will be described. The processing time of each step in the camera group and the integration processing is shown in Table 2 and Table 3, respectively. Table 4 shows the average time taken to acquire one frame. For 10 to 30 markers, the one using group B alone is the fastest, while for 40 markers, the integrated system is the fastest. This is thought to be because the load of the integration processing becomes relatively smaller as the number of markers increases as compared to the reconstruction.

[Table 2] Table 2: Computation time for each step at the camera groups [ms].

[Table 3]

Table 3: Computation time for each step at integration process (ms.

[Table 4]

Table 4: Average time per frame for various combinations of

The tracking performance evaluation will be described. Table 5 shows the percentage of markers that could be tracked to the end in each exercise. We can see that in most cases, the integrated system can track the most effort.

[Table 5] Table 5: Ratio of markers tracked throughout the motion

[0038] The real-time joint angle calculation will be described. In this system, markers are tracked and recorded in real time, so if a marker ID and a label are first matched and then displayed, real-time labeling and joint angle calculation can be performed. UTPoser (Yamane, Nakamura: "Cooperative Structure for Generating Whole Body Motion of Human-figure Yua," developed a GUI for labeling of initial frames and developed a flexible inverse kinematics calculation algorithm. Vol.20, no. 3, ρρ.113-121, 2002), real-time joint angle calculation was realized for an arbitrary marker arrangement. Since the inverse kinematics calculation took 70-80 ms, it was observed that the human figure on the screen moved with a time delay.

[0039] Experiments using two types of cameras have confirmed that high-speed data can be obtained and that mar-power can be tracked with high probability. In addition, real-time labeling and joint angle calculation were realized using the marker tracking function.

Industrial applicability

[0040] According to the present invention, the movement of a human or an animal can be measured as numerical data, and the data obtained thereby can be used in a wide range of fields such as biomechanics, humanoid movement generation, and CG animation generation. Can be used.

Brief Description of Drawings FIG. 1 is a schematic diagram of a motion capture system using two types of camera groups.

FIG. 2 is a diagram illustrating local tracking in a marker detection process and a reconstruction process. FIG. 3 is a diagram illustrating interpolation using a polynomial.

Claims

The scope of the claims

[1] a marker detection step for detecting two-dimensional position information of an image force marker acquired by each camera;

A three-dimensional reconstruction step of acquiring three-dimensional position information of the marker using the two-dimensional position information of the marker detected by each camera;

A tracking step of searching for the three-dimensional position information of the marker in time series and assigning the same ID to the three-dimensional position information of the marker determined to be the same,

The marker detection step and Z or the three-dimensional reconstruction step determines the identity of the reference marker position provided with the ID with the detected marker's two-dimensional position information or Z and three-dimensional position information, and Marker tracking in an optical motion capture system characterized by including a step of assigning the same ID as the reference marker position to the two-dimensional position information or Z and three-dimensional position information determined to be the same as the position Method.

[2] The marker detecting step is characterized in that the ID is given by determining the identity between the two-dimensional coordinate value of the reference marker position to which the ID is attached and the two-dimensional position information of the detected marker. The marker tracking method according to claim 1, wherein the marker tracking method is performed.

3. The marker tracking method according to claim 2, wherein the three-dimensional reconstruction step acquires three-dimensional position information of a marker based on two-dimensional position information to which the same ID is assigned.

[4] In the three-dimensional reconstruction step, the ID is given by determining the identity between the reference marker position to which the ID is added and the three-dimensional position information of the reconstructed marker. 4. The marker tracking method according to claim 1, wherein

[5] Claims 1 to 4 wherein the determination of whether or not the markers have the same force is performed based on the distance between the position information of the detected marker or the position information of the three-dimensionally reconstructed marker and the reference marker position. The marker tracking method according to any one of the above.

[6] The claim according to any one of claims 1 to 5, wherein the reference marker position is a marker prediction position having the same ID calculated based on the measurement marker position with ID obtained in the tracking step. Or the marker tracking method described in

[7] The marker tracking method according to any one of claims 1 to 5, wherein the reference marker position is the latest measurement force position with ID obtained in the tracking step.

[8] The three-dimensional reconstruction step includes:

Calculating a distance between a plurality of marker estimation vectors assigned the same ID in the marker detection step;

Determining whether or not vectors equal to or greater than a predetermined number intersect at a distance equal to or less than a threshold value;

8. The marker tracking method according to claim 1, further comprising:

[9] The marker tracking method according to claim 8, wherein if the judgment is positive, the three-dimensional position of the marker is calculated.

[10] If the determination is negative, a set that intersects with a strong marker estimation vector used for calculation of the three-dimensional position of the marker at a distance equal to or less than the threshold is searched, and a newly found marker estimation vector marker 9. The marker tracking method according to claim 8, wherein a new ID is assigned to a three-dimensional position of a marker composed of the marker.

[11] The method according to any one of claims 1 to 10, wherein the method comprises:

An image is acquired by photographing a subject marked with a heterogeneous camera group having different camera powers at different frame rates and resolutions,

The marker detection step and the three-dimensional reconstruction step are performed for each camera group, and the tracking step includes:

An integration step of integrating the three-dimensional position information of the markers with the same ID acquired by each camera group in a time-series manner,

A marker tracking method, characterized in that:

12. The marker tracking method according to claim 11, wherein the heterogeneous camera group includes a high-precision camera group and a high-speed camera group.

[13] The integrating step includes a step of obtaining an interpolation polynomial for interpolating a marker trajectory, which is a time-series three-dimensional position information force of a marker to which the same ID is assigned over a plurality of frames measured by each camera group. The marker according to any one of claims 11 and 12, Force tracking method.

[14] The marquee tracking method according to claim 10, wherein the interpolation is performed by the following polynomial.

[Number 1]

Jm {t) = a _Q r + ai t '" ¹ +'.. + A _r —it + a _r where f (t) is the interpolation formula between times t and t, a---a Is the coefficient of the polynomial.

m m m + 1 0 r

15. The marker tracking method according to claim 14, wherein in calculating the coefficients of the polynomial, weighting is performed based on an average distance between marker estimation vectors at the time of three-dimensional reconstruction.

16. The marquee tracking method according to claim 13, wherein a marker position in the next frame is predicted using an interpolation polynomial, and the predicted marker position is used as a reference marker position.

[17] a step of capturing an image of a subject marked with a heterogeneous camera group having different camera powers at different frame rates and resolutions, and acquiring an image;

A marker detection step for detecting the two-dimensional position information of the marker also for the image power acquired by each camera group;

A three-dimensional reconstruction step of obtaining three-dimensional position information of the marker using the two-dimensional position information of the marker detected in each camera group;

An integration step of integrating the three-dimensional position information of the marker acquired by each camera group, and the integration step includes:

A tracking step of searching for three-dimensional position information of the marker in a time series and assigning the same ID to the three-dimensional position information of the marker determined to be the same;

Interpolating the three-dimensional position information of the marker with the same ID acquired by each camera group;

An optical motion capture method comprising:

18. The motion capture method according to claim 17, wherein the heterogeneous camera group includes a high-precision camera group and a high-speed camera group.

[19] The interpolation step is for obtaining an interpolation polynomial for interpolating a marker trajectory, which is a time-series three-dimensional position information force of a marker to which the same ID is assigned over a plurality of frames measured by each camera group. 19. The motion capture method according to claim 17, wherein:

[20] The motion capture method according to claim 18, wherein the interpolation is performed by the following polynomial.

[Number 2]

J _m (t) = aof + α \ ί ~ '+.. + A _r ^ \ t + a _r where f (t) is the interpolation formula between times t and t, a---a Is the coefficient of the polynomial.

m m m + 1 0 r

21. The motion capture method according to claim 20, wherein in calculating the coefficients of the polynomial, weighting is performed based on an average distance between marker estimation vectors at the time of three-dimensional reconstruction.

22. The motion capture method according to claim 17, further comprising: predicting a marker position in a next frame of a camera of each camera group based on the interpolation.

[23] An ID is assigned to the marker prediction position, and the marker detection step and Z or the three-dimensional reconstruction step include a marker prediction position having an ID, two-dimensional position information of the detected marker, or Determining the identity with the Z and three-dimensional position information, and assigning the same ID as the marker predicted position to the two-dimensional position information or the Z and three-dimensional position information determined to be the same as the marker predicted position. The motion capture method according to claim 22, characterized in that the motion capture method is characterized in that:

[24] The marker detecting step is characterized in that the ID is assigned by determining the identity between the two-dimensional coordinate value of the marker predicted position to which the ID is assigned and the two-dimensional position information of the detected marker. 24. The motion capture method according to claim 23.

25. The motion capture method according to claim 24, wherein in the three-dimensional reconstruction step, three-dimensional position information of a marker is obtained based on two-dimensional position information to which the same ID is assigned.

[26] In the three-dimensional reconstruction step, the marker prediction position to which the ID is assigned and the reconstructed marker are 26. The motion capture method according to claim 23, wherein the ID is assigned by determining the identity of the force with the three-dimensional position information.

27. The motion capture according to claim 23, wherein the determination of whether or not the markers have the same force is performed based on a distance between the detected marker or the reconstructed marker and the predicted marker position. Method.

[28] The three-dimensional reconstruction step includes:

The method according to any one of claims 23 to 27, comprising:

29. The motion capture method according to claim 28, wherein if the judgment is positive, the three-dimensional position of the marker is calculated.

[30] If the determination is negative, a set that intersects with a strong marker estimation vector used for calculating the three-dimensional position of the marker at a distance equal to or less than the threshold value is searched, and a newly found marker estimation vector marker 29. The motion capture method according to claim 28, wherein a new ID is assigned to a three-dimensional position of a marker configured by the method.

[31] The method according to any one of claims 17 to 30, wherein the ID assigned to the marker in the tracking step is associated with a predetermined part of the subject in advance, and includes a step of performing real-time labeling. Motion capture method.

32. The motion capture method according to claim 31, wherein the joint angle calculation is performed in real time based on real time labeling.

[33] Imaging means for capturing an image of a subject marked with a heterogeneous camera group having different camera powers depending on the frame rate and resolution, and acquiring an image.

Marker detecting means for detecting the two-dimensional position information of the marker, the image force acquired by each camera group,

Using the two-dimensional position information of the markers detected in each camera group, Three-dimensional reconstruction means for acquiring original position information;

Integrating means for integrating the three-dimensional position information of the markers acquired by each camera group, and

Tracking means for searching for the three-dimensional position information of the marker in time series and assigning the same ID to the three-dimensional position information of the marker determined to be the same;

Means for interpolating the three-dimensional position information of the markers with the same ID acquired by each camera group,

An optical motion capture system comprising:

[34] The motion capture system according to claim 33, wherein the heterogeneous camera group includes a high-precision camera group and a high-speed camera group.

[35] The interpolating means obtains an interpolation polynomial for interpolating a marker trajectory, which is a time-series three-dimensional position information force of a marker assigned the same ID over a plurality of frames measured by each camera group. 35. The motion capture system according to claim 33, wherein:

36. The motion capture system according to claim 35, wherein the interpolation is performed by the following polynomial.

[Equation 3] fmi) = aof + a ' ^-1 + ·''+ a it + a _r where f (t) is the interpolation formula between times t and t, and a---a is the polynomial It is a coefficient.

m m m + 1 0 r

37. The motion capture method according to claim 36, wherein in calculating the coefficients of the polynomial, weighting is performed based on an average distance between marker estimation vectors at the time of three-dimensional reconstruction.

38. The motion capture system according to claim 33, further comprising means for predicting a marker position in a next frame of a camera of each camera group based on the interpolation.

[39] The marker prediction position is provided with an ID, and the marker detection means and Z or the three-dimensional reconstruction means provide a marker prediction position provided with an ID and a two-dimensional position of the detected marker. Information or the identity with Z and 3D position information, and 4. The apparatus according to claim 3, wherein the same ID as the force prediction position is assigned to the two-dimensional position information or z and three-dimensional position information determined to be the same as the measured position.

8. The motion capture system according to item 8.

[40] The marker detection means assigns an ID by determining the identity between the two-dimensional coordinate value of the marker predicted position to which the ID is assigned and the two-dimensional position information of the detected marker. 40. The motion capture system of claim 39.

41. The motion capture system according to claim 40, wherein the three-dimensional reconstructing means obtains three-dimensional position information with a strong force based on the two-dimensional position information to which the same ID is assigned. .

[42] The three-dimensional reconstructing means assigns an ID by judging the identity between the marker predicted position to which the ID is added and the three-dimensional position information of the reconstructed marker. 42. The motion capture system according to any one of 39 to 41.

43. The motion capture according to claim 39, wherein the determination of whether or not the markers have the same force is performed based on a distance between the detected marker or the reconstructed marker and the predicted marker position. Ya system.

[44] The three-dimensional reconstruction means,

Means for calculating a distance between a plurality of marker estimation vectors to which the same ID is assigned in the marker detecting means;

Means for determining whether or not vectors equal to or greater than a predetermined number intersect at a distance equal to or less than a threshold value;

The motion capture system according to any one of claims 39 to 43, comprising:

45. The motion capture system according to claim 44, wherein if the judgment is positive, the three-dimensional position of the marker is calculated.

[46] If the judgment is negative, a set that intersects with a strong marker estimation vector used for calculating the three-dimensional position of the marker at a distance equal to or less than a threshold is searched for, and a newly found marker estimation vector 46. The motion capture system according to claim 44, wherein a new ID is assigned to a three-dimensional position of the marker configured by the motion capture system.

47. The method according to any one of claims 39 to 46, wherein the ID assigned to the marker by the tracking means corresponds to a predetermined part of the subject in advance, and includes means for performing real-time labeling. Motion capture system.

48. The motion capture system according to claim 47, wherein the joint angle calculation is performed in real time based on real time labeling.