JP2002236924A - Movement tracking method and its device or the like for three dimensional object using two dimensional continuous image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method or the like capable of properly tracking a movement of a three dimensional object inside a two dimensional continuous image. SOLUTION: The method of tracking the movement of the three dimensional object by using the continuous two dimensional image is provided with a first step of calibrating by using a marker point of the three dimensional object and a corresponding marker point inside an initial frame in the two dimensional continuous image, and a second step of determining a projection matrix of the two dimensional image in a predetermined frame after the initial frame. It is characterized by that the projection matrix in the second step is determined so that a position of the marker point of the two dimensional image determined by the projection matrix is optimal.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention belongs to the technical field of analyzing and tracking the movement of a three-dimensional object from a two-dimensional continuous image.

[0002]

2. Description of the Related Art Conventionally, the motion of a three-dimensional object has been obtained by obtaining the shape and orientation of the three-dimensional object from a two-dimensional image or a two-dimensional continuous image having different viewpoints. However, this method requires a two-dimensional image of a three-dimensional object having a different viewpoint, and thus has a problem in that the configuration and the apparatus are complicated. The present applicant has made an invention relating to a method and an apparatus that has solved this problem, and is currently filing a patent application (Japanese Patent Application No. 2000-1886013). In the invention of the present application, there may be a case where the mark point obtained in the two-dimensional image does not approximate the position of the correct mark point. For this reason, the movement of the three-dimensional object may not be obtained correctly.

[0003]

SUMMARY OF THE INVENTION The present invention has been made under the above-mentioned background, and has as its object to provide a method, an apparatus, and the like capable of correctly tracking the movement of a three-dimensional object.

[0004]

The present invention employs the following means to solve the above-mentioned problems. That is, claim 1
The described invention is directed to a method for tracking the movement of a three-dimensional object using a continuous two-dimensional image, wherein the calibration is performed using a landmark point of the three-dimensional object and a corresponding landmark point in an initial frame in the two-dimensional continuous image. And a second step of determining a projection matrix of a two-dimensional image in a predetermined frame after the initial frame.
The determination of the projection matrix in the second step is characterized in that the arrangement of the landmark points of the two-dimensional image obtained by the projection matrix is determined to be optimal.

[0005] The invention according to claim 2 is based on claim 1.
In the invention described in (1), the landmark point of the three-dimensional object is a point in which six or more points are selected from characteristic points of the shape or color of the three-dimensional object.

According to a third aspect of the present invention, in the first aspect of the present invention, the projection matrix from the three-dimensional object to an arbitrary frame in a two-dimensional continuous image is the same as the first one. It is a projection matrix from a three-dimensional object for calibration in one step to an image coordinate system of the continuous image. According to a fourth aspect of the present invention, in the invention according to any one of the first to third aspects, the determination of the projection matrix for the predetermined frame in the second step is performed by a search unit. And the projection matrix is determined such that an evaluation function such as the sum of the distances or the square of the distance between the landmark points obtained by the search and the landmark points obtained by the projection matrix is minimized. And

According to a fifth aspect of the present invention, in the fourth aspect of the invention, the search means predicts a projection matrix using a Kalman filter, and determines a starting point of the search based on the predicted projection matrix. And searching. The invention according to claim 6 is characterized in that the search means performs a search by using the mark point obtained in the immediately preceding frame as a starting point.

[0008] The invention according to claim 7 is the first invention.
7. The method according to claim 6, wherein a distance or a square value of the distance between the landmark point in the frame of the two-dimensional image obtained by using the landmark point obtained by the search and the optimized projection matrix is a predetermined value. If it is larger than the above, the landmark point is deleted and the optimal projection matrix is determined.

According to an eighth aspect of the present invention, in the method according to any one of the first to seventh aspects, in the method for tracking the movement of a three-dimensional object, when a plurality of three-dimensional object to be tracked exist. The present invention according to claim 9, wherein an optimum projection matrix is obtained for each of the plurality of three-dimensional objects by programming the tracking method according to any one of claims 1 to 8. The program is recorded. According to a tenth aspect of the present invention, there is provided an apparatus for tracking and displaying the movement of a three-dimensional object using a continuous two-dimensional image, the apparatus comprising: Means for optimally determining a projection matrix for the three-dimensional object, and display means for obtaining a mark point of the three-dimensional object by using the projection matrix and displaying the movement of the three-dimensional object, such as its position and orientation. And

[0010]

FIG. 1 is a flowchart showing the flow of a method according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a relationship between a three-dimensional object and a continuous image. FIG. 3 shows an apparatus according to an embodiment of the present invention. Hereinafter, embodiments of the present invention will be described with reference to the drawings.

In FIG. 2, the three-dimensional object 10 is a rigid body, and a plurality of (about 20) mark points are selected and determined so that the overall shape is determined from characteristic points representing the shape and characteristic points in color. A coordinate system (Xb, Yb, Zb) 12 is an object coordinate system, and represents a real three-dimensional space where the object 10 exists. In the three-dimensional space, the three-dimensional object 10 moves such as rotation and movement. The coordinates of each mark point of the three-dimensional object 10 in the object coordinate system 12 are given by actual measurement or other means at the time of calibration. The upper diagram shows the discrete time Δt =
0,... N, (n + 1)}. Since the three-dimensional object 10 is a rigid body, it is not deformed. The lower diagram shows the discrete time Δt = 0,
.. N, (n + 1)}, in a continuous image taken by a video camera, frame 0 (FLM (0)), frame n (FLM (n)), and frame (n + 1) (FLM
(N + 1)) is illustrated. An image coordinate system (u, v) 13 is defined on the image plane.

Projection matrix 射 Φ (0),... Φ (n), Φ
(N + 1)} is a discrete time {t = 0,... N, (n +
1) A projection matrix representing the projection from the object coordinate system (Xb, Yb, Zb) 12 to the image coordinate system (u, v) 13 in｝. Here, it is assumed that the three-dimensional object 10 is fixed and the video camera 10 is relatively moved around the three-dimensional object 10 instead of fixing and installing the video camera. That is, the projection matrix ｛Φ (0),... Φ (n), Φ
(N + 1)} is a discrete time {t = from the coordinate space of the three-dimensional coordinate system 12 at time t = 0 (at the time of calibration).
0,... N, (n + 1)} is a projection matrix of a continuous image onto the image coordinate system (u, v) 13. When there are a plurality of three-dimensional objects whose motions are to be tracked, a projection matrix is determined for each three-dimensional object.

The above assumption of the projection matrix is based on the assumption that the video camera is fixedly installed and the projection matrix is discrete time Δt = 0,.
n, (n + 1)} in the object coordinate system (Xb, Yb, Z
b) This is equivalent to the case where the projection from 12 to the image coordinate system (u, v) 13 is represented. That is, the projection matrix ｛Φ (0),.
By knowing Φ (n), Φ (n + 1)}, the motion of the three-dimensional object in the object coordinate system (Xb, Yb, Zb) 12 in the discrete time {t = 0,. You can know.

A procedure for obtaining the projection matrix Φ (i) (i = 0, 1, 2,...) And a method for tracking the movement of the object 10 will be described below with reference to the flowchart shown in FIG. In FIG. 1, in step 1, calibration is performed to obtain unknown parameters of the projection matrix Φ (0). Calibration methods are well known and again utilize conventional techniques. First, at least six or more (preferably about twenty) landmark points are determined on the three-dimensional object 10. The mark point may be a feature point of the shape or a feature point of color. Next, coordinate values of these landmark points in the object coordinate system 12 are determined. Next, FLM (0)
, The unknown parameters of the projection matrix Φ (0) can be determined by solving the relational expression with the coordinates of the mark points.

Incidentally, it is also possible to program the method of solving the relational expression and determine the unknown parameter by operation on the screen. That is, the landmark points of the three-dimensional object 10 are programmed so as to be directly displayed on the image coordinate system, and when these landmark points are moved to coincide with the landmark points of FLM (0), the projection matrix Φ ( It can be programmed so that the unknown parameter of 0) can be determined. Using this method, the projection matrix Φ (0) can be determined by a simpler operation.

Next, in step S2, the frame FLM
Using the data obtained in (n), the frame FLM (n
The procedure for obtaining the projection matrix Φ (n + 1) of (+1) will be described. First, in step S21, coordinates of a starting point for efficiently searching for a landmark point on an image plane are predicted. For this prediction, a conventionally used method can be used. For example, linear prediction (prediction by a linear expression) using the position and speed of the mark point obtained before the frame FLM (n), or 0-order prediction (movement of the immediately preceding frame) when the movement of the three-dimensional object is slow. The obtained landmark point may be a prediction starting point in the next frame). Alternatively, as described in the invention (Japanese Patent Application No. 2000-242565) filed by the present applicant, a projection matrix is estimated using a Kalman filter, a landmark point is predicted based on the projection matrix, and the point is searched for. It may be a starting point. By using an appropriate prediction method, the search time is reduced (the number of searches is reduced), and the coordinates of the correct landmark point can be obtained.

In step S22, a search for a landmark is performed from the starting point obtained in step S21. This search is performed for all the valid landmark points (the landmark points deleted from the list are omitted). The search method uses a (8 × 8) bit mask, for example, and moves this mask back and forth from left to right and back from the starting point, and matches the bit mask pattern at the mark point obtained in the immediately preceding frame FLM (n) best. Is regarded as a landmark point in the frame FLM (n + 1), and its coordinates (u1, v1) are read. By repeating this, the coordinates of all the mark points are obtained.
Note that, for example, when there are no landmark points that match the frame FLM (n + 1), those landmark points are omitted from the list.

When the coordinates of all the mark points in the frame FLM (n + 1) are obtained by the search, step S23 is performed.
Then, an optimal projection matrix Φ (n + 1) is obtained by the following procedure. That is, it is assumed that the optimal projection matrix Φ (n + 1) has been obtained, and this is defined as “Φm”. By multiplying the landmark point of the three-dimensional object at time t = 0 by the optimal projection matrix Φm, the coordinates (u1m,
v1m) is obtained. It is desirable that the coordinates (u1, v1) of the landmark point obtained by the search match the coordinates (u1m, v1m) of the landmark point obtained by the above method. Therefore, the coordinates (u1m, v1m) and the coordinates (u
1, v1) and the sum of squares (or sum of distances)
The value of each element of the projection matrix Φ (n + 1) is determined so that the evaluation function J is minimized. As a calculation method, a method of obtaining an extreme value by a conventionally known partial differential method or a hill-climbing method can be applied. The optimal projection matrix Φ (n
+1) is determined.

In step S24, the optimal projection matrix Φ (n +
Check if 1) is a truly optimal projection matrix. That is, the projection matrix Φ (n +
1) may include a large error. For example, there are cases where the coordinates (u1, v1) of the landmark point obtained by the search are significantly apart from the coordinates of the true landmark point. Therefore,
For each landmark point, the value of the distance (or square) between the coordinates (u1, v1) of the landmark point obtained by the search and the coordinates (u1m, v1m) of the landmark point obtained by the above method is a predetermined value. If it is larger than (for example, a distance corresponding to 10 pixels), the landmark point (u1, v1) is regarded as an erroneous landmark point and is deleted from the landmark point list. Alternatively, the coordinates (u1m, v1m) of the mark point may be searched again as a starting point. This check is performed for all landmark points, and the projection matrix Φ (n + 1) is again calculated from the coordinate relationship of the remaining landmark points excluding the deleted landmark points (step S2).
3).

The landmark point in the frame FLM (n + 1) is obtained by multiplication using the optimal projection matrix Φ (n + 1), and the landmark point (u1, v1) is searched again by using the coordinates of the landmark point as a starting point. And the projection matrix Φ (n +
1) is determined, a landmark point (u2, v2) is obtained again, and a series of steps (steps S21 to S24) of checking the landmark point are repeated a plurality of times to obtain an optimized projection matrix Φ (n + 1). May be obtained.

Next, the motion of the three-dimensional object is determined in step S3. In step S2, the optimal projection matrix Φ (n + 1) is obtained, and in step S1, the projection matrix Φ (0) is obtained. From the coordinates of the three-dimensional object at time t = 0, time t =
The transformation matrix of the coordinates of i (i = 1, 2,...) is (Ti)
Then, the relational expression “Ti * Φ (0) = Φ (i), (i
= 1, 2,...). The conversion matrix (Ti) can be determined from these relationships. Therefore, the time (t) of the three-dimensional object 10 placed in the object coordinate system 12
The coordinates of the landmark point of i) can be obtained. Thereby, movement such as movement or rotation of the three-dimensional object in the object coordinate system 12 is also obtained. In the last step S4, the movement of the three-dimensional object is displayed on the screen.

FIG. 3 is a block diagram of an apparatus for performing the method of the present embodiment. In FIG. 3, photographed image data is transmitted from the video camera 21 to the device 20. The device 20 includes a processing unit 22, input buffers 23a and 23b, an output buffer 24, a memory 25, and the like. The device 20 also includes a display 26, a keyboard 27, a mouse 2
8 are connected. The operation of the flowchart is executed as follows by the above device.

First, an image from the video camera 21 is taken into the device 20 online or offline.
Recorded in the memory 25. When the program recorded in the memory 25 is started, first, a screen for calibration is displayed on the display 26. The operator operates the screen with the mouse 28 or inputs data from the keyboard 27 to execute the calibration program (the program in step S1). next,
When the program in step S2 including the search program and the projection matrix optimization program is executed, the image data fetched from the memory 25 is processed, and the projection matrix Φ is obtained.

A motion determination program (program of step S3) calculates a coordinate change (movement, rotation, etc.) of the three-dimensional object from the obtained projection matrix, sends the motion data to the display 26, and sends the motion data to the display 26. The movement is displayed. Hereinafter, steps S2 to S4 are repeatedly executed until a stop signal is input.

As can be understood from the above description,
The program of the present embodiment can optimize the projection matrix,
At this time, mark points that are no longer present in the captured image or mark points that are erroneously recognized are deleted from the list and are not considered. This has the advantage that the tracking is performed accurately and that the calculation does not diverge on the way. In addition, the search for a landmark point predicts a starting point, and does not search the entire screen. For this,
There is an advantage that the calculation time can be reduced. Also,
Since the movement of the three-dimensional object is grasped by attaching the mark points to the surface of the three-dimensional object, there is an advantage that the processing is simplified as compared with the case where the entire shape is handled.

The embodiments and examples of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to the examples, and changes in design and the like may be made without departing from the gist of the present invention. Even if there is, it is included in the present invention. For example, although the case where the number of three-dimensional objects for tracking motion is one has been described, the case where there are a plurality of three-dimensional objects can be tracked simultaneously. Further, a recording medium on which a program obtained by programming the method of the present invention is recorded is also included in the technical scope of the present invention.

[0027]

As described above, according to the structure of the present invention, the present invention (the invention as set forth in claims 1 to 10)
Since optimization is performed when obtaining the projection matrix, 3
An effect is obtained that correct tracking of a three-dimensional object becomes possible. In the inventions according to claims 4 to 6, since the starting point is predicted when the landmark point is searched for and found,
The effect of reducing the number of searches and shortening the calculation time is also obtained.

[Brief description of the drawings]

FIG. 1 shows a flowchart of an embodiment of the present invention.

FIG. 2 is a diagram illustrating a relationship between projection matrices.

FIG. 3 shows an apparatus diagram of an embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 3D object 12 Object coordinate system (3D space coordinate system) 13 Image coordinate system (2D plane coordinate system) 20 Processing device 21 Video camera 26 Display (display means)

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Takao Ichihashi 5-19-9 Hiroo, Shibuya-ku, Tokyo Hiroo ON Bldg. QQ24 RR07 SS13 UU05 5B057 AA16 AA19 BA02 BA24 DA07 DB02 DC19 DC33 5L096 BA02 CA04 CA24 DA01 FA38 FA66 FA67 HA05 HA08

Claims (10)

[Claims] 1. A method of tracking the movement of a three-dimensional object using a continuous two-dimensional image, wherein the calibration is performed using a landmark point of the three-dimensional object and a corresponding landmark point in an initial frame of the two-dimensional continuous image. And a second step of determining a projection matrix of a two-dimensional image in a predetermined frame after the initial frame. The determination of the projection matrix in the second step is obtained by the projection matrix. A method for tracking the movement of a three-dimensional object using two-dimensional continuous images, characterized in that the arrangement of landmark points in the two-dimensional image is determined to be optimal. 2. The method according to claim 1, wherein the landmarks of the three-dimensional object are points selected from six or more points out of characteristic points of the shape or color of the three-dimensional object. A motion tracking method for a three-dimensional object using the continuous two-dimensional image described above. 3. A projection matrix from the three-dimensional object to an arbitrary frame in a two-dimensional continuous image is a projection matrix from the three-dimensional object for calibration in the first step to an image coordinate system of the continuous image. The method for tracking the movement of a three-dimensional object using a continuous two-dimensional image according to any one of claims 1 and 2, characterized in that: 4. The step of determining a projection matrix for a predetermined frame in the second step is performed by searching for a landmark point in the predetermined frame by a search means, and the landmark point obtained by the search and a landmark obtained by the projection matrix are determined. 4. The projection matrix according to claim 1, wherein the projection matrix is determined such that an evaluation function such as a sum of distances of points or a sum of squares of distances is minimized.
A method for tracking the movement of a three-dimensional object using the continuous two-dimensional image according to any one of the above. 5. The continuous search method according to claim 4, wherein said search means predicts a projection matrix using a Kalman filter, determines a starting point of the search based on the predicted projection matrix, and performs a search. A method for tracking the movement of a three-dimensional object using a three-dimensional image. 6. The method according to claim 4, wherein the search means searches starting from a mark point obtained in the immediately preceding frame as a starting point. . 7. When the distance or the square of the distance between the landmark point in the frame of the two-dimensional image obtained by using the landmark point obtained by the search and the optimized projection matrix is larger than a predetermined value, the landmark is displayed. The method for tracking the movement of a three-dimensional object using a continuous two-dimensional image according to any one of claims 1 to 6, wherein points are deleted and the optimal projection matrix is determined. 8. A three-dimensional object motion tracking method, wherein when there are a plurality of three-dimensional objects to be tracked, an optimum projection matrix is obtained for each of the plurality of three-dimensional objects. A method for tracking the movement of a three-dimensional object using a continuous two-dimensional image according to any one of claims 1 to 7. 9. A recording medium on which the tracking method according to claim 1 is programmed and the program is recorded. 10. An apparatus for tracking and displaying the movement of a three-dimensional object by using a continuous two-dimensional image, the apparatus comprising: a projection matrix for a landmark point of a two-dimensional image in a predetermined frame after an initial frame; Two-dimensional continuity comprising: means for optimally determining; and display means for obtaining a mark point of the three-dimensional object by using the projection matrix and displaying a movement of the three-dimensional object such as a position and an orientation. A three-dimensional object motion tracking device using images.