JP2003178308A - Calculation method and unit, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically track a shot dynamic image even when a feature point is hidden behind an object or the like and it cannot be tracked in tracking (locus tracking) of the dynamic image. <P>SOLUTION: First positional information which is information of projection positions of feature points of a first group of the object shot in a plurality of images is used as an input, and second positional information which is information of projection positions of feature points of a remaining second group is determined by a calculation process circuit 11. In the calculation process circuit 11, a first parameter relevant to a weight in a representation by a weighted average of positions of the feature points of the first group and a second parameter relevant to the projection positions of the feature points of the first group and positions, orientations and focal lengths of an imaging device shooting the image at a particular time and a target time are used as unknown variables, the second positional information and the unknown variables are determined by a relational expression of the unknown variables and the projection positions of the feature points of the first and second groups on the images, and the determined second positional information is outputted. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a calculation method and apparatus for obtaining a locus of feature points of an object projected in image data, a program, and a recording medium recording the program.

[0002]

2. Description of the Related Art For example, it is assumed that a moving image (hereinafter referred to as an original moving image) is taken by using a camera.
It is assumed that one rectangular object is captured in the image of the original moving image at each time. The four vertices of this rectangle are referred to as the first to fourth vertices, generally the first to fourth feature points. The loci of the projected images at the four corners (four vertices) of the rectangle projected on this original moving image are tracked (trajectory tracing). Let (U1 (t), V1 (t)) be the result of tracking the projected image of the first vertex (that is, the first feature point).
That is, the first vertex at time t is (U
1 (t), V1 (t)). Similarly, the second, third,
The results of tracking the projected images of the four vertices (that is, the second, third, and fourth feature points) are (U2 (t), V2
(t)), (U3 (t), V3 (t)), and (U4 (t), V4 (t)). By tracking these four corners, the position of the projected image of the entire rectangle at each time can be found. Therefore, by using this information, a separately prepared texture (pattern) can be pasted at the position of this rectangular projected image. As a result, it is possible to obtain a moving image captured as if the above-mentioned texture was present in the rectangular portion from the beginning.

More generally, in the image of the original moving image at each time, the projected images of the four feature points of the imaged object (including the case of a plurality of objects, a group of objects, etc.) are tracked. The result is (U1 (t), V1 (t)),
(U2 (t), V2 (t)), (U3 (t), V3 (t)), (U4 (t), V4
(t)).

Thus, a certain object (or a group of objects)
Tracking feature points (eg, object corners) is a common practice in the field of image processing,
And it is necessary.

[0005]

By the way, for example, in the above-mentioned rectangular example, not all vertices are visible at all times. So at one time,
It may happen that a certain vertex cannot be photographed because it is hidden by the shadow of some other object. In this case, conventionally, it was necessary for a human (operator) to specify where the image was projected if it was not hidden by the shadow of the object. If there are many images in which the vertices could not be captured in the moving image, it is necessary for the operator to manually specify all these images one by one. It was hanging. And, it was not possible to answer the request to request the designation in a short time without human intervention.

The present invention has been made in view of such a situation, and when a captured moving image is being tracked (trajectory tracking), the feature point cannot be tracked because it is hidden behind the object. Even if it happens, a calculation method and device, a program, which enables automatic tracking,
In addition, the purpose is to provide a recording medium.

[0007]

According to the calculation method of the present invention, the first position information, which is the information of the projected position of the feature points of the first group of the object photographed in a plurality of images, is input, and the remaining In the calculation method for obtaining the second position information which is the information of the projected position of the feature point of the second group, the first position related to the weight when the weighted average of the positions of the feature points of the first group is expressed.
And a second parameter related to the position, orientation, and focal length of the imaging device that captured the image at the projection position of the first group of feature points and the specific time and the time of interest. And the unknown variable and the first
It has a calculation step of obtaining the second position information and the unknown variable from a relational expression with the projected image position on the image of the feature points of the group and the second group, and the calculation step obtained in the calculation step. The above-mentioned problem is solved by a feature that the second position information is output.

Here, in the calculation step, the error under the condition that the relational expression between the unknown variable and the projected image position of the first and second feature points on the image is satisfied in consideration of the error. The second position information and the unknown variable are calculated so as to minimize the evaluation function, and the error relating to the second position information is weighted more than the error relating to the first position information. Is made lighter, and is evaluated within the above error evaluation function.

Next, the arithmetic unit according to the present invention receives as input the first position information which is the information on the projected position of the feature points of the first group of the object photographed in the plurality of images, and the remaining second group. In a computing device for obtaining second position information, which is information of the projected position of the feature point, a first parameter associated with the weight when expressed by a weighted average of the positions of the feature points of the first group, The unknown positions are the projection positions of the characteristic points of the first group and the second parameter related to the position, orientation, and focal length of the imaging device that has captured the image at a specific time and the time of interest. From the relational expression between the variable and the projected image position on the image of the feature points of the first group and the second group,
The above problem is solved by having a calculation circuit for calculating the second position information and the unknown variable, and outputting the second position information calculated by the calculation circuit.

Here, the calculation circuit has an error under the condition that the relational expression between the unknown variable and the projected image positions of the first and second feature points on the image is satisfied in consideration of the error. The second position information and the unknown variable are calculated so as to minimize the evaluation function, and the error relating to the second position information is weighted more than the error relating to the first position information. Is made lighter, and is evaluated within the above error evaluation function.

Next, a program according to the present invention is characterized by causing a computer to execute the processing procedure of the arithmetic method having the above characteristics.

A computer-readable recording medium according to the present invention is characterized by recording a program for causing a computer to execute the processing procedure of the arithmetic method having the above characteristics.

[0013]

BEST MODE FOR CARRYING OUT THE INVENTION In the present invention, a plurality of feature points (feature points of the first group) of an object (which may be a group of objects) to be tracked as well as other feature points (feature points of the second group) are selected. I also tried to track. This allows
Even if the projected image positions of the characteristic points of the first group cannot be tracked for some reason, the projected image positions of the characteristic points of the first group can be automatically obtained.

By the way, in the example of the rectangle, the four vertices of the rectangle are referred to as first, second, third and fourth feature points. Then, the results of tracking them on the original moving image are (U1 (t), V1 (t)), (U2 (t), V2 (t)),
(U3 (t), V3 (t)) and (U4 (t), V4 (t)). At a certain time, (U1 (t), V1 (t)), (U2 (t), V
One or more data of (2 (t)), (U3 (t), V3 (t)), (U4 (t), V4 (t)) are undefined. The final goal is to finalize these indefinite data.

Further, as mentioned above, the rectangle 4
Tracking is also performed for feature points other than vertices. These are the 0th feature point and the 5th, 6th ,. ． ． , N-1 feature points, N-1 feature points in total. Then, the tracked result is (U0 (t), V0 (t)) and (U5 (t), V5 (t)), ((U6 (t), V6
(t)) ,. ． ． , ((UN-1 (t), VN-1 (t)). These tracked results may also be indefinite at a specific time.

In an example of such a rectangle, the first
The feature points of the group are three vertices of the four vertices of the rectangle, for example, the first, second, and third feature points, and one point not on the plane including the rectangle, for example, the zeroth feature point. There are four feature points consisting of and. The remaining 4th to N-1 feature points are the feature points of the second group.

Since the method of tracking the projected images of these characteristic points can be automatically performed by a method such as block matching, it is a burden on the operator to track N characteristic points as a whole. Don't

The point of the present invention is to automatically determine indefinite data that cannot be tracked by inputting track data of these characteristic points on the original moving image.

The first embodiment of the arithmetic method according to the present invention will be described in detail below.

The position of the i-th feature point in the three-dimensional space is defined as (X
i, Yi, Zi). i is 0 to N-1. For example, in the example of the above rectangle, the 0th feature point (X0, Y0, Z0)
However, when it is not on a plane including the above rectangle, the 0th to 3th (that is, 0th to 3rd) feature points are set as the feature points of the first group, and the remaining 4th to N-1 (that is, 4th to Nth). The (-1st) feature point is defined as the feature point of the second group. In the original moving image, the projected image position of the i-th feature point at time t is (Ui (t), Vi (t)), so if the projective transformation matrix at time t is {rij (t)}, The following Equation 1 should be satisfied.

[0021]

[Equation 1]

Here, the above equation 1 is a homogeneous coordinate system (homoge
neous coordinates), and Wit is a variable added to represent in the homogeneous coordinate system.

Four feature points 0 to 3 (X0, Y0, Z
0), (X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3,
It is assumed that the projection position of Z3) is fixed on the original moving image at t = 0. That is, (U0 (0), V0
(0)), (U1 (0), V1 (0)), (U2 (0), V2 (0)), (U3
(0), V3 (0)) is observed. And the above 4
It is assumed that two feature points are not on one plane in the three-dimensional space. With appropriate m0j, m1j, and m2j, other feature points (Xj,
Yj, Zj) satisfies the following expression 2.

[0024]

[Equation 2]

Here, j is 4 to N-1. In addition,
Particularly, in the example of the above rectangle, m04 = 0, m14 = 1, m24
= -1, m34 = 1. This is because "the position of the second feature point" + "the position of the fourth feature point" = "the position of the first feature point" + "the position of the third feature point" = "the center of the rectangle" is there.

By modifying the above expression 1 by the expression,
The following Equation 3 is obtained.

[0027]

[Equation 3]

However, Kij and R0 (t), R1 (t), R
2 (t) is a value represented by the following Equation 4 and Equation 5, respectively. In addition, in particular, in the example of the above rectangle, since m04 = 0,
K04 = 0.

[0029]

[Equation 4]

[0030]

[Equation 5]

The time t takes a value from 0 to T-1. It is assumed that the moving image is captured from T times from time 0 to T-1. Since the position of the camera at the first time is used as a reference, the projective transformation matrix at time t = 0 is given by the following Expression 6.

[0032]

[Equation 6]

Here, r00 (0) is the focal length at t = 0 in the U-axis direction on the image, r11 (0) is the focal length at t = 0 in the V-axis direction on the image, Coordinates (r02
(0), r12 (0)) is the intersection of the optical axis center and the imaging surface at t = 0. As can be seen from Equations 5 and 6, R0 (0) = 0, R1 (0) = 0, and R2 (0) = 0.

Position of each projected image at each time (observed value)
Since (Ui (t), Vi (t)) contains an error, this error is defined as (εi (t), δi (t)). As described above, the position of a specific feature point at a specific time may not be projected on the original image. Therefore,
A flag Fi (t) for determining whether (Ui (t), Vi (t)) is a fixed observation value or indefinite is introduced. This value is input from the outside. Fi (t) = 0 means that (Ui (t), Vi (t)) is indefinite and Fi
(t) = 1 means that (Ui (t), Vi (t)) is defined input data (that is, an observable value).
For example, if a certain feature point is hidden by a shadow of another object at a certain time and is not captured by the camera, Fi
(t) = 0.

Here, when the problems are sorted out, "Fi (t) = 1
(Ui (t), Vi (t)) ”is given, Kij, R0 (t), R that minimizes the following expression 8 under the condition of the following expression 7 is given.
1 (t), R2 (t) and "Fi (t) = 0 (Ui (t), Vi
(t)) ”.

[0036]

[Equation 7]

[0037]

[Equation 8]

In the above equation 8, Ci (t) is the accuracy of the measurement of the i-th feature point at time t and is input from the outside. When the observed values (Ui (t), Vi (t)) are reliable, a large value is input as Ci (t).

When obtaining the state that minimizes the above equation 8,
It is difficult to obtain Kij and R0 (t), R1 (t), R2 (t) and “Fi (t) = 0 (Ui (t), Vi (t))” all as variables at once. is there. Therefore, (Step SA) Kij and R0
By fixing (t), R1 (t) and R2 (t), "Fi (t) = 0 (Ui (t), Vi (t))" is obtained. (Step SB) R0
(t), R1 (t), R2 (t) and "Fi (t) = 0 (Ui (t), V
i (t)) ”is fixed and Kij is calculated. (Step SC)
Fixing Kij and "(Fi (t) = 0 (Ui (t), Vi (t))", R0 (t), R1 (t), and R2 (t) are obtained. The above three steps are repeated to converge.

Further, the above equation 7 is approximated to the following equation 9. Where A0j (t), A1j (t), A2j (t), A3j (t)
Is a value obtained from the following Expression 10 using the data before updating the data in each step described above. If the above steps are repeated (εi (t), δi (t)) → (0,0)
Then, the following expression 9 is equivalent to the above expression 7. Therefore,
There is no problem by using the following Expression 9 as the conditional expression instead of Expression 7 above.

[0041]

[Equation 9]

[0042]

[Equation 10]

The above processing is summarized in the flow chart of FIG.

First, in step S1 of FIG. 1, data is input. That is, Fi (t) that indicates whether or not the i-th feature point was confirmed as an observation value at time t is input. When Fi (t) = 1, the observed value (U
i (t), Vi (t)) is also input. Furthermore, the "probability of observation value Ci (t)" for all i and t is also input.

Unobservable (Fi (t) = 0) means that (Ui (t), Vi (t)) is set to an appropriate value, and this value includes a considerable error. If Fi (t) = 0, then as Ci (t), a very small value (for example, 1/10000 of the value of Ci (t) when Fi (t) = 1) Enter a value). This has the effect of improving the convergence in steps SB and SC described above.

As described above, at least time t =
It is assumed that the projection positions of the 0th to 3rd feature points at 0 can be observed. That is, F0 (0) = 1, F1 (0) = 1, F2 (0) =
1 and F3 (0) = 1. When the input in step S1 is completed, the process proceeds to step S2.

In step S2, appropriate values are set to Kij, R
Set to 0 (t), R1 (t), R2 (t). For example, Kij =
0.25, R0 (t) = 0, R1 (t) = 0, and R2 (t) = 0. Also, in the example of the above rectangle, K04 = 0, so 0.2
Instead of 5, K04 = 0, K14 = 1, K12 = -1, K13
It may be = 1. When the process of step S2 ends, the process proceeds to step S3.

In steps S3 and S4, the processing in step SA described above is performed. That is, in step S3, (U0 (0), V0 (0)) and (U1 (0), V input in step S1 are input.
1 (0)), (U2 (0), V2 (0)), (U3 (0), V3 (0)) and current Kij, R0 (t), R1 (t), R2 (t) , A0j (t), A1j (t), A2j (t), A3j (t) defined by the above equation 10 are obtained. Then, the process proceeds to step S4. In step S4,
A0j (t), A1j (t), A2j (t), A3 obtained in step S3
Using j (t), we minimize Eq. 8 under the condition of Eq.
i (t) = 0 (Ui (t), Vi (t)) "is obtained. After the process of step S4, the process proceeds to step S5.

In steps S5 and S6, the process of step SB described above is performed. That is, in step S5, (U0 (0), V0 (0)) and (U1 (0), V input in step S1 are input.
1 (0)), (U2 (0), V2 (0)), (U3 (0), V3 (0)) and current Kij, R0 (t), R1 (t), R2 (t) , A0j (t), A1j (t), A2j (t), A3j (t) defined by the above equation 10 are obtained. Then, the process proceeds to step S6. In step S6,
A0j (t), A1j (t), A2j (t), A3 obtained in step S5
Using j (t), Kij that minimizes Equation 8 above is obtained under the condition of Equation 9 above. By the way, A0j (t), A1j (t), A2j (t),
A3j (t) is the same as the value in the immediately preceding step S3, and the value in step S3 may be used as it is without being recalculated in step S5. Also, in the example of the above rectangle, since K04 = 0, the processing in step S6 is
The processing is "to find Kij (excluding K04) that minimizes Expression 8 under the condition that Expression 9 is satisfied and K04 = 0." After the process of step S6, the process proceeds to step S7.

In steps S7 and S8, the process of step SC described above is performed. That is, in step S7, (U0 (0), V0 (0)) and (U1 (0), V input in step S1 are input.
1 (0)), (U2 (0), V2 (0)), (U3 (0), V3 (0)) and current Kij, R0 (t), R1 (t), R2 (t) , A0j (t), A1j (t), A2j (t), A3j (t) defined by the equation 10 are obtained.
Then, the process proceeds to step S8. In step S8, A0j (t), A1j (t), A2j (t), A3j (t) obtained in step S7
R0 that minimizes the above equation 8 under the condition of the above equation 9
Calculate (t), R1 (t), R2 (t). R0 (t), R1 (t) and R2 (t) obtained in step S8 are t = 1 to T-1. t
= 0, the values are R0 (0) = 0 and R1 (0) =
It is fixed at 0 and R2 (0) = 0. After the process of step S8, the process proceeds to step S9.

The series of processing from step S3 to step S8 is repeatedly executed, but the number of times of repeated processing (the number of loops) exceeds a predetermined number, or the equation 8 is calculated in steps S4, S6 and S8. Although the data is updated so as to minimize it, it is determined in step S9 whether or not the value (that is, the error) of the equation 8 calculated at that time is within a predetermined range, and it is true (YES). If there is, go to step S10. If false (NO),
Proceed to step S3, and again from step S3 to step S
A series of processes up to 8 is performed.

When the processing proceeds to step S10, Kij, R0 (t), R1 (t), R2 (t) with sufficient accuracy and "Fi (t) = 0 (Ui (t ), Vi (t)) ”is required. At this point, “Fi (t) = 0 (Ui (t), Vi (t))” that has been finally desired has already been obtained, but it can be further observed (Ui (t), Vi (t)) (that is, (Ui (t), Vi (t)) where Fi (t) = 1) is also corrected. That is, further, step S1
0, S11, Kij, R0 (t), R1 (t), R2
Using (t) and (Ui (t), Vi (t)), (Ui (t), Vi
An estimated value (Ui (t) + εi (t), Vi (t) + δi (t)) of (t)) is obtained.

In step S10, R0 (0) = 0, R1 (0)
Since = 0 and R2 (0) = 0, (εi (0), δi (0)) that minimizes the following expression 12 is obtained under the condition that the following expression 11 is satisfied. That is, in step S10, the estimated value (Ui (0) + εi (0), Vi (0) + δi (0)) at t = 0 is obtained.
Then, the process proceeds to step S11.

[0054]

[Equation 11]

[0055]

[Equation 12]

In step S11, the estimated value (Ui (t) + εi at each time t (t = 1 to T-1) other than t = 0 is obtained.
(t), Vi (t) + δi (t)) is calculated. That is, step S
Kij, R0 (t), R1 (t), R2 (t) obtained up to 9 and the estimated value (Ui (0) + ε) at t = 0 obtained in step S10.
i (0), Vi (0) + δi (0)) is used to obtain (εi (t), δi (t)) that minimizes the following equation 14 under the condition that the following equation 13 is satisfied. That is, in step S11, the estimated values (Ui (t) + εi (t), Vi (t) + δi (t)) at t = 1 to T-1 are obtained.

[0057]

[Equation 13]

[0058]

[Equation 14]

In this way, Kij, R0 (t), R1 (t), R2 (t) and (Ui (t) obtained by step S11 are obtained.
+ Εi (t), Vi (t) + δi (t)) satisfies Expression 15.

[0060]

[Equation 15]

Incidentally, steps S4, S6, S8 and S1
The optimal solutions obtained in 0 and S11 are Lagrange's Method of Indeterminate Mu.
ltiplier).

After completion of step S11, the process proceeds to step S12. In step S12, (Ui (t) + εi (t), Vi (t)
+ Δi (t)) is output and this series of processing ends.

In the above-described embodiment of the present invention, the case where a rectangular object is photographed and the four corners thereof are tracked has been described in detail. However, the present invention is limited to the case of four rectangular vertices. Needless to say, it can be widely and generally used regardless of its shape.

The present invention is implemented by, for example, the apparatus shown in FIG. In FIG. 2, the arithmetic unit 10 includes an arithmetic processing circuit 11, a program memory 12 in which a processing program of the arithmetic processing circuit 11 is stored, and processing data (original moving image and projection position information on the image of feature points on the image). ) For storing data, a display device 14 for displaying data to a user, an input device 15 such as a mouse and a keyboard, and a bus 16 for connecting each circuit and transmitting a program or data. ing.

Using the image data stored in the data memory 13 and the data input by the input device 15, the arithmetic processing circuit 11 performs a series of processes shown in the flowchart of FIG.

The program memory 12 stores a program for executing each procedure of the arithmetic method as described above. In addition, the CD on which this program is recorded
It is also possible to provide a recording medium such as an optical disk such as a ROM, a magnetic disk, a magnetic tape, and a semiconductor memory.

Next, as an arithmetic method according to the present invention, the following examples will be further cited. The example shown below is a second embodiment in which the above-described first embodiment of the present invention is further improved.

For example, consider the case shown in FIG.
FIG. 3 shows an image at time t in the original moving image obtained by photographing a rectangular object. The rectangular projection image (20 in the figure) is hidden by the projection image (23) of another object and is not entirely projected. Since the projected images of the two corners are shown in the image, they can be automatically tracked. That is, (U1 (t), V1 (t)), (U2 (t),
V2 (t)) is determined (points 21 and 22 in the figure). on the other hand,
The remaining two corners are hidden and invisible by the projected image (23) of another object. However, the projection image of at least one hidden vertex from the direction of the edge of the rectangle is
It can be said that it is on the straight line 24 in the figure. And it can be said that the other projected image of the hidden vertex is on the straight line 25 in the figure.

Since a method of tracking an edge is conventionally known, it is possible to track an edge using that method. Therefore, even if the position of a certain characteristic point cannot be known, it is possible to obtain information that "a certain characteristic point is somewhere on a specific straight line". Therefore, in the second embodiment of the present invention, a more accurate solution can be obtained by also using this information as input data.

The second embodiment of the present invention as described above will be described below. In the first embodiment, the error of each observed value is Ci (t) in both the U-axis direction and the V-axis direction.
Was even with the weight of. This is variable in the second embodiment.

The projected image of each feature point i at each time t (U
Data of i (t), Vi (t)) is input, but this input data is divided into three cases. That is: Case C1: The value determined as the projected image position of the feature point is input. -Case C2: The projected image position of the feature point is unknown and is not input. That is, it is input as indefinite. -Case C3: The projected image position of the feature point is Vi (t) = tan (θ
i (t)) Ui (t) + αi (t) Only the information that it is on the straight line is input. It is divided into three cases.

In order to distinguish C1, C2 and C3,
Introduce the variable Fi (t). Fi (t) to represent C1
= 1, Fi (t) = 0 to represent C2, and Fi (t) = 2 to represent C3.

The following expression 16 is introduced as an error function instead of the above expression 8 which is an error function, and the following conditional expression 17 is introduced instead of the above expression 9.

[0074]

[Equation 16]

[0075]

[Equation 17]

Here, for example, when Fi (t) = 1, C
0i (t) = 1, C1i (t) = 1, θi (t) = arbitrary value (for example, 0). When Fi (t) = 0, C0i (t) = 0.00
01, C1i (t) = 0.0001, θi (t) = arbitrary value (for example, 0). When Fi (t) = 2, C0i (t) = 0.
0001, C1i (t) = 1, θi (t) = “input θi (t)”
Is. Further, εi (t) and δi (t) in the formula represent the error of the input measurement value, but unlike the first embodiment, they form an angle θi (t) with the U-axis direction. The direction error is εi (t),
The error in the direction orthogonal to that is defined as Δi (t).

Explaining the case of Fi (t) = 2, C
Since 0i (t) = 0.0001 and C1i (t) = 1, as can be seen from Equation 16, the error εi (t) in the direction forming the angle θi (t) with the U-axis direction is the error in the direction orthogonal thereto. 10 than δi (t)
This means that the error is set to include an error of about 000 times. This is exactly the same as C3 in the above case (the position of the characteristic point is on the straight line of Vi (t) = tan (θi (t)) Ui (t) + αi (t)).

When Fi (t) = 1 and Fi (t) = 0, the above equations 16 and 17 have the same values as the above equations 8 and 9 described in the first embodiment, and the description thereof will be omitted. To do.

As described above, in the second embodiment, since the variable that minimizes the above equation 16 is obtained under the condition that the above equation 17 is satisfied, a specific direction (U axis direction and angle θ) is obtained.
The variable that minimizes the measurement error can be obtained under the condition that the error may be large in the direction of forming i (t).

The above processing is summarized in the flowchart of FIG. Each process of the flowchart of FIG. 4 is basically the same as that of FIG.

First, in step S21 of FIG. 4, data is input. That is, Fi (t) that indicates whether or not the i-th feature point was confirmed as an observation value at time t is input. Then, when Fi (t) = 1, the observed values (Ui (t), Vi (t)) are also input. When Fi (t) = 2, variables θi (t) and αi (t) representing straight lines are input. Furthermore, at all i and t, the "Probability of observed value C
0i (t), C1i (t) ”is also input. The values described above may be input for C0i (t) and C1i (t). Of course, if the data is more ambiguous than others, C0i (t), C1i (t)
Is smaller than the above-mentioned value, and if more certain, C
0i (t) and C1i (t) may be set to values larger than the above-mentioned values.

In step S22, similar to step S2 in FIG. 1, appropriate values are set to Kij, R0 (t), R1 (t), R2.
Set to (t). For example, Kij = 0.25, R0 (t) = 0,
R1 (t) = 0 and R2 (t) = 0. When the process of step S22 ends, the process proceeds to step S23.

In steps S23 and S24, the same processes as steps S3 and S4 in FIG. 1 are performed. That is, in step S23, the input (U0
(0), V0 (0)), (U1 (0), V1 (0)), (U2 (0), V2
(0)), (U3 (0), V3 (0)) and the current Kij, R0 (t), R
From 1 (t) and R2 (t), A0j (t), A1 defined in the above equation 10
j (t), A2j (t), A3j (t) are obtained, and the process proceeds to step S24. In step S24, A0j obtained in step S23
Using (t), A1j (t), A2j (t), and A3j (t), "Fi (t) = 0 or 2 (Ui (t ), Vi (t)) ”. Step S2
After the process of 4, the process proceeds to step S25.

In step S25, what is not performed in the first embodiment is performed. Of the results obtained in step S24, Fi (t) = 2 (Ui (t), Vi (t)) is
It is not always on the straight line Vi (t) = tan (θi (t)) Ui (t) + αi (t). Therefore, "(Fi (t) = 2 (Ui (t), Vi (t))" obtained in step S24 is
(t), Vi (t)) The straight line closest to the position Vi (t) = tan
(θi (t)) Ui (t) + αi (t) The point is moved to a new point (Ui (t), Vi (t)). This is the process of step S25. After the process of step S25, the process proceeds to step S26.

Steps S26, S27, S28, S29
Then, the same processing as steps S5, S6, S7, and S8 in FIG. 1 is performed. The only difference is that the above equation 8 is not minimized under the condition of the above equation 9, but the variables are calculated so that the above equation 16 is minimized under the condition of the above equation 17,
The description is omitted. After the process of step S29, the process proceeds to step S30.

The series of processing from step S23 to step S29 is repeatedly executed. Whether the number of times of repeated processing (loop count) exceeds a predetermined number or whether the error is within a predetermined range. Is determined in step S30, and if true, the process proceeds to step S31. If false, the process proceeds to step S23, and the series of processes from step S23 to step S29 is performed again.

When the process proceeds to step S31, Kij, R0 (t), R1 (t), R2 (t) with sufficient accuracy and "Fi (t) = 0 (Ui (t ), Vi (t)) ”,“ Fi
That is, (Ui (t), Vi (t)) where (t) = 2. At this point, I want to finally obtain "Fi (t)
= 0 (Ui (t), Vi (t)) "and" Fi (t) = 2 (Ui (t), Vi (t)) "are required, but they could be observed. Error correction is also performed including (Ui (t), Vi (t)) (that is, (Ui (t), Vi (t)) where Fi (t) = 1). That is, in step S31, the calculated Kij, R0
(t), R1 (t), R2 (t) and (Ui (t), Vi (t)),
Further, the estimated value (Ui (t), Vi (t)) (Ui (t) + εi (t),
Vi (t) + δi (t)) is calculated. This method is the same as steps S10 and S11, and a description thereof will be omitted.

After the processing of step S31 is completed, the process proceeds to step S32. In step S32, (Ui (t) + εi
(t), Vi (t) + δi (t)) is output, and this series of processing is completed.

The second embodiment of the present invention described above can also be applied to the arithmetic unit of FIG. 2 as described above. That is, the program for executing each procedure of the flowchart shown in FIG. 4 may be stored in the program memory 12 of the arithmetic unit shown in FIG. Further, it is possible to provide a recording medium such as an optical disk such as a CD-ROM in which this program is recorded, a magnetic disk, a magnetic tape, or a semiconductor memory.

As described above, according to the present invention, even if a featured point cannot be tracked due to the obscuration of an object or the like while the captured moving image is being tracked (trajectory tracking), it can be automatically tracked. It becomes possible to track.

Note that the present invention is not limited to the above-described embodiments. For example, in the above-described embodiments, an example in which a rectangular object is photographed and the four corners are tracked has been described. However, the present invention is not limited to the case of four vertices of a rectangle, and it goes without saying that any shape can be used. Besides, various modifications can be made without departing from the scope of the present invention.

[0092]

As is apparent from the above description, according to the present invention, the first position information, which is the information on the projected position of the feature points of the first group of the object photographed in the plurality of images, is input. , The second which is information of the projection positions of the remaining feature points of the second group.
When determining the position information of the first group of feature points, a first parameter associated with the weight when represented by a weighted average of the positions of the first group of feature points, the projected position of the feature points of the first group, and a specific parameter The unknown variable and the characteristic points of the first group and the second group are defined with the time and the second parameter related to the position, orientation, and focal length of the imaging device that captured the image at the time of interest as the unknown variable. From the relational expression with the projected image position on the image, the second position information and the unknown variable are obtained,
By outputting the obtained second position information,
Even if the feature point cannot be tracked because it is hidden behind the object, it can be automatically tracked (trajectory tracking).

[Brief description of drawings]

FIG. 1 is a flowchart for explaining an operation of a first embodiment of a calculation method according to the present invention.

FIG. 2 is a block circuit diagram showing a schematic configuration example of an embodiment of an image processing apparatus according to the present invention.

FIG. 3 is a diagram showing an example of an image for explaining a second embodiment of a calculation method according to the present invention.

FIG. 4 is a flow chart for explaining the operation of the second embodiment of the arithmetic method according to the present invention.

[Explanation of symbols]

10 arithmetic unit, 11 arithmetic processing circuit, 12 program memory, 13 data memory, 14 display device, 15 input device, 16 bus

Claims (8)

[Claims] 1. Inputting first position information, which is information of projection positions of feature points of a first group of an object captured in a plurality of images, and inputting information of projection positions of remaining feature points of a second group. In a calculation method for obtaining a certain second position information, a first parameter associated with the weight when represented by a weighted average of the positions of the feature points of the first group, and the projection of the feature points of the first group. The position and the specific time, and the second parameter related to the position, the orientation, and the focal length of the image capturing device that captured the image at the time of interest are set as unknown variables,
A calculation step of obtaining the second position information and the unknown variable from a relational expression of the unknown variable and the projected image positions of the feature points of the first group and the second group on the image, A calculation method characterized in that the second position information obtained in the calculation step is output. 2. The error evaluation under the condition that in the calculation step, the relational expression between the unknown variable and the projected image position of the first and second feature points on the image is satisfied in consideration of the error. The second position information and the unknown variable are calculated so as to minimize the function, and moreover, the error concerning the second position information is weighted more than the error concerning the first position information. The calculation method according to claim 1, wherein the calculation is made lighter and evaluated within the error evaluation function. 3. The first position information, which is the information on the projected positions of the characteristic points of the first group of the objects photographed in the plurality of images, is input, and the information on the projected positions of the remaining characteristic points of the second group is input. In an arithmetic device for obtaining a certain second position information, a first parameter related to the weight when expressed by a weighted average of the positions of the feature points of the first group, and the projection of the feature points of the first group. The position and the specific time, and the second parameter related to the position, the orientation, and the focal length of the image capturing device that captured the image at the time of interest are set as unknown variables,
A calculation circuit for obtaining the second position information and the unknown variable from a relational expression between the unknown variable and the projected image positions of the feature points of the first group and the second group on the image, An arithmetic unit that outputs the second position information obtained by a calculation circuit. 4. The error evaluation under the condition that the calculation circuit satisfies the relational expression between the unknown variable and the projected image position of the first and second feature points on the image in consideration of an error. The second position information and the unknown variable are calculated so as to minimize the function, and further, the first position information is calculated.
4. The arithmetic unit according to claim 3, wherein the error relating to the second position information is evaluated with a lighter weight than the error relating to the position information, and evaluated in the error evaluation function. 5. The first position information, which is the projection position information of the feature points of the first group of the object photographed in the plurality of images, is input, and the projection position information of the remaining feature points of the second group is input. In a program for obtaining a certain second position information by an arithmetic device, a first parameter related to the weight when expressed by a weighted average of the positions of the feature points of the first group, and the first group As the unknown variables, the projection position of the feature point and the specific time and the second parameter related to the position, orientation, and focal length of the image capturing device that captured the image at the time of interest,
A calculation step for obtaining the second position information and the unknown variable from the relational expression between the unknown variable and the projected image positions of the feature points of the first group and the second group on the image is provided to the arithmetic unit. A program characterized by being executed. 6. The error evaluation under the condition that in the calculation step, the relational expression between the unknown variable and the projected image position of the first and second feature points on the image is satisfied in consideration of the error. The second position information and the unknown variable are calculated so as to minimize the function, and moreover, the error concerning the second position information is weighted more than the error concerning the first position information. The program according to claim 5, wherein the program is lightened and evaluated within the error evaluation function. 7. The first position information, which is the information on the projected positions of the characteristic points of the first group of the objects photographed in the plurality of images, is input, and the information on the projected positions of the remaining characteristic points of the second group is input. In a computer-readable recording medium in which a program for obtaining a certain second position information by a computing device is recorded, a second number related to the weight when the weighted average of the positions of the characteristic points of the first group is expressed. The first parameter and the projection position of the feature point of the first group and the second parameter related to the position, orientation, and focal length of the image capturing device that captured the image at the specific time and the time of interest are unknown. As a variable
A calculation step for obtaining the second position information and the unknown variable from the relational expression between the unknown variable and the projected image positions of the feature points of the first group and the second group on the image is provided to the arithmetic unit. A computer-readable recording medium characterized by being executed. 8. The error evaluation under the condition that, in the calculation step, the relational expression between the unknown variable and the projected image position of the first and second feature points on the image is satisfied in consideration of the error. The second position information and the unknown variable are calculated so as to minimize the function, and moreover, the error concerning the second position information is weighted more than the error concerning the first position information. The recording medium according to claim 7, wherein the recording medium is lightened and evaluated within the error evaluation function.