JP2000113194A - Method and device for extracting three-dimensional shape and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly accurately extract the three-dimensional(3D) shape of an object even when a defective value occurs in a tracking matrix composed of moving image or continuous still picture streams. SOLUTION: Shaping processing 502 of track matrix, construction 506 of estimate matrix and estimating operation of defective value are repeated while increasing an estimate matrix size little by little (512). When the construction of estimate matrix is disabled, the track matrix is vertically inverted (520) and similar defective value estimating processing is repeated. In the shaping processing, feature points are grouped corresponding to the tracking conditions of feature points and the feature points of the track matrix are rearranged in prescribed group order.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for extracting and restoring a three-dimensional shape of a target object from an image sequence such as a moving image obtained by photographing the target object by a digital video camera or the like.

[0002]

2. Description of the Related Art A technique for estimating a three-dimensional shape of a target object from a sequence of moving images and still images captured by digital media such as a digital video camera and a digital still camera is an important subject in the field of computer vision research. one of. In addition, there is great interest in application fields such as robot vision, autonomous vehicles, three-dimensional shape input using a video camera, image coding, and three-dimensional modeling.

In the problem of extracting three-dimensional information from a time-series two-dimensional moving image, a so-called “structure estimation from motion (St.
ructure From Motion) ”
According to the procedure of (n) → distance (Depth) → shape (Shape), it is common to estimate the shape by first calculating the motion of the camera and then calculating the distance from the camera center of the feature point on the object. is there. However, in the time-series moving image, since the movement of the corresponding point between each frame is small, it is almost impossible to specify the movement by the parallel movement or the rotation movement. Eventually, the solution of the obtained depth (Depth) becomes impossible, and the reconstruction of the shape information does not work well. Conversely, if the time-series sampling interval is set large, that is, if the movement of the corresponding point between each frame is large, the reliability of the association of the feature points will be reduced.

As a method for stabilizing a solution by calculating motion and shape simultaneously without calculating depth, Tomasi
And Kanade's proposed factorization
n) "(C. Tomasi and T. Kanade," Shape and mot
ion from image streamunder orthography: A factoriz
ation method '', International Journal of Computer V
ision, vol. 9, 1992, pp. 137-154). This method performs a linear formulation based on an orthographic model and uses a singular value decomposition of a matrix that is numerically stable.
The feature is that the solution is extremely stable.

Furthermore, a pseudo center projection closer to the center projection which is an actual camera model while maintaining the linearity of the formulation
(paraperspective)
proposed by lman and Kanade (CJPoelman and TK
anade, `` A paraperspectivefactorization method for
shape and motion recovery '', IEEE Transactionon Pa
ttern Analysis and Machine Intelligence, vol.19, no.
3, pp. 206-218).

Here, the pseudo center projection model and a factor decomposition method using the model will be described.

First, a pseudo center projection model will be described.
The model uses a central projection scaling effect (making nearby objects appear larger than distant objects) and a position effect (objects at the edges of the image appear at different angles than objects near the center of projection). ) While maintaining the linearity of the orthographic model. The projection of the object onto the image plane by the pseudo center projection model is performed by the following steps. (1) A plane (virtual plane) that passes through the center of gravity of the object and is parallel to the image plane is defined. (2) A point on the object is projected onto a virtual plane along the direction of a straight line connecting the center of the camera and the center of gravity of the object. (3) A point on the virtual plane is projected on the image plane by central projection.

This operation will be specifically described with reference to FIG. In FIG. 1, 1 is the camera center, 2 is the camera center 1
This is an image plane that is separated from the image by a focal length. C is the center of gravity (the center of gravity of the object) of a set of feature points of the object captured by the camera (a part of which is represented by the ■ mark). An imaginary plane 3 passes through the center of gravity C and is parallel to the image plane 2. Taken the origin of the world coordinate system to the center of gravity C, and the 3-dimensional coordinates in the world coordinate system of the feature point p and s _p ∈R ^3.

[0009] Regarding an image frame f in a time-series image,
And the coordinates of the camera center 1 in the world coordinate system are represented by t_f, Painting
Let the base vector of the two-dimensional local coordinate system of the image plane 2 be i_f,
j_f∈R^Three (However, ‖i_f‖ = ‖J_f‖ = 1, i_f× j_f=
0), the optical axis direction of the camera is k_f= I_f× j_f∈R^ThreeAnd
In image frame f, image plane 2 and vector k_fof
Intersection O_fAt the origin, a set of unit orthogonal vectors (i_f, J
_f), The two-dimensional local coordinate system Σ_f= (O_fI_f, J
_f).

In the pseudo center projection, as described above, the projection of the feature point p on the image plane 2 is performed in the following two steps. In a first step, a feature point p is projected onto a virtual plane 3,
This projection is performed in parallel with a straight line from the camera center 1 to the center of gravity C. Then, in a second step, the point projected on the virtual plane 3 is centrally projected on the image plane 2. Image plane 2 of feature point p
Let (u _fp , v _fp ) be the coordinates of the projection point on Σ _f = (O _f ; _if , j _f ). However, the focal length of the camera is 1
And This (u _fp , v _fp ) is expressed as follows.

[0011]

(Equation 1) However,

[0012]

(Equation 2) Here, z _f distance from the camera center 1 to the virtual plane 3, which is projected to (x _p, y _p) is the image plane 2 by central projection of the object centroid C (the origin of the world coordinate system). Also,
(U _fp , V _fp ) are coordinates of a projection point when the feature point p is centrally projected on the image plane 2 and is represented by the following equation.

[0013]

(Equation 3) By Taylor-expanding this (U _fp , V _fp ) around z _f , the pseudo-central projection model calculates the central projection

[0014]

(Equation 4) It can be seen that this is an approximation (first-order approximation) under the assumption of.

Next, the factor decomposition method will be described. In the factorization method, P feature points are tracked over F image frames, and as a result, the image plane 投影
_{f = (O f; i f} , j f) 2 -dimensional local coordinate on _{(u fp, v fp),} f = 1,2, ..., F; p = 1,2, ..., P 2F × P matrix W (“tracking matrix”)
) Is defined.

[0016]

(Equation 5) The upper half of this tracking matrix represents the x coordinate value _ufp of the feature point,
The lower half represents the y coordinate value _vfp of the feature point. Each column of the tracking matrix corresponds to the tracking result of one feature point, and each row corresponds to the x coordinate value or y coordinate value of all the feature points in a single frame.

Next, the average value y _f of the average value x _f and y coordinate values of the x-coordinate values of all feature points in each frame

[0018]

(Equation 6) Ask for. Then, a matrix W ^* (referred to as a “measurement matrix”) is created by subtracting x _f and y _f from each element of the tracking matrix.

[0019]

(Equation 7) This measurement matrix can be decomposed as follows since the number of floors is at most 3 even if the number of frames F and the number of feature points P are increased.

[0020]

(Equation 8) When this is compared with the above equation (1), the matrix R of 2F × 3 is obtained by the camera posture vector ｛(m _f , n _f ): f = 1,
, F}, and the 3 × P matrix S is a matrix in which the position vectors {s _p : p = 1, 2,.

In general, since the measurement matrix contains noise, the rank of the matrix is not always 3; however, even in that case, only three large singular values are maintained by using singular value decomposition. The decomposition yields an optimal decomposition in the sense of minimizing the square error. Similarly, for the pseudo-central projection model,
The measurement matrix can be decomposed into a camera attitude matrix and a feature point shape matrix. The decomposition method for such a measurement matrix is called a “factor decomposition method”.

The basic algorithm for factorization of the measurement matrix will be described. For the factorization of the measurement matrix, a technique of singular value decomposition of the matrix is used. That is, using singular value decomposition, the measurement matrix can be decomposed into the product of three matrices as follows.

[0023]

(Equation 9) Here, U is a 2F × P orthogonal matrix, Σ is a P × P diagonal matrix composed of singular values (σ ₁ , σ ₂ ,..., Σ _P ) of the measurement matrix, and V
Is a P × P orthogonal matrix. If the rank of the measurement matrix is 3
Then, the singular values after σ ₄ become small values close to 0. Here, when the measurement matrix is decomposed assuming that σ _{4 and} subsequent are 0,

[0024]

(Equation 10) Becomes Therefore,

[0025]

[Equation 11] So, one decomposition

[0026]

(Equation 12) Is obtained. However, the decomposition of equation (12) is not unique. In fact, by any regular matrix Q,

[0027]

(Equation 13) Like, there are countless solutions. Therefore, the following constraint condition is introduced, and Q satisfying the constraint condition is obtained.

[0028]

[Equation 14]

[0029]

(Equation 15)

[0030]

(Equation 16) And using this Q

[0031]

[Equation 17] If so,

[0032]

(Equation 18)Can be uniquely decomposed into R is a 2F × 3 matrix and the camera
Show momentum. S is a 3 × P matrix and the three-dimensional coordinates of each feature point
Is shown. Details are given in the above-mentioned Poelman and Kanade reference
Is the matrix R, ie, ｛(m_f, n_f): F = 1,2, ..., F｝
And (x) calculated by equation (6)_f, Y_f) And from the camera
Direction ｛(i_f, j_f, k_f): Finding f = 1,2, ..., F｝
Can be. Next, from equation (14), z_fFrom equation (2)
Camera position t_f Can be calculated.

[0033]

Regardless of whether an orthographic projection model or a pseudo-central projection model is used, the factorization method assumes that all selected feature points can be tracked over a series of image frames. And That is, it is assumed that the feature points that are visible in the first image frame are hidden in the middle and cannot be seen, or that new feature points are not introduced in the middle image frame. However, in a situation where the camera goes around the object, some of the feature points seen in the first image frame are occluded by the object on the way. This is not the case with such moving images. As a result, a significant number of missing values are created in the tracking matrix, and it is necessary to supplement missing values by estimation.

The factorization is not unique, and there are two solutions. That is, irregularities on the surface of the target object cannot be distinguished.

Accordingly, a general object of the present invention is to provide a high-precision extraction of a three-dimensional shape of a target object even when a missing value occurs in a tracking matrix created from a moving image or a series of still images. Is to make it possible. One of the more specific objects of the present invention is to enable efficient estimation of missing values in the tracking matrix, and another object is to reduce the three-dimensional shape of the target object to the surface irregularities. It is to be able to extract accurately including it. The above and other objects will be apparent from the following description.

[0036]

According to the first to fourth aspects of the present invention, an improved three-dimensional shape extracting method is provided. This three-dimensional shape extraction method includes a step of creating a tracking matrix by tracking a feature point of a target object using an image sequence obtained by photographing the target object as input data. Estimating using an estimation matrix, and replacing missing values with the estimated values,
Forming a measurement matrix from the tracking matrix that has completed the missing value estimation step, and converting the measurement matrix into three-dimensional data representing the shape and motion of the target object, respectively. It is characterized in that estimation of missing values is repeated while gradually increasing the size of the estimation matrix. The feature of the invention according to claim 2 is that, in the missing value estimation step, the feature points are classified into a plurality of groups according to the tracking situation with reference to a tracking matrix,
This is to perform a shaping process for rearranging the feature points in a predetermined group order on the tracking matrix. A feature of the invention according to claim 3 is that, in the missing value estimating step, when the missing values that cannot be estimated remain in the tracking matrix but the estimation matrix cannot be constructed, the tracking matrix is inverted. Then, the estimation of the missing value is continued. A feature of the invention according to claim 4 is that, while rotating the target object in a predetermined direction, the moving direction of the feature points of the target object represented by the three-dimensional data converted from the measurement matrix, and the object being photographed The object of the present invention is to add a step of comparing the rotation direction of the object with the three-dimensional data and performing an operation of inverting the concavities and convexities of the feature points that are not compared with each other.

According to the fifth, sixth or seventh aspect of the present invention,
An improved three-dimensional shape extraction device is provided. This three-dimensional shape extraction method includes means for inputting an image sequence obtained by photographing a target object, means for generating a tracking matrix by tracking feature points of the target object in the input image sequence, and features by referring to the tracking matrix. The points are classified into a plurality of groups according to their tracking status, and a shaping process for rearranging the feature points of the tracking matrix in a predetermined group order, and an estimation matrix for missing value estimation from the tracking matrix after the shaping process are constructed. Means for repeating the process and the process of estimating missing values from the constructed estimation matrix and replacing the missing values with the estimated values while gradually increasing the size of the estimation matrix; And a means for converting the measurement matrix into three-dimensional data representing the shape and the motion of the target object, respectively. A feature of the invention according to claim 6 is that, in the missing value estimating means, when a missing value that cannot be estimated remains in the tracking matrix but the estimation matrix cannot be constructed, the tracking matrix is inverted. After that, the shaping process, the process of constructing the estimation matrix, and the process of estimating the missing value are repeated while gradually increasing the size of the estimation matrix. A feature of the invention according to claim 7 is that a moving direction of a feature point of the target object represented by three-dimensional data converted from a measurement matrix is obtained by rotating the target object in a predetermined direction, This is to add a means for comparing the rotation direction of the object with the three-dimensional data and performing an operation for inverting the concavities and convexities of the feature point in which the comparison is inconsistent.

[0038]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, as one embodiment of the present invention, a method for extracting a three-dimensional shape of an object with high accuracy from a moving image or a series of still images obtained by photographing the object with a camera by a factorization method will be described. A three-dimensional shape extraction system will be described. Note that the above-described pseudo center projection model is assumed as a camera model.

FIG. 2 is a configuration diagram of an example of the three-dimensional shape extraction system. In FIG. 2, reference numeral 100 denotes a turntable on which a target object 102 is placed. Turntable 10
A plurality of marks M are provided at predetermined intervals on the upper surface of the outer peripheral portion of the zero. Reference numeral 104 denotes a rotation drive mechanism of the turntable 100, which is driven by the drive unit 106. Reference numeral 108 denotes a digital video camera, which rotates the turntable 100 in a predetermined direction at a predetermined speed, and outputs an object 1 on the turntable 100.
02 is continuously photographed and an image stream is output. During shooting, the mark M on the turntable 100 is
, The positional relationship between the turntable 100 and the camera 108 is determined.

The processing system for extracting the three-dimensional shape of the target object based on the image stream output from the digital video camera 108 includes an image input unit 110 and a two-dimensional operation unit 1
12, and a three-dimensional operation unit 114, which operates in cooperation with the personal computer 116. Digital video camera 10
The image stream output from 8 is input to the image input unit 110. The image input unit 110 creates a set of still images from the image stream and writes it into the memory of the personal computer 116. The two-dimensional operation unit 112 includes the personal computer 11
A measurement matrix for factorization is constructed based on the set of still images in the memory of No. 6 and written in the memory of the personal computer 116. The three-dimensional calculation unit 114 extracts a three-dimensional shape of the target object from the measurement matrix. The extracted three-dimensional shape data is written to the memory of the personal computer 116. Preferably, in the personal computer 116, 3
The dimensional shape is restored and displayed on the display screen, and the result of the shape extraction can be confirmed.

The driving unit 106 outputs rotation direction data indicating the rotation direction of the turntable 100. The rotation direction data is input to the three-dimensional operation unit 114. When the rotation direction of the turntable 100 is used for feature point tracking in the two-dimensional calculation unit 112, the rotation direction data is also input to the two-dimensional calculation unit 112. The personal computer 116 provides a memory for storing the intermediate data of the processing and the final result data as described above, and also controls the operation of the driving unit 106 and the three-dimensional shape extraction processing system (110, 112, 114). All or a part of the image input unit 110, the two-dimensional operation unit 112, and the three-dimensional operation unit 114 may be realized by software on hardware of the personal computer 116.

The processing contents of the image input unit 110 will be described.
FIG. 3 is a flowchart showing the processing flow. The image input unit 110 captures an image stream output from the digital video camera 108 (Step 30).
0). Each frame of the captured image stream is converted into an appropriate still image format as needed (step 302), and the obtained time series of still image frames is written to the memory of the personal computer 116 (step 304).

The processing contents of the two-dimensional operation unit 112 will be described. FIG. 4 is a flowchart showing the processing flow. As shown in the figure, the two-dimensional calculation unit 112 first takes in an image frame from the memory of the personal computer 116 and extracts a small image region where the density changes sharply as a feature point of the target object (step 400). At this time, the feature points are preferably ranked according to the intensity of the density change. In other words, a feature point with a sharp density change is assigned a higher rank.

Next, the two-dimensional calculation unit 112 tracks the feature points by associating the feature points of the frames immediately before and after (step 402). When the feature points are ranked in the previous step, tracking is performed by selecting a certain number of feature points by priority from feature points having a higher rank. When tracking a feature point, pattern matching is performed between the feature point on the current frame and the feature point on the next frame, and the feature points determined to match the pattern are associated with each other. If there are many, there is a possibility that the association of the feature points, that is, the tracking is erroneous.

In order to reduce such tracking errors, preferably, the rotation direction of the turntable 100 indicated by the rotation direction data input from the driving unit 106 is used as a limiting condition for feature point tracking. If the moving direction of the associated feature point is inconsistent, the tracking result is likely to be incorrect, and the tracking result is discarded.
If another appropriate feature point corresponding to the feature point is not found in the next frame, the position coordinate becomes a missing value. In order to further reduce tracking errors, it is preferable to use the pattern of the mark M provided on the outer peripheral portion of the turntable 100. That is, the position coordinates of the feature point are set to (x, y) (where x is the horizontal direction and y is the vertical direction), and in the pattern of the mark M (referred to as a mark pattern) in the current frame and the next frame. The mark pattern having the y coordinate closest to the y coordinate of the associated feature point is selected as the corresponding mark pattern. Then, the direction of the motion vector of the associated feature point is opposite to the direction of the motion vector of the corresponding mark pattern, or the difference between the magnitude of the motion vector of the feature point and the magnitude of the motion vector of the corresponding mark pattern is predetermined. If the threshold value is exceeded, it is determined that tracking has failed, and the tracking result is discarded.

Next, based on the tracking result of the previous step, the two-dimensional operation unit 112 calculates the 2F × P
(Step 404). The data of this tracking matrix is written into the memory of the personal computer 116.
The data obtained in steps 400 and 402 are also temporarily stored in the memory of the personal computer 116.

As described above, an element in which the position coordinates of a feature point is missing (missing value) in the tracking matrix due to concealment of the feature point by the target object itself, or fluctuation of illumination at the time of photographing or tracking failure. Is generally included. 2
Next, the dimension calculation unit 112 estimates the missing value of such a feature point (step 406). Then, a measurement matrix shown in the above equation (7) is created based on the tracking matrix supplemented with the missing values (step 408). The measurement matrix is stored in the memory of the personal computer 116.

Next, the missing value estimation processing in step 406 will be described. Prior to the description of the specific processing content, the basic concept of missing value estimation will be described.

FIG. 5 is an example of the result of tracking feature points 1 to 7 over frames 1 to 8. In the figure, the mark “•” indicates that the tracking of the feature point was successful and the position coordinates have been obtained, and the mark “?” Indicates that the tracking of the feature point failed and the position coordinates are missing values. To indicate that If such a missing value remains in the measurement matrix, it cannot be used for the factorization method as it is, so it is necessary to estimate the missing value and complement it. Estimation of a missing value uses known position coordinates around the missing value. First, one missing value to be estimated is selected. Then, a sub-matrix (called an estimation matrix) of the tracking matrix including the selected missing values is created. In this estimation matrix,
All elements other than the element to be estimated need to be known (position coordinates have been obtained).

For example, in FIG. 5, it is assumed that the missing value of the frame 5 of the feature point 6 is selected as an estimation target. In this case, for example, an estimation matrix corresponding to the inside of the frame shown in FIG. 5 is created. Missing values are estimated from the estimation matrix, and the missing values in the tracking matrix are rewritten with the estimated values. Next, for example, a missing value of the frame 6 of the feature point 6 in FIG. 5 is selected as an estimation target, and an estimation matrix for the estimation is created. For example, an estimation matrix corresponding to feature points 1 to 6 and frames 1 to 6 is created, the missing value is estimated, and the missing value in the tracking matrix is rewritten with the estimated value. Next, for example, feature point 7 in FIG.
Is selected as an estimation target, for example, an estimation matrix corresponding to feature points 1 to 7 and frames 2 to 6 is created, and the missing value is estimated. In this way, missing values are sequentially estimated one by one, and finally all missing values in the tracking matrix are complemented.

Here, a method called “row expansion” and a method called “column expansion” will be described as a method of estimating missing values from the estimation matrix. Consider an estimation matrix as shown in the following equation. Here, u _fp and v _fp are missing values of the estimation target.

[0052]

[Equation 19] First, the row expansion method will be described. A sub-matrix is created by removing the rows of missing values of the estimation matrix of the equation (19). And
This submatrix is decomposed by the factor decomposition method as follows.

[0053]

(Equation 20) M _2mx3 is a matrix representing the attitude of the camera in 2 m frames, and S _{3x (n + 1)} is a matrix representing the three-dimensional coordinates of (n + 1) feature points. The (n +) th frame in the (m + 1) th frame
1) It can be seen that the feature points (u _fp , v _fp ) satisfy the following equation set.

[0054]

(Equation 21) Since equations (21) and (22) have four unknowns (u _fp , m _f ^T ) and (v _fp , n _f ^T ), respectively, the number of known feature points is determined in order to solve these unknowns. (= N + 1) is at least 4, that is, n is 3 or more. But,
Considering the influence of noise and the like, it is appropriate to set the number of known feature points to 4 or more, in other words, to solve the equations (21) and (22) with an over-constrained least squares solution. .

First, the equation (21) is expanded as follows.

[0056]

(Equation 22) Except for (22.n + 1), arrange as follows.

[0057]

(Equation 23) M _f is obtained from equation (23). And (22.n + 1)
Substitute in the expression,

[0058]

(Equation 24) The coordinate value u _fp is obtained as follows. _Similarly , _ufp is determined.

[0059]

(Equation 25) Next, the column extension method will be described. Contrary to the line expansion method,
A sub-matrix is created by removing the columns of missing value locations from the estimation matrix of equation (19). This submatrix is decomposed by the factor decomposition method as follows.

[0060]

(Equation 26) M2 _{(m + 1) x3} is a matrix representing the attitude of the camera in 2 (m + 1) frames, and _S'3xn is a matrix representing the three-dimensional coordinates of n feature points. It can be seen that the (n + 1) -th feature point (u _fp , v _fp ) in the (m + 1) -th frame satisfies the following two equations.

[0061]

[Equation 27] Equation (26) holds for up to n feature points, but (2)
Equations 7) and (28) hold for up to (n + 1) feature points. That is, for S ₁ '+ S ₂ ' + ... + S _n '= 0,
S ₁ + S ₂ +... + S _n + S _{n + 1} = 0.
Here, if S _{n + 1} = −nC,

[0062]

[Equation 28] It is. That is,

[0063]

(Equation 29) Note that (27) and (28) are the x _f and y _f must be modified as follows.

[0064]

[Equation 30] Substituting equations (30) and (31) into equations (27) and (28)

[0065]

(Equation 31) Is obtained. From equation (32), S _p , and u _fp ,
v _fp is required.

The larger the size of the estimation matrix used for the estimation is, the higher the estimation accuracy is. However, on the other hand, the calculation cost for estimating one missing value increases. Therefore, in the present invention, as described above, an estimation matrix of an appropriate size is created, a missing value is estimated, and when the estimation fails, the size of the estimation matrix is gradually increased to a certain size. , Repeat the estimation. In order to smoothly proceed with the process of repeating the estimation operation while increasing the size of the estimation matrix little by little, in the present invention, "shaping process" of the tracking matrix is performed in the process of the estimation process. Further, when the missing values remain and the estimation matrix cannot be constructed any more, a process of inverting the tracking matrix upside down is performed, and the estimation process is continued.

As described above, according to the present invention, by estimating missing values while gradually increasing the size of the estimation matrix, it is possible to suppress an increase in the calculation cost of missing value estimation. By performing the tracking matrix shaping process and further inverting the tracking matrix, such missing value estimation can be performed smoothly, and more missing values can be accurately and efficiently estimated as compared with the related art. Therefore, even when there are many missing values in the tracking matrix, accurate three-dimensional shape extraction becomes possible.

Here, "shaping process" and "vertical inversion" of the tracking matrix will be described. FIG. 6 shows a tracking matrix for an image stream. In the figure, a shaded portion indicates that the position coordinates of the feature point exist, and a blank portion indicates that the position coordinates of the feature point do not exist (becomes a missing value). When observing this tracking matrix, it is noticed that feature points can be grouped according to the tracking situation.

According to the present invention, for example, according to the time (frame) in which feature points are continuously present (tracked),
The feature points are divided into the following four groups. Group A: exists continuously from the first frame to the last frame (successfully tracked without any ray)
Feature points, B group: feature points that exist in the first frame but disappear from the middle frame (tracking failed halfway), C group: feature points that appear in the middle frame but disappear after a while, D Group: A feature point that appears in an intermediate frame and exists continuously up to the last frame. Then, the columns corresponding to the feature points of group A to group D of the tracking matrix are rearranged in the order shown in FIG. This is the shaping process. In FIG. 7, the shaded portion is an element having position coordinates (measured values), and the blank portion is an element having no position coordinates (missing value).

The missing value is estimated by using the tracking matrix shaped as described above. Since the estimation is not always successful at one time, if the estimation is not successful, the estimation is repeated while gradually increasing the size of the estimation matrix to a predetermined maximum size. If the missing values are successfully estimated,
The missing value is rewritten with the estimated value, treated as the position coordinates existing from the beginning, and the tracking matrix is reshaped.

In summary, the procedures of the shaping process of the tracking matrix, the construction of the estimation matrix, and the estimation operation are repeated while gradually increasing the size of the estimation matrix until the stop condition is satisfied. The stop condition is, specifically, that all missing values have been estimated, or that missing values that cannot be estimated remain, but no further estimation matrix can be constructed. And
If the missing values remain even after such an estimation procedure, the tracking matrix is inverted upside down, and then the same estimation procedure is repeated.

FIG. 8 shows an example of the flow of the missing value estimating process in the two-dimensional operation unit 112 described above with reference to FIG. First, a tracking matrix is fetched from the memory of the personal computer 116 (step 500). The above-mentioned shaping process is performed on this tracking matrix (step 502). It is checked whether it is possible to construct an estimation matrix for one or more missing values in the shaped tracking matrix (step 504). If possible, create an estimation matrix for the missing values of interest (step 50).
6), an estimation calculation is performed (step 508), and the success or failure of the estimation is checked (step 510). If successful, replace the missing value with the estimated value. In the case of failure, the size of the estimation matrix is increased (step 512), and it is checked whether the size of the estimation matrix after the increase is equal to or smaller than a predetermined maximum size (step 514). If the size is equal to or smaller than the maximum size, an estimation matrix having the increased size is created and the estimation operation is performed again (step 508). If the estimation is successful or the size of the estimation matrix exceeds the maximum size, it is checked whether the estimation of all missing values has been completed (step 516). If there are no missing values, the process ends. Tracking matrix is PC1
16 memories.

If the estimated value remains, step 5
Returning to 02, the processing for the next missing value is continued.

If it is impossible to construct the estimation matrix although the missing values remain, the process proceeds from step 504 to step 51.
Proceeding to step 8, it is determined whether the tracking matrix has been flipped upside down. If not, the tracking matrix is turned upside down (step 520), and the process returns to step 502 to continue. If the missing values remain even after the tracking matrix is turned upside down and the estimation matrix cannot be constructed, the process proceeds from step 504 to step 518, and further to step 522, where the tracking matrix is turned upside down again. Return to the original line order and end the process. The tracking matrix is stored in the memory of the personal computer 116.

Here, the construction of the estimation matrix will be further described. It is preferable that the estimation matrix be constructed by collecting measurement values (position coordinates) of feature points that sufficiently satisfy the assumptions of the linear approximation projection model, in order to increase estimation accuracy. That is,
As described above, the pseudo-central projection model is a model that is developed around the distance from the optical center of the camera to the center of gravity of the target object in the central projection model and takes a first-order approximation,
The following two approximate conditions are assumed.

[0076]

(Equation 32)

[0077]

[Equation 33] (34) represents the assumption that is sufficiently small size of the object as compared to the shooting distance z _f, the assumption that (33) is very small projection of the optical axis of each feature point cameras respectively ing. If each feature point is on the virtual plane of the model, it can be said that the pseudo central projection matches the central projection. Therefore, equation (33) suggests the construction condition of the estimation matrix. However, since the virtual plane is always parallel to the image plane of the camera, the posture of the camera in each frame is different except when the camera movement is a parallel movement. If the camera moves heavily, it is difficult to satisfy the assumption of equation (33).
However, in the case of a moving image captured by a video camera, since the camera pose conversion between consecutive frames is small, it may be considered that a set of locally concentrated feature points satisfies the assumption of Expression (33). Therefore, when constructing an estimation matrix from the tracking matrix, first, a feature point closest to a feature point that has failed in tracking and is a missing value (missing feature point) is selected to construct an estimation matrix. However, there is a problem that the position of the missing feature point, that is, the missing value of the estimation target is unknown. Therefore, it is assumed that the motion of the feature point is a parallel motion between adjacent frames, and the measurement value (called "assumed measurement value") in the frame immediately before the missing feature point is used instead of the missing value. I do. That is, a measurement value having the smallest distance from the assumed measurement value is selected to construct an estimation matrix. This method is suitable for a moving image in which the motion of a feature point between frames is small.

Next, the processing contents of the three-dimensional operation unit 114 will be described. FIG. 9 is a flowchart showing the processing flow.

First, the three-dimensional operation unit 114
The measurement matrix created by the two-dimensional operation unit 112 is fetched from the 16 memories (step 600). Next, the above-described factorization method is applied to this measurement matrix (step 602), and the three-dimensional coordinates (shape data) 611 of the feature points of the target object and the relative three-dimensional motion of the feature points and the camera are calculated. (Exercise data) 610 is obtained.

The factor decomposition is not unique, and there are two types of solutions, and there is a problem that it is not possible to distinguish between irregularities in the shape. Therefore, in the present invention, the irregularities of the shape are determined and the necessary irregularities are corrected (step 604), and the shape data 606 with accurate irregularities is obtained.

FIG. 10 shows the contents of step 604. That is, three-dimensional (movement / shape) data 610, 611
And the rotation direction data 612 input from the drive unit 106
Are checked to see if the rotation direction of the turntable 100 matches the motion direction of the feature point on the target object (step 702).
The shape data relating to the feature points having the same motion direction is output as it is as it is (step 706). However, the shape data of the feature points whose motion directions do not match is inverted with respect to the z-axis, that is, the unevenness inversion operation is performed
(Step 704), the inverted data is output as accurate shape data (Step 706).

As described above, according to the present invention, when the direction of rotation of the target object being photographed and the direction of motion of the feature point represented by the three-dimensional data obtained from the measurement matrix do not match, the unevenness is reversed. By performing the operation, it is possible to accurately extract the shape including the irregularities on the surface of the target object.

The functions of the image input unit 110, the two-dimensional operation unit 112 and the three-dimensional operation unit 114 shown in FIG. 2, or the processing contents described in connection with FIGS. 3, 4, 8, 9 and 10 Can be realized by software using the personal computer 116 or another personal computer or another computer. For example, a CPU 80 as schematically shown in FIG.
0, a memory 801, an auxiliary storage device 802 such as a hard disk, a drive 803 for reading and writing a recording medium 804 such as a floppy disk, an input interface 805 and an output interface 8 with an external input device
06, etc., connected by a system bus 807 can be used. A program 810 for executing the processing contents described in relation to FIGS. 3, 4, 8, 9 and 10 is placed on the memory 801 and executed by the CPU 800. The program 810 is loaded into the memory 801 via the drive 803 from a recording medium 804 such as a floppy disk on which the program 810 is recorded, or is stored in the auxiliary storage device 802.
And is loaded into the memory 801 when the process is executed. The image stream output from the camera 108 and the rotation direction data output from the driving unit 106 are captured via the input interface 805, for example. The final shape data is stored in, for example, the auxiliary storage device 802 or written out to a floppy disk or the like via the drive 803.

[0084]

As is clear from the above description, according to the present invention, the missing value is estimated while the size of the estimation matrix is gradually increased, thereby increasing the calculation cost of missing value estimation. In addition, the tracking matrix shaping process and the tracking matrix are vertically inverted so that such missing value estimation is performed smoothly, and more missing values are accurately estimated more efficiently than before. be able to. Therefore,
Accurate 3 even when there are many missing values in the tracking matrix
A dimensional shape can be extracted. In addition, when the rotation direction of the target object being photographed and the motion direction of the feature point represented by the three-dimensional data obtained from the measurement matrix do not match, the unevenness of the surface of the target object is inverted. Can also be accurately extracted.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram of a pseudo center projection model.

FIG. 2 is a configuration diagram of a three-dimensional shape extraction system according to the present invention.

FIG. 3 is a flowchart illustrating a processing flow of an image input unit.

FIG. 4 is a flowchart illustrating a processing flow of a two-dimensional operation unit.

FIG. 5 is an explanatory diagram of a method of estimating a missing value of a tracking matrix.

FIG. 6 is a diagram illustrating an example of a tracking matrix.

FIG. 7 is an explanatory diagram of a tracking matrix sorting process.

FIG. 8 is a flowchart illustrating an example of a missing value estimation processing flow.

FIG. 9 is a flowchart illustrating a processing flow of a three-dimensional operation unit.

FIG. 10 is a flowchart illustrating a shape unevenness determination / estimation process.

FIG. 11 is a block diagram illustrating an example of a computer used to implement the present invention with software.

[Explanation of symbols]

 Reference Signs List 100 turntable 102 target object 104 rotation drive mechanism 106 drive unit 108 digital video camera 110 image input unit 112 two-dimensional operation unit 114 three-dimensional operation unit 116 personal computer

Continuation of the front page F term (reference) 2F065 AA04 AA53 BB05 FF01 FF04 JJ03 JJ19 JJ26 KK01 MM04 PP13 QQ17 QQ18 QQ23 QQ31 QQ38 QQ45 5B057 CA08 CA13 CA16 CB08 CB13 CB16 CD06 CD14 CH08 DA20 DB02 FA09 DC22 DC09 LA04 LA05 MA00

Claims (10)

[Claims] 1. A step of creating a tracking matrix by tracking a feature point of a target object using an image sequence obtained by photographing the target object as input data, and estimating a missing value of the tracking matrix as a submatrix of the tracking matrix. Estimating using a matrix, replacing missing values with estimated values, creating a measurement matrix from a tracking matrix that has completed the missing value estimation step,
And converting the measurement matrix into three-dimensional data respectively representing the shape and the motion of the target object. In the missing value estimation step, repeating the estimation of the missing value while gradually increasing the size of the estimation matrix is performed. A featured three-dimensional shape extraction method. 2. In the missing value estimation step, a shaping process of classifying feature points into a plurality of groups according to the tracking situation with reference to the tracking matrix and rearranging the feature points in the tracking matrix in a predetermined group order is performed. The method according to claim 1, wherein the method is performed. 3. In the missing value estimating step, when there is a missing value that cannot be estimated in the tracking matrix but it becomes impossible to construct the estimation matrix, the tracking matrix is inverted upside down to calculate the missing value. 3. The method according to claim 2, wherein the estimation is continued.
Dimensional shape extraction method. 4. A target object is photographed while being rotated in a predetermined direction, and a motion direction of a feature point of the target object represented by three-dimensional data converted from a measurement matrix, and a rotation direction of the target object being photographed. 4. The three-dimensional shape extraction method according to claim 1, further comprising the step of comparing the three-dimensional data with each other, and performing an operation for inverting concavities and convexities of the feature points having the mismatch. . 5. A means for inputting an image sequence obtained by capturing an image of a target object, a means for generating a tracking matrix by tracking feature points of the target object in the input image sequence, and extracting feature points by referring to the tracking matrix. Classifying into a plurality of groups according to the tracking situation, a shaping process of rearranging the feature points of the tracking matrix in a predetermined group order, and a process of constructing an estimation matrix for missing value estimation from the tracking matrix after the shaping process, Means for estimating a missing value from the constructed estimation matrix and replacing the missing value with the estimated value while gradually increasing the size of the estimation matrix, a tracking matrix after processing by the missing value estimating means. 3. A three-dimensional shape extracting apparatus, comprising: a means for constructing a measurement matrix from R and a means for converting the measurement matrix into three-dimensional data representing the shape and motion of the target object, respectively. 6. The missing value estimating means, when missing values that cannot be estimated remain in the tracking matrix but the estimation matrix cannot be constructed, the tracking matrix is turned upside down, and then the shaping process is performed. 6. The three-dimensional shape extracting apparatus according to claim 5, wherein the process of constructing the estimation matrix and the process of estimating the missing value are repeated while gradually increasing the size of the estimation matrix. 7. A target object is photographed while being rotated in a predetermined direction, and a moving direction of a feature point of the target object represented by three-dimensional data converted from a measurement matrix, and a rotation direction of the target object being photographed. 7. The three-dimensional shape extracting apparatus according to claim 5, further comprising: means for comparing the three-dimensional data with each other, and performing an operation of inverting the concavities and convexities for the feature points having the mismatch. 8. A step of creating a tracking matrix by tracking feature points of a target object in an image sequence obtained by capturing the target object, and classifying the feature points into a plurality of groups according to the tracking status with reference to the tracking matrix. A shaping process of rearranging the feature points of the tracking matrix in a predetermined group order, a process of constructing an estimation matrix for missing value estimation from the tracking matrix after the shaping process, and a process of forming missing values from the constructed estimation matrix. Repeating the process of estimating and replacing missing values with estimated values while gradually increasing the size of the estimation matrix, constructing a measurement matrix from the tracking matrix that has completed the missing value estimation step, and measuring A computer-readable program recorded with a program for causing a computer to execute a step of converting a matrix into three-dimensional data respectively representing a shape and a motion of a target object. Readable recording medium. 9. In the missing value estimating step, if there is a missing value that cannot be estimated in the tracking matrix but it becomes impossible to construct the estimation matrix, the tracking matrix is turned upside down, and then the shaping process is performed. 9. The computer-readable recording medium according to claim 8, wherein the processing of constructing an estimation matrix and the processing of estimating missing values are repeated while gradually increasing the size of the estimation matrix. 10. A target object is photographed while being rotated in a predetermined direction. The program includes: a moving direction of a feature point of the target object represented by three-dimensional data converted from a measurement matrix;
Comparing the rotation direction of the target object being photographed with the rotation direction of the target object, and performing an operation of inverting the unevenness on the three-dimensional data for the feature point in which the comparison is inconsistent with the three-dimensional data. The computer-readable recording medium according to claim 8 or 9, wherein: