JPH10255053A - Method for associating subject characteristic point in 3d reconstruction and 3d reconstruction method using it - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for associating a subject characteristic point which easily solves the correspondence problem of a characteristic point without needing a special device that measures view point movement nor special skill such as image processing and an object 3D reconstruction method using it. SOLUTION: In a method for associating subject characteristic points between 2D images when a subject is undergone 3D reconstruction through plural 2D images which are taken by continuously moving a view point, the view point is moved so that the distance between the view point and the subject may be larger than the distance between an initial coordinate and a coordinate after movement, and in this case, one optional point of characteristic points is positioned on the image origins of a 2D image of the initial coordinate and a 2D image of the coordinate after movement. The distance between a 2D position of the characteristic point of one optional point of each 2D image and the 2D image origin is introduced and the movement locus of the characteristic point of one optional point is made a coefficient only for a three-dimensional translational vector H. The movement of the characteristic point of one optional point is traced by using other 2D images, the movement locus in this case is contrasted with the movement locus, and the one that has the higher correlativity is selected.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for associating feature points of a subject between 2D images when the subject is 3D-reconstructed using a continuous 2D image taken while the viewpoint is continuously moved, and a method thereof. More specifically, the present invention relates to a 3D restoration method used, and more specifically, a point (feature point) representing a feature on the shape of a subject when a 3D moving image is restored using a continuous image such as a video image.
And a method for appropriately and automatically associating each 2D image and a method for restoring a 3D moving image using the method.
According to the associating method and the 3D restoration method of the present invention, modeling and rendering operations when creating a virtual reality video are simplified.

[0002]

2. Description of the Related Art In recent years, a method of using a plurality of 2D images having parallax with respect to a subject and restoring the shape of the subject in 3D based on the principle of triangulation (hereinafter, referred to as a "stereo method") has attracted attention. . In such a stereo method, it is necessary to indicate corresponding feature points in two target 2D images, and such correspondence of feature points (hereinafter, referred to as “solution of correspondence problem”) is performed. It is a difficult matter.

[0003] For this reason, conventionally, the operator solves the problem by visually observing and comparing two 2D images.
The stereo corresponding point was indicated. Recently, a method of solving a correspondence problem using a spatio-temporal image in which continuous images are densely arranged on a time axis has been proposed.

[0004]

However, in such a conventional method, in the case of a visual method,
Not only is the burden on the operator heavy, and it is difficult to indicate many feature points, but also the 2D images have poor correspondence with other 2D images and are unsuitable for 3D reconstruction. Therefore, knowledge and skill in image processing are required to determine the authenticity of a feature point, and there is a problem that even an amateur or an experienced person cannot easily perform the determination. Also,
If you want to perform precise 3D reconstruction from two 2D images,
Although it is necessary to increase the parallax as much as possible, there is also a problem that as the parallax increases, the possibility of specifying a false feature point increases.

On the other hand, in the method using a spatiotemporal image, a special device for measuring the viewpoint movement is required since the movement of the viewpoint is known, and there is a problem that the cost is increased. Further, a device for measuring the viewpoint motion outdoors is extremely large-scale, and such a measuring device cannot be obtained depending on the surrounding environment of the subject. Therefore, the subject to which the spatiotemporal imaging method can be applied is mainly limited to indoor objects. There was a problem that it could be.

The present invention has been made in view of the above-mentioned problems of the prior art, and an object of the present invention is to eliminate the need for a special device for measuring a viewpoint movement or special skills such as image processing. Another object of the present invention is to provide a method of associating subject feature points, which can easily solve the feature point correspondence problem, and a 3D restoration method of a subject using the method.

[0007]

Means for Solving the Problems As a result of intensive studies to achieve the above object, the present inventors have imposed a specific constraint on each 2D image, and further focused on a specific amount, thereby achieving the above-mentioned measures. The inventors have found that the problem can be easily solved, and have completed the present invention.

That is, the method for associating feature points according to the present invention is as follows.
When 3D reconstruction of the three-dimensional shape of the subject is performed using a plurality of 2D images taken while continuously moving the viewpoint, the plurality of 2D images are used.
A method for associating a feature point of the subject between D images, wherein (A) a distance between the viewpoint and the subject is sufficiently larger than a distance between initial coordinates of the viewpoint and coordinates after movement. Then, the viewpoints are moved, and at this time, any one of the feature points is located at the image origin of the 2D image based on the initial coordinates and the 2D image based on the coordinates after the movement. Introducing the distance between the 2D position of the arbitrary one feature point and the origin of the 2D image in the above, the motion trajectory of the arbitrary one feature point accompanying the movement of the viewpoint is set to 3
(C) The above arbitrary 1
2 except that the motion of the feature points of the point was used in step (A).
Tracking using the D image, comparing the obtained motion trajectory with the motion trajectory obtained in step (B), adopting a highly correlated motion trajectory as a true feature point motion trajectory, and using this Thus, another feature point in each 2D image is associated.

The 3D reconstruction method of the present invention is used for 3D reconstruction of an object using a plurality of 2D images photographed while continuously moving the viewpoint. The method is applied to complete the association of the feature points between the 2D images. Then, at least one pair of 2D images to be subjected to 3D restoration is selected from the plurality of 2D images, and the corresponding 2D images are selected. The viewpoint coordinates are set as initial coordinates and post-movement coordinates, respectively. (2) The three-dimensional translation vector indicating the movement of the viewpoint by introducing the distance between the 2D position of an arbitrary one feature point and the origin of the 2D image at each of the coordinates. H
And (3) after that, using the association obtained in the step (1) and the three-dimensional translation vector obtained in the step (2), obtaining a 3D image by the at least one pair of the 2D images. Features.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail by embodiments with reference to the drawings. FIG. 1 shows an example in which the viewpoint (camera) is arbitrarily and continuously moved so that a continuous
FIG. 3 is a perspective view showing a state in which a D image is being shot. In addition,
The 3D restoration is performed by focusing on the feature points of the subject. As such feature points, generally, an end point, a branch point, an intersection, and the like on the outer shape of the subject are extracted. In addition, the precision of the obtained 3D image changes depending on the method of extracting the feature points, which will be described in detail later.

As described above, the present invention mainly has a gaze point restriction (step A) in a minute movement section of a viewpoint and a study of a correlation between feature points (step C). Note that the minute movement section of the viewpoint in step A does not necessarily need to start from the first image (still state) of the continuous 2D image, and it is sufficient to extract a specific section of the entire viewpoint movement. .

[Steps A and B] In the present invention, as shown in FIG. 1, it is necessary that the subject 2 is always captured on the image plane of the camera 1 at an arbitrary camera position. It is necessary to control (constrain the gazing point) such that an arbitrary specific feature point (gazing point) of the subject 2 is located at the image origin of the 2D image being moved by the camera. The method can be easily performed by using a method of rotating the coordinate system in a pseudo manner (standard conversion).

FIG. 2 shows a moving relationship of 3D coordinates in a fixed section of continuous camera shooting. In the figure, the initial coordinates are X ^{k + 1} −Y ^{k + 1} −Z ^{k + 1} , and the coordinates after movement are X ^{k + M} −Y
^{k + M−} Z ^{k + M} , and the camera position is the origin O ^{k + 1} of each coordinate.
And O ^{k + M} , and each Z axis coincides with the optical axis of the camera. Also, the origin of the 2D image coordinates based on each 3D coordinate is located at a focal distance f from the origin of each 3D coordinate.
The 2D image coordinates based on the initial coordinates are represented by x ^{k + 1} −y ^{k + 1} , and the X axis and Y axis of each 3D coordinate are parallel to the x axis and y axis of the corresponding 2D image coordinate. .

The movement between the camera positions O ^{k + 1} -O ^{k + M} is defined by the combination of a three-dimensional translation vector H and a rotation matrix R, and the rotation matrix R has X, Y, and Z axes as shown in the figure. Rotation angle θ _X (tilted angle), θ _Y (pan angle), θ
_{Expressed by Z} (roll angle). Note that the number of all feature points of the subject is N, and the i-th feature point, which is an arbitrary one of them, is indicated by a symbol i.

Here, it is assumed that the total number of 2D images is L, and the number of 2D images photographed by slightly moving the camera is M from the (k + 1) th image to the (k + M) th image. (Where k is a positive integer and represents 0, 1,..., LM, i is a natural number, and is 1,..., N.).

The (k + 1) -th image and the (k + m) -th image (m is a natural number, and 1,
2, ..., M are shown. ), In FIG.
The (k + 1) -th image corresponds to the initial coordinates, since the first k + m-th image corresponding to the moved coordinate, when the 3D position of the feature point i in the moving coordinates after the P _i ^{k + m,} the P _i ^{k + m} following It is expressed by an equation. ^{_{P i k + m = (X}} i k + m, Y i k + m, Z i k + m) T = R T (P i k + 1 -H) ··· (1) where, in the formula, R ^T represents the transpose of R, where R ^T =
R _Z ^T is ^{_{R Y T R X T, R}} Z, R Y, each R _{X X} ^{k +} 1,
This is a rotation matrix around the Y ^{k + 1} and Z ^{k + 1} axes.

Here, if the above R _X is expressed as a component, it can be expressed as the following equation 1.

[0018]

(Equation 1)

Further, 2D image coordinates corresponding to the 3D position P _i ^{k + m} of the feature point ^{_{(x i k + m, y}} i k + m) , when displayed using the projective transformation, the following equations . ^{_{x i k + m = f ·}} X i k + m / Z i k + m, y i k + m = f · Y i k + m / Z i k + m ··· (2)

Next, in order to obtain the motion trajectory of the feature point, a gazing point is considered as the 3D position P _i ^{k + m} of the feature point, and this gazing point is P ₁ ^{k + m.} It becomes like equation 3).

[0021]

(Equation 2)

Here, when the equation (3) is multiplied by R _Z from the left, the following equation (4) is obtained. ^{_{R Z P 1 k + m =}} P 1 k + m = R Y T R X T (P 1 k + 1 -H) ··· (4)

Then, the following gazing point constraint equation (5) is obtained by solving equation (4). tan θ _X = H _Y / (Z ₁ ^{k + 1} −H _Z ) (5) tan θ _Y = −H _X / ｛(Z ₁ ^{k + 1} −H _Z ) ² −H _Y ² ｝ ^1/2

Further, when R _XY ^T is obtained from the equation (5), the following is obtained.
Equation (6) is obtained.

[0025]

(Equation 3)

(Step B) As described above, the roll angle (θ _Z ) cannot be obtained only by restricting the gazing point. Therefore, an amount independent of the roll angle is introduced. Feature point P _i
^{k + m} of the 2D image coordinates ^{_{(x i k + m, y}} i k + m) the distance from the image origin (r _i ^{k + m)} does not depend on the roll angle, by using this, (2 ), (3), (5) and (6), the following equation (7) is derived.

[0027] ^{_{(r i k + m / f}} ) 2 = {(α i k + m) 2 + (β i k + m) 2} / (γ i k + m) 2 ··· (7) However, α _i ^{k + m} = (X _i ^{k + 1} −H _X ) + tan θ _X tan θ _{Y
^{(Y i k + 1 -H Y}} ) -tanθ Y (Z i k + 1 -H Z) β i k + m = (Y i k + 1 -H Y) + tanθ X (Z i k + 1 -H Z ^{_{) γ i k + m = (}} Z i k + 1 -H Z) + tanθ Y (X i k + 1 -H X)
−tan θ _X (Y _i ^{k + 1} −H _Y )

Here, since M is sufficiently smaller than the above-described minute movement, the following equation is established. Z ₁ ^{k + m} ≒ Z ₁ ^{k + 1} (m = 2, 3,..., M) In this case, the movement of the camera depends on the distance to the subject (~
Z _i ^{k + 1} ), so that H _X / Z ₁ ^{k + 1} << 1,
Relationships such as H _Y / Z ₁ ^{k + 1} << 1 and H _Z / Z ₁ ^{k + 1} << ₁ are established. Further, from the condition that the subject is always captured on the image plane of the camera, X ^{k + 1} / Z ₁ ^{k + 1} << 1, Y ^{k + 1} / Z ₁ ^{k + 1} << 1
Such a relationship also holds. By performing an approximate calculation using these relationships, the following equation (8) is obtained.

[0029] ^{_{(r i k + m / f}} ) 2 = (r i k + 1 / f) 2 + 2f (1-Z 1 k + 1 / Z i k + 1) (x i k + 1 · H X ^{_{/ Z 1 k + 1 + y}} i k + 1 · H Y / Z 1 k + 1) + 2 (r i k + 1) 2 · Z 1 k + 1 / Z i k + 1 · H Z / Z 1 k ⁺¹・ ・ ・ (8)

On the other hand, as described above, the camera moves so that the subject always enters the screen. However, in a range where the movement of the camera is small and linear motion is considered, the three-dimensional translation vector H has a radius Z ₁ ^{k. +1} can be approximated by a circular motion with a declination δ. Δ takes a discrete value corresponding to the 2D image number m. If this relation is expressed as δ = δ (m), the following equation (9) holds. The approximation by the above-mentioned circular motion is
The moving distance of the camera is 1/10 of the distance between the subject and the camera
It can be preferably applied in the following cases.

[0031]

(Equation 4)

In equation (9), “′” indicates the differentiation at the argument δ. The following equation (10) was obtained by substituting the equation (9) into the equation (8). As a result, the motion trajectory of an arbitrary feature point i could be made a function of only the vector H. ^{_{(R i k + m / f}} ) 2 = (r i k + 1 / f) 2 + a i k + 1 δ (m) ··· (10) ^{_{where, a i k + 1 = 2f}} (1-Z 1 ^{k + 1} / Z _i ^{k + 1} ) (x _i ^{k + 1} · H _X ′
^{_{(0) / Z 1 k +}} 1 + y i k + 1 · H Y '(0) / Z 1 k + 1) +2
(R _i ^{k + 1} ) ² · Z ₁ ^{k + 1} / Z _i ^{k + 1} · H _Z ′ (0) / Z ₁ ^{k + 1}

Here, the motion trajectory of a feature point appropriately used for 3D reconstruction (hereinafter, referred to as a “true feature point”), that is, a true feature point among individual target feature points is Equation (10) is followed for a 2D image. On the other hand, a false feature point caused by occlusion (obstruction by a foreign substance) does not follow the equation (10), so that the difference can be used to determine the true / false of the target feature point. It can be determined whether the solution of the corresponding problem is successful or unsuccessful in at least two 2D images.

In equation (10), r _i ^{k + 1} , a _i ^{k + 1}
It has i dependence as, but leaves affected by the selected feature points, in order to remove such i dependency, an average value with respect to the (10) equation, the r _i ^{k + m} value m 0.0 , And u _i ^{k + m} is the value converted so that the standard deviation is 1.0, (10)
The equation is expressed as the following equation (11).

[0035]

(Equation 5)

In equation (11), it can be seen that the right side does not depend on the feature point number i. This means that all the true feature points satisfy Expression (11). Therefore, the true feature points have a high correlation, and the true / false determination of the feature point and the success / failure determination of the corresponding problem by the above equation (10) are affirmed.

[Step C] Hereinafter, using the motion trajectory of the true feature point calculated using the M images from the (k + 1) th image to the (k + m) th image by the above-described camera minute movement,
Solve the correspondence problem in the remaining 2D images. here,
And k = 0 for simplicity, in the same manner as described above with reference to first to M th image, motion trajectory r _i ^m of the true feature points (see (10) Formula)
Suppose you asked for The 2D image used for 3D restoration is
, M, M + 1,..., L.

[0038] In the motion trajectory r _i ^m, an average value for m 0.0, when the converted value for the standard deviation 1.0 and u _i ^m, as described above, the motion locus of the true feature points are Since the mutual correlation is high, it is possible to determine whether the feature point in the first image is true or false using this correlation, and at the same time, from the first image to the Mth image for the true feature point. Will be solved.

Next, in the above-mentioned first to M-th images, the same processing as that for obtaining the characteristic point trajectory is performed by the remaining 2D image.
Image, i.e., until the (L-M) th to the L th image, performed while shifting the image start number by one, determines the correlation by using the u _i ^m. As a result, the authenticity of the feature points from the first image to the LM-th image can be determined, and as a result, the problem of correspondence between the first image and the L-th image can be solved. In the C step, the first to L-th images may be taken with a large camera movement.

[Extraction of Feature Points] Next, extraction of the feature points described above will be described in detail. In performing 3D restoration from a plurality of 2D continuous images, what point is selected (extracted) as a feature point is important in improving the accuracy of the obtained 3D image. The extraction of such feature points can be performed by focusing on the directional variance value, that is, a scale indicating the complexity of the texture information, and specifically, by applying a Moravec operator. Although it is possible, first, the extraction of the feature points in the first 2D image will be described, and then the extraction of the feature points in the second and subsequent 2D images will be described.

(Extraction of Feature Points in First Image) First, as preprocessing, contrast enhancement and smoothing of a 2D image are performed. Such contrast enhancement and the like can be performed using a generalized histogram conversion or a median filter. Then, from the pre-processed 2D image,
Boundaries (hereinafter, referred to as “edges”) on the shape of the subject, for example, end points, branch points, and intersections are extracted and used as feature points. For example, by using a Sopel operator, High quality edge information can be obtained.

Next, binarization processing of the obtained edge is performed by, for example, a differential histogram method, and further, the edge is thinned and disconnection recovery processing is performed. Thereafter, noise is removed by removing fine particles of a specified size or less, for example, one pixel or less.

FIG. 3 shows an example of thinned edge information. In the figure, the numbers entered in the squares are
It indicates the number of pixels in contact with the edge among the eight pixels existing near the edge. In particular, an edge whose number in the square is 3 or 4 has a large directional variance value and is suitable as a feature point. As described above, the thinned edge has an end point, a branch point, an intersection, and the like. In the present invention, first, the branch point and the like are used as feature points, and further, the branch point and the like are deleted. , Segment the edges and label them.

Next, the following processing is performed on the segmented and labeled edges in the order of label numbers. That is, as shown in FIG. 4, a straight line connecting the start point and the end point of the edge of a certain label number is obtained, and the distance B from the straight line is obtained for all the pixels that come into contact with the edge of the label number. At this time, if the maximum value _{Bmax of the} distance is a threshold value, for example, 5 pixels or more, a pixel corresponding to this maximum value is set as a feature point. Then, the feature point extracted as described above is set as a new end point, the same processing as above is recursively repeated, another feature point is extracted, and the extraction of the feature point in the first image is completed.

(Tracking of Feature Points in Second and Subsequent Images) Since the feature points extracted as described above include fake feature points, it is necessary to improve the accuracy of the obtained 3D reconstruction. It is preferable to delete such false feature points to some extent. Such false feature points can be deleted by template matching performed on the second and subsequent images. Hereinafter, template matching will be described.

This template matching focuses on a fixed area (template) near the feature points extracted as described above, traces this template in a plurality of images, and calculates the minimum value of the directional variance value in the template. ,
This is performed by setting the value of the feature point. Then, threshold processing is performed on the obtained minimum value of the directional variance value, and a feature point having a small directional variance value at an end point is deleted, and the remaining feature points are extracted as feature points. Thus representing the extracted position of the i-th feature point in the feature point among the first image as "r _i ^1" (where, i = 1,2, ···, a N, N is the direction This shows the number of feature points that have been subjected to threshold processing using a variance value.)

FIG. 5 shows an example of template matching. In the figure, reference numeral 3 denotes a k-th image of a continuous 2D image, reference numeral 4 denotes a coarse cursor (21 × 21 pixel window) representing a part of the k-th image, and reference numeral 5 denotes a cursor.
Is a fine cursor (11 × 11 pixel window) representing a part of the (k−1) th image, and the fine cursor 5 corresponds to the template.

In the template matching as shown in FIG. 5, a feature point position is estimated with an accuracy of one pixel in the coarse cursor 4. In this case, the fine cursor 5
The feature point position is calculated by using the differential image in. In FIG. 5, the feature point position is at the center of the fine cursor 5, but the approximate position of the feature point in the k-th image is the position (fine cursor 5) calculated in the (k-1) th image which is the previous image. (The center of the cursor).

In such a state, the fine cursor 5 and the course cursor 4 are made to correspond to each other, and the similarity shown in the following Expression 6 within the course cursor 4, that is, the similarity in the texture information between the two cursors is minimized. The position is calculated and set as the position of the feature point in the k-th image.

[0050]

(Equation 6)

In the above equation, F represents a template,
C indicates a differential image. Further, sigma indicates the total sum in the fine cursor 5. Note that this equation is SSDA
This is effective for speeding up processing by the method.

In the template matching, for example, as described above, all feature points extracted from the first
This is done by tracking over the image. Specifically, as shown in FIG. 6, the position of the i-th feature point in the k-th 2D image is predicted by the course cursor size, and the following relationship holds.

R ′ _i ^k = r _i ^k−1 (i = 1, 2,... N),
(K = 1, 2,..., L)

[0054] However, r _'i ^k is the predicted position of the i-th course cursor in the k-th image, r _i ^k-1 is the first (k-1)
The position of the i-th feature point in the n-th image, N indicates the number of all feature points, and L indicates the number of images to be tracked. Also, r ′ _i ^k indicates the center of the course cursor and is the position of the calculated feature point. It is highly possible that the course cursor contains the i-th feature point.

If the correlation between the feature points is poor, the above tracking may not be performed. Also,
A feature point whose direction variance is not so different from that of the vicinity, such as a middle part of an edge, may be extracted (referred to as an “aperture problem”). It may be significantly different from the motion locus of the feature point.

[Clustering] Next, another method for examining the correlation between feature points described in step C will be described. As described above, the motion trajectory of the true feature point can be calculated by the predetermined method of the present invention, but the motion trajectory of the false feature point cannot be calculated. Therefore, the motion trajectories of the true feature points have a high correlation with each other, and the motion trajectories of the false feature points have a low correlation with any other motion trajectories. Therefore, by clustering the motion trajectories of the feature points as follows, false feature points can be eliminated, and the resolution accuracy of the correspondence problem can be further improved.

First, the gazing point constraint described in step A and the introduction of the distance from the image origin of the feature point 2D image coordinates described in step B are executed. indicating the position of the i-th feature point (r _i ^k)
Ask for ² . Next, a cross-correlation value ρ _ij is obtained according to the following definition equation (Equation 7).

[0058]

(Equation 7)

[0059] In the above equation, u _i ^k denotes a material obtained by normalizing the (r _i ^{k) 2,} the cross-correlation value [rho _ij between u _i ^k and u _j ^k of [-1,1] by definition Take a value. Next, in the calculated cross-correlation value ρ _ij matrix, a dendrogram process for collectively illustrating components having the same identity is performed, and the classification based on the correlation of each feature point is examined (see FIG. 7).

In this dendrogram processing, a cross-correlation value ρ _ij is used as an index of similarity. Most ideally, ρ
_{Although ij} is 1.0, clusters are formed as the threshold is lowered from 1.0. At this time, cluster integration is performed by performing calculation (d _max ) shown in the following Expression 8 between all clusters (C _m , C _n ) and integrating the cluster pair having the maximum value.

[0061]

(Equation 8)

By observing the degree of cluster growth while performing cluster integration as described above, it is possible to consider a solution to the correspondence problem. That is, the motion trajectories of the feature points for which the correspondence problem has been successfully solved are similar, but the motion trajectories of the failed feature points are different. In other words, a feature point that has been successfully associated is a large cluster early, but a failed feature point is a small cluster.

As described above, when the threshold is gradually increased from 0.5 to 1.0, the clusters are decomposed, and the true feature points are divided into a large cluster group and a small cluster group. Since they are separated, if a large cluster group is selected, the correspondence problem can be solved. If the threshold value of the cross-correlation value ρ _ij is set to about 0.9, a reliable solution to the problem can be achieved.

Next, the 3D reconstruction method of the subject according to the present invention will be described. The 3D restoration method of the present invention is a 3D restoration method using the above-described association method. In this 3D restoration method, since the association is completed, the 3D shape of the subject may be known. It is a prerequisite.

In the following, a specific description will be given. First, the association method described above is applied to complete the association of the feature points between the images of the continuous 2D images, and then, the 3D reconstruction from these 2D images is performed. At least one pair of target 2D images is selected, and viewpoint coordinates corresponding to each 2D image are set as initial coordinates and post-movement coordinates, respectively (step (1)).

Further, any one of the initial coordinates and the viewpoint coordinates
By introducing the distance between the 2D position of the feature point of the point (gaze point) and the origin of the 2D image, a three-dimensional translation vector H (hereinafter, referred to as “camera movement parameter”) indicating the movement of the viewpoint is calculated (( 2) Step). Thereafter, using the association obtained in step (1) and the camera motion parameters obtained in step (2), a 3D image of at least one pair of 2D images targeted for 3D restoration is obtained. Then, by applying such processing to other 2D images, a 3D moving image corresponding to a continuous 2D image can be obtained.

In this 3D reconstruction method, (2)
In the step, when calculating a camera motion parameter, a 3D position of a viewpoint in an intermediate image existing between the pair of 2D images is estimated by applying a spline curve to an arbitrary pair of 2D images, and the estimated 3D position is calculated. A position shift between the position and the 3D position of the viewpoint in the real image (the real image of the intermediate image) is calculated, and the camera movement parameter can be corrected using the position shift.
The accuracy of the obtained 3D moving image can be improved. In the processing using such a spline curve,
The arbitrary 2D image pair described above may be the same as or different from at least one pair of images selected in step (1).

[0068]

EXAMPLES Hereinafter, the present invention will be described in more detail with reference to examples, but the present invention is not limited to these examples. In the following Examples 1 and 2, a dedicated high-speed image processor was used for image processing, and a workstation (SUS4) was used for calculation processing.

Example 1 Using a black-and-white CCD camera (shutter speed 1/500 sec), a moving image was taken while always capturing the subject as an image, and the validity of the above-mentioned expression (11) was judged. The camera characteristics and shooting conditions of the used CCD camera are shown below.

[0070] [Camera properties] - focal length (f): 25.0 (mm) · pixel conversion ratio _{(E X, E Y) :(} 81.0,80.0)
[Pixel / mm] Image center (i ₀ , j ₀ ): (273, 235) [pixel]

[Shooting Conditions] As shown in FIG. 8, at the first camera coordinate, Z ₁ ^l = 500.0 m in the Z ^l direction.
The origin of the object coordinate system was set at the position of m, and the object was rotated by 45 ° around the _Xobj axis and the _Yobj axis. As shown in the figure, the number of feature points (N) was set to nine from # 1 to # 9. The translational motion of the camera motion is H = (Z ₁ ^l δ, 0,0) ^t .
However, δ = δ (k) = (k−1 + w (k)) / (2 (L
-1)), and k = 1, 2,..., L. In addition,
w (k) is an initial value w (1) = 0, and is a random walk variable whose value randomly increases / decreases by 1 in each image frame k. Further, the number of frames M used for the correlation coefficient (ρ _ij ) is 20, the number of tracking frames L is 101,
The threshold (D) for the length of the movement locus was 0.1 mm.

[0072] a value obtained by substituting the u _i ^k and δ in (11) of each feature point in the first frame, second M frames by the operations described above, shows the theoretical values graphed 9 Was.
In the graph shown in the figure, the data is shifted in the vertical direction to facilitate observation, but for feature point numbers # 5 and # 8, the length of the trajectory between M frames is equal to or smaller than the threshold (D). For this reason,
It can be seen that these trajectories are not similar to the trajectories of other feature points. In addition, it is clear that u _i ^k of each feature point has a high correlation with the theoretical value and further has a high correlation with each other.

Example 2 [Shooting] As shown in FIG. 10, the camera was fixed, and the subject 10 was placed on a manipulator and rotated to shoot a moving image (VTR shooting). At this time, the subject 10 is always captured on the screen, and the rotation interval is set to 0.3 °.
Photographed to 0.0 °. The total number of frames (L) is 1
01, and the threshold (D) of the locus length was 0.1 mm.

[Extraction of Feature Points] Next, feature points of the subject 10 were extracted using the first frame image. At this time, the threshold (B) of the inflection point described with reference to FIG. 4 was set to 5 pixels, and the threshold of the direction variance was set to 50.0. As a result, the feature point is 1
Six points were extracted (see FIG. 10). The time required for extracting these feature points was about 2 minutes.

[Extraction of Feature Point Motion Locus] The above-described template matching was performed to track feature points. Tracking of feature points having a similarity of 0.25 or more was stopped. If the threshold value of the similarity is too high, tracking is stopped up to a point to be tracked visually, and if the threshold value is too low, almost all points are tracked. FIG. 11 is a diagram illustrating tracking of a feature point, and illustrates an example in which tracking of a seventh feature point has failed halfway. The time required for tracking all the feature points was about 10 minutes.

[Solution of Correspondence Problem] The number of frames (M) used for judging the correlation was set to 20, the above-described clustering was performed, and the authenticity of the feature points in each frame was determined. FIG. 12 shows the obtained determination result.

In FIG. 12, the horizontal axis indicates the frame number, and the vertical axis indicates the feature point number. 10 and 11
As shown in (1), the point with little motion such as the feature point 1, specifically, the length of the trajectory between M frames is a threshold (D = 0.
1 mm) or less, the judgment was suspended. Tracking of the feature point 7 failed after 72 frames.

Table 1 shows the results of the present embodiment in comparison with the results of the visual inspection for solving the corresponding problem in the frames 1 to 101. The visual inspection was performed by the operator checking the tracking results for each feature point for each frame.

In Table 1, ○ in the operator judgment indicates that the visual inspection was successful, and X indicates that the visual inspection failed. In the computer judgment according to the present embodiment, ○ indicates frames 1 to 1.
01 indicates any of a determination of true or pending, no determination of false, and no determination of tracking failure, and other than that indicates x. From Table 1, the true / false judgment according to the present embodiment is that the correct answer is 14 out of 16 points, and the correct answer rate is 88.
%.

[0080]

[Table 1]

(Embodiment 3) [Correction of Camera Motion Parameter by Spline Estimation] In this embodiment, correction of camera motion parameter using the above-described spline curve will be described. FIG. 13 is an explanatory view showing the concept of spline estimation, in which reference numeral 6 denotes a subject, 6a denotes a gazing point, 7 denotes an actual viewpoint position, and 7 ′ denotes an estimated viewpoint position. Further, as a premise of the following processing, the association of the feature points in each continuous 2D image has been completed, and the 3D shape of the subject 6 is also known.

In spline estimation, first, an arbitrary pair of 2D images is selected, and a spline curve is applied to the pair of 2D images to calculate an estimated 3D of a viewpoint in an intermediate image existing between the two 2D images. Next, the actual 3D position of the viewpoint in the intermediate image is determined, and the trajectory of the viewpoint (camera motion parameter) is determined in consideration of the positional deviation between the two 3D positions.
Is corrected.

Specifically, the 3D positions of the viewpoints in the above-mentioned arbitrary image pair are designated as key 0 and key 1, respectively.
And the 3D position of the key 1 is obtained by an ordinary method. Next, a spline curve is applied to keys 0 and 1, and a viewpoint position 7 'in an intermediate image existing between keys 0 and 1 is estimated (see FIG. 13A). Next, the viewpoint position 7 in the real image of the intermediate image is determined, and the estimated 3D position 7 ′ is determined.
And the actual 3D position 7 is determined, and it is determined whether or not the positional deviation h is within a cube having a side Δl (see FIG. 13B).

If the displacement h exists in the cube, the camera motion parameters are not calculated in 3D.
If it is not in the cube, set the camera motion parameter to 3
D is calculated and obtained, this is adopted as a new key, and the same processing as described above is repeated until a positional shift exists in the cube in relation to another key 0 or key 1,
It estimates camera motion parameters.

Hereinafter, spline estimation is performed using the subject 8 shown in FIG. As shown in FIG.
It has 25 feature points, the length of one side is 100 mm, and the depth distance between the subject 8 and the camera (the symbol Z ₀ shown in FIG. 13A) is 500 mm. In addition, the camera motion was an arc motion centering on the subject 8, and the rotation width of the arc motion was set to 0 ° to 30 °, and photographing was performed every 0.3 ° to obtain a total of 101 continuous 2D images. FIG. 15 is a graph showing the values of the components H _X , H _Y , and H _Z of the camera motion parameter H when the camera motion is a true value (the above-described arc motion).

Next, a spline estimation process was performed with the number of keys set to 3 and Δl set to 5 mm, and the positional deviation from the case where the camera motion was a true value was graphed with respect to the components H _X , H _Y , and H _Z. Those are shown in FIG. FIG. 17 shows a case where the number of keys is 5 and Δl is 1 mm. FIG.
17 and FIG. 17, it can be seen that the positional deviation is smaller in FIG. In FIG. 17, the positional deviation is extremely small. Therefore, under the above-described shooting conditions, it is possible to perform highly accurate 3D reconstruction by specifying about five key points and correcting the camera motion parameters. I understand.

Although the present invention has been described in detail with reference to the embodiments, the present invention is not limited to these embodiments, and various modifications can be made within the scope of the present invention.
For example, the description has been given on the assumption that a 3D moving image is obtained for a continuous 2D image. However, such a continuous 2D image can be captured by a video device or the like, and such a video image is processed by the present invention. As a result, a highly accurate 3
D moving images can be easily obtained.

Also, the case where a continuous 2D image is obtained while moving the viewpoint has been described as an example.
It goes without saying that the subject can be moved.

[0089]

As described above, according to the present invention,
A specific constraint is imposed on each 2D image and attention is paid to a specific amount. This eliminates the need for a special device for measuring the viewpoint movement or special skills such as image processing, and easily solves the problem of corresponding feature points. It is possible to provide a method of associating subject feature points and a 3D restoration method using the same.

[Brief description of the drawings]

FIG. 1 is a perspective view showing a state in which a camera is arbitrarily and continuously moved to capture a continuous 2D image of the same subject.

FIG. 2 is an explanatory diagram showing a moving relationship of 3D coordinates in one section of continuous camera movement shooting.

FIG. 3 is an explanatory diagram showing an example of thinned edge information.

FIG. 4 is an explanatory diagram showing an example of a method of extracting a bending point.

FIG. 5 is an explanatory diagram illustrating an example of template matching.

FIG. 6 is an explanatory diagram showing prediction of a feature point position by template matching.

FIG. 7 is an explanatory diagram illustrating an example of a dendrogram process.

FIG. 8 is an explanatory diagram illustrating a shooting state according to the first embodiment.

FIG. 9 is a graph showing calculated values and theoretical values obtained by substituting u _i ^k and δ of each feature point into equation (11).

FIG. 10 is a perspective view illustrating a photographing state in the second embodiment.

FIG. 11 is a perspective view showing a state in which a feature point is being tracked.

FIG. 12 is an explanatory diagram showing a result of a feature point authenticity determination.

FIG. 13 is an explanatory diagram showing a positional shift in spline estimation.

FIG. 14 is a front view and a side view showing a subject used for spline estimation.

FIG. 15 is a graph showing camera motion parameters when the camera motion is a true value.

FIG. 16 is a graph showing a difference between camera motion parameters between a spline estimation and a true value.

FIG. 17 is a graph showing a difference between camera motion parameters between a spline estimation and a true value.

[Explanation of symbols]

 Reference Signs List 1 camera 2 subject 3 k-th image 4 course cursor 5 fine cursor 6 subject 6a fixation point 7 viewpoint position 7 'viewpoint position (estimated) 10 subject

 ──────────────────────────────────────────────────の Continuing on the front page (72) Inventor Shinji Ozawa 3-14-1 Hiyoshi, Kohoku-ku, Yokohama, Kanagawa Prefecture Inside Keio University Faculty of Science and Technology

Claims (10)

[Claims] 1. A method of associating a feature point of a subject between the plurality of 2D images when the stereoscopic shape of the subject is 3D-reconstructed using a plurality of 2D images photographed while the viewpoint is continuously moved. (A) moving these viewpoints so that the distance between the viewpoint and the subject is sufficiently larger than the distance between the initial coordinates and the moved coordinates of the viewpoint; Any one of these points is located at the image origin of the 2D image based on the initial coordinates and the image origin of the 2D image based on the coordinates after movement, and (B) the 2D position of the feature point of the arbitrary one point in each of the 2D images and the origin of the 2D image And the motion trajectory of the one arbitrary feature point accompanying the movement of the viewpoint as a function of only the three-dimensional translation vector H,
(C) The motion of the one arbitrary feature point is tracked using a 2D image other than the one used in (A) step, and the obtained motion trajectory is compared with the motion trajectory obtained in (B) step. And
A feature point associating method, wherein a feature having a high correlation is adopted as a motion trajectory of a true feature point, and this is used to associate another feature point in each 2D image. 2. The motion trajectory obtained in the step (B) is normalized by setting the average value of the number of 2D images accompanying the viewpoint movement in the step (A) to 0 and the standard deviation to 1. 2. The method for associating feature points according to claim 1, wherein: 3. The method according to claim 1, wherein in the step (A), a distance between the initial coordinates and the coordinates after the movement is 1/10 or less of a distance between the viewpoint and the subject. How to associate feature points. 4. The method according to claim 1, wherein the 2D image used in the step (C) is taken with a movement larger than the movement of the viewpoint in the step (A). The method of associating the feature points described in the two sections. 5. The method according to claim 1, wherein the 2D image based on the initial coordinates is a first image of the plurality of 2D images, and the feature points are extracted from a boundary line on texture information of the subject. A method for associating feature points according to any one of items 1 to 4. 6. The method according to claim 5, wherein in the second and subsequent images of the plurality of 2D images, the feature points are extracted by focusing on a directional variance value in a certain area in the vicinity thereof. Method for associating feature points. 7. The method according to claim 1, wherein the viewpoint is fixed, and the subject is continuously moved instead of the viewpoint, and the method according to claim 1 is executed. How to match. 8. The method according to claim 1, wherein the three-dimensional reconstruction of the subject is performed using a plurality of 2D images captured while the viewpoint is continuously moved. Applying the point associating method to complete the associating of the feature points between the 2D images, then selecting at least one pair of 2D images to be 3D-recovered from the plurality of 2D images, The viewpoint coordinates corresponding to the image are set as initial coordinates and coordinates after movement, respectively.
(2) Introduce the distance between the 2D position of one arbitrary feature point at each coordinate and the origin of the 2D image, calculate the three-dimensional translation vector H indicating the movement of the viewpoint, and (3) after that,
A 3D reconstruction method of a subject, characterized in that a 3D image of at least one pair of 2D images is obtained using the association determined in (1) step and the three-dimensional translation vector H determined in (2) step. . 9. When calculating the three-dimensional translation vector H in the step (2), a spline curve is applied to at least one pair of 2D images, and a viewpoint of an intermediate image existing between the pair of images is determined. The 3D position is estimated, and the estimated 3D position and the 3D of the viewpoint in the actual intermediate image are estimated.
9. The method according to claim 8, wherein a positional deviation from the position is calculated, and the three-dimensional translation vector H is corrected using the positional deviation. 10. A method of restoring a 3D object, wherein the viewpoint is fixed, and the object is continuously moved instead of the viewpoint, and the method according to claim 8 or 9 is executed.