JP2000067216A - Method and device for acquiring three-dimensional structure and recording medium for the method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire the structure of an object or accurate structure of a shape from two-dimensional(2D) measured data (measurement matrix) at a feature point provided from time sequential image data photographing the object. SOLUTION: Concerning this 3D structure acquiring method, the coordinate values of feature points constituting the structure of the object are measured in the photographed 2D image data, the feature points are time sequentially traced and concerning the provided 2D measured data of time sequential feature points, the structure or shape of the object is acquired while using a fractorizing method. In this case, the data are decomposed into action information of the camera removing a noise component and structure information of the object removing a noise component and in this decomposing process, a sensor to be operated synchronously with the physical motion of the camera measures the angle value of rotation around the horizontal axis of coordinates of the camera on a coordinate system with the viewpoint of the camera as an origin, the angle value of rotation around the vertical axis, and the angle value of rotation around an optical axis as the motion information of the camera. While using this measured motion information, the relevant motion information is corrected and the structure component of the fractorized object is restored/ reproduced.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for constructing a structure or a shape of an object from two-dimensional measurement data tracked in a time series for feature points constituting the structure or the shape of the object measured from a time-series two-dimensional image. The present invention relates to a three-dimensional structure obtaining method for obtaining three-dimensional coordinate values of feature points to be performed, a computer-readable recording medium storing a program for causing a computer to execute the processing procedure of the three-dimensional structure obtaining method, and an apparatus for implementing the same.

[0002]

2. Description of the Related Art A method of acquiring three-dimensional information for restoring and reproducing the structure or shape of an object from a feature point on a two-dimensional image obtained by photographing the object is a multi-view image matching corresponding points of feature points from a plurality of images. Accordingly, a method of acquiring three-dimensional information has been conventionally performed. FIG. 15 is a diagram illustrating an example of a conventional three-dimensional structure acquisition method by stereoscopic viewing of the camera 1 and the camera 2, where 01 is an object, 02 is three-dimensional coordinate values,
03 is the optical axis of camera 1, 04 is the optical axis of camera 2, 05 is the image plane of camera 1, 06 is the image plane of camera 2,
07 is the corresponding point (X ₁ , Y ₁ ) on camera ₁ and 08 is camera 2
, The corresponding point (X ₂ , Y ₂ ), 09 is the base line length.

On the other hand, the physical motion of an image input device and the structure of an object are obtained by a feature point data obtained by tracking feature points in time series, that is, a three-dimensional structure acquisition method called a factorization method employing singular value decomposition from a measurement matrix. Alternatively, a method has been proposed in which the shape can be determined simultaneously and robustly. In recent years, various applications and improvements of the factor decomposition method have been made.

A three-dimensional structure obtaining apparatus using the factor decomposition method is disclosed, for example, in Japanese Patent Application Laid-Open No. H10-111934.

[0005]

However, in the above-described conventional factorization method or the like, when measuring a measurement matrix obtained from a two-dimensional image, the influence of noise due to quantization errors on the image coordinates and measurement errors. Accordingly, there has been a problem that erroneous detection of physical movement of the image input device and distortion of the structure or shape of the object are affected.

An object of the present invention is to provide a method for obtaining a three-dimensional structure of a structure or a shape of an object constituted by a feature point from two-dimensional measurement data of the feature point obtained from a time-series image of the object. An object of the present invention is to provide a technique capable of improving the accuracy of the structure or shape of an object to be reproduced.

[0007] The above and other objects and novel features of the present invention will become apparent from the description of the present specification and the accompanying drawings.

[0008]

The outline of a typical invention among the inventions disclosed in the present application will be briefly described as follows.

(1) An object (object) by the image input device
Is captured, and two-dimensional coordinate points (hereinafter, referred to as “structures”) constituting the structure or shape of the object are obtained from the captured two-dimensional image data.
"Feature points") are measured, the feature points are tracked in a time series, and the obtained time-series feature point data (input data of the singular value decomposition method) is subjected to a factor decomposition method to obtain an object structure or A three-dimensional structure acquiring method for acquiring a shape, wherein motion information representing physical motion of the image input device from which noise components have been removed from two-dimensional measurement data composed of the feature point data by a factorization method. And decompose into structural information or shape information representing the structure or shape of the object from which the noise component has been removed, and in the process of decomposing, the sensor device that operates in synchronization with the physical motion of the image input device, In a coordinate system (the coordinate system of the image input device, the coordinate system of the camera (viewpoint)) with the viewpoint of the input device as the origin, the rotation angle value of the coordinates of the image input device around the horizontal axis and the rotation angle around the vertical axis rotation A degree value, and a rotation angle value around the optical axis are measured as motion information of the image input device, the motion component is corrected using the measured motion information, and a structural component or a shape component of the factorized object is measured. Is restored and reproduced.

(2) 2 composed of the feature point data
The dimension measurement data is decomposed by a factorization method into motion information representing the physical motion of the image input device from which the noise component has been removed, and structural information or shape information representing the structure or shape of the object from which the noise component has been removed. In the process of performing the translation, the sensor device operating in synchronization with the physical movement of the image input device translates the coordinates of the image input device along a horizontal axis in a coordinate system with the viewpoint of the image input device as the origin. The component value, the translation component value along the vertical axis, and the translation component value along the optical axis are measured as motion information of the image input device, and the motion component is corrected using the measured motion information, In addition, the structural or shape components of the factorized object are restored and reproduced.

(3) 2 composed of the feature point data
The dimension measurement data is decomposed by a factorization method into motion information representing the physical motion of the image input device from which the noise component has been removed, and structural information or shape information representing the structure or shape of the object from which the noise component has been removed. In the process of, by a sensor device or the like that operates in synchronization with the physical movement of the image input device, in a coordinate system with the origin of the viewpoint of the image input device, around the horizontal axis of the coordinates of the image input device. A rotation angle value, a rotation angle value around a vertical axis, and a rotation angle value around an optical axis, and a translation component along a horizontal axis and a translation component along a vertical axis of the coordinates of the image input apparatus. , And the translation component value along the optical axis is measured as motion information of the image input device, the motion component is corrected using the measured motion information, and the structural component or shape of the factorized object is measured. To restore and reproducibility of the components.

(4) 2 composed of the feature point data
The dimension measurement data is decomposed by a factorization method into motion information representing the physical motion of the image input device from which the noise component has been removed, and structural information or shape information representing the structure or shape of the object from which the noise component has been removed. In the process of, from the two-dimensional measurement data composed of the feature point data, by a sensor device that operates in synchronization with the physical motion of the image input device, measure the physical motion state information of the image input device, The motion component is corrected using the measured physical motion state information, and the structural component or shape component of the object is restored and reproduced.

(5) An object is photographed by the image input device,
In the photographed two-dimensional image data, the structure or feature points of the object are measured, the feature points are tracked in time series, and the obtained time-series feature point data is subjected to a factorization method to determine the structure or feature of the object. A computer-readable storage medium storing a program for causing a computer to execute a processing procedure of a three-dimensional structure obtaining method for obtaining a shape, the two-dimensional measurement data including the feature point data being subjected to a factor decomposition method. A motion information expressing the physical motion of the image input device from which the noise component has been removed, a structural information or a shape information expressing the structure or shape of the object from which the noise component has been removed, and the decomposing procedure so,
By a sensor device that operates in synchronization with the physical movement of the image input device, a rotation angle value around a horizontal axis of the coordinates of the image input device in a coordinate system with the origin of the viewpoint of the image input device, A procedure for measuring a rotation angle value about a direction axis and a rotation angle value about an optical axis as motion information of the image input device, a procedure for correcting the motion component using the measured motion information, and a factor decomposition A computer-readable recording medium on which a program for causing a computer to execute a procedure for restoring and reproducing a structural component or a shape component of an object is recorded.

(6) 2 composed of the feature point data
The dimension measurement data is decomposed by a factorization method into motion information representing the physical motion of the image input device from which the noise component has been removed, and structural information or shape information representing the structure or shape of the object from which the noise component has been removed. In the coordinate system having the origin at the viewpoint of the image input device, the translation along the horizontal axis of the coordinates of the image input device by the sensor device operating in synchronization with the physical movement of the image input device. A procedure of measuring component values, translation component values along a vertical axis, and translation component values along an optical axis as motion information of the image input device, and correcting the motion component using the measured motion information Computer-readable program that stores a program for causing a computer to execute a procedure for restoring and reproducing a structural component or a shape component of a factorized object. It is a recording medium.

(7) 2 composed of the feature point data
The dimension measurement data is decomposed by a factor decomposition method into motion information representing the physical motion of the image input device from which noise components have been removed and structure or shape information representing the structure or shape of the object from which noise components have been removed. In a procedure, by a sensor device or the like that operates in synchronization with the physical movement of the image input device, rotation of the coordinates of the image input device around a horizontal axis in a coordinate system with the viewpoint of the image input device as the origin. An angle value, a rotation angle value about a vertical axis, and a rotation angle value about an optical axis, and a translation component along a horizontal axis of the coordinates of the image input device, a translation component along a vertical axis, And a procedure for measuring a translation component value along the optical axis as motion information of the image input device, a procedure for correcting the motion component using the measured motion information, and a structural component of the factorized object. Is a computer readable recording medium recording a program for executing the procedure for restoring and reproducing a shape component to the computer.

(8) 2 composed of the feature point data
The dimension measurement data is decomposed by a factor decomposition method into motion information representing the physical motion of the image input device from which noise components have been removed and structure or shape information representing the structure or shape of the object from which noise components have been removed. A step of measuring physical motion state information of the image input device from the two-dimensional measurement data composed of the feature point data by a sensor device that operates in synchronization with the physical motion of the image input device. A computer-readable program storing a program for causing a computer to execute a procedure for correcting the motion component using the measured physical motion state information and a procedure for restoring and reproducing a structural component or a shape component of the object. It is a recording medium.

(9) An image input device such as a camera for photographing an object, measuring means for measuring feature points constituting the structure or shape of the object in two-dimensional image data of the object, and Means for tracing a feature point in a series and acquiring the structure or shape of the object by using a factorization method on the obtained time-series feature point data. The two-dimensional measurement data composed of is decomposed by a factor decomposition method into a component that represents the physical motion of the image input device from which the noise component has been removed and a component that represents the structure or shape of the object from which the noise component has been removed. Means, a sensor device operating in synchronization with the physical motion of the image input device, means for correcting a motion component using an output of the sensor device, and a structural component or shape of the factorized object It comprises a means for restoring or reproducing a minute.

(10) The rotation angle value of the coordinates of the image input device around the horizontal axis, the rotation angle value around the vertical axis of the coordinates of the image input device in the coordinate system whose origin is the viewpoint of the image input device, And a means for measuring a rotation angle value about the optical axis as sensor information of physical movement of the image input device.

(11) A translation component value along the horizontal axis and a translation component along the vertical axis of the coordinates of the image input device in the coordinate system whose origin is the viewpoint of the image input device by the sensor device. Means for measuring the value and the translation component value along the optical axis as sensor information of the physical movement of the image input device.

(12) The rotation angle value of the coordinates of the image input device around the horizontal axis, the rotation angle value around the vertical axis of the coordinates of the image input device in the coordinate system whose origin is the viewpoint of the image input device, And the rotation angle value around the optical axis, and the translation component value along the horizontal axis, the translation component value along the vertical axis, and the translation component value along the optical axis of the image input device coordinates are measured. Means.

(13) There is provided means for measuring physical motion state information of the image input device from the two-dimensional measurement data composed of the feature point data by the sensor device.

That is, the present invention provides a factorization method for acquiring a three-dimensional structure from two-dimensional measurement data (measurement matrix) of feature points obtained from a time-series image, and a physical resolution of an image input device for photographing an object. It is characterized in that the structure or shape of an object is accurately restored / reproduced using sensor information obtained by sensing a motion state.

Using the method of the present invention, a highly accurate structure relating to the structure or shape of an object is obtained from two-dimensional measurement data (measurement matrix) of feature points obtained from a time-series image of the object. Becomes possible.

[0024]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The embodiments (examples) of the present invention will be described below in detail with reference to the drawings.

In all the drawings for describing the embodiments (examples), those having the same functions are denoted by the same reference numerals, and their repeated description will be omitted.

In the following, an example will be described in which a parallel perspective transformation type factorization method is used as a three-dimensional structure acquisition means, and rotation and translation are performed while the camera motion is kept constant in the optical axis direction. I do. Further, the number of time-series images by the feature point tracking, that is, the number of frames is F, and the number of feature points in each frame is P.

(Embodiment 1) FIG. 1 shows Embodiment 1 of the present invention.
FIG. 4 is an overall processing flow diagram of a three-dimensional structure acquisition method of
S11 is a step of moving (moving) the camera, S12 is a step of photographing an object, S13 is a step of acquiring sensor information, S14 is a step of tracking feature points, S15 is a step of inputting data of the factorization method, S16 is a step of inputting data of the factorization method. A step of performing processing by the factor decomposition method, and step S17 is a step of outputting processing data by the factor decomposition method.

FIG. 2 is a diagram for explaining the relationship between a target object and an image input device. Reference numeral 21 denotes a target object; 22, an image input device including a camera 22A;
Is a camera (viewpoint) coordinate system, 24 is a sensor for detecting camera rotational movement, 25 is time-series image data of an image of a target object, and 26 is a sensor information storage device that records the movement of the image input device 22.

FIGS. 3A and 3B are schematic diagrams showing the appearance of a camera for acquiring a three-dimensional structure and an 8 mm camera for acquiring a three-dimensional structure as one example of the image input device of the first embodiment. FIG.
FIG. 3B is an external view of a camera 22A for acquiring a three-dimensional structure, and FIG. The camera 22A for acquiring a three-dimensional structure includes a three-dimensional structure acquiring means 22A1 for applying a camera rotational movement to a factorization method and a sensor (camera attitude sensor) 22A for detecting the camera rotational movement.
2 are provided. Also, 8mm for obtaining the three-dimensional structure
The camera 22B is provided with a three-dimensional structure acquiring means 22B1 for giving the camera rotational movement to the factorization method and a sensor (camera attitude sensor) 22B2 for detecting the camera rotational movement. The three-dimensional structure acquiring means 22A1 or 22
The three-dimensional structure data acquired in B1 can be used for walk-through from another viewpoint to be reproduced later. Less than,
The image input device 22 is simply referred to as a camera 22.

The overall processing procedure of the three-dimensional structure acquisition method according to the first embodiment will be described with reference to FIG. First, camera 2
2 is rotated and translated (S11), and a target object is photographed (S12) to obtain time-series image data (FIG. 2). At this time, in response to the rotational movement of the camera 22, a camera rotation detection sensor (camera attitude sensor) 22A2 or 2A using a gyro or the like attached to the camera 22
2B2, when the camera 22 is rotationally moved in time series with reference to the coordinate system at the viewpoint of the camera 22 in the initial state (the initial state of the movement of the camera 22 when the first image of the image data is captured) The rotation parameter value of the camera 22 is obtained as sensor information (S13).

Here, the parameter value of the camera 22 is X
An angle value α _f (radian) rotated around the axis, an angle value β _f (radian) rotated around the Y axis, and an angle value γ _f (radian) rotated around the Z axis. Suffix f
Is added in the sense that a rotation value exists for each frame.

On the other hand, as the camera 22 shown in FIG. 3 moves, a digital image of the number F of frames is recorded for the photographed time-series image data (hereinafter, referred to as time-series image data), as shown in FIG. To the frame f (f = 1,
The coordinate values of the feature points on the two-dimensional image plane in the images (2,..., F) are measured. In FIG. 4, 311 is time-series image data, 312 is a frame image of NO.f (f = 1,
2,..., F), 313 is a coordinate system Xd, 314 is a coordinate axis Y
d and 315 are coordinate axes (Xfp, Yfp) of the feature point p (p = 1, 2,... p).

For this measurement, the frame number (N
The image of Off) is displayed on an image output device such as a display, and points on the image such as corners and edges of the object constituting the structure or shape of the object are manipulated by operating a mouse pointer or the like. Coordinate system (Xd, Y
The coordinate values (Xfp, Yfp) of the feature points in d) are sequentially measured from feature points No. 1 to NO.P. The coordinate points, that is, the coordinate values (Xfp, Yfp) of the feature points are arranged according to the format of the measurement matrix as shown in FIG. The measurement matrix is a matrix of 2F × P size, and is input data of the factorization method as described later.

The area of the F × P matrix in the upper half is the X coordinate value of the feature point in each frame, and the F × P matrix in the lower half is
The area of the matrix is the Y coordinate value of the feature point in each frame. In addition, in the case of a feature point on frame No. f,
The P feature points are arranged in order in the row direction, and frames NO. 1 to NO.F are arranged in order in the column direction. Such a series of feature point tracking processing is performed according to a processing flow as shown in FIG. 6, and finally, input data (two-dimensional measurement data of feature points) of the factor decomposition method shown in FIG. Obtain the matrix [A].

FIG. 6 is a flowchart showing a processing procedure for measuring the feature points to be tracked in time series. S31 is a step of reading the f-th frame image. S32 is whether or not the current frame number does not exceed F. Determining step S33
Is a step of measuring the P-th feature point coordinates, S34 is a step of determining whether the number of feature points does not exceed P, S35 is a step of writing the coordinate values of the feature point coordinates Xfp, S36 is a feature point This is the step of writing the coordinate value of the coordinate Yfp.

FIGS. 7A and 7B are diagrams showing the relationship between camera motion, sensor information, and camera vector according to the first embodiment. Reference numeral 41 denotes a camera (image input device) in an initial state;
42 is the X axis in the camera (viewpoint) coordinate system in the initial state, 43 is the Y axis in the camera (viewpoint) coordinate system in the initial state, and 44 is the Z axis in the camera (viewpoint) coordinate system in the initial state.

Reference numeral 45 denotes α _f rotation around the X axis, and β _f rotation around the Y axis.
Rotation, camera (image input device) when rotated by γ _f around the Z axis, 46 denotes a camera coordinate system when rotated by α _f around the X axis, β _f around the Y axis, and γ _f around the Z axis , The Y axis in the camera coordinate system when α _f rotation around the X axis, β _f rotation around the Y axis, and γ _f rotation around the Z axis, 48 is α _f rotation around the X axis, and Y Z axis in the camera coordinate system when rotated β _f around the axis and γ _f around the Z axis, 49 rotated α _f around the X axis, β _f around the Y axis, and γ _f around the Z axis The unit direction vector of the X axis in the camera coordinate system at the time, 410 is the unit direction of the Y axis in the camera coordinate system when rotated by α _f around the X axis, β _f around the Y axis, and γ _f around the Z axis. Vector, 411
Is a unit direction vector of the Z axis in the camera coordinate system when rotated by α _f around the X axis, β _f around the Y axis, and γ _f around the Z axis.

Next, a flow from acquisition of sensor information to a factor decomposition method in FIG. 1 will be described with reference to FIG. The initial state of the camera motion is a physical state of the camera 22 at the time of the first image taken when the object is imaged, with the camera viewpoint as the origin, the Z axis in the optical axis direction, and the horizontal direction for convenience. Assuming the X-axis and the Y-axis in the vertical direction, this is set to the initial state, and the unit direction vectors i ₁ , j ₁ , and k ₁ in the X-axis direction, the Y-axis direction, and the Z-axis direction are set in the initial state. Let it be a camera vector. The rotation parameter value during camera movement is
A rotary component about the unit vector i _1, j _1, k _1, the camera movement of an object photographed, if the rotation about i ₁ occurs, detects sensor information as the rotation parameter values alpha _f, j ₁ If rotation around occurs, detects sensor information as the rotation parameter value beta _f, if rotation about k ₁ has occurred, at the same time as the time-series images by detecting the sensor information as a rotation parameter value gamma _f, The detected rotation parameter values α _f , β _f , and γ _f are recorded for each frame f. Further, in the factorization method, the rotation parameter value cannot be input as it is, and as shown in FIG.
Calculate the rotation matrix [R] around the axis, Y-axis, and Z-axis by the calculation formula of Equation 1.

[0039]

(Equation 1)

The unit direction vectors i _f , j _f , and k _f in the X-axis direction, the Y-axis direction, and the Z-axis direction in the camera viewpoint coordinates in the frame NO.f are sequentially calculated by the equation (4). Input data as sensor information.

FIG. 8 is a flow chart showing the processing procedure of the factor decomposition method in the first embodiment. S111 is a step of inputting a measurement matrix, S112 is a step of calculating the center of gravity of the object, and S113 is a processing of singular value decomposition. Step S11
4 is a step of separating noise components due to rank 3 (see FIG. 4), S115 is a step of calculating a camera vector from camera rotation parameter values, S116 is a step of calculating a camera motion matrix from the camera vector, S11
7 is a step of calculating a matrix [Q], and S118 is a step of calculating a matrix [S].
Is the step of calculating

The calculation formula used in the processing of the factor decomposition method in the first embodiment is shown below.

[0043]

[A] = [U] [W] [V]

[0044]

[A] = [U ₁ ] [W ₁ ] [V ₁ ] + [U ₂ ] [W ₂ ] [V ₂ ] ≒
[U ₁ ] [W ₁ ] [V ₁ ]

[0045]

(Equation 4)

[0046]

(Equation 5)

[0047]

(Equation 6)

[0048]

(Equation 7)

Next, the factor decomposition method in FIG. 1 will be described with reference to FIG.

A measurement matrix 2F × P size [A] composed of coordinate points of feature points on the two-dimensional image plane is converted into a matrix [U] (2F × P
P), [W] (P × P diagonalization matrix), and [V] (P × P) (S113). Where rank 3 gives the matrix
[U] is converted into a 2F × 3 size matrix [U ₁ ] and a 2F × (P-3) size matrix [U ₂ ], and the matrix [W] is changed into a 3 × 3 size diagonal matrix [W ₁ ]. The matrix [V] is transformed into a matrix [V ₁ ] of 3 × P size and a matrix [V ₂ ] of (P-3) × P size in a matrix [W ₂ ] of (P-3) × (P-3) size. Separation (S114) (see Equation 2, FIG. 9).

FIG. 9 is a diagram for explaining noise component removal by rank 3 after singular value decomposition processing. The diagonal element of [W ₂ ] is compared with the diagonal element of [W ₁ ]. Very small,
[U ₂ ] [W ₂ ] [V ₂ ] It is regarded as a noise component and deleted.

On the other hand, from the camera vectors i _f , j _f , and k _f input as sensor information to the camera motion matrix [M],
(S116). This calculation formula is a relational formula derived from the parallel perspective transformation camera model. Note that the values of X _f and Y _f in this formula are obtained from the measurement matrix [A] as the calculation of the physical center of gravity (S112), and the movement in the optical axis direction is not considered for Z _f (= 1). Is set as a fixed value.

Steps S115 and S116 in FIG.
The details of the processing procedure of S117 are shown in FIG. 10 (data processing flowchart). In FIG. 10, S201 is a step of reading a rotation component for each frame, S202 is a step of generating a camera base vector, S203 is a step of generating camera vectors m _f and n _f , and S204 is a measurement matrix.
[A] is input, S205 is a step of reading P feature point coordinates in the f-th frame, S206 is a step of calculating the body weight in the f-th frame from the measurement matrix, and S207 is a step of processing singular value decomposition. , S2
08 is a step of removing noise components, S209 is a step of determining whether or not processing has been completed for all frames, S209
210 is a step for moving to the processing of the next frame when processing for all frames is not completed, S211 is a step for generating a camera motion matrix [M], S212 is a step for generating a camera action matrix [Q], and S213 is a structure. Alternatively, this is a step of performing a shape restoration process.

The detailed internal calculation formula is shown below.

As a formula for calculating the basis vector of the camera, the above formula (4) is used.

The formula for calculating the camera vectors m _f and n _f is the formula of the above equation (5).

The formula for generating the camera motion matrix [M] uses the following formula (8).

[0058]

(Equation 8)

The following formula 9 is used to calculate the body weight centers X _f and Y _f in the f-th frame from the measurement matrix.

[0060]

(Equation 9)

In the factorization method, input data of the factorization method as shown in FIG. 11 is input, and noise components [U ₁ ] [W ₁ ]
From the matrix from which [V ₁ ] is deleted, ([U ₁ ] [Q] × [Q] ~ ¹ [W ₁ ]
[V ₁ ]), the former matrix [U ₁ ] [Q] is converted to a camera motion matrix [M] (FIG. 12), and the latter matrix [Q] ~ ¹ [W ₁ ] [V ₁ ] is converted to an object. Is a matrix [S] in which the three-dimensional coordinate values of the feature points constituting the structure or the shape are arranged.

In the present invention, the matrix [M] is the camera motion matrix described above, and the matrix [Q] is obtained from the pseudo inverse matrix of [U ₁ ] and [M] (Equation 6). Calculation is performed using the matrix [Q] (Equation 7). The matrix [S] is a three-dimensional coordinate value of a coordinate point measured as a feature point on the two-dimensional image plane. The matrix format is as shown in FIG. The coordinates X, Y, and Z coordinate values are arranged, and are arranged in the row direction in the order of the feature points. The elements of this matrix are the output data of the factorization method obtained as the reconstruction and reproduction of the structure or shape of the object according to the present invention.

(Embodiment 2) In Embodiment 2 of the present invention, since there is a difference in the factor decomposition method in Embodiment 1, only a portion having a difference will be described.

FIG. 14 is a flowchart showing the processing procedure of the factorization method according to the second embodiment of the present invention. S121 is a step of inputting a measurement matrix, S122 is a step of calculating the body weight, and S123 is a camera rotation parameter value. Calculating a camera vector from the camera vector, S124 calculating a camera motion matrix from the camera vector, S125
Is a step of calculating the matrix [S].

From the measurement matrix 2F × P size [A] consisting of the coordinates of the feature points on the two-dimensional image plane, X _f and Y _f are calculated from the total reward matrix [A] as (Equation 9). Calculate. Next, the camera vector input as sensor information
The camera motion matrix [M] is calculated from i _f , j _f , and k _f by using the equation (5). The calculation formula of Expression 5 is a relational expression derived from the parallel perspective transformation camera model. In addition, this number 5
As for Z _f (= 1) in the calculation formula, a fixed value is set because the movement in the optical axis direction is not considered. The matrix [S] in which the three-dimensional coordinate values of the feature points constituting the structure or shape of the object are arranged is calculated using the matrix [M] and the input data [A] of the factorization method (Equation 10). .

[0066]

(Equation 10)

The matrix [S] is a three-dimensional coordinate value of a coordinate point measured as a feature point on a two-dimensional image plane, and is obtained as information describing the structure or shape of an object as in the first embodiment. Can be.

As described above, the invention made by the present inventor is:
Although specifically described based on the embodiment (example),
The present invention is not limited to the above-described embodiment (example), and it is needless to say that various changes can be made without departing from the gist of the present invention.

[0069]

As described above, according to the present invention, the structure and object of the object to be restored and reproduced are photographed by the means for acquiring the three-dimensional structure of the structure or shape of the object constituted by the feature points. From the two-dimensional measurement data (measurement matrix) of the feature points obtained from the time-series images, it is possible to improve the accuracy of acquiring the three-dimensional structure or shape of the object.

[Brief description of the drawings]

FIG. 1 is an overall processing flowchart of a three-dimensional structure acquisition method according to a first embodiment of the present invention.

FIG. 2 is a diagram for explaining a relationship between an object to be processed and an image input device according to the first embodiment.

FIG. 3 shows a camera and a camera for acquiring a three-dimensional structure according to the first embodiment.
It is a schematic diagram which shows the external appearance of the 8 mm camera for dimension structure acquisition.

FIG. 4 is a diagram illustrating an example of measuring coordinate values of feature points in each frame image from time-series image data according to the first embodiment.

FIG. 5 is a diagram illustrating an arrangement of feature point coordinate values in a measurement matrix according to the first embodiment.

FIG. 6 is a flowchart showing a processing procedure of tracking and measuring the coordinate values of the feature points in time series according to the first embodiment.

FIG. 7 is a relationship diagram of camera motion, sensor information, and a camera vector according to the first embodiment.

FIG. 8 is a flowchart illustrating a processing procedure of a factor decomposition method according to the first embodiment.

FIG. 9 shows rank 3 after the singular value decomposition processing according to the first embodiment.
FIG. 4 is a diagram for explaining noise component removal by the following method.

FIG. 10 shows steps S115, S116 and S11 in FIG.
7 is a detailed data processing flowchart of the processing procedure of FIG.

FIG. 11 is a diagram showing a format of input data of the factorization method according to the first embodiment.

FIG. 12 is a diagram illustrating a format of a camera motion matrix according to the first embodiment.

FIG. 13 is a diagram illustrating a format of output data of the factorization method according to the first embodiment.

FIG. 14 is a flowchart showing a processing procedure of a factor decomposition method according to the second embodiment of the present invention.

FIG. 15 is a diagram illustrating a conventional method of acquiring a three-dimensional structure by stereoscopic viewing of cameras 1 and 2.

[Explanation of symbols]

01: target object, 02: three-dimensional coordinate value, 03: optical axis of camera 1, 04: optical axis of camera 2, 05: camera 1
Image plane by the image plane by 06 ... camera 2, the corresponding point of 07 ... camera _{1 (X 1, Y 1)} , the corresponding point in the 08 ... camera 2 (X _2, Y _2), 09 ... base length, S11 ... camera movement, S12 ... object shooting, S13 ... sensor information acquisition, S1
4: Feature point tracking, S15: Input data of factorization method, S
16: Factor decomposition method processing, S17: Output data of the factor decomposition method, 21: Target object, 22: Image input device (camera), 23: Camera (viewpoint) coordinate system, 24: Sensor for detecting camera rotational movement .., 25... Time-series image data of an object to be photographed, 26... Sensor information storage device that records the motion of the image input device, 311. Time-series image data, 312. Xd, 314: coordinate axes Yd, 315: feature point p, 41: image input device (camera) in initial state, 42: X axis in camera (viewpoint) coordinate system in initial state, 43 ...
Y axis in the camera (viewpoint) coordinate system in the initial state,
44: Z axis in the camera (viewpoint) coordinate system in the initial state, 45: α _f rotation around the X axis, β _f rotation around the Y axis,
Image input device (camera) when rotated γ _f around the Z axis, 46 .alpha. _F rotation around the X axis, β _f rotation around the Y axis,
X in the camera coordinate system when rotated by γ _f around the Z axis
Axis, 47: α _f rotation around X axis, β _f rotation around Y axis, Z
Y in the camera coordinate system when rotated by γ _f around the axis
Axis, 48: α _f rotation around X axis, β _f rotation around Y axis, Z
Z in the camera coordinate system when rotated by γ _f around the axis
Axis, 49: α _f rotation around X axis, β _f rotation around Y axis, Z
Unit direction vector of the X axis in the camera coordinate system when rotated by γ _f around the axis, 410... Α _f rotation around the X axis, Y
Unit direction vector of Y axis in camera coordinate system when β _f rotation around axis and γ _f rotation around Z axis, 411 ... α _f rotation around X axis, β _f rotation around Y axis, around Z axis γ The unit direction vector of the Z axis in the camera coordinate system when rotated by _f .

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Norio Nakamura 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo Japan Telegraph and Telephone Corporation (72) Inventor Toshiaki Sugimura 3-192 Nishishinjuku, Shinjuku-ku, Tokyo No. Nippon Telegraph and Telephone Corporation F-term (reference) 2F065 AA03 AA04 AA09 AA12 AA17 AA32 AA39 AA53 BB05 DD03 DD04 FF04 FF65 FF67 JJ03 JJ13 MM08 PP04 PP05 QQ01 QQ21 QQ24 QQ27 QQ28 RR05 SS13 UU08 CC13 DA17 DB02 DB09 DC09

Claims (13)

[Claims] 1. An object (object) is photographed by an image input device, and two-dimensional coordinate points (hereinafter, referred to as “feature points”) constituting a structure or a shape of the object in the photographed two-dimensional image data. Measures and tracks feature points in a time series, and acquires the structure or shape of an object using a factorization method on the obtained time-series feature point data (input data of the singular value decomposition method). A method, comprising: moving information representing physical movement of the image input device from which noise components have been removed by a factorization method using two-dimensional measurement data composed of the feature point data; and an object from which noise components have been removed. Is decomposed into structural information or shape information expressing the structure or shape of the image input device, and in the process of decomposing, the sensor device that operates in synchronization with the physical movement of the image input device sets the viewpoint of the image input device as the origin The rotation angle of the coordinates of the image input device around the horizontal axis, the rotation angle around the vertical axis, and the optical axis in the coordinate system (the coordinate system of the image input device, the camera (viewpoint) coordinate system) Measuring the surrounding rotation angle value as motion information of the image input device, correcting the motion component using the measured motion information, and restoring / reproducing the structural component or shape component of the factorized object. A three-dimensional structure acquisition method characterized by the following. 2. An object is photographed by an image input device, characteristic points constituting the structure or shape of the object are measured in the photographed two-dimensional image data, and the characteristic points are tracked in time series. A three-dimensional structure acquiring method for acquiring a structure or a shape of an object by using a factorization method for feature point data of a series, wherein two-dimensional measurement data composed of the feature point data is subjected to noise reduction by a factorization method. Motion information representing the physical motion of the image input device with the component removed,
The image input device is decomposed into structural information or shape information expressing the structure or shape of the object from which the noise component has been removed, and in the process of decomposing, the sensor device that operates in synchronization with the physical movement of the image input device. In the coordinate system with the viewpoint of the origin as the origin, the translation component value along the horizontal axis, the translation component value along the vertical axis, and the translation component value along the optical axis of the coordinates of the image input device are displayed in the image. A three-dimensional structure acquisition characterized by measuring as motion information of an input device, correcting the motion component by using the measured motion information, and restoring and reproducing a structural component or a shape component of the factorized object. Method. 3. An object is photographed by an image input device, characteristic points constituting the structure or shape of the object are measured in the photographed two-dimensional image data, and the characteristic points are tracked in time series. A three-dimensional structure acquiring method for acquiring a structure or a shape of an object by using a factorization method for feature point data of a series, wherein two-dimensional measurement data composed of the feature point data is subjected to noise reduction by a factorization method. Motion information representing the physical motion of the image input device with the component removed,
The image input is decomposed into structural information or shape information representing the structure or shape of the object from which the noise component has been removed, and in the process of decomposing, the image input is performed by a sensor device or the like that operates in synchronization with the physical motion of the image input device. A rotation angle value about a horizontal axis, a rotation angle value about a vertical axis, and a rotation angle value about an optical axis of the coordinates of the image input device in a coordinate system with the viewpoint of the device as an origin; The translation component along the horizontal axis, the translation component along the vertical axis, and the translation component value along the optical axis of the coordinates of the input device were measured as motion information of the image input device, and the measurement was performed. A method for acquiring a three-dimensional structure, wherein the motion component is corrected using the motion information, and the structural component or the shape component of the factorized object is restored and reproduced. 4. An object is photographed by an image input device, characteristic points constituting the structure or shape of the object are measured in the photographed two-dimensional image data, and the characteristic points are tracked in time series. A three-dimensional structure acquiring method for acquiring a structure or a shape of an object by using a factorization method for feature point data of a series, wherein two-dimensional measurement data composed of the feature point data is subjected to noise reduction by a factorization method. Motion information representing the physical motion of the image input device with the component removed,
In the process of decomposing into structural information or shape information representing the structure or shape of the object from which the noise component has been removed, and in the process of decomposing, the physical movement of the image The physical motion state information of the image input device is measured by a sensor device that operates in synchronization with the motion, and the motion component is corrected using the measured physical motion state information, and the structure or shape component of the object is calculated. A three-dimensional structure acquisition method characterized by restoring and reproducing. 5. An object is photographed by an image input device, the structure or feature points of the object are measured in the photographed two-dimensional image data, the feature points are tracked in time series, and the obtained time-series features are obtained. A computer-readable recording medium storing a program for causing a computer to execute a processing procedure of a three-dimensional structure acquisition method of acquiring a structure or a shape of an object by using a factorization method for point data,
Two-dimensional measurement data composed of the feature point data,
A factorization method for decomposing into motion information representing a physical motion of the image input device from which the noise component has been removed, and structural information or shape information representing the structure or shape of the object from which the noise component has been removed; In the disassembling procedure, the sensor device that operates in synchronization with the physical movement of the image input device, in a coordinate system whose origin is the viewpoint of the image input device, around a horizontal axis of the coordinates of the image input device. A procedure of measuring a rotation angle value, a rotation angle value around a vertical axis, and a rotation angle value around an optical axis as motion information of the image input device, and correcting the motion component using the measured motion information. A computer-readable record of a program for causing a computer to execute a procedure and a procedure for restoring and reproducing a structural component or a shape component of a factorized object. Media. 6. An object is photographed by an image input device, characteristic points constituting the structure or shape of the object are measured in the photographed two-dimensional image data, and the characteristic points are tracked in chronological order. A computer-readable storage medium storing a program for causing a computer to execute a processing procedure of a three-dimensional structure obtaining method for obtaining a structure or a shape of an object using a factorization method on time-series feature point data, , Two-dimensional measurement data composed of the feature point data, by a factorization method, motion information representing the physical motion of the image input device from which noise components have been removed,
A step of decomposing the structure or shape information representing the structure or shape of the object from which the noise component has been removed, and a step of decomposing the image by a sensor device that operates in synchronization with the physical motion of the image input device; A translation component value along the horizontal axis, a translation component value along the vertical axis, and a translation component value along the optical axis of the coordinates of the image input device in a coordinate system whose origin is the viewpoint of the input device. To the computer, a procedure for measuring the motion information as the motion information of the image input device, a procedure for correcting the motion component using the measured motion information, and a procedure for restoring and reproducing the structural component or the shape component of the factorized object. A computer-readable recording medium that stores a program to be executed. 7. An object is photographed by an image input device, characteristic points constituting the structure or shape of the object are measured in the photographed two-dimensional image data, and the characteristic points are tracked in time series. A computer-readable storage medium storing a program for causing a computer to execute a processing procedure of a three-dimensional structure obtaining method for obtaining a structure or a shape of an object using a factorization method on time-series feature point data, , Two-dimensional measurement data composed of the feature point data, by a factorization method, motion information representing the physical motion of the image input device from which noise components have been removed,
A step of decomposing the structure or shape information representing the structure or shape of the object from which the noise component has been removed, and a step of decomposing the object by a sensor device or the like that operates in synchronization with the physical motion of the image input device; In a coordinate system whose origin is the viewpoint of the image input device, a rotation angle value around a horizontal axis, a rotation angle value around a vertical axis, and a rotation angle value around an optical axis of the coordinates of the image input device, A step of measuring a translation component along the horizontal axis of the coordinates of the image input device, a translation component along the vertical axis, and a translation component value along the optical axis as motion information of the image input device; ,
A computer-readable recording medium storing a program for causing a computer to execute a procedure for correcting the motion component using the measured motion information and a procedure for restoring and reproducing a structural component or a shape component of the factorized object. . 8. An object is photographed by an image input device, characteristic points constituting the structure or shape of the object are measured in the photographed two-dimensional image data, and the characteristic points are tracked in a time series. A computer-readable storage medium storing a program for causing a computer to execute a processing procedure of a three-dimensional structure obtaining method for obtaining a structure or a shape of an object using a factorization method on time-series feature point data, , Two-dimensional measurement data composed of the feature point data, by a factorization method, motion information representing the physical motion of the image input device from which noise components have been removed,
A step of decomposing the structure or shape information representing the structure or shape of the object from which the noise component has been removed, and a step of decomposing the two-dimensional measurement data composed of the feature point data from the physical data of the image input device. Measuring the physical motion state information of the image input device by a sensor device operating in synchronization with the dynamic motion, correcting the motion component using the measured physical motion state information, A computer-readable recording medium on which a program for causing a computer to execute a procedure for restoring and reproducing a structural component or a shape component of the computer is recorded. 9. An image input device such as a camera for photographing an object, a measuring means for measuring feature points constituting the structure or shape of the object in two-dimensional image data of the object, and a time series of the measured values. Means for acquiring the structure or shape of an object by using a factorization method on the obtained time-series feature point data using a factorization method. Means for decomposing the constituted two-dimensional measurement data into a component expressing the physical motion of the image input device from which the noise component has been removed and a component expressing the structure or shape of the object from which the noise component has been removed by a factor decomposition method. And a sensor device that operates in synchronization with the physical motion of the image input device, means for correcting a motion component using an output of the sensor device, and a structural component or a shape component of the factorized object. A three-dimensional structure acquisition device comprising means for restoring and reproducing. 10. The three-dimensional structure acquiring apparatus according to claim 9, wherein the sensor device rotates the rotation angle of the coordinates of the image input device around a horizontal axis in a coordinate system whose origin is the viewpoint of the image input device. A three-dimensional structure acquiring apparatus comprising: means for measuring a value, a rotation angle value around a vertical axis, and a rotation angle value around an optical axis as sensor information of physical motion of an image input device. 11. The three-dimensional structure acquiring apparatus according to claim 9, wherein the sensor device translates the image input device coordinates along a horizontal axis in a coordinate system whose origin is the viewpoint of the image input device. A three-dimensional structure acquisition unit comprising means for measuring component values, translation component values along a vertical axis, and translation component values along an optical axis as sensor information of physical movement of the image input device. apparatus. 12. The three-dimensional structure acquisition device according to claim 9, wherein the sensor device is configured to rotate the image input device coordinates around a horizontal axis in a coordinate system whose origin is the viewpoint of the image input device. Values, rotation angle values about the vertical axis, and rotation angle values about the optical axis, and translation component values along the horizontal axis of the image input device coordinates, translation component values along the vertical axis, and And a means for measuring a translation component value along the optical axis. 13. The three-dimensional structure acquiring device according to claim 9, wherein the sensor device measures physical motion state information of the image input device from two-dimensional measurement data composed of the feature point data. A three-dimensional structure obtaining apparatus comprising: means for obtaining a three-dimensional structure.