JPH08210815A - Method and device for extracting three-dimensional motion information - Google Patents

Abstract

PURPOSE: To provide a method and device for extracting three-dimensional motion information for measuring the motion vector of a motional object. CONSTITUTION: First, the multi-directional image information of a measuring object X is collected and processed by image input means A1 -A3 and an image accumulation means B and then the multi-directional collection processing image information is integrated by a two-dimensional motion vector extraction means C and a three-dimensional position coordinate extraction means D. After that, an association processing in time direction of integration processing information is performed by a two-dimensional motion vector inverse-projection means E and further the three-dimensional motion information of a motional measuring object X is extracted and processed from the association processing information by a three-dimensional motion vector extraction means F and an output means G.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is used to extract three-dimensional behavior information of a structure moving in a three-dimensional space such as a machine, a vehicle, a person, an animal, etc. by using a captured moving image. In particular, the present invention relates to a three-dimensional motion information extraction method that can be used in the field of image recognition of a moving object, image analysis field, and visual control of a robot, and a device directly used for its implementation.

[0002]

2. Description of the Related Art Conventionally, there are various methods for measuring the three-dimensional coordinates of a measuring object having a three-dimensional structure. One of them is a stereo vision method using two cameras (for example, Taniuchida: 3D position / shape measurement using vision and its application, system and control, Vol.29, No.10, pp.631-638, 1985)
Is proposed. This method is a method of measuring the position of an object from the images of the object projected on two images captured from two different positions, using the principle of triangulation.

As another method, there is a method of extracting an edge on an image and using it to backproject to the scene space to restore the edge of a three-dimensional object (for example,
T.Hamano, T.Yasuno, K.Ishii: “Direct Estimation
of Struature from Non-linear Motion by Voting Algo
rithm without Tracking and Matching "Proc. ICP
R, Vol. 1, pp. 505-508, 1992., and 1994 patent application No. 137254).

[0004]

By the way, in the above-mentioned conventional stereoscopic measurement and backprojection measurement, it is difficult to track the three-dimensional motion because the time directions cannot be associated with each other. There was a point.

The present invention has been made to solve the above problems, and the main objects to be solved by the present invention are as follows. That is, a first object of the present invention is to provide a three-dimensional motion information extraction method and device for measuring a motion vector of a moving object.

A second object of the present invention is to provide a three-dimensional motion information extraction method and device capable of associating in the time direction.

A third object of the present invention is to provide a three-dimensional motion information extraction method and device which enables tracking of three-dimensional motion.

Other objects of the present invention will be apparent from the description and the accompanying drawings, especially the description of the claims.

[0009]

In order to solve the above-mentioned problems, the above-mentioned object is achieved by adopting the novel characteristic methods and means listed in the following by the present invention. That is, the first feature of the method of the present invention is that the multi-directional information of the measurement object is first collected, and then the multi-directional collected processing information is integrated.
Subsequently, the integrated processing information is associated with the time direction association processing, and the extraction processing is performed on the three-dimensional motion information of the moving measurement object from the association processing information 3
There is a method of extracting dimensional movement information.

A second feature of the method of the present invention is a three-dimensional motion information extraction method in which the multidirectional information in the first feature of the method of the present invention is image information.

A third feature of the method of the present invention is that the multi-directional information collecting process in the first or second feature of the method of the present invention captures image information from a plurality of directions of the object to be measured, This is a three-dimensional motion information extraction method in which step II for accumulating the captured continuous image information and step S are sequentially executed.

The fourth feature of the method of the present invention is that the integration processing in the first, second or third feature of the method of the present invention is
Step I of extracting the two-dimensional motion vector information on the two-dimensional image of each part of the measurement object accumulated in step II
In the three-dimensional motion information extraction method, step II and step IV of extracting the three-dimensional position coordinate information of the measurement object from the plurality of image information accumulated in step II are sequentially executed.

A fifth feature of the present invention is that the process of associating in the time direction in the first, second, third or fourth feature of the method of the present invention is the two-dimensional motion vector information in step III and step IV. In the method for extracting three-dimensional motion information, the step V of back-projecting the two-dimensional motion vector into the virtual three-dimensional space is executed by using the three-dimensional position coordinate information.

The sixth feature of the method of the present invention is 3 in the first, second, third, fourth or fifth feature of the method of the present invention.
The step of extracting the three-dimensional motion vector information from the virtual three-dimensional space in which the two-dimensional motion vector is backprojected in step V, and the step of outputting the extracted three-dimensional motion vector information VII,
In the three-dimensional motion information extraction method.

The seventh characteristic of the method of the present invention is the step III in the fourth, fifth or sixth characteristic of the method of the present invention.
And step IV are the three-dimensional motion information extraction method which is performed by parallel processing at the same time.

An eighth feature of the method of the present invention is that step VI in the sixth feature of the method of the present invention uses a fixed viewpoint two-dimensional motion information on a two-dimensional image and moving position information to measure an object to be measured. 3 by extracting the three-dimensional motion vector information of
There is a method of extracting dimensional movement information.

The ninth feature of the method of the present invention is that the step V in the fourth, fifth, sixth, seventh or eighth feature of the method of the present invention first detects one point on the measuring object X. If P

[Outside 3]

[Outside 4] To obtain the three-dimensional coordinate value of the point P ′, and further, from the coordinate values of the points P and P ′, the three-dimensional motion vector V = P′−
It is a three-dimensional motion information extraction method in which P is calculated and extracted.

The apparatus of the present invention is characterized by an image storing means for storing image information of images taken from a plurality of directions of an object, and a two-dimensional movement of an object on a two-dimensional image from a plurality of images input from the image storing means. Two-dimensional motion vector information extraction means for extracting vector information, three-dimensional position coordinate extraction means for extracting three-dimensional position coordinate information of an object from a plurality of images input from the image storage means, and the two-dimensional motion vector A two-dimensional motion for back-projecting the two-dimensional motion vector information of the object into a three-dimensional virtual space using the two-dimensional motion vector information and the three-dimensional position coordinate information input from the information extracting means and the three-dimensional position coordinate extracting means, respectively. Vector backprojection means and three-dimensional motion for extracting three-dimensional motion vector information from the three-dimensional virtual space input from the two-dimensional motion vector backprojection means. A vector extracting means, in the three-dimensional movement information extracting device including an output means for outputting the 3-dimensional motion vector information extracted in the 3-dimensional motion vector extracting means.

[0019]

Since the present invention has taken the novel method and means as described above, the two-dimensional motion vector information extracting means for executing step III of the integration processing performs the image processing for executing step II of the collection processing. Two-dimensional motion vector information on a two-dimensional image of an object is extracted from a plurality of images as multidirectional information input from the storage means.

On the other hand, the three-dimensional position coordinate extraction means for executing the step IV of the integration processing extracts the three-dimensional position coordinate information of the object from the plurality of images input from the image storage means. Then, the two-dimensional motion vector back-projecting means that executes the step V of the time-direction associating processing extracts the two-dimensional motion vector information input from the two-dimensional motion vector information extracting means and the three-dimensional position coordinate extraction. The two-dimensional motion vector information of the object is back projected onto the three-dimensional virtual space by using the three-dimensional position coordinate information input from the means.

The three-dimensional motion vector extraction means for executing step VI of the three-dimensional motion information extraction force processing is the three-dimensional motion vector from the three-dimensional virtual space input from the two-dimensional motion vector backprojection means. Extracting information. Therefore, the three-dimensional motion vector information of the moving object can be extracted. Therefore, the three-dimensional motion vector information of the moving object can be accurately integrated by integrating the information from multiple directions and can be associated with the time direction.

[0022]

【Example】

(Apparatus Example) Hereinafter, an apparatus example of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a three-dimensional motion information extraction device of this device example.

In FIG. 1, A1, A2 and A3 are measurement objects X.
Is an image input means such as a camera, a pick-up tube, and a photographing device for inputting an image of an object. B is an image storage unit that stores a plurality of images input from the image input units A1 to A3. C is a two-dimensional motion vector extraction means for extracting the two-dimensional motion vector information on the two-dimensional image of the measurement object X and the background object from the plurality of images input from the image storage means B. Reference numeral D is a three-dimensional position coordinate extracting means for extracting the three-dimensional position coordinate information of the measuring object X using a plurality of images input from the image storage means B.

E is a three-dimensional position coordinate of the two-dimensional motion vector using the two-dimensional motion vector information extracted by the two-dimensional motion vector extracting means C and the three-dimensional position coordinate information extracted by the three-dimensional position coordinate extracting means D. 3 extracted by the extraction means D
It is a two-dimensional motion vector back-projection means that back-projects a two-dimensional motion vector into a three-dimensional virtual space using the three-dimensional position coordinate information. F is a three-dimensional motion vector extracting means for extracting three-dimensional motion vector information from the three-dimensional virtual space in which the two-dimensional motion vector back-projecting means E back-projects the two-dimensional motion vector. Reference symbol G denotes an output unit that outputs the three-dimensional motion vector information extracted by the three-dimensional motion vector extraction unit F.

(Example of Method) The execution processing procedure of the example of the method of the present invention applied to the three-dimensional information extracting apparatus of the present example will be described in detail with reference to FIG. FIG. 2 is a flowchart of the execution processing procedure of this method example. First, as an execution procedure of the collection processing of multi-directional image information, the measurement target X is input to the image input means A1 to A3.
Image information obtained from a plurality of directions is input (step I). The image input means A1 to A3 input the image to the image storage means B to store the continuous image information of the measurement object X (step II).

Next, as the execution procedure of the integration processing of the accumulated multi-directional image information, the two-dimensional motion vector extraction means C selects each part of the measurement object X from the plurality of images input from the image storage means B. 2D motion vector information on the 2D image is extracted (step III). In the two-dimensional motion vector extraction means C, a method of obtaining two-dimensional motion vector information on a two-dimensional image, that is, an optical flow from a plurality of images is described in, for example, “EJ Gibson,
JJ Gibson, O.I. W. Smith and H.M. Flock: Mot
ion Parallax as a Determinant of Perceived D
epth, Journal of Experimental Psychology, Vol.
8, No. 1, pp. 40-51, 1959 “.

Simultaneously with step III, the three-dimensional position coordinate extracting means D uses the plurality of images input from the image accumulating means B and photographed from a plurality of directions to measure the measurement object X.
The three-dimensional position coordinate information of is extracted (step IV). Three
The method of extracting the three-dimensional position coordinate information of the measurement object X from the plurality of images photographed from a plurality of directions in the dimensional position coordinate extracting means D is, for example, “Japanese Patent Application No. 137, 1994”.
No. 254 ".

Subsequently, as an execution procedure of a time-direction associating process with respect to the integrated process information processed in parallel and simultaneously,
The two-dimensional motion vector backprojection means E uses the two-dimensional motion vector information input from the two-dimensional motion vector extraction means C and the three-dimensional position coordinate information input from the three-dimensional position coordinate extraction means D to generate a two-dimensional motion vector. Is back projected onto the virtual three-dimensional space (step V). At this time, the coordinate system is defined as shown in FIG. In the coordinate system, P is one point on the measurement object X. Here, the two-dimensional motion vector extraction hand

[Outside 5]

[Outside 6]

[Outside 7] According to the method described in Japanese Patent No. 137254 ", the three-dimensional coordinate of P'is obtained by back-projecting into the virtual three-dimensional space.

Further, the three-dimensional motion vector extraction means F has two steps as an execution procedure of the extraction force processing of the three-dimensional motion information of the measuring object X from the correlation processing information in the time direction.
A three-dimensional motion vector V = P'-P is calculated from the coordinates of the points P and P'input from the three-dimensional motion vector backprojecting means E (step VI). That is, using the fixed viewpoint two-dimensional motion information and the movement position information on the two-dimensional image, the measurement object X is measured.
3D motion vector information is extracted. Then, the output unit G outputs the three-dimensional motion vector information extracted by the three-dimensional motion vector extraction unit F (step VII).

As can be seen from the above description, according to the present embodiment, in the measurement of the three-dimensional movement of the object by a plurality of cameras or the like, it is possible to measure the movement of the measurement object X in time. Further, it becomes possible to extract highly accurate three-dimensional motion vector information by integrating information from multiple directions.
Although the present invention has been specifically described based on the embodiments, the present invention is not limited to the above embodiments, and it goes without saying that various modifications can be made without departing from the scope of the invention.

[0031]

As described above, according to the present invention, in measuring the three-dimensional movement of an object by a plurality of cameras, etc.,
It is possible to measure the motion of the measurement target object from multiple directions in correspondence with the time direction, and it is possible to extract highly accurate three-dimensional motion vector information.

Further, according to the present invention, for computer animation and three-dimensional motion analysis, it is possible to accurately input three-dimensional motion information of a moving object such as an animal such as a human being or a car, and further, the car automatically. It has the advantage that it can be applied to driving and visual control of robots.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of a three-dimensional motion information extraction device according to an example of the present invention.

FIG. 2 is a flowchart showing an execution processing procedure of a method example of the present invention.

FIG. 3 is a diagram showing a coordinate system used in the execution processing of the above.

[Explanation of symbols]

 A1 ... Image input means A2 ... Image input means A3 ... Image input means B ... Image storage means C ... Two-dimensional motion vector extraction means D ... Three-dimensional position coordinate extraction means E ... Two-dimensional motion vector back-projection means F ... Three-dimensional motion Vector extraction means G ... Output means X ... Measurement object

Claims (10)

[Claims] 1. First, multi-directional information of a measurement object is collected, then the multi-directional collected processing information is integrated, and subsequently, the integrated processing information is associated with the time direction, Furthermore, the three-dimensional motion information extraction method is characterized in that the three-dimensional motion information of the moving measurement target is subjected to extraction force processing from the association processing information. 2. The three-dimensional motion information extraction method according to claim 1, wherein the multidirectional information is image information. 3. A multi-directional information collecting process includes: step I of capturing image information of a measurement object from a plurality of directions; and step II of accumulating the captured continuous image information.
3. The three-dimensional motion information extracting method according to claim 1 or 2, wherein the steps are sequentially executed. 4. The integration process is a step I of extracting two-dimensional motion vector information on a two-dimensional image of each part of the measurement object accumulated in step II.
II, and step I of extracting three-dimensional position coordinate information of the measurement target from the plurality of image information accumulated in step II.
4. The three-dimensional motion information extraction method according to claim 1, wherein V and are sequentially stepped to perform the execution process. 5. The process of associating in the time direction includes a step V of back-projecting the two-dimensional motion vector into a virtual three-dimensional space by using the two-dimensional motion vector information in the step III and the three-dimensional position coordinate information in the step IV. The three-dimensional motion information extracting method according to claim 1, 2, 3 or 4, wherein the method is executed. 6. The three-dimensional motion information extraction force processing includes a step VI of extracting three-dimensional motion vector information from a virtual three-dimensional space obtained by back-projecting the two-dimensional motion vector in step V, and the extracted three-dimensional motion. The step VII of outputting vector information and the step are sequentially executed to execute the processing. 3. The method according to claim 1, 2, 3, 4 or 5.
Dimensional motion information extraction method. 7. The three-dimensional motion information extraction method according to claim 4, 5 or 6, wherein step III and step IV are performed in parallel. 8. The step VI extracts the three-dimensional motion vector information of the measurement object using the fixed viewpoint two-dimensional motion information on the two-dimensional image and the movement position information. The three-dimensional motion information extraction method described. 9. In step V, first, a point on the measurement object X is set to P, and a photographing table of the point P on the two-dimensional image is obtained. [Outside 2] 6. A three-dimensional coordinate value of the point P'is obtained, and a three-dimensional motion vector V = P'-P is calculated and extracted from the coordinate values of the point P and the point P '. , 6, 7 or 8 three-dimensional motion information extraction method. 10. An image storage unit for storing image information of images of an object taken from a plurality of directions, and extracting two-dimensional motion vector information on a two-dimensional image of the object from a plurality of images input from the image storage unit. Two
Dimensional motion vector information extraction means, three-dimensional position coordinate extraction means for extracting three-dimensional position coordinate information of an object from a plurality of images input from the image storage means, the two-dimensional motion vector information extraction means and the three The two-dimensional motion vector information and the three-dimensional position coordinate information input from the three-dimensional position coordinate extraction means are used to determine the object 2
Backproject 3D motion vector information to 3D virtual space 2
Dimensional motion vector backprojection means, three-dimensional motion vector extraction means for extracting three-dimensional motion vector information from the three-dimensional virtual space input from the two-dimensional motion vector backprojection means, and extraction by the three-dimensional motion vector extraction means An output unit configured to output the generated three-dimensional motion vector information, and a three-dimensional motion information extraction device.