JPH09212652A - Method for extracting three-dimensional structure information and three-dimensional motion information of moving image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce and stabilize the description amount of parameters for moving image description. SOLUTION: By performing a frame memory processing S1 for input moving image data, an area division processing S2 and a first parameter estimation processing S3, the parameters for respective frames using each two frames are estimated. By a parameter initial value calculation processing S4 and a second parameter estimation processing S5, based on the parameter of a selected reference frame, the estimated parameters are successively reestimated and stabilized. In a parameter judgement processing S6, when the result of re- estimation satisfies set conditions, the frame corresponding to an optimum parameter is selected as the reference frame. By a parameter correction processing S7, based on three-dimensional structure information in the parameter of the reference frame, the three-dimensional structure information and three- dimensional motion information of the other frame are obtained and the result is outputted by an output processing S8.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to three-dimensional structure information and three-dimensional structure of a moving image for extracting three-dimensional structure information (depth information) of a target object and three-dimensional motion information captured from input moving image data. The present invention relates to a method of extracting motion information.

[0002]

2. Description of the Related Art Conventionally, techniques in such a field include:
For example, there was one shown in the following documents. Literature; IEEE Conference on Computer Vision and Patte
rn Recognition (1994) Michal Irani et al. “Recovery of
Ego-Motion Usiug Image Stabilization "P.454-460 In the conventional technique disclosed in the above document, two temporally consecutive frames are used for an input moving image, and a single motion showing the same motion between the frames is used. One motion area is extracted, two-dimensional motion information of the extracted single motion area is assumed to be a plane, and the two-dimensional motion information is used to perform alignment between two frames. By doing so, the rotational motion component of the single motion region is removed and only the translational motion component is extracted.The conventional technique estimates and detects the focus position (Focus of Expansion) of the camera from the extracted translational motion component. 3 of the single motion area based on the focal position
It seeks three-dimensional structure information and three-dimensional motion information. A series of processing for two consecutive frames is sequentially performed to sequentially output three-dimensional motion information and three-dimensional motion information of a moving image.

[0003]

However, the conventional methods for extracting three-dimensional structure information and three-dimensional motion information have the following problems. Since the analysis is performed using only two frames which are temporally continuous, the analysis result tends to be unstable. The estimation results of the three-dimensional structural information analyzed independently for every two frames do not always match,
It is necessary to describe both 3D structural information and 3D motion information of a single motion region for each frame. Therefore, the amount of descriptive information is large, which is inconvenient for describing continuous movement. Moreover, when there are three or more different motions in the moving image, the focus positions of the cameras estimated in the three or more single motion regions corresponding to them do not match, and it is difficult to obtain an accurate focus position. is there. Therefore, the final estimated 3
The three-dimensional structure information and the three-dimensional motion information were insufficient in accuracy.

[0004]

In order to solve the above problems, the first invention takes the following processing in a method of extracting three-dimensional structure information and three-dimensional motion information of a moving image. That is, the input moving image data obtained from the moving image is
A frame memory process of accumulating each in the form of a plurality of frames that make up a temporal section of the moving image, and based on the accumulated moving image data, a single region showing a common motion in the moving image A three-dimensional structure information and a three-dimensional motion information of the moving image for each frame regarding the single moving region are used by using a region extracting process for dividing and extracting as a moving region and moving image data between the two temporally consecutive frames. And a first parameter estimation process for estimating respectively. Then, in the extraction method of the first invention, a parameter initial value calculation process for calculating an initial value for re-estimation for the estimated 3D structure information and 3D motion information for each frame is provided. Re-estimation is performed on the estimated 3D structural information and 3D motion information for each frame using the respective initial values,
A second parameter estimation process for stabilizing the three-dimensional structure information and the three-dimensional motion information, and whether or not the re-estimated plurality of three-dimensional structure information and three-dimensional motion information satisfy the set conditions. And if not satisfied, a parameter determination process for repeatedly executing the re-estimation; and if the parameter determination process determines that the condition is satisfied, the re-estimated three-dimensional structure for each frame A parameter correction process for recalculating and correcting the information and the three-dimensional motion information, and an output for outputting the three-dimensional structure information and the three-dimensional motion information obtained by the parameter correction process as parameters for describing a moving image. Processing and so on.

According to a second aspect of the present invention, the parameter initial value calculation process and the second parameter estimation process of the first aspect are cyclically performed. Then, in the parameter initial value calculation process, first, a frame corresponding to the optimum 3D structure information and 3D motion information of the estimated 3D structure information and 3D motion information for each frame is set as a reference frame, Using the three-dimensional structure information and the three-dimensional motion information of the reference frame, the initial value for the frame that is temporally continuous with the reference frame is calculated and passed to the second parameter estimation process, and thereafter, the second parameter estimation process is performed.
3 for each frame obtained by the parameter estimation process of
Using the re-estimation results of the three-dimensional structure information and three-dimensional motion information,
The initial values for the temporally consecutive frames are calculated in order. In a third aspect of the invention, the parameter correction process in the first or second aspect uses the three-dimensional structure information of a reference frame among the plurality of re-estimated three-dimensional structure information, and uses the three-dimensional structure information of another frame. The three-dimensional structure information and the three-dimensional motion information are recalculated and corrected.

According to the first to third aspects of the invention, as described above, the method of extracting the three-dimensional structure information and the three-dimensional motion information of a moving image is configured, so that the frame memory processing and the area extraction for the input moving image data are performed. By the processing and the first parameter estimation processing, the three-dimensional structure information and the three-dimensional motion information of the moving image for each frame are estimated. These estimated three-dimensional structure information and three-dimensional motion information are unstable because they are estimated from two frames each, but they are re-estimated by the parameter initial value calculation process and the second parameter estimation process, Stabilize. Whether or not the re-estimated three-dimensional structure information and three-dimensional motion information satisfy the set conditions is determined by the parameter determination process. Only when the condition is satisfied, the re-estimated three-dimensional structure information and three-dimensional motion information are corrected by the parameter correction processing, and output by the output processing as a parameter describing the moving image. Therefore, the above problem can be solved.

[0007]

2 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention. This embodiment estimates three-dimensional motion information and three-dimensional structure information necessary to describe a three-dimensional motion and a three-dimensional structure of an object from a moving image. Hereinafter, these three-dimensional motion information and three-dimensional structure information will be described. Information such as dimensional structure information is called a parameter. This image processing device
An image input unit 1 such as a camera for inputting moving image data, and a frame memory 2 connected to the image input unit 1 are provided. The output side of the frame memory 2 is connected to the area extraction unit 3, and the output side of the area extraction unit 3 is connected to the parameter estimation unit 4. The output side of the parameter estimation unit 4 is
The parameter initial value calculation unit 5 is connected, and the output side of the parameter initial value calculation unit 5 is connected to the parameter estimation unit 6. The parameter estimation unit 6 has two output destinations. One of the two is a parameter initial value calculation unit 5, and the other is a parameter determination unit 7. The parameter determination unit 7 also has two output destinations. One of the output destinations of the parameter determination unit 7 is the parameter initial value calculation unit 5, and the other is the parameter correction unit 8. The parameter output unit 9 is connected to the output side of the parameter correction unit 8. FIG. 1 is a flowchart showing a method of extracting the three-dimensional structure information and the three-dimensional motion information, which is implemented by FIG.

FIG. 3 shows the concept of the extraction method of FIG. 1 (No. 1).
FIG. 4 is an explanatory diagram showing the concept (part 2) of the extraction method of FIG. 1. A method of extracting three-dimensional structure information and three-dimensional motion information of a moving image will be described with reference to FIGS. 1, 3, and 4. The image input unit 1 picks up an image of an object, and the moving image data is given to the frame memory 2. In the frame memory 2, the frame memory process S1 of FIG. 1 is performed. In the frame memory processing S1, the frame memory 2 is a frame that represents a moving image in a time section with respect to the given moving image data ... i-2, i-
1, i, i + 1, i + 2, ... (i; arbitrary integer number) are sequentially accumulated. The area division unit 3 performs area division processing S2 based on the accumulated moving image data, and the moving image is divided into single moving areas. Each area that shows a common motion in a moving image is a single motion area. In this area division processing S2, a known method such as an area division method using an optical flow that is a distribution of movement vectors or an area division method using a spatiotemporal image is used.

Next, the parameter estimation unit 4 performs a first parameter estimation process S3 using two temporally consecutive frames for each of the divided single-motion regions, and three-dimensional motion information for each frame is obtained. And estimate three-dimensional structural information.
An example of the estimation method will be described below. The motion of each region is assumed to be a rigid body motion. In the case of a rigid body motion, the three-dimensional motion information includes two components, a translational motion component representing a translation amount and a rotational motion component. Here, the three-dimensional structure information to be obtained is a depth value based on the focal length of each point on the two-dimensional projection plane. A translational motion component at a certain time at a point X ^* = (X, Y, Z) ^T in a three-dimensional space is t ^* = (t _X , t _Y , t _Z ).
_Let ^{T be} the rotational motion component w ^* = (w _X , w _Y , w _Z ) ^T. However, " ^* " shows a vector symbol and " ^T " shows transposition. The rigid body motion model U ^* can be expressed by the following (1). U ^* = t ^* + w ^** X ^* ... (1) U ^* = (U, V, W) ^T is a three-dimensional movement at the point X ^* . Point x on the plane of the two-dimensional projection image with three-dimensional coordinates X ^*
The movement in ^* = (x, y) u ^* = (u, v) ^T is x
= FX / Z, y = fY / Z (f; focal length)
It is expressed by the following equation (2) by time differentiation. However,
"'" Represents time differentiation.

[0010]

[Equation 1] The parameter estimation unit 4 describes the movement of the points in the area using the equation (2), and uses a nonlinear iterative minimization method such as the Gauss-Newton method so that the inter-frame difference using the movement is minimized. Therefore, the three-dimensional motion information t ^* , w ^* and the depth information 1 / Z (x ^* ), which are parameters, are estimated. It is also possible to give a condition to the depth information 1 / Z (x ^* ) such that the target object is a plane or a sphere, and apply the restriction to the three-dimensional structure information for estimation.

According to the method described above, the parameter estimation unit 4 sets 2
Using three temporally continuous frames, the three-dimensional motion information t ^* , w ^* and the three-dimensional structure information 1 / which are the parameters P are calculated.
The process of obtaining Z (x ^* ) is performed on all the frames stored in the frame memory. Here, as an initial value for performing this calculation, a general value without any precondition is set. For example, the initial depth information 1 / Z
(X ^* ) is set to infinity and the initial motion is set to none, and then estimated. If an iterative algorithm is used when estimating the parameters in the subsequent processing, the result may change significantly depending on the initial value given first, and it may be difficult to obtain a stable result. Therefore, the parameter initial value calculation unit 5
Is the parameter of the most stable frame among the parameters of FIG. 3 estimated by the parameter estimation unit 4, ..., P _i-2 , P _i-1 , P _i , P _{i + 1} , P _{i + 2} ,. Choose. The most stable parameter has the smallest error obtained by the estimation using the equation (2). Parameter initial value calculation unit 5
Uses the parameters of the selected reference frame and calculates initial values for re-estimation of the parameters corresponding to the temporally consecutive frames. This process is the parameter initial value calculation process S4, and the calculation result is given to the parameter estimation unit 6. The parameter estimation unit 6 performs the second parameter estimation process S5 to re-estimate the parameters of the frame next to the reference frame. This re-estimation result is given to the parameter initial value calculation unit 5 again. The parameter initial value calculation unit 5 calculates the parameter initial value of the next consecutive frame using the re-estimated frame parameters. In this way, the parameter initial value calculation process S4 and the second parameter estimation process S5 are repeated to obtain the parameters for all the frames. These stabilize the parameter estimation of all frames.

For example, the parameters estimated by the parameter estimation unit 4 ... P _i-2 , P _i-1 , P _i , P _{i + 1} , P _{i + 2} ,
When the error of the parameter of the frame j (j; arbitrary integer) is the smallest and is the most stable among the ..., The parameters are re-estimated using the frame j as the reference frame. The parameter initial value calculation unit 5 calculates the initial value of the parameter of the frame j + 1 from the parameter P _j of the frame j and gives it to the parameter estimation unit 6, and the parameter estimation unit 6 estimates the parameter p _{j + 1} of the frame j + 1. . Further, the initial value of the frame j + 2 is calculated using the parameter p _{j + 1} of the frame j + 1 obtained by the parameter estimation unit 6. In this way, the estimation and the initial value calculation are alternately performed to sequentially re-estimate the parameters of consecutive frames. As for the parameters of the frame before the reference frame j, three-dimensional motion information is described starting from the reference frame j, so that the direction opposite to the direction estimated by the parameter estimation unit 6, that is, from the frame at the time later in time, It will estimate the motion parameters for the previous time frame. Therefore,
The parameter p _{j of the} frame j is the three-dimensional motion information parameter from time t _j to t _j-1 and 3 at the time t _j-1 .
Means dimensional structure information. Furthermore, the initial value of the previous frame is calculated, and the parameter estimation in the reverse direction is repeated.
In this way, the initial value of the re-estimation is calculated using a certain frame as the reference frame j, and the parameter estimation of other frames and the setting of the initial value are sequentially repeated, so that stable parameters of all frames, ..., P _{j -2} ,
It is possible to obtain p _j-1 , p _j , p _{j + 1} , p _{j + 2} , ....

Here, the parameter estimation unit 6 uses the initial values given by the parameter initial value calculation unit 5 as described above, and in the same manner as the parameter estimation unit 4, the three-dimensional motion information t ^* , w ^*. And 3D structure information 1 / Z (x ^* )
Is estimated. By giving a stable initial value, the convergence of the calculation result of parameter estimation is accelerated, and a stable result can be obtained. Re-estimated parameters ..., p
_{The j-2} , p _j-1 , p _j , p _{j + 1} , p _{j + 2} , ... _Are given to the parameter determination unit 7, and the parameter determination unit 7 performs the parameter determination process S6. . First, the parameter determination unit 7 determines the parameters re-estimated by the parameter estimation unit 6 ... P _j-2 , p _j-1 , p _j , p _{j + 1} ,
It is determined whether or not the set stability condition is satisfied for p _{j + 2} , .... If not satisfied, the parameter initial value estimation unit 5 re-estimates the parameters ...
Among p _j-2 , p _j-1 , p _j , p _{j + 1} , p _{j + 2} , ..., the most stable frame k with the smallest error (k; any integer)
Parameter p _k is sent and the parameter stabilization process S5 is performed again. When the set stability condition is satisfied, the parameter determination unit 7 determines the parameters ... _Pj-2 , p.
_{Of the j-1} , p _j , p _{j + 1} , p _{j + 2} , ..., The parameter p _k of the frame k having the smallest error and the most stable is sent to the parameter correction unit 8.

The parameter correction unit 8 performs the parameter correction processing S7, and the re-estimated parameters ... P _j-2 p
_j-1 , p _j , p _{j + 1} , p _{j + 2} , ... Are corrected by recalculation.
That is, in the parameter correction unit 8, the frame k corresponding to the parameter p _k output from the parameter determination unit 7 is used as the reference frame. Three-dimensional structure information Z of reference frame k
_{Based on k} (= 1 / Z _k (x ^* )), the parameter correction unit 8 receives each frame, ..., K-2, k-1, as shown in FIG.
Translational motion information, which is a motion parameter of k, k + 1, k + 2, ..., t ^* _k-2 , t ^* _k-1 , t ^* _k , t ^* _{k + 1} , t.
^* _{k + 2} , ... and rotational motion information ..., w ^* _k-2 , w ^* _k-1 , w
^{Recalculates *} _k , w ^* _{k + 1} , w ^* _{k + 2} , ... And outputs the result to the parameter output unit 9. The method of recalculation uses the same method as that of the parameter estimation unit 4, but the three-dimensional structure information Z _k of the frame _k is not updated with the given initial value during the iterative calculation and the translation of the three-dimensional motion information is performed. Motion information ..., t ^* _k-2 , t ^* _k-1 , t ^* _k , t ^* _{k + 1} ,
t ^* _{k + 2} , ... and rotational movement information ..., w ^* _k-2 ,
Only w ^* _k-1 , w ^* _k , w ^* _{k + 1} , w ^* _{k + 2} , ... Are updated to calculate the parameters. The correction result given to the parameter output unit 9 is the three-dimensional structure information Z _k of one reference frame _k and each frame ..., K-2, k-1, k +.
Translation information of 1, k + 2, ..., t ^* _k-2 ,
t ^* _k-1 , t ^* , t ^* _{k + 1} , t ^* _{k + 2} , ... And rotational motion information ..., W ^* _k-2 , w ^* _k-1 , w ^* _k , w ^* _{k + 1.} , W ^*
_{k + 2} , and so on.

When the three-dimensional structure information Z _k of the frame k is given, and if the three-dimensional motion information t ^* _{k + 1} and w ^* _{k + 1} from the frame k to the frame k + 1 are given, 3 of the frame k + 1 is given. The dimensional structure information Z _{k + 1} can be calculated. Also regarding the three-dimensional structure information Z _{k + 2} of the frame k + 2, the three-dimensional motion information t ^* from the frame k + 1 to the frame k + 2 is obtained ^.
_It can be calculated if _{k + 2} and w ^* _{k + 2} are given. On the contrary, for the frame kl direction, similarly to the parameter estimation unit 6, it is possible to sequentially calculate the three-dimensional structural information by using the three-dimensional motion information in the temporally opposite directions. By obtaining the three-dimensional motion information based on the three-dimensional structure information Z _k of one reference frame k, the 3
Can describe 3D structure information and 3D motion information,
It is possible to reconstruct the moving image of the extracted area. The parameter output unit 9 performs the output process S8 to generate the three-dimensional structure information Z.
_k and translational motion information ..., t ^* _k-2 , t ^* _k-1 , t ^* , t
^* _{k + 1} , t ^* _{k + 2} , ... and rotational motion information ..., w ^* _k-2 ,
w ^* _k-1 , w ^* _k , w ^* _{k + 1} , w ^* _{k + 2} , ... Are output as parameters for describing a moving image. As described above, according to the present embodiment, the parameters of the three-dimensional structure information and the three-dimensional motion information, estimated from two frames each, ..., P _i-2 , P _i-1 , P _i , P _{i +. Since} the parameter initial value calculation process S4 to the output process S8 are performed for ₁ , P _{i + 2} , ..., Stable three-dimensional motion information and three-dimensional structure information using a plurality of consecutive frames are obtained. You can Also, conventionally, 3D structure information and 3
It was necessary to describe the dimensions motion information, based on the 3-dimensional structural information Z _k of frame k of the l one criterion, the translational motion information of the three-dimensional motion information _{^{..., t * k-2,}} t * k _{- 1,}
t ^* , t ^* _{k + 1} , t ^* _{k + 2} , ... And rotational motion information ..., w
^{Since *} _k-2 , w ^* _k-1 , w ^* _k , w ^* _{k + 1} , w ^* _{k + 2} , ... _Are obtained by recalculation, one three-dimensional structural information Z
_{With k} and the 3D motion information for each frame, it is possible to easily describe the 3D structure and 3D motion of a moving image with a small amount of description.

The present invention is not limited to the above embodiment, and various modifications can be made. For example, there are the following modifications. (1) In FIG. 2, the area extraction unit 3 to the parameter output unit 9
However, one or an arbitrary number of CPUs may be combined and operated by software. (2) In the present embodiment, a nonlinear iterative minimization method based on the Gauss-Newton method is used.
Although an example of estimating parameters has been described, the estimation may be performed using a non-linear minimization method using another gradient method or the like.

[0017]

As described in detail above, the first to third embodiments
According to the invention, the three-dimensional structure information of the moving image for each frame and the 3
The dimensional motion information is estimated and re-estimated by the parameter initial value calculation process and the second parameter estimation process to stabilize. Then, in the parameter determination processing re-estimation,
It is determined whether or not the re-estimation result satisfies the set condition, and only when the condition is satisfied, the re-estimated three-dimensional structure information and three-dimensional motion information are corrected, and the output process is performed. , Outputs parameters that describe a moving image. Therefore, stable three-dimensional motion information and three-dimensional structure information can be obtained. Conventionally, 1
It was necessary to describe the 3D structure information and the 3D motion information for each frame, but by recalculating the 3D motion information based on the 3D structure information of one reference frame, there are few descriptions. It is possible to easily describe the three-dimensional structure and three-dimensional motion of a moving image by using the amount, that is, the three-dimensional structure information of the reference frame and the three-dimensional motion information of each frame.

[Brief description of drawings]

FIG. 1 is a flowchart showing a method for extracting 3D structure information and 3D motion information, which is implemented in FIG.

FIG. 2 is a functional block diagram of the image processing apparatus according to the embodiment of the present invention.

FIG. 3 is an explanatory diagram showing the concept (1) of the extraction method of FIG. 1.

FIG. 4 is an explanatory diagram showing the concept (No. 2) of the extraction method of FIG.

[Explanation of symbols]

 1 Image Input Section 2 Frame Memory 3 Area Extraction Section 4, 6 Parameter Estimation Section 5 Parameter Initial Value Calculation Section 7 Parameter Judgment Section 8 Parameter Correction Section 9 Parameter Output Section S1 Frame Memory Processing S2 Area Division Processing S3 First Parameter Estimation Processing S4 parameter initial value calculation processing S5 second parameter estimation processing S6 parameter determination processing S7 parameter correction processing S8 output processing

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Masahide Kaneko 23-1319, Shiba, Minato-ku, Tokyo Communication / Broadcasting Organization (72) Takahiro Saito Inventor, Takahiro Saito 2-31-19, Shiba, Minato-ku, Tokyo Broadcasting system

Claims (3)

[Claims] 1. A frame memory process for accumulating input moving image data obtained from a moving image in the form of a plurality of frames, each of which constitutes a temporal section of the moving image, and the accumulated moving image. Based on the data, a region extraction process for dividing and extracting a region showing a common motion in the moving image as a single-moving region, and moving image data between two temporally consecutive frames are used to perform the single moving region. Parameter estimation processing for respectively estimating the three-dimensional structure information and the three-dimensional motion information of the moving image for each frame, and the initial value for re-estimation of the estimated three-dimensional structure information and the three-dimensional motion information Parameter initial value calculation processing for calculating each of the frames, and the estimated three-dimensional structure information for each frame and 3 using the given initial values. Second parameter estimation processing for performing re-estimation on the original motion information and stabilizing the three-dimensional structure information and the three-dimensional motion information, and the plurality of re-estimated three-dimensional structure information and three-dimensional motion information Determines whether or not the set condition is satisfied, and if not satisfied, a parameter determination process that repeatedly executes the re-estimation, and if it is determined that the condition is satisfied in the parameter determination process, Parameter correction processing for recalculating and correcting the three-dimensional structure information and three-dimensional motion information for each frame re-estimated, and the three-dimensional structure information and three-dimensional motion information obtained by the parameter correction processing, An output process for outputting a moving image as a parameter to describe the moving image, and extracting the three-dimensional structure information and the three-dimensional motion information of the moving image. Method. 2. The parameter initial value calculation process and the second parameter estimation process are cyclically performed, and the parameter initial value calculation process is initially performed by the estimated three-dimensional structure information for each frame. And a frame corresponding to the optimum three-dimensional structure information and three-dimensional motion information among the three-dimensional motion information is used as a reference frame, and the three-dimensional structure information and three-dimensional motion information of the reference frame are used to temporally change the reference frame. Calculating the initial value for successive frames and passing it to the second parameter estimation process,
Using the result of re-estimation of the three-dimensional structure information and the three-dimensional motion information for each frame obtained by the second parameter estimation processing, the initial value for temporally consecutive frames is sequentially calculated. The method for extracting 3D structure information and 3D motion information of a moving image according to claim 1. 3. The parameter correction processing uses the three-dimensional structure information of a reference frame among the plurality of re-estimated three-dimensional structure information, and uses the three-dimensional structure information and the three-dimensional motion information of another frame. 3. The method for extracting three-dimensional structure information and three-dimensional motion information of a moving image according to claim 1 or 2, further comprising: