JPH1115953A - Camera operation estimating method, and recording medium recorded with camera operation estimation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correctly decide the validity/invality of a camera parameter regard lessly of the degree of complexity of an image by making an estimated camera parameter invalid when the ratio of an error amount after camera operation compensation to an error amount before compensation is not the given threshold or below. SOLUTION: A variable t that stores a frame number is initialized to one and a variable v that stores plural camera parameters is initialized to null (ϕ) (S101). A parameter b(t)=(bx (t), by (t)) is estimated from two images ft (x and y) and ft-1 (x, y) (S102). An error amount MET(t) after camera operation compensation is performed by using the estimated parameter b(t) is calculated with a prescribed expression (S103). Further, an error amount MSE0 (t) when camera compensation is not performed is calculated through the prescribed expression. When all prescribed conditions are satisfied, a camera parameter rid decided as valid (S104 to S106).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for estimating a camera operation from a sequence of image data constituting a video, and a recording medium storing a camera operation estimation program. The estimated camera parameters are stored in a video database and are used when searching or displaying.

[0002]

2. Description of the Related Art Various methods have been disclosed for estimating camera parameters in order to analyze a camera operation from an image sequence. By storing the estimated camera parameters in the database in association with the video,
It is possible to respond to a search request such as “show the camera panned”.

Further, a panoramic image can be constructed by synthesizing an image sequence while performing conversion in accordance with the estimated camera parameters (Japanese Patent Laid-Open No. 5-304675,
"Method and apparatus for obtaining high-resolution still images using multiple images with different focal lengths or different viewing zones").

FIG. 4 is a diagram for explaining a panoramic image synthesizing process. Four images 91 to 9 as shown in FIG.
Consider four. Comparing the image 91 and the image 92, it can be seen that the image has shifted to the left because the camera has moved slightly to the right. Here, it is assumed that there is a movement of 10 pixels.
This translation amount b (1) = (10, 0) is called a camera parameter. In this way, the camera operation of rotating in the horizontal direction (x-axis direction) while keeping the optical axis of the camera fixed is called "pan"
Call.

The parallel movement vector b (2) = (10, 0) can be similarly estimated between the image 92 and the image 93. The camera moves upward between the image 93 and the image 94, and the camera parameter b (3) = (0,
5). The camera operation of rotating the camera in the vertical direction (y-axis direction) while keeping the optical axis of the camera fixed is called "tilt".

The estimated three camera parameters b
By synthesizing the images while shifting them according to (t), a panoramic image 95 can be configured. A conventional technique of camera parameter estimation will be described.

(1) Camera Model First, it is necessary to set a camera model in order to define camera parameters. The camera model is a mathematical model that indicates to which point in the image at the next time a point in the two-dimensional image moves.

[0008] translation model translation model, a point on one image (x, y) point of another image (x ', y') _{to, x '= x + b x} , y' = y + b y, in conversion It is to be done. This is a model with a point in the image by moving the camera, all (b _x, b _y) only moves in parallel. Assuming this model, there are two camera parameters (b _x , b _y ).

Projection Model A projection model is a model more faithful to an actual camera (Brown, LG: A Survey of Image Registration Te
chniques. ACM Computing Surveys, Vol.24, No.4, pp.
325-376).

X ′ = (a ₁ x + a ₂ y + a ₃ ) / (a ₄ x + a ₅ y +
1), y ′ = (a ₆ x + a ₇ y + a ₈ ) / (a ₄ x + a ₅ y +
1), This model has eight parameters.

Other examples include an affine model and a bilinear model. (2) Camera Parameter Estimation Method Error Scale The following mean square error (MSE) is often used to estimate the degree of coincidence (error) between two images.

MSE (t) = Σx _{, y} ｛f (x, y) -f '
(x ′, y ′)｝ ² / N where the sum (Σ _{x, y} ) is calculated for all corresponding pixels, where N is the number of corresponding pixels. In addition to MSE, there is also an M estimation method that introduces the concept of robust statistics.

Optimization Method The problem of estimating camera parameters from actual two images can be reduced to an optimization problem of finding camera parameters that minimizes the amount of error. A simple method of simply calculating an error amount for all possible combinations of camera parameters and outputting a parameter that achieves the minimum value may be used, but the problem is that the processing cost is enormous.

Therefore, an image with a stepwise reduced resolution (called a pyramid image) is created, and camera parameters are estimated using the image with the lowest resolution. There is a method for efficiently and accurately estimating parameters by repeating the process of obtaining the optimal solution for high-resolution images while gradually increasing the resolution (HSSawhne
y and S. Ayer: CompactRepresentation of Videos Thro
ugh Dominant and Multiple Motion Estimation. IEEE
Transactions on Pattren Analysis and Machine Intel
ligence, Vol. 18, No. 8, pp. 814-830, 1996).

[0015]

The above-described camera operation estimating method is used when an image includes a change that is not assumed by a camera model, such as a movement of a subject or a change of a scene.
May output wrong camera parameters.

For example, if a small insect flies from the left to the right of the screen in an image of a deep blue sky with the camera fixed, the camera may have moved without actually moving. Incorrect recognition may output meaningless camera parameters.

As described above, in the conventional camera operation estimation method, there is no means for knowing whether or not the output camera parameters are reliable. If the camera parameters that were erroneously estimated are trusted and used for applications such as panoramic image synthesis, a problem arises as a result. For example, there is a problem that a distorted image is output when a panoramic image is synthesized using camera parameters that are erroneously estimated.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and a camera operation estimating method and a recording medium storing a camera operation estimating program capable of determining whether or not an estimated camera parameter is valid. The purpose is to provide.

[0019]

According to a first aspect of the present invention, a camera parameter is estimated, and an error amount (ERROR) calculated after performing a camera operation compensation is calculated based on a camera parameter output as a result. If the ratio ERROR / ERROR0 is smaller than a certain threshold value, the camera parameter is regarded as valid. Otherwise, the camera parameter estimation is erroneously performed. Consider it to be. This is because an error change ERROR tends to take a large value when an image change such as a large motion of a subject that is not assumed in the camera model is included and an effective camera parameter cannot be obtained.

By performing threshold processing on ERROR / ERROR0, it is possible to determine whether camera parameters are valid or invalid. The error amount ERROR increases or decreases in accordance with the complexity (the degree of texture) of the image. However, by performing threshold processing on the ratio with ERROR0, it is possible to make a determination independent of the complexity of the image.

According to a second aspect of the present invention, when the condition that the estimated camera parameter smoothly changes with time is not further satisfied, the estimated camera parameter is invalidated. In general, the movement of a camera utilizes the property of smoothly changing with time, whereby an unintended camera movement such as camera shake can be distinguished from an intended camera operation.

The invention according to a third aspect is characterized in that the error amount is calculated only in a partial area excluding the area of the subject. By calculating the error amount only in the screen area excluding the area where the “moving subject is reflected” in the video, the validity / invalidity of the camera parameter can be determined regardless of the movement of the subject.

According to a fourth aspect of the present invention, when the ratio of the error amount after camera operation compensation and the error amount before compensation does not satisfy a condition that the ratio is equal to or less than a given threshold, the estimated camera parameters are invalidated. A camera operation estimating program for causing a computer to execute the processing to be performed, on a computer-readable recording medium. Thus, the camera operation estimating method according to claim 1 can be realized by a computer.

According to a fifth aspect of the present invention, there is provided a camera operation estimating program for causing a computer to execute processing for invalidating the estimated camera parameter when the condition that the estimated camera parameter smoothly changes with time is not satisfied. Is recorded on a computer-readable recording medium. Thus, the camera operation estimation method according to claim 2 can be realized by a computer.

According to a sixth aspect of the present invention, a camera operation estimation program for causing a computer to execute a process of calculating an error amount limited to a partial area excluding an area of a subject is recorded on a computer-readable recording medium. It is characterized by the following. Thus, the camera operation estimation method according to claim 3 can be realized by a computer.

[0026]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a processing flow chart for explaining an embodiment of the present invention.

(1) Camera Parameter Estimation In this embodiment, the translation model described in the prior art is used as a camera model. Therefore, the camera parameters to be obtained are the translation vectors (b _x (t), b
_y (t)). Here, it is assumed that t is a frame number designating an image in a video, and t = 0, 1, 2, 3,.... As a camera parameter estimation method, any of the methods described in the related art may be used.

In step S101, a variable t for storing a frame number is initialized to 1, and a variable v for storing a plurality of camera parameters is initialized to empty (φ). Step S1
02, two images f _t (x, y) and f _t−1 (x,
y), the camera parameters b (t) = (b _x (t), b
_y (t)).

In step S103, the error amount MSE (t) after performing the camera operation compensation using the estimated parameter b (t) is calculated by the following equation. _{MSE (t) = Σ x,} y {f t-1 (x, y) -f t (x + b x (t), y + b y
(t))｝ ² / N Furthermore, the error amount MSE when camera operation compensation is not performed
₀ (t) is calculated by the following equation.

MSE ₀ (t) = Σ _{x, y} ｛f _t-1 (x, y) −f _t (x, y)｝ ² / N where た だ_{し x, y} is The sum, N, is the number of corresponding pixels.

(2) Determination of validity of camera parameters Steps S104 to S106 are steps for determining whether or not the camera parameters are valid. When all of the following three conditions are satisfied, it is determined that the camera parameters are valid.

Condition 1) b (t) ≠ 0 (Step S104) Condition 2) b (t) · b (t−1)> 0 (Step S104)
105) Condition 3) MSE (t) / MSE ₀ (t) <T (Step S106) “Condition 1” is to check whether or not the estimated camera parameter is 0. Invalid because there is no. Note that it may be determined to be valid depending on the purpose.

"Condition 2" is to check whether the camera parameters are changing smoothly. here,
The inner product between the estimated camera parameter b (t) and the camera parameter b (t-1) estimated immediately before is taken, and when the inner product is larger than 0, that is, as shown in FIG.
If the angle θ is less than 90 degrees, it is determined that “camera parameters smoothly change over time”.

"Condition 3" is for confirming whether the error amount is sufficiently reduced by the camera operation compensation. Error MSE when camera operation compensation is performed
The ratio between (t) and the error amount MSE ₀ (t) when not performed is taken and compared with a threshold value T (0 <T <1). The meaning of the condition 3 will be described later.

When all of the above three conditions are satisfied, it is determined that the camera parameter b (t) is likely to be effective, and the camera parameter b (t) is added to the variable v and stored ( Step S107).

One is added to the variable t (step S10).
8) If the end of the video has not been reached (step S1)
09), the process returns to step S102. If any one of the above three conditions is not satisfied, it is determined that the camera parameter b (t) is invalid, and step S11 is performed.
Move to the process of 0.

In step S110, if the variable v stores at least the threshold L camera parameters, the plurality of camera parameters stored in the variable v are output as valid ones (step S111). Otherwise, it is highly likely that the camera has momentarily moved due to camera shake or the like, so it is determined to be an invalid parameter.

In step S112, the variable v is set to empty (φ).
Then, the process shifts to the process of step S108, and shifts to the process of the next frame. (3) Meaning of Condition 3 The meaning of “condition 3” will be described in detail.

MSE (t) represents the amount of error after performing camera operation compensation. The fact that the error amount MSE (t) is small means that the change of the image can be well explained by the set camera model. MSE
Although it seems good that (t) is simply subjected to threshold processing (for example, if MSE (t) <T, processing for validating camera parameters), there is actually the following problem.

MSE (t) has the property of increasing or decreasing depending on the complexity of the image (fineness of the texture), and tends to take a high value when the texture is fine. Therefore, it was found through experiments that the correct decision cannot be made by the fixed threshold processing. By performing threshold processing on the ratio between MSE (t) and MSE ₀ (t), the validity / invalidity of the camera operation can be determined regardless of the complexity of the image.

(4) Calculation of Error Amount In the above embodiment, the error amount MSE is set for the entire screen.
Although (t) is calculated, the error amount may be calculated only for a certain screen area.

For example, as shown in FIG. 3, the error amount is calculated only for the partial area 33 of the screen. That is, FIG.
In step S103, the sum is limited to the partial area 33. By setting the screen area and calculating the error amount in this way, the following effects are produced. That is, when it is known in advance that the subject moves at the center of the screen, the width Δx and the height Δy are set so that the partial region 33 in FIG. 3 does not overlap the subject region.

When the error amount MSE (t) is calculated only for the partial area 33 set as described above, the error amount MSE (t) is calculated.
E (t) is less affected by the movement of the subject. Therefore, there is an effect that the validity of the camera parameter can be confirmed regardless of the movement of the subject. (When the error amount is calculated for the entire screen, the error amount increases due to the movement of the subject, and it is determined that the camera parameter is not valid.) In this case, the camera parameter b (t) in step S102 in FIG. , The same width Δ
It is desirable to obtain the camera parameters for a partial area set by x and height Δy.

The present invention is not limited to the embodiment described above. In the above embodiment, the translation model is set as the camera model. However, an arbitrary model such as an affine model or a projection model can be used. The mean square error (MSE)
Although the example of (t)) has been described, it is also possible to use various error amounts developed in the field of absolute value sum of errors and robust statistics.

As a condition for confirming that “camera parameters smoothly change over time”, “condition 2”
Was used, but other conditions can also be used.
For example, a condition that the magnitude of the camera parameter vector does not change significantly can be added, and smoothness can be defined by an operation other than the inner product operation.

Further, various modifications are conceivable for the setting of the partial area for which the error amount is to be calculated. As described in the above embodiment, in addition to the method of setting a partial area in advance, a subject area is cut out by image processing, and a screen area excluding the area is set as a partial area for which an error amount is to be calculated. You can also.

Note that the camera operation estimation procedure described above is realized as a computer program, and the program is provided by being recorded on a recording medium such as a floppy disk, a compact disk, or a tape.

[0048]

According to the first aspect of the present invention, when the ratio of the error amount after the camera operation compensation and the error amount before the compensation does not satisfy a condition that the ratio is equal to or less than a given threshold, the estimated camera parameters are changed. Since the setting is made invalid, there is an effect that the validity / invalidity of the camera parameter can be correctly determined regardless of the complexity of the image.

According to the second aspect of the present invention, when the estimated camera parameters do not satisfy the condition of smoothly changing with time, the estimated camera parameters are determined to be invalid, so that an unintended camera such as a camera shake is obtained. There is an effect that it is possible to determine that the movement is invalid.

According to the third aspect of the present invention, the error amount is calculated only in the partial area excluding the object area, so that it is possible to correctly determine whether the camera parameter is valid or invalid regardless of the movement of the object or the like. There is an effect that can be done.

The inventions of claims 4 to 6 have the effect that the methods of claims 1 to 3 can be realized by a computer by causing the computer to execute the camera operation estimation program.

[Brief description of the drawings]

FIG. 1 is a process flow for explaining an embodiment of the present invention.

FIG. 2 is a diagram for explaining the geometric meaning of “smooth change of camera parameters”;

FIG. 3 is a schematic diagram for explaining a screen area in the present embodiment.

FIG. 4 is a diagram for explaining a panoramic image synthesizing process.

[Explanation of symbols]

 S101 to S112 Processing steps of the embodiment

Claims (6)

[Claims] 1. A camera operation estimating method for estimating a camera parameter from image sequence data, comprising calculating an error amount after camera operation compensation and an error amount before camera operation compensation, and calculating the error amount after compensation and the error amount before compensation. A camera operation estimating method, wherein an estimated camera parameter is invalidated when a condition that a ratio with respect to an error amount is not more than a given threshold value is not satisfied. 2. The camera operation estimating method according to claim 1, wherein the estimated camera parameters are invalidated when the condition that the estimated camera parameters smoothly change with time is not satisfied. Operation estimation method. 3. The camera operation estimating method according to claim 1, wherein the error amount is calculated only for a partial area excluding a subject area. 4. A recording medium storing a camera operation estimation program for estimating camera parameters from image sequence data by a computer, wherein an error amount after camera operation compensation and an error amount before camera operation compensation are calculated. Recorded a program that causes the computer to execute processing to invalidate the estimated camera parameters when the ratio between the error amount after compensation and the error amount before compensation does not satisfy the condition that the error amount is equal to or less than a given threshold value. A recording medium on which a camera operation estimation program is recorded. 5. A recording medium storing the camera operation estimation program according to claim 4, wherein the estimated camera parameter is invalidated when the condition that the estimated camera parameter smoothly changes with time is not satisfied. A recording medium recording a camera operation estimation program, characterized by recording a program for causing a computer to execute processing. 6. A recording medium on which the camera operation estimating program according to claim 4 or 5 is recorded, wherein a program for causing a computer to execute a process of calculating the error amount limited to a partial area excluding a subject area is recorded. A recording medium storing a camera operation estimating program characterized by the following.