JP4729632B2 - Vanishing point estimation device, vanishing point estimation method, and vanishing point estimation program - Google Patents

Description

  The present invention relates to a vanishing point estimation device, a vanishing point estimation method, and a vanishing point estimation program for estimating a vanishing point of a captured image.

  An image taken while the subject and / or the camera is moving has a more noticeable geometric distortion caused by the aberration of the camera lens than when it is stationary, causing a radial distortion toward a certain point in the depth direction. It is known.

  At present, there are various methods for detecting a vanishing point that has disappeared due to distortion using the captured image. Specifically, grasping the image structure around the vanishing point and estimating the intersection point of the straight line segment toward the vanishing point as the vanishing point, or estimating the vanishing point from the moving direction and moving amount of the subject or camera Is used (see Non-Patent Document 1).

Akihiro Minagawa, 1 other, “Sequential detection method of vanishing point sequence with time variation”, IEICE Transactions D-II, Vol. J86-D-II, No. 6, June 2003, p. .856-868 “Lucas-Kanade Optical Flow Method”, [online], [Search May 11, 2009], Internet <http://en.wikipedia.org/wiki/Lucas_Kanade_method> Chiao-Fe Shu and Ramesh C. Jain, “Vector field analsys for oriented patterns”, IEEE Trans. PAMI, vol.16, no.9, 1994, p.946-950

  However, if the field of view is unclear and the change in the gray value is poor, it is unstable to detect the characteristics of the subject for structures, roads, etc. in the image. There was a problem that it was difficult. Specifically, when shooting under bad weather such as low contrast, snowfall, and heavy rain, detection of the vanishing point is likely to be hindered. Also, when underwater photography is taken with a submarine, it is very difficult to detect the vanishing point because microorganisms and suspended matter called marine snow diverge in front of the camera like rain. So far, vanishing points have not been detected in underwater photography.

  In addition, in the method disclosed in Non-Patent Document 1, several tens or more images are required to improve the estimation accuracy of vanishing points, and therefore it is difficult to estimate vanishing points that change sharply in a timely manner. There was also a problem.

  This invention is made | formed in view of the said subject, and makes it a subject to provide the vanishing point estimation apparatus, vanishing point estimation method, and vanishing point estimation program which are resistant to a weather change.

  The present invention described in claim 1 is a vanishing point estimation device that estimates a vanishing point that has disappeared due to distortion of an image captured during movement of a subject and / or a camera, and an image input unit that inputs a time-series image; Image storage means for storing the input time-series image, and velocity vector estimation for reading out two temporally continuous images from the image storage means and estimating the velocity vector of each pixel Means and a linear system equation that calculates a stationary point with a time derivative of zero as a singular point, and substitutes the velocity vector to derive an estimation equation for estimating parameters of the linear system equation, and the estimation equation is derived from a nonlinear system. The parameter is obtained by applying it to an equation, and the singular point calculated by substituting the obtained parameter into the linear system equation is estimated as a vanishing point. It characterized by having a a cannonball estimating means.

The present invention according to claim 2 is that the estimation formula is expressed by the following z, and the nonlinear system equation is expressed by the following E (a, b, c, d, e, f). Features.

  According to a third aspect of the present invention, speed vector re-estimating means for re-estimating a speed vector of each pixel by substituting the obtained parameters into a linear system equation, and the subject and / or the re-estimated speed vector The image processing apparatus further includes motion estimation means for estimating the motion of the camera and display means for displaying the estimated motion of the subject.

  The present invention according to claim 4 is characterized in that the velocity vector estimating means estimates a velocity vector of each pixel using a cross-correlation method or an optical flow method.

  According to a fifth aspect of the present invention, a first step of inputting a time-series image by a vanishing point estimation device that estimates a vanishing point that has disappeared due to distortion of an image captured during movement of a subject and / or a camera. A second step of storing the input time-series image in the image storage means; and reading out two temporally continuous images of the time-series images from the image storage means; Substituting the velocity vector into a linear system equation that calculates a stationary point with a time derivative of zero as a singular point, and deriving an estimation equation for estimating parameters of the linear system equation, The parameter is obtained by applying the estimation formula to a nonlinear system equation, and the singular point calculated by substituting the obtained parameter into the linear system equation is defined as a vanishing point. A fourth step of estimating, and having a.

The present invention according to claim 6 is that the estimation equation is expressed by the following z, and the nonlinear system equation is expressed by the following E (a, b, c, d, e, f). Features.

  According to a seventh aspect of the present invention, the obtained parameter is substituted into a linear system equation to re-estimate the velocity vector of each pixel, and the motion of the subject and / or the camera from the re-estimated velocity vector And a step of displaying the estimated movement of the subject.

  The present invention according to claim 8 is characterized in that the third step estimates a velocity vector of each pixel using a cross-correlation method or an optical flow method.

  The present invention described in claim 9 is characterized by causing a computer to execute each step described in any one of claims 5 to 8.

  ADVANTAGE OF THE INVENTION According to this invention, the vanishing point estimation apparatus, vanishing point estimation method, and vanishing point estimation program which are resistant to a weather change can be provided.

It is a figure which shows the function structure of the vanishing point estimation apparatus which concerns on one embodiment. It is a figure which shows the mode (one screen) when it image | photographs with the camera which moves in the real environment of marine snow, snowfall, and heavy rain, and the estimation result of a motion speed field. It is a figure explaining the method to estimate the moving speed of a to-be-photographed object or a camera. It is a figure explaining the re-estimated velocity vector. It is an example of estimation of a vanishing point when the camera is moving while rotating. This is an estimation example of vanishing points obtained by a camera moving under bad weather such as snowfall, heavy rain, and marine snow.

<Functional configuration of vanishing point estimation device>
FIG. 1 is a diagram illustrating a functional configuration of a vanishing point estimation apparatus according to an embodiment. The vanishing point estimation apparatus 1 includes an image input unit 11, an image storage unit 12, a velocity vector estimation unit 13, a vanishing point estimation unit 14, a velocity vector re-estimation unit 15, a motion estimation unit 16, and a display unit. 17. The vanishing point estimation device 1 can be configured by a computer including an arithmetic processing device such as a CPU and a storage device such as a memory, and the processing of each functional unit is executed by a program. This program is stored in a storage device, and can be recorded on a recording medium or provided through a network. Hereinafter, functions and processes in each unit will be described in detail.

  The image input unit 11 has a function of inputting a time series image composed of a plurality of temporally continuous images, and the image storage unit 12 has a function of storing the input time series image. This time-series image is an image taken while the subject and / or the camera is moving. For example, when taken in the sea, it is an image of the ocean current that flows while the camera is fixed, and it may be an image that is taken while moving the camera while the ocean current is not flowing. It may be an image taken when both of them are moving.

  The speed vector estimation unit 13 reads two temporally continuous images from the time-series images stored in the image storage unit 12, and the motion speed field of the subject photographed in the images (that is, for each pixel). (Velocity vector) is estimated. Specific estimation calculation will be described later.

  The vanishing point estimation unit 14 substitutes the velocity vector estimated by the velocity vector estimation unit 13 into a linear system equation that calculates a stationary point with a time derivative of zero as a singular point, and estimates parameters of the linear system equation. A function to estimate the singular point as a vanishing point by substituting the calculated parameter into the previous linear system equation by calculating the above parameter and applying the derived estimation formula to the nonlinear system equation. Have. Specific estimation calculation will be described later.

  The velocity vector re-estimation unit 15 has a function of substituting the parameters obtained by the vanishing point estimation unit 14 into the linear system equation and estimating the velocity vector of each pixel again.

  The motion estimation unit 16 has a function of estimating the motion of the subject and / or the camera from the speed vector re-estimated by the speed vector re-estimation unit 15.

  The display unit 17 has a function of displaying the motion of the subject estimated by the motion estimation unit 16.

<Processing of the velocity vector estimation unit>
An estimation calculation method of the velocity vector estimation unit 13 will be described. FIG. 2A is a diagram showing a state (one screen) taken with a camera that moves in an actual environment of marine snow, snowfall, and heavy rain. FIG. 2A shows a state where snowfall, heavy rain, etc. cross in a divergent manner.

The velocity vector estimation unit 13 is photographed as an image from two temporally continuous images among the photographed time-series images using a cross-correlation method or an optical flow method (see Non-Patent Document 2). Estimate the rough movement (velocity vector) of the subject. For example, when the estimation is performed using the optical flow method, the following equation (1) is used as an objective function for estimating and calculating the velocity vector. Incidentally, I _x, I _y is x in a case where the image luminance of the image was I, the spatial first derivative of y, I _t is the time first derivative. U and v are estimated velocity vectors.

  By solving Equation (1) as a minimization problem for unknowns (u, v), the motion of the subject can be calculated as the velocity vector of the pixel, and the motion velocity field as shown in FIG. An estimation result can be obtained. Note that the necessary condition for minimization when calculating the equation (1) is (∂E / ∂u) = (∂E / ∂v) = 0. In this case, since it becomes a linear simultaneous equation, the equation of the number according to the number of pixels stands.

<About processing of vanishing point estimation unit>
Next, the estimation calculation method of the vanishing point estimation unit 14 will be described. The vanishing point estimator 14 estimates the vanishing point using the linear system equation, but calculates the parameters of the linear system equation, which are secondarily estimated in the vanishing point estimation process, using the nonlinear system equation. Conventionally, even when estimating this parameter, it was calculated using a linear system equation, but since the vanishing point estimation unit 14 is calculated using a nonlinear system equation, the speed of calculating the parameter can be increased. Even when the weather changes suddenly, the vanishing point can be estimated more accurately.

First, let the singular point to be found be (x, y), P is the time-varying system for the first derivative x ^ of the singular point x (dot “•” on x), and the first derivative y ^ (y of y Let Q be the time-varying system for the dot “•” above (see equations (2) and (3)).

Since the theoretical singularity corresponding to the vanishing point may be calculated as a stationary point, the time variations of P and Q are set to zero as shown in equations (4) and (5).

Here, Equations (2) and (3) are linear models like Equations (6) and (7). V _x and V _y are the speeds of the x and y components, and correspond to the speed vector (u, v) estimated by the speed vector estimation unit 13. X ₀ and y ₀ are equilibrium points, and a, b, c, and d (and e and f described later) are parameters.

Furthermore, Expression (6) and Expression (7) are expanded as Expression (8) and Expression (9) using the parameters of e and f. F (x, y) and G (x, y) indicate functions when the entire system is a linear system.

Here, when the angle of the vector at the point (x, y) is θ and the magnitude of the vector is m, the angle θ and the magnitude m are obtained from the equations (8) and (9) from the equation (8). 10) and equation (11).

Further, Expression (10) is expanded as Expression (12). Ζ is a real value, and the velocity vector estimated by the velocity vector estimation unit 13 from the relationship (V _x , V _y ) = (u, v) shown in the equations (6) and (7). Can be calculated by dividing the y component of x by the x component.

Thereafter, using equation (12), an estimation equation for estimating the parameters (a, b, c, d, e, f) of the linear system equations shown in equations (2) to (12) Derived as shown in 13).

Then, the six parameters are estimated by evaluating and calculating Expression (13) using the objective function represented by Expression (14) and the Lorentzian robust function represented by Expression (15). Note that n is the number of pixels, μ is a constraint condition for the parameters a, b, c, and d, and σ is a variance value.

Since the objective function of Equation (14) is a nonlinear function, a necessary condition is imposed as shown in Equation (16), and iterative calculation is performed for six parameters using the steepest descent method as shown in Equation (17). To do. Since π is a parameter of any one of a, b, c, d, e, and f, Expression (17) is six expressions corresponding to each parameter. For example, 0 can be given as the initial value of each parameter. η is an adjustment coefficient and can be given, for example, 0.0001. p is the number of iterations.

  After that, the estimated parameters are substituted into the equations (8) and (9), and (x ^, y ^) = (0, 0), so that the singular point (x, y) at the stationary point is obtained. Simultaneous linear equations can be obtained. The vanishing point estimation unit 14 uses a singular point obtained by solving the simultaneous linear equations as a vanishing point.

In addition, by estimating the parameters, it is possible to calculate the vorticity curl, the divergence div, and the determinant def having the parameters under the conditions of Node (a) to Spiral (f). V _e is a velocity vector.

The above is a method for estimating a parameter using a nonlinear system equation and estimating a vanishing point based on the estimation result. Instead of the equations (13) to (17), a conventional linear system equation is used. A method for estimating parameters will be described. First, equation (12) is transformed into the equation (18).

And a parameter is estimated by the least squares method using Formula (19). Note that the constraint condition in Expression (19) corresponds to the second term on the right side of Expression (14).

Here, as a method of solving the equation (19), as shown in the equations (21) to (23), two vectors L2 and L4 including parameters and λ (Lagrange multiplier) are represented by the equation (20). The parameters can be calculated using simultaneous equations obtained by arranging the equations by imposing necessary conditions for minimizing the objective function Γ. Note that λ is an empirical coefficient and can be given, for example, 0.001. Ψ is a matrix.

  The above is a method for estimating parameters using a conventional linear system equation. However, as shown in equation (20), it is necessary to calculate a matrix when calculating the objective function Γ. The number of components constituting the matrix is large and complicated, and it can be easily understood that a considerable time is required to obtain the calculation result. On the other hand, the vanishing point estimation unit 14 is simpler than the linear system equations of Equations (19) to (23) as in the conventional method when estimating the parameters necessary for estimating the vanishing point. Since the nonlinear system equations of Expressions (14) to (17) are used, it is possible to estimate parameters with higher speed and accuracy than in the past, and as a result, it is possible to estimate the vanishing point more accurately. Become. In addition, when the variation of the velocity vector estimated by the velocity vector estimation unit 13 is large, it is known that the reliability of the parameter is greatly reduced by the influence of the loss value in the conventional least square method ( Since the vanishing point estimation unit 14 does not use the least square method, the reliability of the estimated parameter can be prevented from being lowered.

<About the processing of the motion estimation unit>
Next, a motion estimation calculation method of the motion estimation unit 16 will be described. FIG. 3 is a diagram for explaining a method of estimating the movement of the subject or camera (moving speed). As already described, the vanishing point is estimated from two consecutive images, and at the same time, the parameters of the linear system equation are estimated. Moreover, the velocity vector can be reconstructed by using the velocity vector re-estimator 15. Here, one velocity vector in the reconstructed velocity field is shown in FIG. At this time, in the vicinity of the vanishing point, the result of the reconstructed velocity vector is symmetrical in the left, right, up and down directions. Using this, the angle formed by one velocity vector and the x axis is calculated. As the moving speed of the camera increases, the adjacent angle tends to be smaller, so the moving speed of the camera can be estimated from the calculated angle. Actually, since the subject may also be moving, it can be estimated as the relative speed between the subject and the camera motion.

  In addition, when the camera is stationary, the divergent optical flow does not occur, but whether the light is diverging or not diverging is determined based on the conditions for Node (a) to Spiral (f). Can do.

  Note that there is no vanishing point when the video moves parallel to the camera line of sight. In this case, if the velocity vector is used as it is, the movement of the camera can be estimated.

<About reestimated velocity vectors, etc.>
Finally, the velocity vector re-estimated by the velocity vector re-estimator 15 will be described. FIG. 4A shows a velocity field generated by simulation. FIG. 4B is a diagram showing a velocity field after artificially adding noise to FIG. From the velocity field shown in FIG. 4 (b), a set of parameters (a, b, c, d, e, f) is estimated using the nonlinear system equations of equations (14) to (17), By substituting this set into Equation (8) and Equation (9) for each pixel, the vector on each pixel is reconstructed. FIG. 4C is a diagram showing the reconstructed velocity field. Comparing FIG. 4A and FIG. 4C, the velocity field error was 0.001%, and it was possible to reconstruct a velocity field substantially the same as the velocity field generated by the simulation.

  FIG. 5 is an example of vanishing point estimation when the camera is moving while rotating. When the camera is moving, a vortex field is obtained, but even when the camera moves in such a complicated manner, the vanishing point can be estimated stably.

  FIG. 6 is an estimation example of vanishing points obtained by a camera that moves under bad weather such as snowfall, heavy rain, and marine snow. Although the optical flow is disturbed by the influence of noise and environmental disturbance, it can be seen that the vanishing point is well obtained near the center of the image.

  The vanishing point estimation apparatus 1 can be widely used in industrial fields that require landscape scene analysis in the multimedia field, coding field, communication field, camera field, and video field.

  According to the present embodiment, the vanishing point estimation unit 14 substitutes the velocity vector estimated by the velocity vector estimation unit 13 into a linear system equation that calculates a stationary point with a time derivative of zero as a singular point, Deriving an estimation formula that estimates the parameters of the linear system equation, applying the derived estimation formula to the nonlinear system equation to obtain the above parameters, substituting the calculated parameters into the previous linear system equation, and calculating the singular point Is estimated as a vanishing point, so it is possible to quickly and robustly estimate the parameters estimated secondary in the vanishing point estimation process, and tolerate vanishing point estimation even for images with steep weather changes. An apparatus can be provided. Moreover, since the vanishing point is calculated as an eigenvalue problem even if the estimated velocity vector is incomplete, the vanishing point estimation accuracy can be improved.

DESCRIPTION OF SYMBOLS 1 ... Vanishing point estimation apparatus 11 ... Image input part 12 ... Image memory | storage part 13 ... Speed vector estimation part 14 ... Vanishing point estimation part 15 ... Speed vector re-estimation part 16 ... Motion estimation part 17 ... Display part

Claims (9)

In a vanishing point estimation device that estimates a vanishing point that has disappeared due to distortion of an image captured during movement of a subject and / or a camera,
Image input means for inputting time-series images;
Image storage means for storing the input time-series image;
Speed vector estimation means for reading out two images of the time-series images from the image storage means and estimating a speed vector of each pixel;
Substituting the velocity vector into a linear system equation that calculates a stationary point with a time derivative of zero as a singular point, derives an estimation equation that estimates the parameters of the linear system equation, and applies the estimation equation to a nonlinear system equation And the vanishing point estimating means for estimating the singularity calculated as a vanishing point by substituting the obtained parameter into the linear system equation,
The vanishing point estimation apparatus characterized by having. The estimation formula is expressed by the following z,
The vanishing point estimation apparatus according to claim 1, wherein the nonlinear system equation is expressed by the following E (a, b, c, d, e, f).
Speed vector re-estimation means for substituting the obtained parameter into a linear system equation and re-estimating the speed vector of each pixel;
Motion estimation means for estimating motion of the subject and / or the camera from the re-estimated velocity vector;
Display means for displaying the estimated movement of the subject;
The vanishing point estimation apparatus according to claim 1, further comprising: The velocity vector estimation means includes
The vanishing point estimation apparatus according to any one of claims 1 to 3, wherein a velocity vector of each pixel is estimated using a cross-correlation method or an optical flow method. By a vanishing point estimation device that estimates a vanishing point that has disappeared due to distortion of an image taken while the subject and / or the camera is moving,
A first step of inputting a time-series image;
A second step of storing the input time-series image in an image storage means;
A third step of reading out two images of the time-series images from the image storage means and estimating a velocity vector of each pixel;
Substituting the velocity vector into a linear system equation that calculates a stationary point with a time derivative of zero as a singular point, derives an estimation equation that estimates the parameters of the linear system equation, and applies the estimation equation to a nonlinear system equation And calculating the parameter, and substituting the calculated parameter into the linear system equation to estimate the singular point calculated as a vanishing point;
A vanishing point estimation method characterized by comprising: The estimation formula is expressed by the following z,
The vanishing point estimation method according to claim 5, wherein the nonlinear system equation is expressed by the following E (a, b, c, d, e, f).
Substituting the obtained parameters into a linear system equation to re-estimate the velocity vector of each pixel;
Estimating the motion of the subject and / or the camera from the re-estimated velocity vector;
Displaying the estimated movement of the subject;
The vanishing point estimation method according to claim 5 or 6, further comprising: The third step includes
The vanishing point estimation method according to any one of claims 5 to 7, wherein a velocity vector of each pixel is estimated using a cross-correlation method or an optical flow method.   A vanishing point estimation program that causes a computer to execute the steps described in any one of claims 5 to 8.