JP4845998B2 - Animation method, animation apparatus, and animation program - Google Patents

Description

  The present invention relates to a technique for generating a moving image from one image.

  There are various methods and tools for editing one digital image taken by a digital camera or the like, or one digital image cut out from a video image. A method of transforming a digital image into a moving image is known as a method for converting the digital image into a moving image.

  On the other hand, in computer graphics, simple three-dimensional moving image expression using many particles is known.

Anat Levin, Dani Lischinski, and Yair Weiss, "A Closed Form Solution to Natural Image Matting", IEEE CVPR, 2006 p.61-68 Antonio Criminisi, Patrick Perez, and Kentaro Toyama, "Region Filling and Object Removal by Exemplar-Based Image Inpainting", IEEE Trans. Image Processing, September 2004, vol. 13 No. 9, p.1200-1212 "Perspective (graphical)", [online], [Search April 28, 2009], Internet <URL: http://en.wikipedia.org/wiki/Perspective_ (graphical)>

  However, many problems remain to create a moving image from a single image of a natural landscape. A single image of a natural landscape or the like includes a sense of depth and the context of the target position in the line-of-sight direction. For this reason, the conventional two-dimensional deformation operation method clearly has a limit to the realization of moving images.

  For example, when making a video of a waterfall flowing down a cliff in the image, it is possible to apply the waterfall video by clipping the waterfall, but it is cumbersome and cumbersome to cut out the waterfall. It was not enough for physical expression such as water splashing and flowing in the wind. The conventional simple expression using a large number of particles is not an animation method for live action. In the case of animation from live action, most cases are rigid bodies and elastic bodies. Although a certain level of animation can be obtained for a specific image if a large amount of time is spent, a special creation process is not suitable for general use.

  This invention is made | formed in view of the above, and makes it a subject to create a natural moving image easily from one image which copied the natural scenery.

According to a first aspect of the present invention, there is provided a moving-image generating method comprising: inputting an image and storing the image in an accumulating unit; and reading the image from the accumulating unit and inputting the image to a part of a foreground to be animated And a user's instruction input to a part of the background that is not an animation target, the step of separating the image into a foreground and a background, and the missing area of the background separated from the foreground as a texture of the background Using the hole filling interpolation, converting the foreground coordinates (X, Y) into a three-dimensional coordinate system represented by (x, y, z) based on the following equation ;
(F represents the distance from the origin along the z axis to the foreground) In the polar coordinate system (r, θ, z) received from the user, based on the setting of the three-dimensional velocity field and physical conditions defined by the following equations: Setting a three-dimensional velocity vector for the foreground;
(V _r , V _θ , V _z are speeds in the respective directions, and Q is a constant relating to the flow rate.) The foreground converted into the three-dimensional coordinate system is time-developed based on the three-dimensional velocity vector using the following equation: And animating the foreground;
(∇ is the first derivative operator, v is the velocity, ρ is the density, ν is the viscosity coefficient, F is the external force. V is v = v ₁ + v ₂ and v ₁ is the three-dimensional velocity field received from the user. velocity vector, v ₂ is a velocity vector based on the physical phenomenon is defined by the following equation)
(Θ is an angle with respect to the X-axis direction of the wind having the magnitude wv, and g is a gravitational constant) The foreground animated by using the ratio of the foreground and the background mixed in the separating step And a step of synthesizing the interpolated background.

According to a second aspect of the present invention, there is provided an animation device for inputting an image and storing it in an accumulation unit; reading the image from the accumulation unit; and inputting the image to a part of the foreground to be animated Accepting a user instruction and a user instruction input to a part of the background that is not an animation target, separating means for separating the image into a foreground and a background, and a missing region that separates the foreground of the background An interpolation means for performing hole filling interpolation using a texture, a coordinate conversion means for converting the foreground coordinates (X, Y) into a three-dimensional coordinate system represented by (x, y, z) based on the following equation ;
(F represents the distance from the origin along the z axis to the foreground) In the polar coordinate system (r, θ, z) received from the user, based on the setting of the three-dimensional velocity field and physical conditions defined by the following equations: Setting means for setting a three-dimensional velocity vector for the foreground;
(V _r , V _θ , V _z are speeds in the respective directions, and Q is a constant relating to the flow rate.) The foreground converted into the three-dimensional coordinate system is time-developed based on the three-dimensional velocity vector using the following equation: And an animation calculation means for animating the foreground;
(∇ is the first derivative operator, v is the velocity, ρ is the density, ν is the viscosity coefficient, F is the external force. V is v = v ₁ + v ₂ and v ₁ is the three-dimensional velocity field received from the user. velocity vector, v ₂ is a velocity vector based on the physical phenomenon is defined by the following equation)
(Θ is an angle with respect to the X-axis direction of the wind having a magnitude wv, and g is a gravitational constant) The foreground animated by using the ratio of the foreground and the background mixing calculated by the separating means And a synthesis means for synthesizing the interpolated background.

  An animation program according to a third aspect of the present invention causes a computer to execute each step in the animation method.

  ADVANTAGE OF THE INVENTION According to this invention, a natural moving image can be easily created from one image which copied natural scenery.

It is a block diagram which shows the structure of the animation apparatus in one embodiment. It is the schematic explaining a mode that a two-dimensional image is converted into a three-dimensional coordinate system. It is a figure which shows the example of the three-dimensional velocity field pattern which a user selects and sets. It is a figure which shows the example of the simulation of a flow when using the three-dimensional velocity field which shows a uniform pattern. It is a figure which shows the example which set and animated the three-dimensional vortex field, FIG.5 (a)-(c) shows a mode that the three-dimensional vortex field was set, FIG.5 (d)-(f) , Shows the result of animation. FIGS. 6A and 6B are diagrams illustrating a state in which a foreground and a background of one image are separated. FIG. 6A illustrates an original image, and FIG. 6B illustrates a state in which a user specifies a background. 6 (c) shows a state in which a missing area is interpolated as a background, and FIG. 6 (d) shows a state in which it is separated as a foreground. A state in which a three-dimensional velocity field is set in the foreground is shown. FIG. 8A shows an image before moving to a moving image, and FIG. 8B shows an image at a certain time after moving to a moving image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of the animation apparatus in the present embodiment. An animation apparatus 1 shown in FIG. 1 includes an image input unit 11, a data storage unit 12, an optimum separation unit 13, an interpolation unit 14, a coordinate conversion unit 15, a motion setting unit 16, an animation calculation unit 17, an image synthesis unit 18, A display unit 19 and a user interface 20 are provided. The animation apparatus 1 may be configured by a computer including an arithmetic processing device, a storage device, a memory, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the animation apparatus 1, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. Hereinafter, an outline of processing of each unit will be described.

  The image input unit 11 inputs one image taken by a camera or the like or one image cut out from a video and stores it in the data storage unit 12.

  The optimum separation unit 13 reads one image from the data storage unit 12 and separates the read image into the foreground and the background with the subject to be animated as the foreground and the fixed subject as the background. When separating the foreground and background, the user interface 20 accepts a rough position of the foreground or background from the user. Details of the method for separating the image into the foreground and the background will be described later.

  The interpolation unit 14 performs fill-in interpolation on the background missing area obtained by separating the foreground from the image with the background texture image in the vicinity of the missing area. Details of this interpolation processing will be described later.

  The coordinate conversion unit 15 adds three-dimensional information to the foreground, which is a two-dimensional image, and converts it into a three-dimensional coordinate system when moving the separated foreground into a moving image.

  The motion setting unit 16 receives, from the user, physical conditions such as wind and gravity, and three-dimensional velocity field information such as vortex and divergence from the user, and generates a three-dimensional velocity vector for a region to be animated (foreground). Set.

  The moving image calculation unit 17 generates a moving image by calculating the time evolution of the three-dimensional foreground using the partial differential equation based on the three-dimensional velocity vector set for the foreground.

  The image synthesizing unit 18 synthesizes the background obtained by interpolating the missing image area and the generated moving image, and causes the display unit 19 to display the synthesized image.

  The user interface 20 is an input device such as a mouse or a keyboard.

  Next, details of the processing of each unit will be described along the flow of processing of the animation apparatus 1.

  First, a process in which the optimum separation unit 13 separates an image into a foreground and a background will be described.

  In order to extract a specific object, the boundary line between the background of rocks, buildings, etc. and the foreground, such as clouds, waves, waterfalls, etc., is complex, especially in an image of a natural landscape. Many ideas are needed. Although it is possible to extract the foreground by hand, it is necessary to spend enormous time. Therefore, in this embodiment, the efficiency is improved with the support of the computer by the framework of the optimization problem.

As one image model, two objects, foreground and background, are combined at a certain ratio, and the following expression (1) is defined (see Non-Patent Document 1).

  Here, I, F, and B are the pixel intensity value (or luminance), foreground intensity value, and background intensity value in units of pixels, respectively, and α is the ratio (coefficient) of the foreground / background mixture. . This coefficient is obtained and the foreground and background are separated.

First, consider an image model that assumes that α can be approximated by a linear function of a pixel gray value I of a pixel. In the case of this image model, α can be expressed as the following equation (2).

  w is a small area (window area) in the image, and is a set of neighboring pixels. The size of the window is “3 pixels × 3 pixels” to “5 pixels × 5 pixels”. i indicates the position of the pixel in the window region.

Here, the following equation (3) is defined as the objective function. Let F, B, α be unknowns, and reduce the equation (3) to a minimization problem. However, the calculation is performed for each window region of 3 pixels × 3 pixels, and the R, G, and B color channels are handled in the same manner.

  Where ε is a regularization coefficient. j indicates the j-th window region.

  A rough foreground and background designation is input from the user using the user interface 20, and the pixels are labeled. Then, as the constraint condition, the image gray value of the pixel at the position designated as the foreground by the user is set to F, and the image gray value of the pixel at the position designated as the background is set to B. By analyzing the equation (3) using the least square method, α, a, and b are obtained for each pixel, and F and B are obtained by substituting into the equation (2). Then, using F and B, it is determined whether the pixel is a foreground pixel or a background pixel.

  Next, a process in which the interpolation unit 14 interpolates a background missing region will be described.

  When the optimum separation unit 13 separates the image into the foreground and the background, a missing region in which the foreground is cut out remains in the background. This missing area is re-interpreted as a hole filling problem, and interpolation is performed with a texture image close to the background. As a result, when the foreground converted into a moving image is combined and combined, the interpolated background and the foreground are combined to produce a natural result.

The missing area is Ω, the image is Γ, and the background excluding the missing area is Φ. That is, there is a relationship of Φ = Γ−Ω (see Non-Patent Document 2). It is assumed that a certain pixel point p is at the center of the patch Ψ _p . There is always a point p at the boundary line ∀ _p ∈ ∂Ω between the missing region and the background. The order of filling the holes is defined by the following equation (4) consisting of the product of the confidence term C (p) and the data term D (p). The confidence term always has a high value when there is a strong edge. That is, the holes are filled in order from the strong edge (region). Pixel point q belongs to Ψ _p or Φ.

Here, | Ψ _p | is the area of Ψ _p . α is a normalization coefficient. Here, the maximum luminance 255 of the image is assumed.

The pixel for filling the hole is searched from the background (known) based on the similarity of the small image area according to the following equation (5).

Here, d (Ψ _a , Ψ _b ) is the distance between the two patches Ψ _a and Ψ _b, and here the cumulative difference is applied. Find the most similar patch and fill it with the patch. Repeat filling up until all missing areas are filled.

  Next, a method in which the coordinate conversion unit 15 converts a two-dimensional image into a three-dimensional image will be described. Here, a perspective projection transformation model will be described (see Non-Patent Document 3). It is the same coordinate system as a so-called pinhole camera.

In FIG. 2, the three-dimensional coordinate system is (x, y, z). Assume that the two-dimensional image 200 is on the optical axis from the origin O at the position (0, 0, f) of the focal length f, and the coordinates on the two-dimensional image 200 are (X, Y). In such a coordinate system, the point P (x, y, z) indicated by reference numeral 210 in FIG. 2 is associated with the point P (X, Y) indicated by reference numeral 220 on the two-dimensional image 200, and It is given by the relational expression (6).

  A two-dimensional image is converted into a three-dimensional coordinate system based on the relational expression shown in Expression (6). The three-dimensional coordinate system is indispensable as a basic coordinate system that defines various three-dimensional equations, three-dimensional velocities, and the like described later.

  Next, processing of the movement setting unit 16 will be described.

The motion setting unit 16 accepts input of physical conditions and a three-dimensional velocity field pattern through the user interface 20 and sets the foreground to be animated. FIG. 3 shows a three-dimensional velocity field pattern selected and set by the user. Although various patterns are conceivable, FIG. 3 shows an example of springs, uniforms, and vortices from the left. These patterns are defined by the following equations (7) to (9) in the polar coordinate system (r, θ, z).

However, v _r , v _θ and v _z are velocities in the respective directions. Q is a constant relating to the flow rate.

  FIG. 4A is an example of a flow simulation when Expression (7) representing a uniform pattern is used. A uniform velocity field is set three-dimensionally and Equation (7) is applied. The user sets the flow width three-dimensionally. When the width is increased, an expression like another waterfall is possible as shown in FIG.

  Next, the processing of the animation calculation unit 17 will be described.

The animation calculation unit 17 uses the Navier-Stokes equation expressed by Equation (10) in order to calculate the three-dimensional time evolution for the speed and pressure set by the motion setting unit 16. Here, the case of two components is given for the velocity component, and is defined as v = v ₁ + v ₂ . The velocity vector selected by the user from the menu of the three-dimensional velocity field in the motion setting unit 16 is v _1, and v ₂ is a velocity based on a physical phenomenon such as gravity or wind. Of course, other speed models are possible.

However, ∇ = (∂ / ∂x, ∂ / ∂y, ∂ / ∂z), v = (v x, v y, v z) is ^T, x, y, is defined in a three-dimensional system Cartesian of z The first derivative operator, speed. ρ is the density. ν is a viscosity coefficient. F is an external force, and an object force or gravity due to turbulent flow based on perin noise is given.

On the other hand, v ₂ = (v _x2 , v _y2 , v _z2 ) ^T is a three-dimensional velocity component and can be defined as the following equation (11).

  Here, θ is an angle of the wind having the magnitude wv with respect to the X-axis direction. g is gravity, which is directed downward in the Y-axis. In addition to this, various physical laws can be applied.

Using the foreground extracted from the image and converted to 3D, and the 3D velocity vector set in the foreground, the advection equation shown in the following equation (12) is applied to generate an image of the time evolution. The gray value of the image is also given and the temporal development of the gray value is solved. There are various solutions, but in order to prevent image quality degradation due to diffusion errors and quantization errors in the solution (image) in the course of time development, in the case of the finite difference method, the upwind difference method third order And so on.

  Here, I (x, y, z) is a three-dimensional image, and v is a speed.

  Then, the image composition unit 18 synthesizes the foreground moving image generated by the moving image calculation unit 17 with the background interpolated by the interpolation unit 14 to generate a moving image from the input single image. At the time of synthesis, α estimated by Expression (3) is applied to each pixel.

  Next, an embodiment in which an image is animated by the animation apparatus 1 will be described.

  FIG. 5 shows an example in which a three-dimensional vortex field is set on a cloud image near a mountain and animated. 5A to 5C show a state in which a three-dimensional vortex field is set in an image. The three-dimensional vortex field is selected and set by the user with the mouse. As can be clearly seen in FIG. 5B, a vortex field is given to the image in three dimensions.

  The three-dimensional velocity field set by the user is given to the three-dimensional advection equation. In addition, the gray value of the three-dimensional image is also given to solve the temporal development of the gray value. The results are shown in FIGS. From FIG. 5D to FIG. 5F, realistic cloud deformation using natural and live-action textures was realized.

  FIG. 6 is a diagram showing that the optimum separation unit 13 separates the foreground and the background, and the interpolation unit 14 fills the background missing region. FIG. 6A is a natural landscape image including a waterfall and a cliff to be animated. FIG. 6B shows a state in which the background of the image is roughly specified by the user. The curve in FIG. 6B is a portion designated by the user as the background. When the background and the foreground are roughly designated by the user, the optimum separation unit 13 automatically separates the background and the foreground by applying Expression (3).

  FIG. 6C is a diagram illustrating a result of the interpolation unit 14 interpolating the missing area obtained by separating the foreground from the background. As shown in FIG. 6C, the missing region is interpolated with a natural feeling like the texture of the nearby cliff. FIG. 6D is a diagram illustrating a waterfall cut out as a foreground.

  FIG. 7 is a diagram illustrating a state in which a three-dimensional uniform velocity field and a divergent field are set in the foreground illustrated in FIG. A uniform velocity field represents a falling waterfall, and a divergence field represents a water splash on a rock. The velocity field takes into account the effects of gravity and wind. By these, a more natural flow can be expressed.

  FIG. 8A shows an image that has been converted into a moving image, and FIG. 8B shows an image at a certain time as a result of moving into a moving image. The animated waterfall is synthesized sequentially with the background interpolating the missing area. At this time, the cliff and the waterfall are synthesized by applying α estimated by the equation (3) to each pixel.

  In this way, a realistic waterfall was generated as an animation. Since it can be realized within 5 minutes from one image to animation, it is very efficient.

  Therefore, according to the present embodiment, the optimal separation unit 13 separates the image into the foreground to be animated and the background that is not to be animated, and the interpolation unit 14 fills in the missing region in which the background foreground is separated by filling in Then, the coordinate conversion unit 15 converts the foreground into a three-dimensional coordinate system, and the motion setting unit 16 accepts a three-dimensional velocity field and physical conditions from the user, sets a three-dimensional velocity vector for the foreground, and animates it. By calculating the foreground using the partial differential equation based on the three-dimensional velocity vector by the calculation unit 17 and converting the foreground into a moving image, and by synthesizing the foreground and the animated foreground by the image combining unit 18, It is possible to generate a moving image rich in physical expression from one complex natural landscape image. In particular, a natural moving image can be generated by converting the separated foreground into a three-dimensional coordinate system and three-dimensionally solving partial differential equations related to various deformations such as the Navier-Stokes equations.

  In addition, the objective function is generated using the foreground and background image grayscale values at the location specified by the user as constraints, and the objective function is analyzed using the least square method, thereby reducing the user's workload and work time. However, the foreground and background of the image can be separated.

  Further, by interpolating the background missing area with the background texture image, a natural result can be generated when the animated foreground and background are combined. Natural synthesis is possible by using the ratio of the background and foreground mixing ratio calculated when separating the foreground and the background when the foreground and the background are combined.

DESCRIPTION OF SYMBOLS 1 ... Animation apparatus 11 ... Image input part 12 ... Data storage part 13 ... Optimal separation part 14 ... Interpolation part 15 ... Coordinate conversion part 16 ... Setting part 17 ... Animation conversion calculation part 18 ... Image composition part 19 ... Display part 20 ... User interface

Claims (3)

Inputting an image and storing it in a storage means;
The image is read from the storage means, and a user instruction input to a part of the foreground to be animated and a user instruction input to a part of the background that is not to be animated are received. Separating the foreground and background,
Interpolating a missing region that separates the foreground of the background using a texture of the background; and
Converting the foreground coordinates (X, Y) into a three-dimensional coordinate system represented by (x, y, z) based on the following equation:
(F indicates the distance from the origin along the z axis to the foreground)
In the polar coordinate system (r, θ, z) received from the user, a step of setting a three-dimensional velocity vector for the foreground based on the setting of a three-dimensional velocity field and physical conditions defined by the following equations:
(V _r , V _θ , V _z are speeds in the respective directions, and Q is a constant related to the flow rate)
Evolving the foreground into the three-dimensional coordinate system based on the three-dimensional velocity vector using the following formula to animate the foreground;
(∇ is the first derivative operator, v is the velocity, ρ is the density, ν is the viscosity coefficient, F is the external force. V is v = v ₁ + v ₂ and v ₁ is the three-dimensional velocity field received from the user. velocity vector, v ₂ is a velocity vector based on the physical phenomenon is defined by the following equation)
(Θ is an angle with respect to the X-axis direction of a wind having a magnitude wv, and g is a gravitational constant)
Combining the animated foreground and the interpolated background using the foreground and background mixing ratio calculated in the separating step;
An animation method characterized by comprising: Input means for inputting an image and storing it in the storage means;
The image is read from the storage means, and a user instruction input to a part of the foreground to be animated and a user instruction input to a part of the background that is not to be animated are received. Separating means for separating the foreground and the background;
Interpolating means for interpolating the missing region obtained by separating the foreground of the background using the background texture; and
Coordinate conversion means for converting the foreground coordinates (X, Y) into a three-dimensional coordinate system represented by (x, y, z) based on the following equation:
(F indicates the distance from the origin along the z axis to the foreground)
In the polar coordinate system (r, θ, z) received from the user, setting means for setting a three-dimensional velocity vector for the foreground based on the setting of the three-dimensional velocity field and physical conditions defined by the following equations:
(V _r , V _θ , V _z are speeds in the respective directions, and Q is a constant related to the flow rate)
An animation calculation means for developing the foreground converted into the three-dimensional coordinate system in time based on the three-dimensional velocity vector using the following formula and converting the foreground into an animation;
(∇ is the first derivative operator, v is the velocity, ρ is the density, ν is the viscosity coefficient, F is the external force. V is v = v ₁ + v ₂ and v ₁ is the three-dimensional velocity field received from the user. velocity vector, v ₂ is a velocity vector based on the physical phenomenon is defined by the following equation)
(Θ is an angle with respect to the X-axis direction of a wind having a magnitude wv, and g is a gravitational constant)
A synthesizing unit that synthesizes the animated foreground and the interpolated background using the ratio of the mixture of the foreground and the background calculated by the separating unit ;
An animation apparatus comprising:   An animation program for causing a computer to execute each step in the animation method according to claim 1.