JP4369951B2 - Image generation method, image generation apparatus, and image generation program - Google Patents

Description

  The present invention relates to a technique for processing a video material and generating an image.

  In recent years, there has been image processing for performing various deformation / movement processes on digital video and images. Most of them are methods relating to the movement of the center of gravity of the object or the limited deformation that makes the area constant, assuming the rigid body of the object. However, when the water surface and clouds change, the rigid body cannot be assumed clearly, and the standard of constant area is broken due to development and decline. Further, when performing deformation such as enlargement / reduction, it is not possible to perform enlargement / reduction deformation while identifying individual objects, and natural deformation such as that caused by an optical lens cannot be applied.

  Therefore, a method based on a fluid model has been proposed as a method of expressing a fluid pattern such as smoke or clouds (see Non-Patent Document 1).

In addition, a method based on an advection equation often used in the field of fluid dynamics has been proposed for deformation / movement of an image into a fluid pattern (see Patent Document 1). The technique described in Patent Document 1 sets a velocity field for vortices and divergence, and sequentially applies an image at an arbitrary time by applying an advection equation including an advection term for each pixel of the image based on the set velocity field. To create.
JP 2004-334694 A Jos Stam, "Stable Fluids", SIGGRAPH 99 Conference Proceedings, 1999, p. 121-128

  However, although the method shown in Non-Patent Document 1 is suitable for expressing fluid-like patterns such as smoke and clouds, there is a problem that image deterioration becomes remarkable as the calculation time is advanced.

  The advection equation is a partial differential equation in which the gray level of an image is a time variable, the velocity field is known, and the image is deformed and moved along the velocity field. There is always a calculation stability condition defined by the number of Courants in between, so there is a certain limit to shortening the calculation time. When an image is composed of hundreds of thousands of pixels, the speed field is set and defined by the number of pixels. Furthermore, unless the advection term is differentiated with high-order approximation accuracy, there is a problem that the image is blurred even under stable conditions.

  Therefore, the inventor of the present application has proposed Japanese Patent Application No. 2006-119516 as a technique for greatly suppressing image degradation by solving an advection equation based on the CIP method. However, a third-order interpolation function and a first-order differential function between lattice points are proposed. There was a problem that the coefficient had to be calculated and the number of codes of the program was slightly increased.

  The present invention has been made in view of the above, and an object of the present invention is to increase the processing speed and improve image degradation when deforming an image based on an advection equation.

The image generation method according to the first aspect of the present invention includes a step of inputting a first image in which a gray value is set for each grid point by an image input unit, storing the first image in the image storage unit, and a speed field setting unit Setting the velocity field information representing the spatial distribution of velocity components, storing the velocity field information in the velocity field accumulating means, and the advection means for each grid point of the second image at a time when a predetermined time has elapsed from the first image Based on a model in which the gray value at the grid point is a velocity component at the grid point and the gray value of the first image at a position retroactive to the past for a predetermined time is converted from the speed field accumulating means. Reading the field information, calculating the position of the grid point retroactive for a predetermined time based on the velocity component at the grid point, and reading the first image from the image storage means by the interpolation means; For each grid point of the image, the gray value of the first image at the position retroactive for a predetermined time of the grid point obtained in the step of calculating the position is used as the density of the grid point of the first image near the position. is calculated by interpolating the values, and generating a second image by setting a gray value at the lattice point of the second image, have a, the second image is calculated by interpolating When the gray value at the grid point becomes a value outside the range of the gray scale value set when the first image is input, the grid point in the fixed area of the first image used for interpolation is displayed. The process of weighting the gray value to correct the gray value in a non-linear manner and the process of obtaining the gray value by performing interpolation again using the corrected gray value, the gray value calculated by interpolation is within the range of the gradation. Japanese to have a repeating until it fits within To.

In the present invention, it is calculated based on the velocity field information at which position in the first image the gray value at the grid point of the second image to be generated is a reference. At the same time, by calculating the gray value by interpolating the gray value set at the grid point of the first image, it is possible to create a deformation and movement expression equivalent to the advection equation, and a numerical value based on the advection equation. Real-time processing can be performed over a wide range from calculation to image deformation and image prediction. In addition, by weighting the gray value of the grid points used for gray value interpolation, the gray value to be interpolated does not take a value outside the range of the predetermined gradation value, so that interpolation without a sense of incongruity is possible. To do.

  The image generation method includes the step of storing the second image generated in the step of generating the second image in the image storage means as the first image, and calculating the speed field information and the position. By repeatedly executing the step and the step of generating an image, an image at a time when a predetermined time has passed is further generated.

  In the present invention, the generated second image is stored as a reference first image, and further, an image at a time when a predetermined time has elapsed is generated, thereby forming a plurality of images that are continuous in time series. A video can be generated.

  In the image generation method, the interpolation method used in the step of generating the second image is a cubic interpolation method.

  In the present invention, since the accumulation of errors is suppressed by using the cubic interpolation method, it is possible to suppress image deterioration.

  In the image generation method, the interpolation method used in the step of generating the second image is a cubic spline interpolation method.

  In the present invention, since the accumulation of errors is suppressed by using the cubic spline interpolation method, the accumulation of errors is suppressed, so that the deterioration of the image can be suppressed.

The image generating apparatus according to the second aspect of the present invention inputs the first image in which the gray value is set for each grid point, stores the image in the image storage means, and expresses the spatial distribution of velocity components. For each grid point of the second image at a time when a predetermined time has elapsed from the first image, the gray value at the grid point is set to the grid. Based on a model in which the gray value of the first image at a position traced back in the past for a predetermined time in the velocity component at the point is translated, velocity field information is read from the velocity field accumulating means, and the velocity at the lattice point is read. Advection means for calculating the position of the grid point retroactively for a predetermined time based on the components, and the grid calculated by the advection means for each grid point of the second image by reading the first image from the image storage means Points for a predetermined time in the past The gray value of the first image at the retroactive position is calculated by interpolating the gray value of the grid point of the first image in the vicinity of the position, and set as the gray value at the grid point of the second image. in an interpolation unit for generating a second image, was closed, interpolation means, a result of interpolating the gray value, the gradation range of the gray value gray value set when entering the first image When the value becomes the value of, the gray value of the grid points in the fixed region of the first image used for interpolation is weighted and the gray value is corrected nonlinearly, and the corrected gray value is used. It has a function of repeating the process of performing the interpolation again to obtain the gray value until the gray value calculated by the interpolation falls within the gradation range .

  The image generating apparatus is characterized in that the second image generated by the interpolating means is stored in the image accumulating means as the first image, and further has a repeating means for generating an image at a time when a predetermined time has passed.

  In the image generation apparatus, the interpolation method used in the interpolation means is a cubic spline interpolation method.

  An image generation program according to a third aspect of the present invention causes a computer to execute each step in the image generation method.

  According to the present invention, when an image is deformed based on an advection equation, it is possible to increase the processing speed and improve image degradation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of an image generation apparatus according to an embodiment. The image generating apparatus 1 shown in the figure has a configuration including an input unit 11, a velocity field setting unit 12, an advection unit 13, an interpolation unit 14, a repetition unit 15, a display unit 16, and a storage device 17. The image generation device 1 can be configured by a computer including an arithmetic processing device, a storage device, a memory, and the like, and the processing of each unit is executed by a program. This program is stored in the storage device 17, and can be recorded on a recording medium or provided through a network.

  As a feature of the image generation apparatus 1, a specific velocity field is set by the velocity field setting unit 12 for one reference image input by the input unit 11, and the advection equation is solved in real time as a fluid model. Therefore, the gray value of the grid point of the image at the time when a predetermined time has elapsed assumes a linear movement model in which the gray value at a certain position of the reference image is translated, and each image generated by the advection unit 13 The position in the reference image corresponding to the grid point is calculated, and the interpolation unit 14 calculates the gray value at that position to generate an image. Hereinafter, processing in each unit will be described in detail.

  The input unit 11 inputs a reference image in which a gray value is set for each grid point (pixel) and stores the reference image in the storage device 17.

  The speed field setting unit 12 sets a speed field such as divergence, rotation, convergence, vibration, and wave for one reference image, and stores the speed field in the storage device 17. The velocity field represents a spatial distribution of velocity components. In the present embodiment, a velocity field (velocity vector) set at each grid point of the image is stored in the storage device 17 as velocity field information. I remember it. The velocity field can be input by a pointing device (not shown) such as a mouse connected to the image generating apparatus 1 or can be generated based on a partial differential equation of a fluid system. When two continuous time-series images are input, a velocity field can be generated by optical flow calculation.

  The advection unit 13 calculates the gray value at each grid point of the image at a time when a predetermined time has elapsed from the reference image, at which position of the reference image the gray value is translated. The position in the reference image is calculated by reading the velocity field corresponding to the image generated from the storage device 17 and referring to the velocity vector set at the lattice point. Details of the calculation method will be described later.

  The interpolation unit 14 reads the reference image from the storage device 17 and calculates the gray value of the position in the reference image calculated by the advection unit 13. Since the position in the reference image calculated by the advection unit 13 does not necessarily coincide with the position of the grid point of the reference image, the gray value is calculated by interpolating the gray value of the grid point in the vicinity of the calculated position. An image is generated by setting the calculated gray value as the gray value at the lattice point. Details of the interpolation method will be described later.

  The repeating unit 15 determines whether or not to generate an image that has further passed, and when further generating an image, causes the storage device 17 to store the image generated by the interpolation unit 14 as a reference image, and The processes in the field setting unit 12, the advection unit 13, and the interpolation unit 14 are repeated again. When the image generation process is repeated, the speed field setting unit 12 may change the speed field with time to generate a new speed field.

  The display unit 16 displays the generated image. In addition, it is also possible to display as a moving image by sequentially displaying the images generated in time series.

Next, calculation of the position of the reference image corresponding to the lattice points of the image generated in the advection unit 13 will be described. The advection equation in which the gray value c (x (t), y (t)) moves with the flow velocity u (t) is expressed by the following equation (1).

Further, as shown in the explanatory diagram of FIG. 2A, the gray value at the position 22 (x0 ′ (t), y0 ′ (t)) of the reference image is the lattice point 21 (x0 (t + 1), advection to y0 (t + 1)). Therefore, as shown in the explanatory diagram of FIG. 2B, the gray value at the lattice point 21 can be considered to have moved by Δt time at the flow velocity u (t + 1) at the lattice point 21 indicated by reference numeral 23. 22 can be obtained by the following equation.

Further, as shown by the following equation (4), the gray value c (x0 ′ (t), t) at the position 22 (x0 ′ (t), y0 ′ (t)) obtained by the equations (2) and (3). The image at time t + 1 is generated by setting y0 ′ (t)) as the gray value c (x0 (t + 1), y0 (t + 1)) of the grid point 21 (x0 (t + 1), y0 (t + 1)) at time t + 1. can do. The gray value is assumed to have arrived unchanged from the past. This corresponds to obtaining an ideal solution without error diffusion in the advection equation.

  As described above, difference approximation is not necessary, complicated mathematical formulas such as the CIP method are unnecessary, and the method is extremely compact, so that the processing speed can be increased. In addition, since there is no stability condition associated with the difference approximation, the time width can be several tens of times larger than that of the CIP method.

  Since the calculated position 22 is not on the grid point where the gray value is set, the gray value c (x0 ′ (t), y0 ′ () is obtained by interpolating the gray value of the neighboring grid point by the interpolation unit 14. t)).

Next, the interpolation processing in the interpolation unit 14 will be described in detail. Here, a linear interpolation method, a cubic interpolation method (cubic equation), and a cubic spline interpolation will be described as methods for interpolating the gray value f _P at the point P not on the grid point.

FIG. 3 is a schematic diagram for explaining the linear interpolation method. When the coordinates of the point P are (dx, dy), the gray value f _{P at the} point P can be obtained by the following equation (5).

Here, f ₀₀ is the gray value of the grid point (0, 0), f ₀₁ is the gray value of the grid point (0, 1), f ₁₀ is the gray value of the grid point (1, 0), and f ₁₁ is the grid point. The gray value of (1, 1) can be obtained by reading the reference image stored in the storage device 17.

  Next, the cubic interpolation method will be described. FIG. 4 is a schematic diagram for explaining the cubic interpolation method. Of the lattice points in the vicinity of the point P shown in the figure, the lattice points adjacent to the point P indicated by the black circle are classified as the primary neighborhood, and the lattice points indicated by the surrounding white circles are classified as the secondary neighborhood.

First, the weights W _x and W _y for the distances Δx and Δy are obtained independently for each lattice point in the x and y directions, and the final weight W is obtained by W = W _x * W _y . The i-direction weight W _i is calculated by the following equation (6). For the lattice points near the first order, the upper calculation formula of Expression (6) is used, and for the lattice points near the second order, the lower calculation expression of Expression (6) is used.

Here, Δi is the distance Δx or Δy. For example, the lattice point (−1, −1) in the second order neighborhood is Δx = 1 + dx, Δy = 1 + dy, so that W _x = dx (dx−1) ² and W _y = dy (dy−1). ² and W = dx (dx−1) ² dy (dy−1) ² .

Subsequently, the gray value is obtained using the weight at each grid point and the gray value obtained from the reference image. The shading value f _{P at the} point P can be obtained by the following equation (7).

  Here, W (i, j) represents the weight of the grid point (i, j), and f (i, j) is the gray value of the grid point (i, j).

  Next, cubic spline interpolation will be described. Here, for the sake of simplicity, a one-dimensional case will be described, but two-dimensional expansion is easy. FIG. 5 shows that gray value data {f [−1], f [0], f [1], f [2]} are set at four points {−1, 0, 1, 2}. FIG. 6 is a schematic diagram for explaining a calculation of a light and shade value f [x] at a point x.

The gray value f [x] at the point x (0 ≦ x ≦ 1) between the intervals [0, 1] is interpolated with a cubic function f (x) represented by the following equation (8).

Expressions that hold for the gray value at the grid point 0 and the grid point 1 and the differentiation of the gray value by the expressions (8) and (9) are summarized as follows. The gray value differentiation is approximated by the center difference.

By calculating the coefficients A, B, C, and D of the cubic function f (x) shown in the equation (8) from the above four equations, the following equation is obtained.

  As described above, the gray value f [x] at the point x can be calculated.

  Next, the effects of using the linear interpolation method and the cubic interpolation method as a method for interpolating gray values will be described by comparison. FIG. 6A is a reference image of a fluid pattern, and FIG. 6B is an image generated when a cubic interpolation method is applied to the image shown in FIG. 6A by several hundred steps. FIG. 6C shows an image generated when the linear interpolation method is applied to the image shown in FIG. When the cubic interpolation method is applied as the gray value interpolation method, as shown in FIG. 6B, the texture of the reference image is almost preserved, whereas the linear interpolation method is applied by several hundred steps. In this case, as shown in FIG. 6C, the image pattern is flat. This is because the cubic interpolation method has higher approximation accuracy than the linear interpolation method, and the accumulation of errors is suppressed.

  Next, a case where weight values are weighted when using a cubic interpolation method and a cubic spline interpolation method will be described. FIG. 7A shows an image obtained by applying a cubic interpolation method and displaying a pixel as shown by a portion denoted by reference numeral 61. This is because, due to the characteristics of the cubic function, the shaded value to be interpolated becomes a negative value or exceeds the upper limit of the shade of the shaded value, for example, 255 when it is an 8-bit tone. The case where the grayscale value deviates from the range of grayscale values is when the grayscale values of the grid points in a certain region are discontinuous and the grayscale values of several grid points take a value greatly deviating from the average.

  In the present embodiment, when the corrected value is not in the gradation range of the gradation value (for example, in the case of 8-bit gradation, 0 or more and 255 or less), the gradation value of the lattice point in the fixed area used for the interpolation is set. The average and variance are obtained, the gray value is weighted, the gray value is corrected nonlinearly, and interpolation is performed again using the corrected gray value to obtain the gray value. This is repeated until the interpolated gray value falls within the gradation range. The weighting is corrected little by little using a small value from 0.001 to 0.1. In this way, by performing weighting on the gray value and performing the interpolation process, as shown in FIG. 7B, the missing portion can be corrected without a sense of incongruity.

  Therefore, according to the present embodiment, the velocity field setting unit 12 uses the advection unit 13 to determine at which position in the reference image the gray value of each grid point of the generated image is translated. By calculating based on the set speed field and calculating the gray value at the calculated position of the reference image by interpolating the gray value at the lattice point in the vicinity of the position by the interpolation unit 14, a simple algorithm is used. Since deformation and movement expression equivalent to the advection equation can be performed, the processing speed can be increased.

  According to the present embodiment, the repeating unit 15 stores the generated image in the storage device 17 as a reference image, and further generates an image at the time when the time has elapsed, whereby a plurality of images that are consecutive in time series are generated. It is possible to generate a video consisting of

  According to the present embodiment, the interpolation unit 14 calculates the gray value by applying the cubic interpolation method or the cubic spline interpolation method, so that the approximation accuracy with respect to the gray value increases and the accumulation of errors is suppressed. Therefore, even if the generated image is used as a reference image and repeated processing is performed, image degradation can be suppressed.

  According to the present embodiment, when the interpolation unit 14 applies the cubic interpolation method or the cubic spline interpolation method to interpolate the gray value, the interpolated gray value is outside the range of the predetermined gradation value. If the value becomes a value, weighting is applied to the gray value of the grid point used for the interpolation, and the gray value is interpolated so that the interpolated gray value does not deviate from the range of the gradation value. Interpolation without any

It is a block diagram which shows the structure of the image generation apparatus in one embodiment. It is a figure which shows a mode that the position in a reference | standard image is calculated | required, The figure (a) is a figure which shows a mode that the light and shade value of a reference | standard image advects, The figure (b) shows the velocity vector and its velocity vector in a lattice point. It is a figure which shows a mode that a position is calculated | required using. It is a schematic diagram which shows a mode that a shade value is calculated | required by applying a linear interpolation method. It is a schematic diagram which shows a mode that a gradation value is calculated | required by applying a cubic interpolation method. It is a schematic diagram for demonstrating a cubic spline interpolation method. FIG. 4A is a diagram illustrating an image generated according to an embodiment, where FIG. 5A is a reference image, FIG. 5B is an image generated by applying a cubic interpolation method, and FIG. 4C is a linear interpolation method. The image produced | generated by applying is shown. It is a figure which shows the image produced | generated by the Example, The figure (a) is an image when the shade value which interpolated deviated from a predetermined gradation value, The figure (b) is the lattice point used for the interpolation. This is an image when the gray value is weighted and the gray value is interpolated.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Image generation apparatus 11 ... Input part 12 ... Speed field setting part 13 ... Advection part 14 ... Interpolation part 15 ... Repeat part 16 ... Display part 17 ... Memory | storage device

Claims (8)

A step of inputting a first image in which a gray value is set for each grid point by the image input means and storing the first image in the image storage means;
Setting speed field information representing a spatial distribution of speed components by the speed field setting means, and storing the speed field information in the speed field storage means;
For each grid point of the second image at a time when a predetermined time has elapsed from the first image, the gray value at the grid point is a velocity component at the grid point and the position at a position retroactive to the past by the advection means. Based on the model that the gray value of the first image is translated, the velocity field information is read from the velocity field accumulating means, and the predetermined time of the lattice point is determined based on the velocity component at the lattice point. Calculating a retroactive position; and
The interpolating means reads the first image from the image accumulating means, and for each grid point of the second image, the position of the grid point obtained in the step of calculating the position is a position retroactive to the past for the predetermined time. By calculating the gray value of the first image by interpolating the gray value of the grid point of the first image in the vicinity of the position, and setting it as the gray value at the grid point of the second image. Generating the second image;
I have a,
When the gray value at the grid point of the second image calculated by interpolation becomes a value outside the range of the gray value gradation set when the first image is input, interpolation is performed. A process of weighting the gray values of the grid points in the fixed region of the first image to be used to nonlinearly correct the gray values, and a process of obtaining a gray value by performing interpolation again using the corrected gray values preparative, image generation method grayscale value calculated by interpolation, characterized in that have a repeating up falls within the range of the gradation. Storing the second image generated in the step of generating the second image as the first image in the image storage means;
The image at the time when a predetermined time has passed is further generated by repeatedly executing the step of setting the velocity field information, the step of calculating the position, and the step of generating the image. Image generation method.   The image generation method according to claim 1 or 2, wherein the interpolation method used in the step of generating the second image is a cubic interpolation method.   3. The image generation method according to claim 1, wherein the interpolation method used in the step of generating the second image is a cubic spline interpolation method. An image input unit that inputs a first image in which a gray value is set for each grid point and stores the first image in the image storage unit;
A velocity field setting means for setting velocity field information representing a spatial distribution of velocity components and storing the velocity field information in the velocity field storage means;
For each grid point of the second image at a time when a predetermined time has elapsed from the first image, the first image at a position where the gray value at the grid point goes back to the past for the predetermined time with the velocity component at the grid point. The speed field information is read from the speed field accumulating means based on the model that the gray value of the grid point is translated, and the grid point is traced back to the predetermined time based on the speed component at the grid point. Advection means for calculating the position;
The first image is read from the image storage means, and for each grid point of the second image, the gray level of the first image at a position retroactive to the grid time calculated by the advection means for the predetermined time in the past. The second image is generated by calculating the value by interpolating the gray value of the grid point of the first image in the vicinity of the position and setting the gray value at the grid point of the second image. Interpolation means to perform,
I have a,
As a result of the interpolation means interpolating the gray value, when the gray value becomes a value outside the range of the gray value set when the first image is input, the first value used for interpolation is used. A process of weighting gray values of grid points in a certain area of one image to nonlinearly correct the gray values, and a process of performing interpolation again using the corrected gray values to obtain a gray value. An image generating apparatus comprising a function of repeating until a gray value calculated by interpolation falls within the gradation range . 6. The apparatus according to claim 5 , further comprising: a repeating unit that stores the second image generated by the interpolation unit as the first image in the image storage unit, and further generates an image at a time when a predetermined time has elapsed. Image generation device. 7. The image generation apparatus according to claim 5 , wherein the interpolation method used in the interpolation means is a cubic spline interpolation method. Image generation program for executing the respective steps in a computer in the image generation method according to any one of claims 1 to 4.