JP6069115B2 - Video generation device, video generation method, and program - Google Patents

Description

  The present invention relates to a video generation device, a video generation method, and a program for generating a video in which a transparent non-rigid object flows.

  A computer graphics (CG) technique is mentioned as a prior art which produces | generates the image | video which a transparent non-rigid body object (for example, water, a beard, etc.) flows. If CG technology is used, an image can be generated through a process of fluid physical simulation and rendering (Non-Patent Document 1). Non-Patent Document 2 discloses that a change in image distortion based on a change in refractive index contributes to the perception of the thickness of a transparent rigid object in a still image. However, Non-Patent Document 2 does not disclose details of image information processing for causing perception of a transparent non-rigid object.

Crane, K., Llamas, I., & Tariq, S. (2007) .Real time simulation and rendering of 3D fluids.In Hubert N. (ed) GPUGems 3, 30, 633-675. Fleming, R.W., Jakel, F. and L. T. Maloney (2011). Visual Perception of Thick Transparent Materials. Psychological Science, 22 (6): 812-820.

  In general, the physical calculations required for the simulation of a fluid including a non-rigid body are complicated and enormous. For example, in order to simulate the flow of a non-rigid object in a CG moving image for several seconds using a physics calculation engine, several hours of calculation time are required. Recently, a technique has been proposed in which a real-time rendering technique introduced in Non-Patent Document 1 is used and the calculation is performed in a short time. When using real-time rendering technology, complex information processing related to the physical properties and refraction of the fluid must be incorporated into the algorithm. For users who do not have knowledge about fluid and refraction, it is difficult to incorporate complex information processing related to physical properties and refraction of fluids into algorithms. It would be convenient if it was possible to generate a moving image of a transparent non-rigid object without introducing the complicated calculation described above, but the image information processing for realizing this was not clear.

  Accordingly, an object of the present invention is to provide an image generation apparatus that can easily generate a moving image of a transparent non-rigid object.

  The video generation device of the present invention includes a modulated image generation unit and a video generation unit.

  The modulated image generation unit modulates the original image using a plurality of different distortion distributions having low spatial frequency components to generate a plurality of modulated images. The video generation unit generates a video configured by arranging a plurality of modulated images in time.

  According to the image generation apparatus of the present invention, it is possible to easily generate a moving image of a transparent non-rigid object.

The figure which shows the result of a 1st subjective evaluation experiment. The figure which shows the result of a 2nd subjective evaluation experiment. The block diagram which shows the structure of the image | video presentation apparatus of Example 1 of this invention. The flowchart which shows operation | movement of the image | video presentation apparatus of Example 1 of this invention. FIG. 3 is a block diagram illustrating a configuration example of a modulated image generation unit according to the first embodiment of the present invention. 5 is a flowchart illustrating an operation example of a modulated image generation unit according to the first embodiment of the present invention. The figure which shows the example of the distortion map produced | generated from the low frequency component of the Gaussian noise image. The figure which shows the example of the modulation | alteration image which modulated distortion to the original image and the original image.

<Principle of the present invention>
In the present invention, a transparent non-rigid object flows in front of an original image by presenting a modulated image group in which the original image is modulated by a plurality of different distortion distributions having low spatial frequency components. Based on the discovery of visual characteristics that are perceived as being. Based on this visual characteristic, an image that is perceived as a transparent non-rigid object such as water or sharks flowing in front of the original image can be seen without using simulation results such as fluid movement and refractive index. Can be generated.

  The first subjective evaluation experiment regarding this visual characteristic will be described below with reference to FIG. FIG. 1 is a diagram showing the results of the first subjective evaluation experiment. In this experiment, we examined how the coarseness (spatial frequency) of the two-dimensional distribution of the strain amount affects the impression formation of the transparent liquid. In this specification, the two-dimensional distribution of the distortion amount is called a distortion map. For the distortion map, for example, two-dimensional bandpass noise can be used. In this case, the distribution of the distortion amount represented by the distortion map corresponds to the luminance value distribution of each pixel of the two-dimensional bandpass noise. In the present specification, the central spatial frequency of the two-dimensional bandpass noise is referred to as the spatial frequency of the distortion map. Note that the spatial frequency of this distortion map is not the spatial frequency of the modulated image generated by modulating the original image using the distortion distribution.

  This experiment is configured by temporally arranging a plurality of modulated images generated by modulating an original image based on a distortion map that has been subjected to bandpass filter processing so as to include many specific spatial frequency bands. Video was presented to the subject. The test subject evaluated the strength of the impression (the impression of the transparent liquid) when he felt that the transparent liquid was present in the video (before the original image). The evaluation value is a five-level scale from 1 to 5. The higher the evaluation value, the stronger the impression of the transparent liquid. The graph of FIG. 1 shows the relationship with the vertical axis representing the five-level evaluation values from 1 to 5, and the horizontal axis representing the spatial frequency of the distortion map used for video generation. Ten subjects and error bars were displayed as ± 1 SEM.

  As shown in FIG. 1, it can be seen that the image generated based on the distortion map including many low spatial frequency bands makes the impression of the transparent liquid perceived more strongly. For example, when the spatial frequency of the distortion map is 0.3, 0.6, 1.2 cpd (cycles per degree), the average value of the evaluation values of the subjects is 4 or more.

  Hereinafter, a second subjective evaluation experiment regarding the visual characteristics will be described with reference to FIG. FIG. 2 is a diagram illustrating a result of the second subjective evaluation experiment. In the graph of FIG. 2, the vertical axis represents the above-described five-stage evaluation values, and the horizontal axis represents the inter-frame time interval (sec) of the video presented. There are two types of frame presentation orders: a normal frame order and a random frame order in which images of each frame are presented at random. Five subjects and error bars were displayed as ± 1 SEM. Unlike the first subjective evaluation experiment, the stimulus presented to the subject is an image of a transparent flowing liquid generated by CG. As shown in FIG. 2, it can be seen that the frame presentation order does not affect the rating value. It can also be seen that the evaluation value decreases when the time interval between frames is 0.05 (sec) (50 milliseconds) or more. For example, the average value of evaluation values at an interframe time interval of 0.02 (sec) is about 3.5, whereas the evaluation value at an interframe time interval of 0.05 (sec) is about 2.1, The evaluation value is about 2.0 when the time interval between frames is 0.10 (sec). This seems to be because if the time interval between frames is long, it is perceived by human beings as a sequence of still images, not as a continuous moving image. In other words, the presentation time of each frame may be adjusted so that the created modulated image sequence can be recognized as a continuous moving image rather than a still image sequence by the viewer, and the frame rate is fixed within this range. Or may be variable. In this specification, a sequence in which modulated images are arranged in time so that the viewer can recognize them as a moving image (video) rather than a sequence of still images is referred to as “a video configured by arranging modulated images in time”. ".

  Based on the result of the first subjective evaluation experiment, in the present invention, the original image is modulated using a distortion in which the spatial frequency of the two-dimensional distribution of the distortion amount becomes low (the distribution becomes coarse). That is, the original image is modulated using two-dimensional distortion in which the spatial frequency of the distortion map becomes small. That is, when the magnitude of distortion applied to each pixel is represented by a luminance value, the original image is modulated using distortion that lowers the spatial frequency of the image composed of the luminance value. Based on the result of the second subjective evaluation experiment, an image obtained by modulating the original image (modulated image) is switched in time and presented. The order of presenting the modulated images at this time does not matter. The presentation interval (frame rate) of each image does not need to be constant, but each image is presented to the presentation device at a time interval that can be recognized as a moving image (video) rather than a series of still images by the viewer. Like that. For example, it is preferable that the presentation time of each image does not exceed 0.05 (sec). When the frame rate is constant, it is more preferable that the video frame rate is 20 Hz or more.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the structure part which has the same function, and duplication description is abbreviate | omitted.

  Hereinafter, the video presentation device 1 according to the first embodiment will be described with reference to FIGS. 3 and 4. FIG. 3 is a block diagram showing the configuration of the video presentation device 1 of the present embodiment. FIG. 4 is a flowchart showing the operation of the video presentation device 1 of the present embodiment. As shown in FIG. 3, the video presentation device 1 of this embodiment includes a video generation device 11 and a presentation device 12. The video generation device 11 includes a modulated image generation unit 111, a video generation unit 112, and an image storage unit 113. The image storage unit 113 can be omitted as appropriate for reasons described later.

<Modulated image generation unit 111>
The modulated image generation unit 111 generates and outputs a plurality of modulated images obtained by modulating the original image based on the distortion distribution (S111). Here, the image data of the original image may be input from the outside, or may be given by reading out the image data stored in the image storage unit 113 as indicated by a broken line in FIG. When the image data of the original image is input from the outside, the image storage unit 113 can be omitted. Although the original image needs to be a still image, it is not always necessary to have only one type. For example, even if the original image is composed of a plurality of pieces of image data having different details, if these are sufficiently similar images, a modulated image generated by modulating each of the original images at a predetermined frame rate. It may be switched and presented. For example, a modulated image may be obtained by modulating each temporally continuous frame image in a moving image using a distortion distribution.

  When all the pixels of the original image have the same luminance value (for example, a white image), the above-described perceptual effect cannot be obtained even if modulation is performed based on the distortion distribution. This is because the images before modulation (before giving distortion) and after modulation (after giving distortion) do not change at all. Further, the more the high-frequency component is included in the original image, the stronger the perception of the transparent fluid liquid can be perceived. An image including a large amount of high-frequency components is an image including many portions called “edges” where the density of the image changes sharply.

  The plurality of modulated images are images that are modulated with respect to the original image using different distortion distributions of low spatial frequency components.

  In order to strongly perceive the impression that the transparent liquid is flowing, it is desirable to modulate the distortion in which the spatial frequency of the distortion distribution (distortion map) is 3 cpd (cycles per degree) or less into the original image. In other words, when rough distortion that reduces the difference in distortion between adjacent pixels is added to the original image, the impression of the transparent liquid can be made stronger. The original image may be modulated using a distortion distribution having the same low spatial frequency component (for example, 3 cpd or less), or different low spatial frequency components (for example, 2 cpd or less for one and 3 cpd or less for the other). You may modulate using the distortion distribution which has.

  In addition, a two-dimensional or more distortion direction is given to the modulated image. Any kind of distortion may be used as long as it is a two-dimensional geometric distortion such as rotational distortion, translation distortion, and random distortion.

<Video generation unit 112>
The video generation unit 112 generates a video based on a plurality of modulation images generated from the original image generated by the modulation image generation unit 111 (S112). For example, the video generation unit 112 may generate a sequence of modulated image sequences as videos so that a plurality of modulated images generated from the original image are temporally switched and presented to the presentation device 12. The modulated image sequence is a sequence in which the presentation time (frame rate) of each image is set within a range in which the modulated image sequence can be viewed as a moving image rather than a series of still images by the viewer, It is a video composed by arranging. Further, for example, the video generation unit 112 may perform control of switching each of the plurality of modulated images generated from the original image to the presentation device 12 and presenting them. The presentation interval of each image may be controlled within a range that can be viewed as a moving image (video) rather than a series of still images by the viewer. For example, the presentation time of each image may be in a range that does not exceed 0.05 (sec) or a frame rate of 20 Hz or higher.

<Presentation device 12>
The presentation device 12 can be realized by a display, a projector, or the like. The presentation device 12 presents the video generated by the video generation unit 112 (S12). Alternatively, the presentation device 12 switches and presents a plurality of modulated images generated from the original image as videos according to the control of the video generation unit 112 (S12).

<Configuration Example of Modulated Image Generation Unit 111>
Hereinafter, a configuration example of the modulated image generation unit 111 will be described with reference to FIGS. 5 and 6. FIG. 5 is a block diagram illustrating a configuration example of the modulated image generation unit 111 of the present embodiment. FIG. 6 is a flowchart illustrating an operation example of the modulated image generation unit 111 of the present embodiment. As shown in FIG. 5, the modulation image generation unit 111 can include, for example, a Gaussian noise image generation unit 1111, a distortion map generation unit 1112, a distortion modulation unit 1113, and a luminance value-distortion table storage unit 1114. . First, the Gaussian noise image generation unit 1111 generates a plurality of Gaussian noise images (S1111). The Gaussian noise image has the same average value and standard deviation. The Gaussian noise image generation unit 1111 may be omitted, and a plurality of Gaussian noise images may be acquired from the outside. Next, the distortion map generation unit 1112 assigns an image (distortion map) obtained by subjecting a plurality of Gaussian noise images to a bandpass filter (low-pass filter) process that passes only a low spatial frequency to each direction. A distortion map is generated (S1112). For example, when the Gaussian noise image generation unit 1111 generates two Gaussian noise images, the distortion map generation unit 1112 performs two low-pass filter processing on these two Gaussian noise images (distortion). One map is generated as a distortion map in the horizontal direction and the other as a distortion map in the vertical direction (S1112).

  An example of the direction-specific distortion map will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of a distortion map generated from a low-frequency component of a Gaussian noise image. FIG. 7A is an example of a horizontal distortion map, and FIG. 7B is an example of a vertical distortion map. The bar on the right side of each image shows the correspondence between the luminance value of the distortion map and the distortion amount. In FIG. 7A, a positive numerical value means rightward distortion, and a negative numerical value means leftward distortion. In FIG. 7B, a positive numerical value means upward distortion, and a negative numerical value means downward distortion. Therefore, in FIG. 7A, the closer the color is to white, the greater the amount of distortion in the right direction, and the closer the color is to black, the greater the amount of distortion in the left direction. In the example of FIG. 7, the maximum distortion amount is set to 8 pixels. Therefore, in FIG. 7A, when the luminance of the distortion map is the minimum value (white), it moves by 8 pixels in the right direction, and when the luminance of the distortion map is the maximum value (black), it moves by 8 pixels in the left direction. It is shown that. The amount of distortion is one of the parameters for strongly perceiving the impression of the transparent liquid. The amount of distortion varies depending on the type of the original image. Therefore, it is necessary to appropriately adjust the distortion amount depending on the type of the original image. It should be noted that the relationship between the luminance of the distortion map and the amount of distortion in the left / right and up / down directions is appropriately determined in advance and stored in the luminance value / distortion table storage unit 1114 as, for example, a luminance value / distortion amount table. In the example of FIG. 7, the light color portion of the distortion map corresponds to the right or upward distortion amount, and the dark color portion corresponds to the left or downward distortion amount. As long as it corresponds, the correspondence may be reversed.

  Finally, the distortion modulation unit 1113 moves each pixel of the original image by the movement direction and movement amount specified by each element of the distortion map corresponding to the pixel based on the direction-specific distortion map generated in step S1112. Thus, a modulated image is generated (S1113). The amount of movement of each pixel in the left-right direction follows the value of each element of the left-right direction distortion map, and the amount of movement of each pixel in the up-down direction follows the value of each element of the up-down direction distortion map.

  For example, if a filter that passes only spatial frequency components of 3 cpd (cycles per degree) or less is used as the low-pass filter, the impression that the transparent liquid is flowing can be strongly perceived. FIG. 8 shows an example of generating an original image and a modulated image generated based on the original image. 8A is a diagram illustrating an example of an original image, and FIG. 8B is a diagram illustrating an example of a modulated image obtained by modulating the original image using the distortion distribution of FIG. 8A.

  In the above description, a distortion map for each direction corresponding to the movement amount (distortion amount) of each of the axes constituting the two dimensions (left and right direction, vertical direction) is generated and combined. However, the distortion map is not necessarily limited to each axis. There is no need to divide the two, and one distortion map corresponding to the movement amount on the two-dimensional plane may be used. However, since the distortion map must express the amount of movement in the orthogonal two-dimensional coordinate system, it includes the amount of movement in each dimension in the two-dimensional coordinate system, or the magnitude of movement and the direction of movement Information must be included. However, the latter makes it difficult to manipulate the spatial frequency of the distortion.

  In the above example, for the sake of easy understanding, the distortion map has been described as an image composed of luminance values corresponding to the movement direction and movement amount of each pixel. However, the distortion map does not necessarily have to be an image. It may be a matrix corresponding to the number of pixels of the image. In this case, it is sufficient that each element of the matrix is configured with values that specify the moving direction and moving amount of the pixels of the original image. When a matrix is used as a distortion map, instead of passing through a bandpass filter, the matrix map is transformed so that the difference between adjacent elements of the matrix is reduced, resulting in a lower spatial frequency of the distortion map. It is also possible to do so.

  In other words, the distortion map has each element corresponding to each pixel of the original image, each element indicates the movement direction and movement amount of each corresponding pixel, and includes many low spatial frequency components (including many coarse spatial changes). ) Anything can be used.

<Comparison with conventional methods>
Generally, techniques such as mosaic processing and a smoothing filter are known as techniques for blurring an original image. While these methods change the luminance of the original image directly, the modulated image of the present invention changes the amount of distortion of the original image and does not change the luminance of the original image directly. There is no difference.

  The mosaic processing is generated by, for example, replacing the luminance of all pixels in the region with an average value or a representative value of the luminance in the region for each rectangular region. The spatial frequency inside each rectangular area of the mosaic is low, and the spatial frequency of the boundary (edge) portion of each lattice of the mosaic is high. If the entire mosaic image is viewed, the spatial frequency distribution is an image having a statistically higher number of lower spatial frequency components than the original image.

  The smoothing filter uses the luminance information of surrounding pixels, and converts the luminance of each pixel by converting the luminance of each pixel so that the pixel component with high luminance decreases in luminance and the pixel component with low luminance increases in luminance. This is a method of generating a blurred image by reducing the difference in luminance between the two. Also in this case, the spatial frequency distribution of the image obtained through the averaging filter has a higher spatial frequency component than the original image, and the lower spatial frequency component is a statistically larger image.

  On the other hand, in the present invention, the distortion map includes many low spatial frequency components. In other words, the distortion distribution used for modulating the original image has a low spatial frequency. However, the spatial frequency of the modulated image generated by moving each pixel of the original image based on the distortion map generated in this way hardly changes compared to the original image (FIGS. 8A and 8B). reference).

  Thus, the modulated image of the present invention is operated so that the spatial distribution of the geometric distortion amount (movement amount and deformation amount) from the original image of the modulated image becomes coarse, and the spatial frequency of the original image It does not change the ingredients greatly. In general image blurring methods such as mosaic processing and averaging filters, the difference in luminance histogram is large when comparing the original image with the converted image, and the high spatial frequency components of the converted image are less than the original image. On the other hand, the difference between the luminance histograms of the original image and the modulated image of the present invention is small, and the components in the high spatial frequency band of the original image are not significantly removed even in the modulated image.

  Since human vision is insensitive to such image distortion, a liquid impression cannot be obtained even when the modulated image in FIG. 8B is viewed in a stationary state. However, by switching between different modulated images and presenting them as images, the brain notices that the images are distorted and tries to attribute the cause to the liquid, so that the liquid flowing in front of the original image The effect is perceived as if it exists.

  Note that the region to be distorted is a predetermined region in the original image. If the predetermined area is the entire original image, it is possible to generate an image that is perceived as if there is a transparent flowing object all over the front of the original image. If the predetermined region is a partial region in the original image, it is possible to generate an image that is perceived as if there is a transparent flowing object in the region in the original image. If the predetermined area is a part of the original image, the pixel values of the original image other than the predetermined area may not be changed (moved) in step S1113.

<Effect>
According to the image generation device 11 of the present invention, it is possible to generate an image perceived as if a transparent object is flowing with a small amount of computation without simulating the movement of the fluid, the refractive index, and the like. The present invention can be used, for example, for visual representations of turbulent liquid flow and dark brown. Since the operation of manipulating the spatial frequency of the motion vector is computationally low, a transparent non-rigid object moving image can be quickly generated. When the time taken to generate a moving image of 90 frames (3 seconds) with 512 × 512 pixels was measured, the average was 10.8804 seconds (standard deviation: 0.3913) in 100 calculations. The computer used was Macbook Pro (registered trademark), and the performance of the computer was as follows.
CPU: 2.7 GHz Intel Core i7, Memory: 16GB 1600 MHz DDR3, Mac OS X ver.10.7.5

  In addition, the various processes described above are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Needless to say, other modifications are possible without departing from the spirit of the present invention.

  Further, when the above-described configuration is realized by a computer, processing contents of functions that each device should have are described by a program. The processing functions are realized on the computer by executing the program on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good.

  Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer). In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

Claims (5)

A modulated image generating unit that generates a plurality of modulated images obtained by modulating the original image using a plurality of different distortion distributions each having a low spatial frequency component;
A video generation unit that generates a video configured by temporally arranging the plurality of modulated images,
The modulated image generation unit
A distortion map generation unit having each element corresponding to each pixel of the original image, indicating a movement direction and a movement amount of each pixel corresponding to each element, and generating a plurality of distortion maps having low spatial frequency components;
For each of the plurality of distortion maps, a distortion modulation unit that generates the plurality of modulated images by moving each pixel of the original image by a movement direction and a movement amount specified by the elements corresponding to the pixel. Including video generation device. The video generation device according to claim 1 ,
The distortion map generation unit
A video generation apparatus that generates, as the distortion map, an image obtained by performing low-pass filter processing on a Gaussian noise image. A video generation method executed by a video generation device,
A modulated image generating step for generating a plurality of modulated images obtained by modulating the original image using a plurality of different distortion distributions each having a low spatial frequency component;
A video generation step of generating a video configured by temporally arranging the plurality of modulated images,
The modulated image generating step includes
A distortion map generating step having each element corresponding to each pixel of the original image, indicating a moving direction and a moving amount of each pixel corresponding to each element, and generating a plurality of distortion maps having low spatial frequency components;
For each of the plurality of distortion maps, a distortion modulation step of generating the plurality of modulated images by moving each pixel of the original image by a movement direction and a movement amount specified by the elements corresponding to the pixel. Including video generation method. The video generation method according to claim 3 ,
The distortion map generation step includes:
A video generation method for generating an image obtained by subjecting a Gaussian noise image to low-pass filter processing as the distortion map. The program which gives the instruction | indication which should perform the image | video production | generation method of Claim 3 or 4 with respect to a computer.