JP6317180B2 - Image generating apparatus and program - Google Patents

Description

  The present invention relates to an image generation apparatus and a program.

  When shooting a still image such as a photograph or a moving image such as a video, an unnecessary object may be shot over the subject to be shot. Still images and moving images shot with unnecessary objects covered by the subject may greatly impair the quality of viewing experience, and it is possible to remove unnecessary images contained in still images and moving images without feeling uncomfortable. Demand is extremely high.

  In a technique called free viewpoint video composition, an image from an arbitrary viewpoint is synthesized from a group of images taken by a plurality of cameras. At this time, part of the synthesized video from an arbitrary viewpoint may be lost due to occlusion by the object. Since such a defect may greatly impair the quality of viewing the image, there is a great demand for a technique that complements the defect without a sense of incongruity.

  Hereinafter, a region to be complemented, such as a region in which a reflection of unnecessary objects in a still image or a moving image is to be removed and a region that has not been observed due to occlusion, is referred to as a missing region. Further, a process of inputting an image in which a defective area is given by a mask and obtaining an image in which the defective area is complemented with an area other than the defective area in the input image without a sense of incongruity is referred to as a completion process. .

  The mask indicating the position and size of the defect area is given manually or by a known technique regardless of whether it is a still image or a moving image. As a known technique for acquiring a mask, for example, there is a technique described in Non-Patent Document 1. Note that a mask is information indicating whether or not an image to be subjected to image processing is an area on which image processing is to be performed.

  FIG. 3 is a diagram illustrating an example of a mask representing a defective region. FIG. 3A shows an example in which a mask is given as a binary image. The mask shown in FIG. 3A is given separately from the image to be subjected to image processing. This mask indicates that image processing is performed on the region indicated by the region 91a, and indicates that image processing is not performed on the region indicated by the region 91b. FIG. 3B is an example in which a mask is given by an image in which a mask is superimposed on an image to be subjected to image processing. The mask shown in FIG. 3B indicates that image processing is performed on the region indicated by the region 92, and that image processing is not performed on other regions. When a mask is provided as shown in FIG. 3B, the region 92 is provided in a color or pattern that can be easily distinguished from other regions.

Xue Bai, Jue Wang, David Simons and Guillermo Sapiro, "Video SnapCut: Robust Video Object Cutout Using Localized Classifiers", ACM Transactions on Graphics (TOG) -Proceedings of ACM SIGGRAPH 2009, Volume 28, Issue 3, August 2009, Article No .70

  However, in a still image including a complicated structure or color change or an image obtained by performing a completion process on a moving image having a complicated structure, color change or intense motion, a strange feeling is observed inside and outside the defective area. There was a problem that there was something.

  In view of the above circumstances, an object of the present invention is to provide an image generation apparatus and a program that reduce an uncomfortable appearance in and out of a defective area in an image that has been subjected to completion processing on the defective area.

  One aspect of the present invention is an image generation apparatus that generates an image in which a defective area in the original image is removed from an original image to be deleted of a defective area and mask information indicating the defective area. An information amount space conversion unit that generates a first image in the different image space by performing conversion on the original image to an image space different from the original image space defined by a parameter indicating each pixel value; Based on mask information and the first image, a completion processing unit that generates a second image by performing a completion process on the missing area in the first image, the original image, and the second image An information amount space inverse transform unit that performs transformation from the different image space to the image space on the second image based on pixel values of the image of It is formed apparatus.

  In addition, according to one aspect of the present invention, in the image generation device, the information amount space conversion unit may be configured to enter the different image space in which a range indicating a value indicating a pixel characteristic is narrower than a pixel value in the original image space. The conversion is performed on the original image.

  According to another aspect of the present invention, in the image generation apparatus, the information amount space conversion unit performs conversion from a color image, which is the original image space, to a grayscale image space, on the original image. To generate a gray scale image as the first image, and the information space inverse transform unit performs conversion from the gray scale image space to the color image space on the second image.

  According to another aspect of the present invention, in the above image generation device, the information amount space conversion unit converts the gray scale image from a gray scale image space to a feature amount space, thereby converting the feature amount image. The completion processing unit generates a third image by performing a completion process on the missing region in the feature image, and generates the mask information, the grayscale image, and the third image. Based on the image, the second image is generated by performing a completion process on the missing area in the grayscale image.

  One embodiment of the present invention is a program for causing a computer to function as the above-described image generation device.

  According to the present invention, the first image to be subjected to the completion process is obtained by performing conversion from the original image space to a different image space on the original image, and the completion process is performed. By performing inverse transformation of the image of 2 into the original image space based on the pixel values of the original image and the second image, the pixel value in the defective area is changed to the pixel value based on the original image, and the pixels in different image spaces Variations in pixel values in the defective region can be suppressed by the restriction by the range of values that can be taken, and it is possible to reduce the uncomfortable appearance in and out of the defective region.

1 is a block diagram illustrating a configuration example of an image generation device 1. FIG. It is a figure which shows an example of the original image and mask information which the image generation apparatus 1 inputs. It is a figure which shows the example of the mask showing a defect | deletion area | region.

[Outline of Embodiment of the Present Invention]
The image generation apparatus is an apparatus that generates a still image or a moving image from which a missing area is removed by complementing an image in a missing area specified in an input still image or moving image. Hereinafter, when a still image and a moving image are not distinguished, the still image and the moving image are referred to as an image. A moving image is a set of a plurality of continuous still images (frames). FIG. 1 is a block diagram illustrating a configuration example of the image generation apparatus 1. As illustrated in FIG. 1, the image generation apparatus 1 includes an information amount space conversion unit 11, a completion processing unit 12, and an information amount space inverse conversion unit 13.

  The image generation apparatus 1 inputs an original image that is a still image or a moving image that is a target of a completion process for complementing an image of a defective area, and mask information indicating the defective area from the outside. FIG. 2 is a diagram illustrating an example of an original image and mask information input by the image generation apparatus 1. As shown in FIG. 2A, the image generation apparatus 1 individually inputs an original image and mask information. Alternatively, the image generation apparatus 1 inputs an image in which a mask is superimposed on the original image, as shown in FIG.

  The information amount space conversion unit 11 converts the input original image into an image in a space different from the space defined by the parameter representing the original image. Hereinafter, a space defined by parameters representing the original image is referred to as an original image space. A space that is a conversion destination of the original image and is different from the original image space is referred to as a post-conversion space.

In addition, when the image on which the mask is superimposed on the original image is input as illustrated in FIG. 2B, the information amount space conversion unit 11 converts the original image into an image in the post-conversion space according to the following procedure. Convert to
(Procedure 1) The information amount space conversion unit 11 generates mask information in which the missing area indicated by the mask and the area other than the missing area in the input image are expressed in binary.
(Procedure 2) The information space conversion unit 11 performs conversion from the original image space to the post-conversion space for an area other than the missing area in the input image, and acquires an image in the post-conversion space.

  The completion processing unit 12 acquires the mask information input from the outside and the image of the converted space acquired by the information amount space conversion unit 11. The completion processing unit 12 performs a completion process on the missing area indicated by the mask information in the image in the converted space. Many studies on the completion processing for still images and moving images have been performed, and the completion processing unit 12 performs the completion processing on the image in the converted space using a known technique. When the image generation apparatus 1 inputs an image in which a mask is superimposed on the original image, the completion processing unit 12 acquires mask information from the information amount space conversion unit 11.

  The information-space inverse transform unit 13 acquires an image that has been subjected to the completion processing by the completion processing unit 12. The information amount space inverse transform unit 13 generates an image in the original image space by performing the inverse conversion to the conversion performed by the information amount space conversion unit 11 on the acquired image. The information-space inverse transform unit 13 outputs the generated image as an image from which the missing area in the original image is removed.

  The image generation device 1 generates an image in which a missing area of an input original image is removed by the information amount space conversion unit 11, the completion processing unit 12, and the information amount space inverse conversion unit 13 performing the above-described operations. .

[First Embodiment]
In the image generation apparatus 1 according to the first embodiment, the information amount space conversion unit 11 performs conversion from a color image space to a grayscale image space. In the first embodiment, the original image is a color image, and the original image space is a color image space. In addition, the information space conversion unit 11 is a gray scale determined by a pixel value Y that is one parameter from a color image space that is determined by a pixel value (R, G, B) that is three parameters in each pixel of the original image. Perform conversion to image space. The information space conversion unit 11 outputs the gray scale image generated by the conversion to the completion processing unit 12.

For example, the information space conversion unit 11 uses the following equation (1) to convert the original image space (color image space) from the pixel values (R, G, B) into the post-conversion space for each pixel of the original image. Conversion to a pixel value Y in (grayscale image space) is performed.
Y = 0.299 × R + 0.587 × G + 0.114 × B (1)

  The completion processing unit 12 performs the completion processing on the missing area indicated by the mask information in the grayscale image generated by the information amount space conversion unit 11. The completion processing unit 12 outputs the grayscale image that has been subjected to the completion processing to the information amount space inverse conversion unit 13.

  The information amount space inverse transform unit 13 transforms an image in the grayscale image space into an image in the color image space. The information-space inverse transform unit 13 acquires the grayscale image that has been subjected to the completion processing by the completion processing unit 12. The information-space inverse transform unit 13 restores a color image having pixel values (R, G, B) in the color image space by performing colorization on the acquired gray scale image.

  When restoring the color image, the information amount space inverse transform unit 13 acquires a combination of pixel values in corresponding pixels of the image subjected to the completion process and the original image as a certain number of samples. The information space inverse transform unit 13 restores the grayscale image to a color image based on the correspondence of the pixel values in the acquired sample. The information-space inverse transform unit 13 outputs the restored color image as an image from which the missing area in the original image is removed.

Note that colorization for an image is based on the assumption that regions of equal brightness among adjacent pixels in the image have the same color component, by optimizing the RGB pixel values of the entire image from RGB sample points that occupy a small number in the image. This is the technology that we want. The information-space inverse transform unit 13 performs colorization on the grayscale image that has been subjected to the completion process using a known technique. For the colorization, for example, the techniques described in Reference 1 and Reference 2 can be used.
[Reference 1] Anat Levin, Dani Lischinski and Yair Weiss, "Colorization Using Optimization,"SIGGRAPH'04 ACM SIGGRAPH 2004 Papers, August 2004, p.689-694
[Reference 2] Tomihisa Welsh, Michael Ashikhmin and Klaus Mueller, "Transferring Color to Greyscale Images,"SIGGRAPH'02 Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 2002, P.277-280

The acquisition of the sample by the information amount space inverse transform unit 13 is performed by the following procedure, for example.
(Procedure A) The feature points in the color image that is the input of the information space inverse transform unit 13 and the gray scale image that is the input of the completion processing unit 12 are acquired, and the feature points in the color image and the gray scale image are obtained. Take matching. When matching feature points, for example, feature point detection / feature amount extraction by SIFT or SURF can be used.
(Procedure B) The pixel value (R, G, B) of the color image is acquired as a sample for the pixel value Y of the grayscale image for the set of matched feature points.

  The image generation apparatus 1 according to the first embodiment uses a gray scale in which a range of possible values of pixel values indicating characteristics of each pixel is smaller than an image space of a color image that is an original image, as a target image to be subjected to completion processing. The image is converted to the image space. By performing the completion process by narrowing the range of values that can be taken by the pixel values indicating the characteristics of each pixel, it is possible to suppress variations in the pixel values assigned to each pixel in the defective area, and when restoring to a color image In addition, it is possible to reduce discomfort in appearance inside and outside the defect region.

[Second Embodiment]
In the image generation apparatus 1 according to the second embodiment, the information amount space conversion unit 11 performs conversion from a color image space to a motion vector space. In the second embodiment, the original image is a color moving image, and the original image space is a color image space. The information space conversion unit 11 calculates a motion vector for each pixel of the original image or each predetermined partial area, and assigns a motion vector to each pixel or partial area of each still image included in the moving image. Generate an image. The information space conversion unit 11 outputs the generated image to the completion processing unit 12.

For the conversion from the color image space to the motion vector space, a known technique for acquiring an optical flow of a moving image using the Lucas-Kanade method can be used. For example, known techniques for acquiring an optical flow include techniques described in Reference 3 and Reference 4.
[Reference 3] Bruce D. Lucas and Takeo Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision,"IJCAI'81 Proceedings of the 7th international joint conference on Artificial intelligence-Volume.2, 1981, P.674 -679
[Reference 4] Jean-Yves Bouguet, "Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the algorithm," In Intel Research Laboratory, Technical Report, 1999

  The completion processing unit 12 performs the completion processing on the missing region indicated by the mask information in each image in the motion vector space generated by the information amount space conversion unit 11. The completion processing unit 12 outputs the image subjected to the completion processing to the information amount space inverse conversion unit 13.

The information-space inverse transform unit 13 acquires an image that has been subjected to the completion processing by the completion processing unit 12. The information space inverse transform unit 13 performs transformation from the motion vector space to the color image space on the acquired image. This conversion is performed by, for example, the technique described in Reference 5, by detecting which pixel in the previous frame has moved to the pixel α from the motion vector information for a certain pixel α and detecting the detected pixel. Is obtained from the previous frame of the original image and converted to the pixel value for the pixel α.
[Reference 5] Takaaki Shiratori, Yasuyuki Matsushita, Sing Bing Kang and Xiaoou Tang, "Video Completion by Motion Field Transfer," In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), Volume 1, 2006

  The information amount space inverse transform unit 13 replaces the motion vector of each pixel or each partial area in the image subjected to the completion processing with the pixel value (R, G, B) by the above-described method, thereby moving the motion vector space. The image at is restored to a color image in the color image space. The information space inverse transform unit 13 outputs the restored color image as an image from which the missing area is removed.

  The image generation apparatus 1 according to the second embodiment uses a range of values that can be taken by a parameter (motion vector) indicating a feature of each pixel from an image space of a color image that is an original image, as a moving image to be subjected to completion processing. Is converted into an image of a narrow motion vector space. A color image is a three-dimensional parameter of pixel values (R, G, B) as described above, whereas a motion vector is a two-dimensional parameter because it is a vector on the image. That is, the information space conversion unit 11 converts the original image into an image in an image space in which the range of values that can be taken by the parameters for each pixel is narrower than the image space of the original image. By converting each still image that makes up a moving image into an image in the motion vector space and complementing the motion vector of each pixel in the defective region, even in moving images that contain intense motion, the visual discomfort inside and outside the defective region can be felt. Can be reduced. Further, when converting an image in the motion vector space into a color image space, the pixel values (R, G, B) are restored by using the pixel values in the still image constituting the moving image that is the original image. Since the range of values that can be taken is reduced, it is possible to reduce discomfort in appearance inside and outside the defect region.

[Third Embodiment]
In the image generation apparatus 1 according to the third embodiment, the information amount space conversion unit 11 performs conversion from the color image space to the grayscale image space on the original image, and further converts the image in the grayscale image space into the feature amount space. Convert. The completion processing unit 12 performs a completion process on the image in the gray scale image space based on the image in the feature amount space. In the third embodiment, the original image is a color image, and the original image space is a color image space.

  As in the first embodiment, the information space conversion unit 11 performs conversion from a color image space to a grayscale image space on the original image to generate a grayscale image. The information space conversion unit 11 duplicates the generated grayscale image. The information amount space conversion unit 11 generates an image of the feature amount space by performing conversion from the gray scale image space to the feature amount space with respect to the copied gray scale image. The information amount space conversion unit 11 outputs the grayscale image and the feature amount space image to the completion processing unit 12. As the feature amount space, for example, a Canny feature amount space, a Hog feature amount space, or the like is used.

  The completion processing unit 12 performs a completion process on the missing region indicated by the mask information in the image of the feature amount space generated by the information amount space converting unit 11. The completion processing unit 12 performs the completion processing on the missing region of the gray scale image using the feature amount space image obtained by the completion processing as prior knowledge. For example, the completion processing unit 12 has a pixel at a position (x, y) = (x2, y2) with respect to a pixel at a position (x, y) = (x1, y1) in an image defect region in the feature amount space. When the completion processing is performed on the pixel at the position (x, y) = (x1, y1) in the missing region of the grayscale image, using the copied value as prior knowledge, the position (x, y) = The pixel value of the pixel (x2, y2) is copied. The completion processing unit 12 outputs the grayscale image that has been subjected to the completion processing to the information amount space inverse conversion unit 13.

  The information amount space inverse conversion unit 13 converts the grayscale image output from the completion processing unit 12 into a color image in the color image space. The information amount space inverse transform unit 13 in the third embodiment performs the conversion from the gray scale image space to the color image space on the gray scale image, similarly to the information amount space inverse transform unit 13 in the first embodiment. By doing so, the color image is restored. The information space inverse transform unit 13 outputs the restored color image as an image from which the missing area is removed.

  The image generation apparatus 1 according to the third embodiment uses a grayscale image in which a range of values that can be taken by pixel values indicating the characteristics of each pixel is narrower than an image space of a color image that is an original image, as a target image for completion processing. The image space is converted. By performing the completion process by narrowing the range of values that can be taken by the pixel values indicating the characteristics of each pixel, it is possible to suppress variations in the pixel values assigned to each pixel in the defective area, and when restoring to a color image In addition, it is possible to reduce discomfort in appearance inside and outside the defect region.

  Furthermore, the image generation apparatus 1 performs a completion process on the grayscale image based on the completion process result for the image in the feature amount space obtained from the grayscale image. That is, the image generation device 1 performs the completion process based on a region having a feature similar to the vicinity of the missing region in the image of the feature amount space, and thus, when the image is restored to the color image, the unnatural appearance in and out of the missing region. Can be further reduced.

  The image generation apparatus 1 described in each of the above embodiments performs a completion process in an image space in which a pixel value of each pixel can be smaller than that in the original image space, so that the original image has a complicated configuration, color change, Even in the case of intense movement, it is possible to suppress the range of pixel values assigned to the pixels in the defective area. As a result, it is possible to remove the missing area of the original image while maintaining the consistency of the image structure and the brightness between the inside and outside of the missing area in the image or between frames in the moving image.

  In addition, by performing completion processing after converting the original image to an image space different from the image space of the original image, many samples that can be used for learning are obtained from a limited number of data, so that learning is performed closely. It can be carried out. In addition, since it is possible to perform highly accurate learning even if the number of data used for learning is small, it is possible to enjoy the advantage that the calculation cost required for learning can be reduced.

  You may make it implement | achieve the image generation apparatus in embodiment mentioned above with a computer. In that case, it is realized by recording a program for realizing the constituent elements of the image generation apparatus on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. Also good. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” is a program that dynamically holds a program for a short time, like a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the above-described constituent elements, and may be realized by combining the above-described constituent elements with a program already recorded in a computer system. It may be realized by using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  In the completion processing for a defective area in a still image or a moving image, the present invention can be applied to a use in which it is indispensable to reduce the uncomfortable appearance inside and outside the defective area.

DESCRIPTION OF SYMBOLS 1 ... Image generation apparatus 11 ... Information amount space conversion part 12 ... Completion processing part 13 ... Information amount space reverse conversion part

Claims (5)

An image generation apparatus that generates an image obtained by removing a defect area in the one original still image from one original still image that is a removal target of the defect area and mask information indicating the defect area,
The first original still image is converted into an image space different from the original image space defined by the parameter indicating each pixel value of the one original still image, whereby the first image in the different image space is changed. An information space conversion unit for generating an image;
A completion processing unit that generates a second image by performing a completion process on the missing region in the first image based on the mask information and the first image;
An information space inverse transform unit that performs transformation from the different image space to the original image space on the second image based on the pixel values of the one original still image and the second image; An image generation apparatus comprising: The information amount space conversion unit performs conversion on the one original still image to the different image space in which a range of values indicating pixel characteristics is narrower than a pixel value in the original image space. The image generation apparatus according to claim 1. An image generating apparatus that generates an image in which a defective area in the original image is removed from an original image to be removed of a defective area and mask information indicating the defective area,
Information amount space conversion for generating a first image in the different image space by performing conversion on the original image to an image space different from the original image space defined by a parameter indicating each pixel value of the original image And
A completion processing unit that generates a second image by performing a completion process on the missing region in the first image based on the mask information and the first image;
An information space inverse transform unit that performs transformation on the second image from the different image space to the original image space based on pixel values of the original image and the second image;
With
The information amount space conversion unit generates a grayscale image as the first image by performing conversion on the original image from a color image that is the original image space to a grayscale image space,
The amount of information space inverse conversion section images generator you and performs conversion to the color image space with respect to the second image from a gray scale image space. The information amount space conversion unit generates a feature amount image by performing conversion from a gray scale image space to a feature amount space with respect to the gray scale image,
The completion processing unit generates a third image by performing a completion process on the missing region in the feature amount image, and based on the mask information, the grayscale image, and the third image. The image generation apparatus according to claim 3, wherein the second image is generated by performing a completion process on the defect area in the grayscale image.   The program for functioning a computer as an image generation apparatus as described in any one of Claims 1-4.