JP6339964B2 - Image generating apparatus and computer program - Google Patents

Description

  The present invention relates to an image generation apparatus and a computer program.

  When shooting a still image such as a photograph or a moving image such as a video, an unnecessary object may be shot over the subject to be shot. Still images and moving images shot with unnecessary objects on the subject may seriously impair the quality of viewing experience, and it is possible to remove unnecessary things contained in still images and moving images without feeling uncomfortable. Demand is extremely high.

  In a technique called free viewpoint video composition, an image from an arbitrary viewpoint is synthesized from a group of images taken by a plurality of cameras. At this time, part of the synthesized video from an arbitrary viewpoint may be lost due to occlusion by the object. Since such a defect may greatly impair the quality of viewing the image, there is a great demand for a method that complements the defect without a sense of incongruity.

  Hereinafter, a region to be complemented, such as a region in which a reflection of unnecessary objects in a still image or a moving image is to be removed and a region that has not been observed due to occlusion, is referred to as a missing region. In addition, an image in which the defect area is given by a mask is input, and the defect area in the input image is complemented without a sense of incongruity with the area other than the defect area (hereinafter referred to as “defect peripheral area”). The process of acquiring the process is referred to as a completion process.

  The mask indicating the position and size of the defect area is given manually or by a known technique regardless of whether it is a still image or a moving image. As a known technique for acquiring a mask, for example, there is a technique described in Non-Patent Document 1. Note that a mask is information indicating whether or not an image to be subjected to image processing is an area on which image processing is to be performed.

  FIG. 5 is a diagram illustrating an example of a mask representing a defective region. FIG. 5A is an example in which a mask is given as a binary image. The mask shown in FIG. 5A is given separately from the image to be subjected to image processing. This mask indicates that image processing is performed on the region indicated by the region 91a, and indicates that image processing is not performed on the region indicated by the region 91b. FIG. 5B is an example in which a mask is given by an image in which a mask is superimposed on an image to be subjected to image processing. The mask shown in FIG. 5B indicates that image processing is performed on the region indicated by the region 92, and that image processing is not performed on other regions. In the case of providing a mask as shown in FIG. 5B, the region 92 is provided in a color or pattern that can be easily distinguished from other regions.

One of the completion methods is a patch-based method. Patch-based completion can be done by the following procedure.
(Procedure 1) A small region (hereinafter referred to as “completion target patch”) including both a defective region and a defective peripheral region is selected from the regions to be completed. Examples of this method include a method of selecting a patch that exists on the outline of a defect region and has a strong peripheral edge (see, for example, Non-Patent Document 2).
(Procedure 2) A similar patch is searched based on the pixels in the defect peripheral region of the completion target patch selected in Procedure 1. Examples of the search range include still images and moving images stored in the same image or moving image or on a storage.
(Procedure 3) The similar patch searched in Procedure 2 is copied to the completion target patch selected in Procedure 1.
The above steps 1 to 3 are repeated until there is no area to be completed.

Xue Bai, Jue Wang, David Simons and Guillermo Sapiro, "Video SnapCut: Robust Video Object Cutout Using Localized Classifiers", ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAPH 2009, Volume 28, Issue 3, August 2009, Article No .70 A. Criminisi, P. Perez, and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 9, pp. 1200-1212, Sept. 2004 .

  However, the result of the completion process varies depending on the feature space in which the completion process is performed. Therefore, it is necessary to perform the completion process in the feature amount space corresponding to the area to be completed. However, the conversion that returns the completion result obtained in the feature amount space different from that of the input image / video to the original feature amount space before the conversion is often a one-to-many conversion, so that the conversion process cannot be performed linearly. There are also processes such as colorization that convert from grayscale space to RGB space, but there are very few variations. Completion processing is performed in every feature space, and the results are converted into RGB feature values. There was a problem that it was difficult to return to space.

  In view of the above circumstances, an object of the present invention is to provide a technique capable of easily performing conversion from a predetermined feature amount space to a feature amount space before conversion.

  One aspect of the present invention is an image generation device that generates a first image by removing an image of the defect area from the original image based on an original image to be removed of the defect area and mask information indicating the defect area. A conversion unit that generates a second image in the different image space by performing conversion on the original image to an image space different from an original image space defined by a parameter representing the original image; Based on mask information and the second image, a completion processing unit that generates a third image by performing a completion process on the missing area in the second image, and the original image and the second image Regions that do not correspond to missing regions are first teacher data and second teacher data, respectively, and based on the first teacher data and the second teacher data, The inverse transformation learning unit for learning the inverse transformation used for the transformation from the different image space in the second teacher data to the original image space in the first teacher data, and the learned inverse for the third image An image generation device comprising: a feature space inverse transform unit that generates the first image by transforming the third image from the different image space to the original image space by using transformation.

  One aspect of the present invention is the above-described image generation apparatus, wherein the inverse transformation learning unit sets all regions of the original image and the second image as the first teacher data and the second teacher data.

  One aspect of the present invention is the above-described image generation device, wherein the inverse transformation learning unit calculates the region of the original image and the region of the second image corresponding to the region of the original image indicated at a predetermined ratio. The first teacher data and the second teacher data are used.

  One aspect of the present invention is the image generation device described above, wherein the inverse transformation learning unit indicates a predetermined ratio of regions in the original image and the second image in which edges are equal to or greater than a predetermined threshold. The areas to be processed are the first teacher data and the second teacher data.

  One embodiment of the present invention is a computer program for causing a computer to function as the above-described image generation apparatus.

  According to the present invention, it is possible to easily perform conversion from a predetermined feature amount space to a feature amount space before conversion.

2 is a block diagram illustrating a configuration example of an image generation device 10. FIG. It is a figure which shows an example of the original image and mask information which the image generation apparatus 10 inputs. It is a figure which shows an example of the combination of the feature-value in the space after conversion. 3 is a flowchart illustrating a processing flow of the image generation apparatus 10. It is a figure which shows the example of the mask showing a defect | deletion area | region.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
The image generation apparatus is an apparatus that generates a still image or a moving image from which an image of a defective area is removed by complementing an image in a defective area specified in an input still image or moving image. Hereinafter, when a still image and a moving image are not distinguished, the still image and the moving image are simply referred to as an image. A moving image is a set of a plurality of continuous still images (frames).

FIG. 1 is a block diagram illustrating a configuration example of the image generation apparatus 10.
The image generation apparatus 10 receives from the outside an original image I_Original that is an image to be subjected to a completion process that complements the missing area Damaged, and mask information Mask that indicates the missing area Damaged. The image generation apparatus 10 generates an image from which the image of the defective area Damaged is removed (first image) by removing the image of the defective area Damaged from the original image I_Original based on the original image I_Original and the mask information Mask.

  As illustrated in FIG. 1, the image generation apparatus 10 includes an image acquisition unit 101, a reference image storage unit 102, a conversion unit 103, a post-conversion image storage unit 104, a completion processing unit 105, an inverse conversion learning unit 106, a feature amount space. An inverse conversion unit 107 is provided. FIG. 2 is a diagram illustrating an example of the original image I_Original and the mask information Mask input by the image generation apparatus 10. As shown in FIGS. 2A and 2B, the image generation apparatus 10 individually inputs the original image I_Original and the mask information Mask. Alternatively, as illustrated in FIG. 2C, the image generation apparatus 10 inputs an image in which a mask is superimposed on the original image I_Original.

  The image acquisition unit 101 acquires an original image I_Original to be subjected to completion, mask information Mask indicating a missing area Damaged, and color information Mask_color indicating a mask area from the outside. The mask information Mask is an image having the same size as the original image I_Original, the pixel value of the defective region Damaged is 1 (white in FIG. 2B), and the pixel value of the defective peripheral region is 0 (in FIG. 2B). Black). However, the pixel value in the mask information may be a value other than the above-described values as long as the defective region Damaged and the defective peripheral region can be distinguished. The color information Mask_color is a three-dimensional vector having RGB values. For example, when magenta is specified as a mask area, it can be expressed as Mask_color = (255, 0, 255). 2A and 2B, when the original image I_Original and the mask information Mask indicating the missing area Damaged are separately provided, the area indicated by the mask is used as the missing area Damaged. At this time, the color information Mask_color indicating the mask area does not need to be specified.

When an image in which a mask is superimposed on the original image I_Original as shown in FIG. 2C is input to the image generation apparatus 10, the image acquisition unit 101 displays color information representing a mask in the input image. Mask information Mask in which a mask area represented by Mask_color and a defect peripheral area are represented by binary values is generated. The image generation apparatus 10 uses the area indicated by the mask area as the defect area Damaged. The missing area Damaged may be represented by a set of pixel positions designated as missing {(dx ₁ , dy ₁ ),..., (Dxn, dyn)}, and has the same size as the original image I_Original. The image may be held as a binary image in which the defect area Damaged is 1 and the defect peripheral area is 0.

The reference image storage unit 102 includes a storage device such as a magnetic hard disk device or a semiconductor storage device. The reference image storage unit 102 stores reference images I_Source = {I ₁ ,..., I _M } that are images used as search candidates for similar patches. Here, M is an integer of 2 or more. The similar patch is a patch similar to a pixel in a defect peripheral region of the completion target patch. Note that the original image I_Original may be included in the reference image I_Source. When the original image I_Original is stored, it may be processed so as not to be included in the reference image I_Source by removing the area of the missing area Damaged.

  The conversion unit 103 converts the input original image I_Original to an image in a space different from the space defined by the parameter representing the original image I_Original. Hereinafter, a feature amount space defined by a parameter representing the original image I_Original is referred to as an original image space. A feature space that is a conversion destination space of the original image I_Original and that is different from the original image space is referred to as a post-conversion space. In the present embodiment, the original image space is an RGB feature amount space. The conversion unit 103 generates a converted original image I_Converted (second image) by converting the RGB feature amount space into a certain predetermined converted space F. Further, the conversion unit 103 generates the converted reference image I_Source_conv by converting the reference image I_Source from the RGB feature amount space to a certain predetermined converted space F in the same manner as the original image I_Original.

  Examples of conversion to the predetermined post-conversion space F include gray scale conversion, conversion to HOG (Histogram of Oriented Gradients) feature quantity, conversion to GIST feature quantity, etc., but the present invention particularly restricts this. It is not a thing. In addition to the feature amount space as described above, a feature amount space in which the resolution of the image is changed, such as ½ resolution, can be included in the converted space F. Further, there is no particular limitation on the method for providing the converted space at the conversion destination and the method for performing the conversion. For example, it can be carried out by the following methods 1 and 2. At this time, it is not necessary for the user to know the generation algorithm of the converted original image I_Converted and the conversion filter from the original image I_Original to the converted original image I_Converted. This is because there is no problem even if information on conversion performed in advance is not obtained.

(Method 1: Method for performing predetermined feature value conversion)
The conversion unit 103 converts the original image I_Original from the RGB feature amount space to a predetermined post-conversion space F, and generates a post-conversion original image I_Converted. The conversion may be performed by a user-specific algorithm, or may be performed by commercially available image processing software.

(Method 2: Method of converting feature quantity into a space selected from pre-converted space candidates)
A candidate F (i) (1 ≦ i ≦ N, N: total number of feature amount candidates) of the converted space F is given in advance by a combination of feature amounts Feature. Specific examples of the feature value Feature to be given include pixel values (RGB, gray scale, etc.), HOG feature values, GIST feature values, and the like. Although there is no particular limitation on the method for giving the combination of feature amounts Feature, for example, it can be given as shown in FIG.

FIG. 3 is a diagram illustrating an example of combinations of feature amounts in the converted space.
In the example illustrated in FIG. 3, whether or not each feature amount is used is indicated for each feature amount space ID. The feature amount space ID represents identification information for identifying a converted space that is a feature amount space different from the original image space. Specific examples of the converted space include an R space, a G space, a B space, a gray scale space, and a HOG feature amount space. For each feature amount space ID, each item of feature amounts R, G, B, Gray, HOG, and GIST is indicated by 0 or 1. The case of 1 means that the corresponding feature amount is used in the converted space F (i), and the case of 0 means that it is not used. For example, the post-conversion space indicated by the feature amount space ID “1” indicates that only the feature amount R is used. That is, the converted space indicated by the feature amount space ID “1” is the R space. Similarly, it is shown that the converted space indicated by the feature amount space ID “4” uses only the feature amount Gray. That is, the converted space indicated by the feature amount space ID “4” is a gray scale space. In the example of FIG. 3, N feature amount space candidates are registered. The user selects one or more candidates from the feature amount space candidates F (i) given as described above, and performs conversion to the converted space F (i).

The post-conversion image storage unit 104 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The post-conversion image storage unit 104 stores the post-conversion reference image I_Source_conv converted to the predetermined post-conversion space F by the conversion unit 103.
The completion processing unit 105 performs the completion processing based on the converted original image I_Converted, the missing region Damaged, and the converted reference image I_Source_conv stored in the converted image storage unit 104, thereby performing the missing region Damaged. A post-conversion image from which the image is removed is generated. Specific processing will be described below.

  Step 1: The completion processing unit 105 selects a completion target patch Damaged_Patch from the converted original image I_Converted. As this method, there is a method of selecting from patches having a strong edge around the outline of the defect region Damaged. Note that the size of the patch may be any size as long as it is the size of a small region that includes both the defect region Damaged and the defect peripheral region.

  Step 2: The completion processing unit 105 searches for the similar patch Similar_Patch of the completion target patch Damaged_Patch from the converted reference image I_Source_conv, and performs the completion process using the similar patch Similar_Patch obtained by the search. A converted image I_Completed_conv (third image) from which the image of the missing area Damaged is removed is generated. The similar patch Similar_Patch is a patch similar to the pixel in the defective peripheral region of the completion target patch Damaged_Patch. In this way, the completion processing unit 105 acquires the converted image I_Completed_conv in the same feature amount space as the converted original image I_Converted.

  Note that many studies on the completion processing for an image have been performed, and the completion processing unit 105 performs the completion processing on the original image I_Original using a known technique. The completion processing unit 105 outputs the image that has been subjected to the completion processing, the converted original image I_Converted, the original image I_Original, and the mask information Mask to the inverse conversion learning unit 106.

  The inverse conversion learning unit 106 acquires the converted image I_Completed_conv, the converted original image I_Converted, the original image I_Original, and the mask information Mask. The inverse transformation learning unit 106 performs transformation from the feature amount space of the converted image I_Completed_conv to the feature amount space of the original image I_Original based on the acquired original image I_Original, converted original image I_Converted, and missing region Damaged. The conversion process is obtained by learning.

  This process is represented by f. f is a filter matrix. Further, the dimension number of the feature amount space of the image subjected to the completion processing and the feature amount space of the original image I_Original is not always the same. Therefore, the inverse transform filter f may be a one-to-many transformation that cannot be expressed by a linear transformation. Therefore, the inverse transformation learning unit 106 obtains the inverse transformation filter f based on learning by the following process.

Step 1: The inverse transformation learning unit 106 sets a portion that does not correspond to the missing region Damaged as learning teacher data in the original image I_Original and the converted original image I_Converted obtained by the conversion unit 103. The teacher data obtained from the original image I_Original and the converted original image I_Converted are set as original image teacher data I_train_orig (first teacher data) and converted original image teacher data I_train_conv (second teacher data), respectively.
Step 2: The inverse transformation learning unit 106 obtains the inverse transformation filter f by learning the transformation from the converted original image teacher data I_train_conv to the original image teacher data I_train_orig. Although the method for obtaining the inverse transform filter f is not particularly limited, for example, the technique of Reference Document 1 below may be used.
[Reference 1] A Hertzmann, CE Jacobs, N Oliver, B Curless, DH Salesin, “Image Analigies”, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp.327-340, 2001.

  A region of an image (original image I_Original and converted original image I_Converted) used as learning teacher data for obtaining the inverse transform filter f is A_train. There are the following three methods for determining A_train.

(Determination method 1)
A method in which all areas of an image are A_train.
In this method, the inverse transformation learning unit 106 sets all areas of the original image I_Original as original image teacher data I_train_orig. Further, the inverse conversion learning unit 106 sets all regions of the converted original image I_Converted as converted original image teacher data I_train_conv.

(Determination method 2)
A method to specify the ratio of images used for learning.
Specifically, the inverse transformation learning unit 106 uses the ratio R of the teacher data used for learning represented by a predetermined real number of 0 or more and 100 or less to calculate R% of the resolution of the original image I_Original. Use as data. R may be determined from a desired calculation time required for learning. For example, when the calculation time order of the learning algorithm is O (n), where n is represented by the number of pixels of the data image and learning is performed in half the time when all regions in the image are A_train, R = 0.5 and the number of pixels of the data image may be specified as half.
In this method, the inverse transformation learning unit 106 sets the original image I_Original area indicated by the designated ratio R as original image teacher data I_train_orig. In addition, the inverse transformation learning unit 106 sets a region corresponding to the region of the original image I_Original obtained above for the converted original image I_Converted as converted original image teacher data I_train_conv.

(Determination method 3)
A method that uses a region having a saliency.
Specifically, first, the inverse transformation learning unit 106 convolves the filter to obtain the edge of the original image I_Original. As a filter, a Laplacian filter or the like can be applied. Next, the inverse transformation learning unit 106 determines that a region whose edge is equal to or greater than a predetermined threshold is highly significant, and the ratio of teacher data used for learning represented by a predetermined real number of 0 or more and 100 or less. A region having high saliency in accordance with R is set as original image teacher data I_train_orig. In addition, the inverse transformation learning unit 106 sets a region corresponding to a region in which the edge of the original image I_Original obtained above is equal to or greater than a predetermined threshold for the converted original image I_Converted as converted original image teacher data I_train_conv.

  As described above, the inverse transformation learning unit 106 determines A_train by any one of the three determination methods. If all regions of the original image I_Original are used as teacher data as in the determination method 1, a high-accuracy conversion filter can be generated. However, the calculation amount can be reduced by selecting a method such as the determination methods 2 and 3. However, efficient learning is possible.

  The feature space inverse transform unit 107 performs an inverse transform filter f based on the inverse transform filter f obtained by the inverse transform learning unit 106 and the image obtained by the completion processing obtained by the completion processing unit 105. By performing a convolution operation between the image and the image subjected to the completion processing, the image subjected to the completion processing is inversely converted to the RGB feature amount space which is the same feature amount space as the original image I_Original, and the RGB feature amount space Completion result I_Completed in is generated. The feature space inverse transform unit 107 outputs the acquired completion result I_Completed as an image from which the image of the missing area Damaged is removed.

  FIG. 4 is a flowchart showing a process flow of the image generation apparatus 10. In the description of FIG. 3, the case where the original image I_Original and the mask information are individually input to the image generation apparatus 10 will be described as an example. In the description of FIG. 3, all of the reference images I_Source stored in the reference image storage unit 102 are converted from the RGB feature amount space into a predetermined post-conversion space F by the conversion unit 103, and the converted image storage is performed. An example in which the converted reference image I_Source_conv is stored in the unit 104 will be described.

  The image acquisition unit 101 individually acquires the original image I_Original and the mask information Mask from the outside (step S101). The image acquisition unit 101 outputs the acquired original image I_Original to the conversion unit 103, and outputs the original image I_Original and the mask information Mask to the completion processing unit 105. The conversion unit 103 generates the converted original image I_Converted by converting the output original image I_Original from the RGB space, which is the feature amount space of the original image I_Original, to a predetermined converted space (step S102). The conversion unit 103 outputs the converted original image I_Converted to the completion processing unit 105.

  The completion processing unit 105 acquires the converted original image I_Converted, the missing area Damaged, and the original image I_Original. The completion processing unit 105 selects a completion target patch Damaged_Patch from the acquired converted original image I_Converted (step S103). Next, the completion processing unit 105 searches for a similar patch Similar_Patch of the completion target patch Damaged_Patch from the converted reference image I_Source_conv stored in the converted image storage unit 104 (step S104).

The completion processing unit 105 performs a completion process on the missing area Damaged indicated by the mask information Mask using the searched similar patch Similar_Patch (step S105). Thereafter, the completion processing unit 105 determines whether or not the completion processing has been performed on all of the completion target regions (step S106). When the completion process has been performed on all the completion target areas (YES in step S106), the completion processing unit 105 converts the image that has been subjected to the completion process into an image that has been subjected to the completion process (post-conversion image I_Completed_conv). ) To the inverse transformation learning unit 106 and the feature amount space inverse transformation unit 107.
On the other hand, when the completion processing is not performed on all the completion target regions (step S106: NO), the image generation apparatus 10 performs steps S103 to S105 until the completion processing is completed for all the completion target regions. Repeat the process.

  The inverse transformation learning unit 106 acquires the image that has been subjected to the completion process, the converted original image I_Converted, the original image I_Original, and the mask information Mask. The inverse transformation learning unit 106 calculates the inverse transformation filter f based on the acquired original image I_Original, converted original image I_Converted, and missing area Damaged (step S107). After that, the feature space inverse transform unit 107 performs a convolution operation on the calculated inverse transform filter f and the image subjected to the completion process, thereby converting the image subjected to the completion process into the original image I_Original. Inverse conversion to the RGB feature amount space, which is the same feature amount space. By this processing, the feature amount space inverse transform unit 107 generates a completion result I_Completed in the RGB feature amount space. The feature space inverse transform unit 107 outputs the generated completion result I_Completed as an image from which the missing area is removed from the original image.

According to the image generating apparatus 10 configured as described above, conversion from a predetermined feature amount space to a feature amount space before conversion can be easily performed. Hereinafter, this effect will be described in detail.
The image generation apparatus 10 performs the completion process after converting the original image to be subjected to the completion process into an image space different from the image space of the color image that is the original image. The image generation apparatus 10 obtains, through learning, a conversion process for inversely converting the image that has been subjected to the completion process (the converted image I_Completed_conv) to the image space before the conversion. Then, the image generation apparatus 10 can perform inverse conversion to the image space before conversion by performing a convolution operation of the inverse conversion filter f on the image subjected to the completion processing based on the learning result. . As described above, the image generation apparatus 10 learns the conversion process for performing the inverse conversion by learning and performs the inverse conversion. Therefore, it is easy to convert the feature amount space after the predetermined conversion into the feature amount space before the conversion. It becomes possible to do.

<Modification>
In the present embodiment, a configuration in which an image to be completed is input to the image generation apparatus 10 from the outside has been described. However, the present invention is not limited to this. For example, the image generation apparatus 10 may further include an image storage unit that stores an image to be completed, and the completion process may be performed on the image stored in the image storage unit. In such a configuration, the mask information is input from the outside.
The functional units of the image acquisition unit 101, the reference image storage unit 102, the conversion unit 103, the post-conversion image storage unit 104, the completion processing unit 105, the inverse transformation learning unit 106, and the feature quantity space inverse transformation unit 107 are different from each other. It may be mounted on a device, or a part of each functional unit may be mounted on another device.

  You may make it implement | achieve the image generation apparatus 10 in embodiment mentioned above with a computer. In that case, it is realized by recording a program for realizing the constituent elements of the image generating apparatus 10 on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. May be. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” is a program that dynamically holds a program for a short time, like a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the above-described constituent elements, and may be realized by combining the above-described constituent elements with a program already recorded in a computer system. It may be realized by using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Image generation apparatus, 101 ... Image acquisition part, 102 ... Reference image memory | storage part, 103 ... Conversion part, 104 ... Conversion image memory | storage part, 105 ... Completion process part, 106 ... Inverse conversion learning part, 107 ... Feature-value Spatial inverse transform unit

Claims (5)

An image generation device that generates a first image by removing an image of the defect area from the original image based on an original image to be removed of the defect area and mask information indicating the defect area,
A conversion unit that generates a second image in the different image space by performing conversion on the original image to an image space different from an original image space defined by a parameter representing the original image;
A completion processing unit that generates a third image by performing a completion process on the missing region in the second image based on the mask information and the second image;
In the original image and the second image, regions that do not correspond to the missing region are set as first teacher data and second teacher data, respectively, and the second teacher data is based on the first teacher data and the second teacher data. An inverse transformation learning unit that learns an inverse transformation used for transformation from the different image space to the original image space in the first teacher data;
Feature space inverse transform that generates the first image by transforming the third image from the different image space to the original image space by using the learned inverse transform for the third image. And
An image generation apparatus comprising:   The image generation apparatus according to claim 1, wherein the inverse transformation learning unit sets all regions of the original image and the second image as the first teacher data and the second teacher data.   The inverse transformation learning unit uses the original image area and the second image area corresponding to the original image area shown at a predetermined ratio as the first teacher data and the second teacher data, respectively. The image generating apparatus according to 1.   The inverse transformation learning unit uses, as the first teacher data and the second teacher data, a region indicated by a predetermined ratio among regions whose edges are equal to or greater than a predetermined threshold in the original image and the second image. The image generation apparatus according to claim 1.   The computer program for functioning a computer as an image generation apparatus as described in any one of Claim 1 to 4.

Images