JP2017058930A - Learning data generation device, learning device, image evaluation device, learning data generation method, learning method, image evaluation method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a learning data generation device, a learning device, an image evaluation device, a learning data generation method, a learning method, an image evaluation method and an image processing program, capable of automatically generating learning data corresponding to an image in which a repair process is performed in a missing region.SOLUTION: A learning data generation device includes an input unit in which first image data and mask data that specifies a region which is to be a missing region in the first image data are inputted, and a learning data generation unit in which noise is added to the region corresponding to the missing region on the basis of the first image data and the mask data, to generate second image data containing the missing region being repaired in simulation, and learning data is generated that contains the first image data and the second image data as well as an evaluation rank order of the first image data and the second image data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a learning data generation device, a learning device, an image evaluation device, a learning data generation method, a learning method, an image evaluation method, and an image processing program.

  When shooting a still image such as a photograph or a moving image such as a video, an unnecessary object covering the subject to be shot may be shot. Still images and moving images shot with unnecessary objects covered on the subject may greatly impair the quality of experience in viewing, and unnecessary images included in still images and moving images can be removed without feeling uncomfortable. The demand for this technique is extremely high.

  In a technique called free viewpoint video synthesis, video from an arbitrary viewpoint is synthesized from a group of videos shot by a plurality of cameras. At this time, part of the synthesized video from an arbitrary viewpoint may be lost due to occlusion by the object. Such deficiencies may greatly impair the quality of viewing the video, so there is a great demand for a method that complements the deficiencies without any discomfort.

  Hereinafter, regions that are desired to be complemented, such as regions that are desired to remove reflections of unnecessary objects and regions that are not observed due to occlusion, in still images and moving images are referred to as missing regions. In addition, an image in which the defect area is given by a mask is input, and the defect area in the input image is complemented without a sense of incongruity with the area other than the defect area (hereinafter referred to as “defect peripheral area”). The repair process for acquiring the image is referred to as a completion process.

  The mask indicating the position and size of the defect area is given manually or by a known technique regardless of whether it is a still image or a moving image. As a known technique for acquiring a mask, for example, there is a technique described in Non-Patent Document 1. Note that a mask is information indicating whether or not an image to be subjected to image processing is an area on which image processing is to be performed.

  FIG. 9 is a diagram illustrating an example of a mask representing a defective region. FIG. 9A shows an example in which a mask is given as a binary image. The mask image 71 shown in FIG. 9A is given separately from the image 70 that is an original image to be subjected to image processing. The mask image 71 indicates that image processing is performed on the region indicated by the region 71a to which the dot pattern is added, and indicates that image processing is not performed on the region indicated by the white region 71b.

  FIG. 9B is an example of specifying a mask region by superimposing a mask 72 indicated by a dot pattern on an image 70 to be subjected to image processing. The mask 72 shown in FIG. 9B indicates that image processing is performed on the area indicated by the mask 72, and that image processing is not performed on other areas. When the mask 72 is superimposed on the image 70 as shown in FIG. 9B, the mask 72 is given in a color or pattern that can be easily distinguished from other regions. For example, a color that is not used in the image 70 is used as the color that can be easily distinguished from other regions.

One of the completion processing methods is a method of performing sequential repairs in units of patches. Completion processing for performing sequential repair in units of patches is performed according to the following procedure.
(Procedure 1) A small region (hereinafter referred to as a “completion target patch”) including both a defective region and a defective peripheral region is selected from the regions in the image to be subjected to the completion process. A patch that exists on the outline of the defect region and has a strong peripheral edge is selected (for example, see Non-Patent Document 2).
(Procedure 2) A similar patch is searched based on the pixels in the defect peripheral region of the completion target patch selected in Procedure 1. The search range is a still image or a moving image stored in the same image or moving image or on a storage.
(Procedure 3) The similar patch searched in Procedure 2 is copied to the completion target patch selected in Procedure 1.
The processes shown in steps 1 to 3 are repeated until there is no missing area indicated by the mask that is the completion target area.

  In the above-described completion processing, an image output as a result changes depending on a method and parameters to be selected. Therefore, in order to set an optimal method and parameters in advance in the completion process, a technique for appropriately evaluating the image quality of the image after the completion process and specifying an image with a good image quality is necessary. SSEQ (Spatial-Spectral Entropy-based Quality) is known as a method for evaluating the naturalness of the appearance of image data (see, for example, Non-Patent Document 3). And the structure which evaluates an image quality using SSEQ with respect to the image which performed the completion process can be considered.

  However, the image evaluation by SSEQ described in Non-Patent Document 3 has a problem that the accuracy of detecting the difference in image quality of images obtained by the completion processing is low. Therefore, in order to specify an image with good image quality, it is necessary for a human to manually observe and select an image after completion processing, and such work has a problem that requires time and effort.

  As a method for dealing with the above problem, a plurality of images that have been subjected to completion processing by a plurality of various methods and parameters, and a plurality of images that have been given an evaluation rank that has been evaluated and ranked in advance by evaluating the image quality of each image. A method of processing an image using ranking learning is conceivable. The process using ranking learning is to learn the correspondence between the feature amount extracted from the plurality of images and the evaluation rank of each image, and to automatically obtain an image with a high evaluation rank. It is. Here, the ranking learning is a method of learning the correspondence between the feature amount of the image and the evaluation rank in advance in the machine learning framework and specifying the evaluation rank of two or more images using the generated learning model. For such ranking learning, SVM (Support Vector Machine) -Rank or SVM described in Non-Patent Document 4 can be used.

Xue Bai, Jue Wang, David Simons and Guillermo Sapiro, "Video SnapCut: Robust Video Object Cutout Using Localized Classifiers", ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAPH 2009, Volume 28, Issue 3, August 2009, Article No .70 A. Criminisi, P. Perez, and K. Toyama, “Region Filling and Object Removal by Exemplar-Based Image Inpainting”, IEEE TRANSACTIONS ON IMAGEPROCESSING, VOL. 13, NO. 9, pp. 1200-1212, Sept. 2004. Lixiong Liu, Bao Liu, Hua Huang, and Alan Conrad Bovik, "No reference image quality assessment based on spatial and spectral entropies", Sig. Proc .: Image Comm., Vol. 29, No. 8, pp. 856-863 , 2014. T. Joachims, “Training Linear SVMs in Linear Time”, Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), 2006

  However, as described above, in the method of specifying the evaluation rank of the image that has been subjected to the restoration process such as the completion process using the ranking learning, the evaluation rank obtained by evaluating and ranking the image quality of each image after the restoration process in advance. It is necessary to prepare learning data including a plurality of assigned images. The work of preparing such learning data is a very time-consuming work.

  In view of the above circumstances, the present invention provides a learning data generation device, a learning device, an image evaluation device, a learning data generation method, and a learning data generation device that can automatically generate learning data corresponding to an image that has been repaired for a defective region. The object is to provide a learning method, an image evaluation method, and an image processing program.

  According to one aspect of the present invention, an input unit to which first image data and mask data for specifying a region to be a defective region in the first image data are input, and the first input to the input unit. Based on the image data and the mask data, noise is added to the area corresponding to the missing area to generate second image data having a simulated missing area, and the first image data and A learning data generation device includes: a learning data generation unit configured to generate learning data including the second image data and the evaluation order of the first image data and the second image data.

  One aspect of the present invention is a first feature amount extraction unit that extracts feature amounts from the first image data and the second image data based on the learning data generated by the learning data generation device; The feature amounts of the first image data and the second image data extracted by the first feature amount extraction unit, and the first image data and the second image data included in the learning data. The learning apparatus includes: a learning model generation unit that generates a learning model indicating a correspondence relationship between the feature amount and the evaluation rank based on the evaluation rank.

  One aspect of the present invention is a second feature in which a feature amount is extracted from a pixel corresponding to an outline of the defect region with respect to fourth image data obtained by repairing the third image data including the defect region. A quantity extraction unit; and an evaluation unit that evaluates the fourth image data based on the feature quantity extracted by the second feature quantity extraction unit and the learning model generated by the learning device. An image evaluation apparatus.

  One aspect of the present invention is an input step in which first image data and mask data for specifying a region to be a defective region in the first image data are input, and the first input in the input step. Generating a second image data having a simulated defective region by adding noise to a region corresponding to the defective region based on the image data and the mask data; and A learning data generation step of generating learning data including the image data and the second image data generated in the generation step, and the evaluation order of the first image data and the second image data. This is a learning data generation method.

  One aspect of the present invention is a first feature amount extraction step of extracting a feature amount from the first image data and the second image data based on the learning data generated by the learning data generation method; The feature amounts of the first image data and the second image data extracted in the first feature amount extraction step, and the first image data and the second image data included in the learning data. A learning model generation step of generating a learning model indicating a correspondence relationship between the feature quantity and the evaluation rank based on the evaluation rank.

  One aspect of the present invention is a second feature in which a feature amount is extracted from a pixel corresponding to an outline of the defect region with respect to fourth image data obtained by repairing the third image data including the defect region. 6. The fourth image data is evaluated based on the feature amount extracted in the amount extraction step, the second feature amount extraction step, and the learning model generated in the learning method according to claim 5. An image evaluation method comprising: an evaluation step.

  One aspect of the present invention is an image processing program for causing a computer to realize each functional unit in any one of the learning data generation device, the learning device, and the image evaluation device.

  According to the present invention, it is possible to automatically generate learning data corresponding to an image obtained by performing repair processing on a defective area.

It is a figure which shows the structural example of the image processing system in this embodiment. It is a figure which shows the relationship between the original image and mask data input into the image input part 200 in this embodiment, and the learning data produced | generated. It is a figure which shows the specific example of the learning data. It is a flowchart which shows operation | movement of the learning model acquisition apparatus 20 in this embodiment. It is a figure which shows the specific example of the parameter list stored in the repair information storage part 102 in this embodiment. It is a figure which shows the specific example which calculates the feature-value paying attention to the outline of the defect | deletion area | region D. FIG. It is a figure which shows the specific example of the information which specifies the edge direction centering on the pixel p. It is a flowchart which shows operation | movement of the image evaluation apparatus 10 in this embodiment. It is a figure which shows the example of the mask showing a defect | deletion area | region.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration example of an image processing system according to the present embodiment. As shown in FIG. 1, the image processing system 1 performs a plurality of types of restoration processing on an input image to generate a plurality of restored images, evaluates the image quality of the plurality of restored images, and evaluates the image quality. An image evaluation apparatus 10 that selects a high-restored image, and a learning model that extracts at least one feature amount from an image that is input learning data and shows a relationship between the extracted feature amount and an evaluation of image quality of the image A learning model acquisition device 20 for acquisition.

<About the structure of the learning model acquisition apparatus 20>
The configuration of the learning model acquisition device 20 will be described.
The learning model acquisition apparatus 20 includes an image input unit 200, a learning data generation unit 201 that generates learning data based on image data input to the image input unit 200, a feature amount extraction unit 202, a learning list DB (database) ) 203 and an evaluation rank learning unit 204.

The image input unit 200 receives the original image (first image data) and mask data indicating a region to be a defective region that is repaired by simulation by adding noise to the original image. FIG. 2 is a diagram illustrating a relationship between the original image and mask data input to the image input unit 200 and generated learning data in the present embodiment. As shown in FIG. 2, the image input unit 200 receives an original image 220 and mask data 221 indicating a region to be a defective region D that is simulated and repaired by adding noise to the original image 220. The The mask data 221 is represented by a set of pixels {(d _x1 , d _y1 ),..., (D _xn , d _yn )} corresponding to the missing region D that has been simulated and repaired by completion processing with noise. May be. The mask data 221 may be a binary image having the same size as the original image 220, with the missing area D being 1 and the other area being 0.

  Based on the original image 220, the learning data generation unit 201 generates noise-added images 222 to 225, which are images to which noise is added to simulate image quality degradation due to the effect of the completion process (restoration process). To do. The learning data generation unit 201 horizontally converts the noise-added image 222 obtained by adding blur (blur) to the original image 220, the noise-added image 223 obtained by moving the original image 220 in the vertical direction to add vertical deviation, and the original image 220. A noise-added image 224 that has been moved in the direction and added with a horizontal shift and a noise-added image 225 that has been rotated in the rotation direction by adding the rotation shift are generated.

  The learning data generation unit 201 performs a mask process based on the mask data 221 on the noise-added images 222 to 225, and only the region corresponding to the missing region D with respect to the original image 220 is the noise-added images 222 to 225. The replaced dummy images (second image data) 226 to 229 are generated. As a result, the dummy images 226 to 229 are images obtained by simulating the state in which the image quality is deteriorated by performing the completion processing on the missing region D. The dummy images 226 to 229 are image data included in the learning data.

  The learning data generation unit 201 assigns an evaluation rank = 1 that ranks the original image 220 including no noise by evaluating the image quality and ranks the evaluation rank = for the dummy images 226 to 229 including the noise. Learning data including the original image 220 and the dummy images 226 to 229 associated with the evaluation rank is generated by giving the second rank.

Here, a specific example of a method for generating the noise-added images 222 to 225 will be described.
The learning data generation unit 201 uses a method of generating a noise-added image 222 by adding blur to the original image 220 by a process of applying a low-pass filter to the original image 220. As a method for generating a noise-added image 223 by adding a vertical shift, the learning data generation unit 201 uses a method of generating the original image 220 by a process of enlarging x% and shifting it by α1 pixels in the vertical direction.

  As a method for generating a noise-added image 224 by adding a horizontal shift, the learning data generation unit 201 uses a method of generating the original image 220 by a process of enlarging x% and shifting it by α2 pixels in the horizontal direction. The learning data generation unit 201 uses a method of generating a noise-added image 225 by adding a rotational deviation by a process of rotating α degrees about the center of the original image 220 as the rotation center.

  Here, α1 and α2 are integer values of 1 or more, and when an object exists in the missing area D of the original image 220, the object goes out of the missing area D in the noise-added images 223 and 224. Set the value so as not to deviate. α degree is an integer that is equal to or greater than 1 and excluding a multiple of 360, and has a pixel value that changes by rotation. For example, if there is an object to be rotated and the center of the object to be rotated is the center of rotation of the rotation deviation, the objects to be rotated overlap (value before rotation). And the same figure after rotation).

  In addition, the image which added the noise which imitated degradation of the image quality by the influence of a completion process to an image is not limited to the noise addition image 222-225 mentioned above. For example, an image with an oblique shift, an image with noise that combines blur and a predetermined shift, or the like may be used.

  In addition, it may be configured to include a detection unit that detects a pixel whose value has changed from the original image 220 by the process of adding noise, or detects a difference between the original image 220 and the noise-added image after the addition of noise. By providing this detection unit, when there is almost no difference, it is assumed that sufficient noise could not be generated, and noise is added to the original image 220 using parameters (α1, α2, α) having different values. can do.

  Specifically, the learning data generation unit 201 generates the dummy images 226 to 229 in the following procedure. An original image 220 and mask data 221 are input to the image input unit 200. Here, the completion processing based on the mask data 221 is not performed, and the mask data 221 is prepared to indicate the defective region D subjected to the completion processing.

  The learning data generation unit 201 generates the noise-added images 222 to 225 by adding the above-described various noises to the original image 220. The learning data generation unit 201 performs the process of adding noise on the entire original image 220. For example, in the process of adding blur, the learning data generation unit 201 generates an image obtained by applying a low-pass filter to the entire original image 220 as the noise addition image 222.

  The learning data generation unit 201 generates dummy images 226 to 229 by replacing only the region D indicated by the mask data 221 with the data of the noise-added images 222 to 225 with respect to the original image 220. For example, in the dummy image 226, only the inside of the rectangular area corresponding to the area D is data of the noise-added image 222, and the other areas are composed of data of the original image 220. The dummy image 226 is an image obtained by simulating a state in which the image quality is deteriorated by performing the completion process on the missing region D. On the other hand, the original image 220 is an image obtained by simulating a state in which the image quality is not deteriorated at all by performing the completion processing on the defective area D.

  A configuration example of learning data generated by the learning data generation unit 201 will be described. FIG. 3 is a diagram illustrating a specific example of the learning data 24. As shown in FIG. 3, the learning data 24 is used when obtaining a group ID 241 indicating a group of images to which a series of evaluation ranks have been assigned, an evaluation rank 242 for the restored image data, and restored image data 244. The list format data includes the mask information 243 and the restored image data 244. The learning data 24 is a variable length table in which the length of the list varies depending on how many image data 244 after restoration are compared.

  As shown in FIG. 3, the learning data 24 associates the mask information 243 with the group ID, the restored image data 244, and the evaluation rank 242 in the restored image data 244 of the same group. . Therefore, the number of restored image data 244 in the same group needs to be two or more. Further, when the number of image data 244 after restoration of the same group ID is K, the number of the same rank among them is K−1 or less so that at least one different evaluation rank can be assigned. Must.

  An example of correspondence between the original image 220 and the dummy images 226 to 229 shown in FIG. 2 and the data shown in FIG. 3 will be described. “Output1.bmp” of the restored image data 244 shown in FIG. 3 is, for example, the original image 220 shown in FIG. 2, and since there is no deterioration in image quality, the evaluation rank is “1” indicating the first place. “Output2.bmp” of the restored image data 244 is, for example, a dummy image 226, and the image quality deteriorates due to the addition of noise, so the evaluation rank is “2” indicating the second place. “Output3.bmp” of the restored image data 244 is, for example, a dummy image 227, and the image quality deteriorates due to the addition of noise, so the evaluation rank is “2” indicating the second place. The mask data 221 in FIG. 2 is, for example, “mask1.bmp” in FIG. As shown in FIG. 3, the learning data 24 associates different original images for each ID, mask data, and dummy images created based on these original images and mask data, and the original image has an evaluation rank = 1. The dummy image is associated with evaluation rank = 2.

The feature quantity extraction unit 202 focuses on the outline of the defect region D and extracts three feature quantities, ie, a color consistency feature quantity F _u , an edge consistency feature quantity F _v, and an edge continuity feature quantity F _c . The feature amount extraction unit 202 extracts feature amounts from the original image 220 and the dummy images 226 to 229 based on the original image 220, the dummy images 226 to 229, and the mask data 221. A detailed method for extracting the three feature amounts of the color consistency feature amount F _u , the edge consistency feature amount F _v, and the edge continuity feature amount F _c will be described later.

  The learning list DB 203 is a database that stores evaluation ranks 242 and three feature amounts in association with group IDs.

  The evaluation rank learning unit 204 refers to the learning list from the learning list DB 203 and generates a learning model by performing ranking learning on the correspondence relationship between the image feature amount and the evaluation rank. Ranking learning is a process of generating a learning model by learning in advance the correspondence between image feature amounts and evaluation ranks in a machine learning framework. If this learning model is used, the evaluation order for these images can be calculated based on the feature amounts of two or more images. Such ranking learning is a known technique, and for example, a method such as SVM (Support Vector Machine) -Rank can be used.

The evaluation rank learning unit 204 generates a learning model indicating a correspondence relationship between the image feature amount and the image evaluation rank, and outputs the learning model to the image evaluation apparatus 10. The evaluation rank learning unit 204 associates the three feature quantities of the color consistency feature quantity F _u , the edge consistency feature quantity F _v, and the edge continuity feature quantity F _c extracted from the image with the evaluation rank of the image. Generate a learning model that shows the relationship.

  Note that the learning model generated in the evaluation rank learning unit 204 is not limited to the configuration for estimating the evaluation rank as long as the data is used for classifying data in multiple stages. For example, a learning model that performs binary classification indicating, for example, image quality evaluation OK and image quality evaluation NG based on the feature amount of the image may be generated by processing the learning list using SVM. The value output by the learning model is not limited to binary, and for example, a multi-level classification of 5 levels may be output.

<Operation of Learning Model Acquisition Device 20>
Next, operation | movement of the learning model acquisition apparatus 20 in this embodiment is demonstrated. FIG. 4 is a flowchart showing the operation of the learning model acquisition apparatus 20 in the present embodiment. In the image input unit 200, the original image 220 and the mask data 221 are input (step S200). Based on the original image 220 and the mask data 221, the learning data generation unit 201 stores the original image 220, the dummy images 226 to 229, the evaluation rank of the original image 220, and the dummy images 226 to 229 as shown in FIGS. Learning data 24 including the evaluation rank is generated (step S201).

The feature amount extraction unit 202 starts loop processing for extracting feature amounts for all of the learning data 24 generated by the learning data generation unit 201 (step S202). The feature amount extraction unit 202 acquires the restored image data 244 from the learning data 24 to be processed (step S203). The feature quantity extraction unit 202 pays attention to the outline of the defect area D with respect to the acquired image data 244 after restoration, and features the color consistency feature quantity F _u , the edge consistency feature quantity F _v, and the edge continuity feature quantity. extracting the three feature amounts of F _c. The feature amount extraction unit 202 stores the extracted feature amount and the evaluation rank 242 of the image data 244 in the learning list DB 203 in association with the group ID (step S204).

  The feature quantity extraction unit 202 returns to step S202 if the learning data 24 to be processed is not the last data, ends the loop processing if the learning data 24 to be processed is the last data, and proceeds to step S206 ( Step S205). Through the above loop processing, a database that stores the evaluation rank 242 and the three feature quantities in association with the group ID is constructed in the learning list DB 203.

  The evaluation rank learning unit 204 refers to the learning list from the learning list DB 203 and generates a learning model by performing learning processing on the correspondence relationship between the image feature amount and the evaluation rank (step S206). The learning model generated by the evaluation rank learning unit 204 is output to the image evaluation device 10 and stored in the learning model DB 105.

<Configuration of Image Evaluation Apparatus 10>
The configuration of the image evaluation apparatus 10 will be described.
The image evaluation apparatus 10 includes an input unit 101, a repair information storage unit 102, a repair processing unit 103, a feature amount extraction unit 104, a learning model DB (database) 105, and an image selection unit 106. The input unit 101 receives an image (third image data) I _in to be subjected to a completion process (restoration process) and a mask image I _mask that specifies a missing area or color information C that represents a mask area. . An image input as the image I _in is a still image or a moving image (video). The input unit 101 obtains a missing area D _in the image I _in based on the mask image I _mask or the color information C. Here, the color information C can be represented by a three-dimensional vector C (R, G, B) having RGB values (R = red, G = green, B = blue), and has, for example, a magenta color. The color information C can be expressed as C (R, G, B) = (255, 0, 255) when designated as a mask area.

Specific processing in the input unit 101 will be described with reference to FIG. When the image 70 which is the original image shown in FIG. 9A is input as the image I _in and the mask image 71 indicating the missing area is input, the input unit 101 displays the area indicated by the area 71b. Is identified as a defect region D. In addition, when the image 70 and the mask 72 that is the color information C superimposed on the image 70 are input as the image I _in that is the original image illustrated in FIG. A mask image in which the area indicated by 72 and the area not shown by binary are represented by binary values is generated and used as the defect area D. The deficient area D may be represented by a set of pixel positions designated as deficient {(d _x1 , d _y1 ), ..., (d _xn , d _yn )}. Further, the defect area D may be an image having the same size as the image I _in, and may be a binary image in which the defect area is 1 and the other area is 0.

  In the repair information storage unit 102, M repair parameters such as a repair method, a patch size, and the number of pyramid layers are registered in a list format. FIG. 5 is a diagram illustrating a specific example of a parameter list stored in the repair information storage unit 102 according to the present embodiment. As shown in FIG. 5, a restoration method indicating a completion processing method and parameters 1, 2,..., N used for the restoration method are stored in association with an ID for identifying each parameter list. Here, for example, parameter 1 is the patch size indicating the size of the completion target patch, and parameter 2 is the number of pyramid hierarchies when the resolution of an image for searching for similar patches is changed stepwise.

  In the specific example of FIG. 5, ID = 1, 2,..., M are set. That is, the repair information storage unit 102 has a parameter list that is repair information of M types (M is an integer of 2 or more). As repair information, repair method = A, parameter 1 = 3, parameter 2 = 3,..., Parameter N = I are stored in association with ID = 1, and repair method = X, in association with ID = M. Parameter 1 = 5, parameter 2 = 2,..., Parameter N = J are stored.

The repair processing unit 103 receives the completion target image I _in and the missing region D from the input unit 101, and acquires a parameter list from the repair information storage unit 102. The restoration processing unit 103 performs restoration processing corresponding to all of the M types of parameter lists, and outputs M images after restoration (fourth image data) I _{out_1 to} I _{out_M} .

Feature quantity extracting unit 104 inputs the image _I out_1 _{~I OUT_M} the repaired inputted from the recovery processing unit 103, and outputs the extracted feature amount of each image _I out_1 _{~I out_M.} Specifically, the feature quantity extraction unit 104, based on the value of the pixel corresponding to the outline of the missing area D, the color consistency feature quantity (first feature quantity) F _u , the edge consistency feature quantity (second features) F _v and edge continuity feature quantity (third feature quantity) to extract the three feature amounts of F _c of. Here, the pixels corresponding to the outline of the defect area D include pixels located around the pixels located on the outline of the defect area D in addition to the pixels located on the outline of the defect area D.

  FIG. 6 is a diagram illustrating a specific example in which the feature amount is calculated by paying attention to the outline of the defect region D. In FIG. 6, a region Ω indicated by a hatched portion indicates a defective region D, and the outline of the defective region D is expressed by δΩ. A patch having a size of 3 × 3 centering on a pixel p on the contour δΩ is represented by P (p).

ここで、ｕ（ｑ）はパッチＰ（ｐ）に含まれる画素ｑのＲＧＢ値である。 << Calculation Method of Color Consistency Feature Quantity _Fu >>
The feature quantity extraction unit 104 includes the color of the pixel included in the defective region D and the periphery of the defective region D not included in the defective region D for each small patch P (p) centered on the pixel p on the contour δΩ. The color consistency feature amount _Fu , which is a feature amount indicating whether or not there is a change in the color of the pixel, is calculated based on (Expression 1) to (Expression 3) described below. First, the feature amount extraction unit 104 calculates the first color feature amount U ₁ based on the following (Equation 1).

Here, u (q) is the RGB value of the pixel q included in the patch P (p).

As shown in (Expression 1) above, the feature amount extraction unit 104 includes pixels located on the contour δΩ and within the contour δΩ among the 3 × 3 pixels included in the patch P (p) (hereinafter referred to as missing region pixels). calculating a first color characteristic amount U ₁ the value obtained by adding the respective RGB values) called.

上記（式２）に示すように、特徴量抽出部１０４は、パッチＰ（ｐ）に含まれる３×３の画素の内、欠損領域画素を除いた画素のＲＧＢ値のそれぞれを加算した値を第２色特徴量Ｕ_２として算出する。 Next, the feature amount extraction unit 104 calculates the second color feature amount U ₂ based on the following (Formula 2).

As shown in the above (Formula 2), the feature amount extraction unit 104 adds a value obtained by adding each of the RGB values of the pixels excluding the defective region pixels among the 3 × 3 pixels included in the patch P (p). calculating a second color characteristic amount _{U 2.}

The feature amount extraction unit 104 calculates a pixel average value obtained by dividing the first color feature amount U ₁ by the pixel number a of the defective area pixel and the second color feature amount U ₂ as the total number of pixels of the patch P (p) (see FIG. Based on the following (Equation 3) that calculates the ratio with the average value divided by the number of pixels b (b = 9−a in FIG. 6) obtained by subtracting the number of pixels a from 3 × 3 = 9 pixels in FIG. The color consistency feature value _Fu is calculated.

As is clear from (Equation 1) to (Equation 3), the color consistency feature amount _Fu is the color of the defective region pixel and the pixels around the defective region D that are not the defective region pixels in the region of the patch P (p). The more consistent the color is, the closer the value is to 1. The color consistency feature value _Fu is greater when the color of the defective region pixel is different from the color of the pixel around the defective region D that is not the defective region pixel in the region of the patch P (p). If 1 color feature value U ₁ > second color feature value U ₂ , the value is large, and if 1 color feature value U ₁ <second color feature value U ₂ , the value approaches 0.

Described above (Equation 1) Color consistency feature amount F _u obtained using to (Equation 3) is one example, but is not limited thereto. For example, the color consistent feature amount F _u shown in equation (3), and divided by a value multiplied by a constant, a value obtained by adding or subtracting a constant, may be color consistent feature amount F _u. The feature amount extraction unit 104 calculates various feature amounts indicating whether or not there is a change in the color of the pixels included in the defective region D and the color of the pixels around the defective region D that are not included in the defective region D. The color consistency feature value _Fu may be calculated using a calculation formula. For example, the feature amount extraction unit 104 calculates the color difference between the color of the pixel included in the defective region D and the color of the pixel around the defective region D not included in the defective region D as the color consistency feature amount _Fu. Also good.

ここで、ｖ（ｑ）はパッチＰ（ｐ）に含まれる画素ｑのエッジ強度を表す。 The method of calculating the << edge consistent feature amount _{F v} >>
The feature amount extraction unit 104 changes the edge strength of pixels included in the defective region D and the edge strength of pixels around the defective region D that are not included in the defective region D for each patch P (p) of the small region. whether a is a characteristic quantity showing the edge consistent feature amount F _v will be described below is calculated on the basis of (equation 4) to (equation 5). First, the feature amount extraction unit 104 calculates the first edge feature amount V ₁ based on the following (Equation 4).

Here, v (q) represents the edge strength of the pixel q included in the patch P (p).

As shown in (Expression 4) above, the feature amount extraction unit 104 uses the value obtained by adding the edge strengths of the missing region pixels among the 3 × 3 pixels included in the patch P (p) as the first edge feature amount V. Calculated as ₁ . The edge strength of a pixel is obtained, for example, by adding the absolute value of the difference in luminance value between the pixel q and surrounding pixels. The luminance value of the pixel is a value obtained by R + G + B using the RGB value of the pixel, for example.

上記（式５）に示すように、特徴量抽出部１０４は、パッチＰ（ｐ）に含まれる３×３の画素の内、欠損領域画素を除いた画素のエッジ強度を加算した値を第２エッジ特徴量Ｖ_２として算出する。 Next, the feature extraction unit 104 calculates a second edge feature quantity V ₂ based on the following equation (5).

As shown in the above (Formula 5), the feature amount extraction unit 104 calculates the second value obtained by adding the edge intensities of the pixels excluding the defective region pixels from the 3 × 3 pixels included in the patch P (p). calculated as an edge feature quantity _{V 2.}

Feature amount extraction unit 104, the average value of the pixel divided by the number of pixels a defective region pixel of the first edge feature quantity V _1, a second edge feature quantity V ₂ from the total number of pixels of the patch P (p) The edge consistency feature amount _Fv is calculated based on the following (Equation 6) that calculates the ratio of the average value of the pixels divided by the pixel number b obtained by subtracting the pixel number a.

As is clear from (Equation 4) to (Equation 6), the edge consistency feature amount _Fv is obtained in the region of the patch P (p), and the edge strength of the defective region pixel and the peripheral region of the defective region D that is not the defective region pixel. The value is closer to 1 as the edge strength of the pixel is more consistent (the edge strength is the same). The edge consistency feature amount _Fv is such that, in the region of the patch P (p), the first edge feature amount V becomes larger as the edge strength of the defective region pixel differs from the edge strength of the peripheral pixel of the defective region D that is not a defective region pixel. _{If 1} > second edge feature value V ₂ , the value is a large value, and if first edge feature value V ₁ <second edge feature value V ₂ , the value approaches 0.

It described above (Equation 4) to the edge consistent feature amount F _v obtained using the equation (6) is an example, but is not limited thereto. For example, the edge consistent feature amount F _v shown in (Equation 6), and dividing a value multiplied by a constant, a value obtained by adding or subtracting a constant, may be an edge consistent feature amount F _v. The feature amount extraction unit 104 indicates whether or not there is a change (level difference) between the edge strength of the pixel included in the defective region D and the edge strength of the pixel around the defective region D not included in the defective region D. the amount of the various formulas may be calculated edge consistent feature amount F _v using to calculate.

For example, the feature amount extraction unit 104 acquires the edge strength of the pixels included in the defective region D together with the edge direction information, and determines the edge strength of the pixels around the defective region D that are not included in the defective region D as the edge direction information. acquired in combination with this, by multiplying a weight coefficient corresponding to the direction of the coincidence of the edge may be calculated edge consistent feature amount F _v by the process of addition. The weighting coefficient according to the degree of coincidence of the edge direction is a coefficient that changes from −1 to 1 according to the degree of coincidence, for example, 1 in the same direction and −1 in the opposite direction. The feature amount extraction unit 104 is configured to obtain the edge intensity based on the luminance value of the pixel, but is not limited to this. For example, the feature amount extraction unit 104 may obtain the edge strength based on the RGB values of the pixels. In this case, the edge strength is obtained for each of R, G, and B colors.

<< method of calculating the edge continuity feature quantity _{F c} >>
The feature amount extraction unit 104 has an edge continuity feature amount F _c that is a feature amount indicating whether or not there is a change in the edge direction of each pixel p on the contour δΩ and the edge direction of the pixel adjacent to the pixel p. Is calculated by the following method.

  The feature amount extraction unit 104 obtains an edge direction for each pixel p located on the contour δΩ. The edge direction may be information that specifies two neighboring pixels in the edge direction centered on the pixel p. FIG. 7 is a diagram illustrating a specific example of information for specifying an edge direction centered on the pixel p. As shown in FIG. 7, the diagonal direction (1) passing through the lower left pixel and the upper right pixel around the pixel p, the horizontal direction (2) passing through the left pixel and the right pixel, and the vertical direction passing through the upper pixel and the lower pixel. (3) and an oblique direction (4) passing through the lower right pixel and the upper left pixel. The feature quantity extraction unit 104 may specify a likely direction among the four directions shown in FIG. For example, the feature amount extraction unit 104 obtains c (p) = dy / dx based on the edge strengths (dx, dy) in the x and y directions of the luminance value of the pixel p (corresponding to obtaining tangent) There is a method for determining which of the four directions corresponds to the inclination.

Next, the feature amount extraction unit 104 similarly obtains c (pp) and c (pn) for the pixels pp and pn adjacent to the pixel p, and averages these values c (p ′) = [c ( pp) + c (pn)] / 2. The feature amount extraction unit 104 obtains c (p) and c (p ′) for all the pixels on the contour δΩ, and the first edge direction specified by c (p) and c (p ′) The second edge direction specified by is acquired. The feature amount extraction unit 104 obtains a correlation value indicating the degree of coincidence between the first edge direction and the second edge direction, and uses this correlation value as the edge continuity feature amount F _c . Note that the positions of the pixels pp and pn adjacent to the pixel p are, for example, the pixel adjacent to the lower side of the pixel p is the pixel pp and the pixel adjacent to the right side of the pixel p is the position of the pixel pn. It is not limited to this.

  The learning model DB 105 stores a learning model indicating the relationship between the image feature amount acquired by the learning model acquisition device 20 and the image quality evaluation of the image. Specifically, the learning model, when at least two or more feature quantities are input, estimates and outputs a ranking indicating whether the image quality of each feature quantity is excellent. Information indicating the relationship.

The image selection unit 106 performs M restorations output from the restoration processing unit 103 based on the feature amounts of the images I _{out — 1 to} I out — _M extracted by the feature amount extraction unit 104 and the learning model stored in the learning model DB 105. An evaluation rank is _assigned to each of the subsequent images I _{out — 1 to} I out — _M , and an image I _best ranked first with an estimated rank that is estimated to have the best image quality is output.

<Operation of Image Evaluation Apparatus 10>
Next, the operation of the image evaluation apparatus 10 in this embodiment will be described. FIG. 8 is a flowchart showing the operation of the image evaluation apparatus 10 in the present embodiment. In the input unit 101, an image I _in to be subjected to completion processing and a mask image I _mask for designating a missing area are input (step S101). The input unit 101 identifies the missing area D _in the image I _in based on the mask image I _mask (step S102).

The repair processing unit 103 performs a repair process corresponding to all of the M types of repair information (parameter list) acquired from the repair information storage unit 102 based on the image I _in and the defect area D input from the input unit 101. performed on _in, and outputs the image _I out_1 _{~I OUT_M} after M pieces of repair (step S103). Feature quantity extracting unit 104 inputs the image _I out_1 _{~I OUT_M} the repaired inputted from the recovery processing unit 103, and outputs the extracted feature amount of each image _I out_1 _{~I out_M} (step S104). Specifically, the feature quantity extraction unit 104 determines the color consistency feature quantity F _u and the edge consistency feature quantity F _v from each of the images I _{out — 1} to I out — _M based on the value of the pixel corresponding to the outline of the missing region D. and extracted three characteristic amounts of edge continuity feature quantity F _c.

The image selection unit 106 performs M restorations output from the restoration processing unit 103 based on the feature amounts of the images I _{out — 1 to} I out — _M extracted by the feature amount extraction unit 104 and the learning model stored in the learning model DB 105. Evaluation ranks are _assigned to the _subsequent images I _{out — 1 to} I out — _M (step S105). Image selecting unit 106 selects the image imparted with evaluation rank = 1 of the most image quality is estimated to be from the images _I out_1 _{~I OUT_M} outputs as an image _{I best} (step S106).

  As described above, the learning model acquisition device 20 according to the present embodiment can automatically generate learning data corresponding to an image obtained by performing a repair process on a defective area. The image processing apparatus 30 according to the present embodiment extracts feature amounts from a plurality of restored image data that has been subjected to completion processing using various methods and parameters, and the restored image based on the extracted feature amounts. Evaluation rank can be given to data. Thus, by selecting the restored image data having the best evaluation order, it is possible to obtain image data that has been automatically subjected to the optimum completion processing. In addition, unlike existing image evaluation methods, a feature amount focused on the boundary of the missing region D is extracted, and an evaluation order based on the feature amount is obtained with reference to the learning model, thereby achieving higher accuracy. Realizes image quality evaluation. In addition, the learning model acquisition device 20 can improve the accuracy of the learning model by generating a learning model based on a learning list in which feature amounts extracted from learning data are associated with evaluation ranks.

  Each function part with which image evaluation device 10 and learning model acquisition device 20 in this embodiment mentioned above are provided is realizable with a computer, for example. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. You may implement | achieve using programmable logic devices, such as FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The learning data generation apparatus, learning apparatus, image evaluation apparatus, learning data generation method, learning method, image evaluation method, and image processing program of the present invention can be used in an image processing apparatus that performs a completion process.

DESCRIPTION OF SYMBOLS 1 ... Image processing system, 10 ... Image evaluation apparatus, 20 ... Learning model acquisition apparatus, 102 ... Repair information storage part, 103 ... Repair processing part, 104, 202 ... Feature-value extraction part, 105 ... Learning model DB, 106 ... Image Selection unit, 200 ... Image input unit, 201 ... Learning data generation unit, 203 ... Learning list DB, 204 ... Evaluation order learning unit

Claims (7)

An input unit to which first image data and mask data for specifying a region to be a defective region in the first image data are input;
Based on the first image data and the mask data input to the input unit, noise is added to a region corresponding to the defect region, and second image data having a defect region repaired in a simulated manner is obtained. A learning data generating unit that generates learning data including the first image data and the second image data, and evaluation ranks of the first image data and the second image data;
A learning data generation apparatus comprising: A first feature amount extraction unit that extracts feature amounts from the first image data and the second image data based on the learning data generated by the learning data generation device according to claim 1;
The feature amounts of the first image data and the second image data extracted by the first feature amount extraction unit, and the first image data and the second image data included in the learning data. A learning model generation unit that generates a learning model indicating a correspondence relationship between the feature amount and the evaluation rank based on the evaluation rank;
A learning apparatus comprising: A second feature quantity extraction unit for extracting feature quantities from pixels corresponding to the outline of the missing area for the fourth image data obtained by repairing the third image data including the missing area;
An evaluation unit that evaluates the fourth image data based on the feature amount extracted by the second feature amount extraction unit and the learning model generated by the learning device according to claim 2;
An image evaluation apparatus comprising: An input step in which first image data and mask data for specifying a region to be a defective region in the first image data are input;
Based on the first image data and the mask data input in the input step, the second image data having a defect region that is simulated and repaired by adding noise to the region corresponding to the defect region. A generation step to generate;
A learning data generation step for generating learning data including the first image data and the second image data generated in the generation step, and the evaluation order of the first image data and the second image data. When,
A learning data generation method comprising: A first feature amount extraction step of extracting a feature amount from the first image data and the second image data based on the learning data generated in the learning data generation method according to claim 4;
The feature amounts of the first image data and the second image data extracted in the first feature amount extraction step, and the first image data and the second image data included in the learning data. A learning model generation step for generating a learning model indicating a correspondence relationship between the feature amount and the evaluation rank based on the evaluation rank;
Learning method. A second feature amount extraction step of extracting feature amounts from pixels corresponding to the outline of the defect region with respect to the fourth image data obtained by repairing the third image data including the defect region;
An evaluation step of evaluating the fourth image data based on the feature amount extracted in the second feature amount extraction step and the learning model generated in the learning method according to claim 5;
An image evaluation method comprising:   An image for realizing each functional unit in any one of the learning data generation device according to claim 1, the learning device according to claim 2, and the image evaluation device according to claim 3 by a computer. Processing program.

Images