JP2017058929A - Image information acquisition method, image evaluation method, image information acquisition device, image evaluation device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image information acquisition method, an image evaluation method, an image information acquisition device, an image evaluation device and an image processing program, for precisely evaluating picture quality of an image in which repair process is performed on a missing region.SOLUTION: An image information acquisition method includes a feature amount extracting step of extracting a feature amount from a pixel corresponding to a profile of a missing region, in an input image. The feature amount includes at least one of a first feature amount that is obtained based on a color of a pixel on the profile and a color of a pixel that is not included in the missing region positioned in a periphery of the pixel corresponding to the profile, a second feature amount that is obtained based on an edge intensity of the pixel on the profile and an edge intensity of the pixel that is not included in the missing region positioned in the periphery of the pixel corresponding to the profile, and a third feature amount that is obtained based on an edge orientation at the pixel on the profile and an edge orientation at the pixel positioned in the periphery of the pixel corresponding to the profile, not included in the missing region.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image information acquisition method, an image evaluation method, an image information acquisition device, an image evaluation device, and an image processing program.

  When shooting a still image such as a photograph or a moving image such as a video, an unnecessary object covering the subject to be shot may be shot. Still images and moving images shot with unnecessary objects covered on the subject may greatly impair the quality of experience in viewing, and unnecessary images included in still images and moving images can be removed without feeling uncomfortable. The demand for this technique is extremely high.

  In a technique called free viewpoint video synthesis, video from an arbitrary viewpoint is synthesized from a group of videos shot by a plurality of cameras. At this time, part of the synthesized video from an arbitrary viewpoint may be lost due to occlusion by the object. Such deficiencies may greatly impair the quality of viewing the video, so there is a great demand for a method that complements the deficiencies without any discomfort.

  Hereinafter, regions that are desired to be complemented, such as regions that are desired to remove reflections of unnecessary objects and regions that are not observed due to occlusion, in still images and moving images are referred to as missing regions. In addition, an image in which the defect area is given by a mask is input, and the defect area in the input image is complemented without a sense of incongruity with the area other than the defect area (hereinafter referred to as “defect peripheral area”). The process of acquiring the process is referred to as a completion process.

  The mask indicating the position and size of the defect area is given manually or by a known technique regardless of whether it is a still image or a moving image. As a known technique for acquiring a mask, for example, there is a technique described in Non-Patent Document 1. Note that a mask is information indicating whether or not an image to be subjected to image processing is an area on which image processing is to be performed.

  FIG. 10 is a diagram illustrating an example of a mask representing a defective region. FIG. 10A shows an example in which a mask is given as a binary image. A mask image 71 shown in FIG. 10A is given separately from the image 70 that is an original image to be subjected to image processing. The mask image 71 indicates that image processing is performed on the region indicated by the region 71a to which the dot pattern is added, and indicates that image processing is not performed on the region indicated by the white region 71b.

  FIG. 10B is an example of specifying a mask region by superimposing a mask 72 indicated by a dot pattern on an image 70 to be subjected to image processing. A mask 72 shown in FIG. 10B indicates that image processing is performed on the area indicated by the mask 72, and that image processing is not performed on other areas. When the mask 72 is superimposed on the image 70 as shown in FIG. 10B, the mask 72 is given in a color or pattern that can be easily distinguished from other regions. For example, a color that is not used in the image 70 is used as the color that can be easily distinguished from other regions.

One of the completion processing methods is a method of performing sequential repairs in units of patches. Completion processing for performing sequential repair in units of patches is performed according to the following procedure.
(Procedure 1) A small region (hereinafter referred to as a “completion target patch”) including both a defective region and a defective peripheral region is selected from the regions in the image to be subjected to the completion process. A patch that exists on the outline of the defect region and has a strong peripheral edge is selected (for example, see Non-Patent Document 2).
(Procedure 2) A similar patch is searched based on the pixels in the defect peripheral region of the completion target patch selected in Procedure 1. The search range is a still image or a moving image stored in the same image or moving image or on a storage.
(Procedure 3) The similar patch searched in Procedure 2 is copied to the completion target patch selected in Procedure 1.
The processes shown in steps 1 to 3 are repeated until there is no missing area indicated by the mask that is the completion target area.

  In the above-described completion processing, an image output as a result changes depending on a method and parameters to be selected. Therefore, in order to set an optimal method and parameters in advance in the completion process, a technique for appropriately evaluating the image quality of the image after the completion process and specifying an image with a good image quality is necessary. SSEQ (Spatial-Spectral Entropy-based Quality) is known as a method for evaluating the naturalness of the appearance of image data (see, for example, Non-Patent Document 3). And the structure which evaluates an image quality using SSEQ with respect to the image which performed the completion process can be considered.

Xue Bai, Jue Wang, David Simons and Guillermo Sapiro, "Video SnapCut: Robust Video Object Cutout Using Localized Classifiers", ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAPH 2009, Volume 28, Issue 3, August 2009, Article No .70 A. Criminisi, P. Perez, and K. Toyama, “Region Filling and Object Removal by Exemplar-Based Image Inpainting”, IEEE TRANSACTIONS ON IMAGEPROCESSING, VOL. 13, NO. 9, pp. 1200-1212, Sept. 2004. Lixiong Liu, Bao Liu, Hua Huang, and Alan Conrad Bovik, "No reference image quality assessment based on spatial and spectral entropies", Sig. Proc .: Image Comm., Vol. 29, No. 8, pp. 856-863 , 2014.

  However, the image evaluation by SSEQ described in Non-Patent Document 3 has a problem that the accuracy of detecting the difference in image quality of images obtained by the completion processing is low. Therefore, in order to identify an image with good image quality, it is necessary for a human to manually observe and select an image after completion processing, and such operations require time and effort, and human image quality This included variations in judgment.

  In view of the above circumstances, the present invention provides an image information acquisition method, an image evaluation method, an image information acquisition device, and an image evaluation device for accurately evaluating the image quality of an image that has been repaired for a defective region. And an image processing program.

  In one aspect of the present invention, a first input step of inputting an image in which a defective area is repaired, and a feature amount from pixels corresponding to the outline of the defective area are input to the image input in the first input step. A feature amount extracting step to extract, wherein the feature amount is a color of a pixel on the contour and a color of a pixel not included in the defective region located around a pixel corresponding to the contour. A second feature value obtained on the basis of a first feature value obtained based on the edge strength of a pixel on the contour and an edge strength of a pixel not included in the defect region located around the pixel corresponding to the contour. And a third feature obtained based on an edge direction of a pixel on the contour, and an edge direction of a pixel located around the pixel corresponding to the contour and not included in the defect region Izu with quantity Including at least image information acquisition method one or.

  One aspect of the present invention is the image information acquisition method, wherein the first feature amount is based on (Expression 1) to (Expression 3) in a small region centered on a pixel on the contour. The second feature value is obtained based on (Expression 4) to (Expression 6), and the third feature value is an edge in a horizontal direction in a pixel corresponding to the contour. The edge direction in the pixel is specified based on the intensity and the edge strength in the vertical direction, and obtained based on the specified edge direction.

  According to one aspect of the present invention, a plurality of second images restored by a plurality of restoration methods with respect to a first image having a defective area, and an evaluation rank that is a rank of image quality evaluation for the plurality of second images, A second input step in which is inputted, an extraction step of extracting the feature amount by applying the image information acquisition method to the plurality of second images inputted in the second input step, The image evaluation method includes a learning step of learning a correspondence relationship between the feature amount extracted in the extraction step and the evaluation rank input in the second input step.

  According to one aspect of the present invention, a third input step in which a plurality of third images repaired by a plurality of repair methods are input to the first image having a defect region is input in the third input step. An extraction step of extracting the feature amount by applying the image information acquisition method to a plurality of the third images, and a plurality of the third images based on the feature amount extracted in the extraction step And an image selection step of selecting a fourth image having the highest image quality evaluation from among them.

  One aspect of the present invention corresponds to the outline of the defect region with respect to the first input unit to which an image in which the defect region is repaired is input and the image including the defect region input to the first input unit. A feature amount extraction unit that extracts a feature amount from a pixel, and the feature amount includes a color of a pixel on the contour and a pixel that is not included in the defect region located around the pixel corresponding to the contour Based on the first feature amount obtained based on the color of the pixel, the edge strength of the pixel on the contour, and the edge strength of the pixel not included in the defective region located around the pixel corresponding to the contour And the edge direction of a pixel on the contour, and the edge direction of a pixel located around the pixel corresponding to the contour and not included in the defective region Any one of the third feature values An image information acquisition device including at least.

  One aspect of the present invention is the image information acquisition device according to the first aspect, wherein the first feature amount is based on (Expression 1) to (Expression 3) in a small region centered on a pixel on the contour. The second feature value is obtained based on (Expression 4) to (Expression 6), and the third feature value is an edge in a horizontal direction in a pixel corresponding to the contour. The edge direction in the pixel is specified based on the intensity and the edge strength in the vertical direction, and obtained based on the specified edge direction.

  According to one aspect of the present invention, a plurality of second images restored by a plurality of restoration methods with respect to a first image having a defective area, and an evaluation rank that is a rank of image quality evaluation for the plurality of second images, Is input, the image information acquisition device that inputs the plurality of second images input to the second input unit and extracts the feature amount, and the extracted feature amount, An image evaluation apparatus comprising: a learning processing unit that learns a correspondence relationship with the evaluation rank input to the second input unit.

  According to one aspect of the present invention, a third input unit that inputs a plurality of third images that have been repaired by a plurality of repair methods with respect to the first image that has a defect region, and the third input unit that is input The image information acquisition device for inputting the plurality of third images and extracting the feature amount, and the fourth image having the highest image quality evaluation among the plurality of third images based on the extracted feature amount And an image selection unit that selects the image.

  One aspect of the present invention is an image processing program for causing each functional unit in the image information acquisition apparatus or the image evaluation apparatus to be realized by a computer.

  According to the present invention, it is possible to accurately evaluate the image quality of an image obtained by performing repair processing on a defective area.

It is a figure which shows the structural example of the image processing system in 1st Embodiment. It is a figure which shows the specific example of the parameter list stored in the repair information storage part 102 in 1st Embodiment. It is a figure which shows the specific example which calculates the feature-value paying attention to the outline of the defect | deletion area | region D. FIG. It is a figure which shows the specific example of the information which specifies the edge direction centering on the pixel p. It is a figure which shows the specific example of the learning data. It is a flowchart which shows operation | movement of the image evaluation apparatus 10 in 1st Embodiment. It is a flowchart which shows operation | movement of the learning model acquisition apparatus 20 in 1st Embodiment. It is a figure which shows the structural example of 10 A of image evaluation systems in 2nd Embodiment. It is a figure which shows the structural example of 20 A of learning model acquisition systems in 3rd Embodiment. It is a figure which shows the example of the mask showing a defect | deletion area | region.

Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram illustrating a configuration example of an image processing system according to the first embodiment. As shown in FIG. 1, the image processing system 1 performs a plurality of types of restoration processing on an input image to generate a plurality of restored images, evaluates the image quality of the plurality of restored images, and evaluates the image quality. An image evaluation apparatus 10 that selects a high-restored image, and a learning model that extracts at least one feature amount from an image that is input learning data and shows a relationship between the extracted feature amount and an evaluation of image quality of the image A learning model acquisition device 20 for acquisition.

<Configuration of Image Evaluation Apparatus 10>
The configuration of the image evaluation apparatus 10 will be described.
The image evaluation apparatus 10 includes an input unit 101, a repair information storage unit 102, a repair processing unit 103, a feature amount extraction unit 104, a learning model DB (database) 105, and an image selection unit 106. The input unit 101 receives an image I _in to be subjected to a completion process (restoration process) and a mask image I _mask that specifies a missing area or color information C that represents a mask area. An image input as the image I _in is a still image or a moving image (video). The input unit 101 obtains a missing area D _in the image I _in based on the mask image I _mask or the color information C. Here, the color information C can be represented by a three-dimensional vector C (R, G, B) having RGB values (R = red, G = green, B = blue), and has, for example, a magenta color. The color information C can be expressed as C (R, G, B) = (255, 0, 255) when designated as a mask area.

Specific processing in the input unit 101 will be described with reference to FIG. When the image 70 which is the original image shown in FIG. 10A is input as the image I _in and the mask image 71 indicating the missing area is input, the input unit 101 displays the area indicated by the area 71b. Is identified as a defect region D. Further, the input unit 101 receives the image 70 as the image I _in which is the original image shown in FIG. 10B and the mask 72 that is the color information C superimposed on the image 70. A mask image in which the area indicated by 72 and the area not shown by binary are represented by binary values is generated and used as the defect area D. The deficient area D may be represented by a set of pixel positions designated as deficient {(d _x1 , d _y1 ), ..., (d _xn , d _yn )}. Further, the defect area D may be an image having the same size as the image I _in, and may be a binary image in which the defect area is 1 and the other area is 0.

  In the repair information storage unit 102, M repair parameters such as a repair method, a patch size, and the number of pyramid layers are registered in a list format. FIG. 2 is a diagram illustrating a specific example of a parameter list stored in the repair information storage unit 102 according to the first embodiment. As shown in FIG. 2, a repair method indicating a completion processing method and parameters 1, 2,..., N used for the repair method are stored in association with an ID for identifying each parameter list. Here, for example, parameter 1 is the patch size indicating the size of the completion target patch, and parameter 2 is the number of pyramid hierarchies when the resolution of an image for searching for similar patches is changed stepwise.

  In the specific example of FIG. 2, ID = 1, 2,..., M are set. That is, the repair information storage unit 102 has a parameter list that is repair information of M types (M is an integer of 2 or more). As repair information, repair method = A, parameter 1 = 3, parameter 2 = 3,..., Parameter N = I are stored in association with ID = 1, and repair method = X, in association with ID = M. Parameter 1 = 5, parameter 2 = 2,..., Parameter N = J are stored.

The repair processing unit 103 receives the completion target image I _in and the missing region D from the input unit 101, and acquires a parameter list from the repair information storage unit 102. The restoration processing unit 103 performs restoration processing corresponding to all the M types of parameter lists, and outputs M images I _{out — 1} to I out — M after restoration.

Feature quantity extracting unit 104 inputs the image _I out_1 _{~I OUT_M} the repaired inputted from the recovery processing unit 103, and outputs the extracted feature amount of each image _I out_1 _{~I out_M.} Specifically, the feature quantity extraction unit 104, based on the value of the pixel corresponding to the outline of the missing area D, the color consistency feature quantity (first feature quantity) F _u , the edge consistency feature quantity (second features) F _v and edge continuity feature quantity (third feature quantity) to extract the three feature amounts of F _c of. Here, the pixels corresponding to the outline of the defect area D include pixels located around the pixels located on the outline of the defect area D in addition to the pixels located on the outline of the defect area D.

  FIG. 3 is a diagram illustrating a specific example in which the feature amount is calculated by paying attention to the outline of the defect region D. In FIG. 3, a region Ω indicated by a hatched portion indicates a defective region D, and the outline of the defective region D is expressed by δΩ. A patch having a size of 3 × 3 centering on a pixel p on the contour δΩ is represented by P (p).

ここで、ｕ（ｑ）はパッチＰ（ｐ）に含まれる画素ｑのＲＧＢ値である。 << Calculation Method of Color Consistency Feature Quantity _Fu >>
The feature quantity extraction unit 104 includes the color of the pixel included in the defective region D and the periphery of the defective region D not included in the defective region D for each small patch P (p) centered on the pixel p on the contour δΩ. The color consistency feature amount _Fu , which is a feature amount indicating whether or not there is a change in the color of the pixel, is calculated based on (Expression 1) to (Expression 3) described below. First, the feature amount extraction unit 104 calculates the first color feature amount U ₁ based on the following (Equation 1).

Here, u (q) is the RGB value of the pixel q included in the patch P (p).

As shown in (Expression 1) above, the feature amount extraction unit 104 includes pixels located on the contour δΩ and within the contour δΩ among the 3 × 3 pixels included in the patch P (p) (hereinafter referred to as missing region pixels). calculating a first color characteristic amount U ₁ the value obtained by adding the respective RGB values) called.

上記（式２）に示すように、特徴量抽出部１０４は、パッチＰ（ｐ）に含まれる３×３の画素の内、欠損領域画素を除いた画素のＲＧＢ値のそれぞれを加算した値を第２色特徴量Ｕ_２として算出する。 Next, the feature amount extraction unit 104 calculates the second color feature amount U ₂ based on the following (Formula 2).

As shown in the above (Formula 2), the feature amount extraction unit 104 adds a value obtained by adding each of the RGB values of the pixels excluding the defective region pixels among the 3 × 3 pixels included in the patch P (p). calculating a second color characteristic amount _{U 2.}

The feature amount extraction unit 104 calculates a pixel average value obtained by dividing the first color feature amount U ₁ by the pixel number a of the defective area pixel and the second color feature amount U ₂ as the total number of pixels of the patch P (p) (see FIG. Based on the following (Equation 3) that calculates the ratio with the average value divided by the number of pixels b (3 = 3 = 9 pixels in 3) minus the number of pixels b (b = 9−a in FIG. 3): The color consistency feature value _Fu is calculated.

As is clear from (Equation 1) to (Equation 3), the color consistency feature amount _Fu is the color of the defective region pixel and the pixels around the defective region D that are not the defective region pixels in the region of the patch P (p). The more consistent the color is, the closer the value is to 1. The color consistency feature value _Fu is greater when the color of the defective region pixel is different from the color of the pixel around the defective region D that is not the defective region pixel in the region of the patch P (p). If 1 color feature value U ₁ > second color feature value U ₂ , the value is large, and if 1 color feature value U ₁ <second color feature value U ₂ , the value approaches 0.

Described above (Equation 1) Color consistency feature amount F _u obtained using to (Equation 3) is one example, but is not limited thereto. For example, the color consistent feature amount F _u shown in equation (3), and divided by a value multiplied by a constant, a value obtained by adding or subtracting a constant, may be color consistent feature amount F _u. The feature amount extraction unit 104 calculates various feature amounts indicating whether or not there is a change in the color of the pixels included in the defective region D and the color of the pixels around the defective region D that are not included in the defective region D. The color consistency feature value _Fu may be calculated using a calculation formula. For example, the feature amount extraction unit 104 calculates the color difference between the color of the pixel included in the defective region D and the color of the pixel around the defective region D not included in the defective region D as the color consistency feature amount _Fu. Also good.

ここで、ｖ（ｑ）はパッチＰ（ｐ）に含まれる画素ｑのエッジ強度を表す。 The method of calculating the << edge consistent feature amount _{F v} >>
The feature amount extraction unit 104 changes the edge strength of pixels included in the defective region D and the edge strength of pixels around the defective region D that are not included in the defective region D for each patch P (p) of the small region. whether a is a characteristic quantity showing the edge consistent feature amount F _v will be described below is calculated on the basis of (equation 4) to (equation 5). First, the feature amount extraction unit 104 calculates the first edge feature amount V ₁ based on the following (Equation 4).

Here, v (q) represents the edge strength of the pixel q included in the patch P (p).

As shown in (Expression 4) above, the feature amount extraction unit 104 uses the value obtained by adding the edge strengths of the missing region pixels among the 3 × 3 pixels included in the patch P (p) as the first edge feature amount V. Calculated as ₁ . The edge strength of a pixel is obtained, for example, by adding the absolute value of the difference in luminance value between the pixel q and surrounding pixels. The luminance value of the pixel is a value obtained by R + G + B using the RGB value of the pixel, for example.

上記（式５）に示すように、特徴量抽出部１０４は、パッチＰ（ｐ）に含まれる３×３の画素の内、欠損領域画素を除いた画素のエッジ強度を加算した値を第２エッジ特徴量Ｖ_２として算出する。 Next, the feature extraction unit 104 calculates a second edge feature quantity V ₂ based on the following equation (5).

As shown in the above (Formula 5), the feature amount extraction unit 104 calculates the second value obtained by adding the edge intensities of the pixels excluding the defective region pixels from the 3 × 3 pixels included in the patch P (p). calculated as an edge feature quantity _{V 2.}

Feature amount extraction unit 104, the average value of the pixel divided by the number of pixels a defective region pixel of the first edge feature quantity V _1, a second edge feature quantity V ₂ from the total number of pixels of the patch P (p) The edge consistency feature amount _Fv is calculated based on the following (Equation 6) that calculates the ratio of the average value of the pixels divided by the pixel number b obtained by subtracting the pixel number a.

As is clear from (Equation 4) to (Equation 6), the edge consistency feature amount _Fv is obtained in the region of the patch P (p), and the edge strength of the defective region pixel and the peripheral region of the defective region D that is not the defective region pixel. The value is closer to 1 as the edge strength of the pixel is more consistent (the edge strength is the same). The edge consistency feature amount _Fv is such that, in the region of the patch P (p), the first edge feature amount V becomes larger as the edge strength of the defective region pixel differs from the edge strength of the peripheral pixel of the defective region D that is not a defective region pixel. _{If 1} > second edge feature value V ₂ , the value is a large value, and if first edge feature value V ₁ <second edge feature value V ₂ , the value approaches 0.

It described above (Equation 4) to the edge consistent feature amount F _v obtained using the equation (6) is an example, but is not limited thereto. For example, the edge consistent feature amount F _v shown in (Equation 6), and dividing a value multiplied by a constant, a value obtained by adding or subtracting a constant, may be an edge consistent feature amount F _v. The feature amount extraction unit 104 indicates whether or not there is a change (level difference) between the edge strength of the pixel included in the defective region D and the edge strength of the pixel around the defective region D not included in the defective region D. the amount of the various formulas may be calculated edge consistent feature amount F _v using to calculate.

For example, the feature amount extraction unit 104 acquires the edge strength of the pixels included in the defective region D together with the edge direction information, and determines the edge strength of the pixels around the defective region D that are not included in the defective region D as the edge direction information. acquired in combination with this, by multiplying a weight coefficient corresponding to the direction of the coincidence of the edge may be calculated edge consistent feature amount F _v by the process of addition. The weighting coefficient according to the degree of coincidence of the edge direction is a coefficient that changes from −1 to 1 according to the degree of coincidence, for example, 1 in the same direction and −1 in the opposite direction. The feature amount extraction unit 104 is configured to obtain the edge intensity based on the luminance value of the pixel, but is not limited to this. For example, the feature amount extraction unit 104 may obtain the edge strength based on the RGB values of the pixels. In this case, the edge strength is obtained for each of R, G, and B colors.

<< method of calculating the edge continuity feature quantity _{F c} >>
The feature amount extraction unit 104 has an edge continuity feature amount F _c that is a feature amount indicating whether or not there is a change in the edge direction of each pixel p on the contour δΩ and the edge direction of the pixel adjacent to the pixel p. Is calculated by the following method.

  The feature amount extraction unit 104 obtains an edge direction for each pixel p located on the contour δΩ. The edge direction may be information that specifies two neighboring pixels in the edge direction centered on the pixel p. FIG. 4 is a diagram illustrating a specific example of information for specifying an edge direction centered on the pixel p. As shown in FIG. 4, an oblique direction (1) passing through the lower left pixel and the upper right pixel around the pixel p, a horizontal direction (2) passing through the left pixel and the right pixel, and a vertical direction passing through the upper pixel and the lower pixel. (3) and an oblique direction (4) passing through the lower right pixel and the upper left pixel. The feature quantity extraction unit 104 may specify a likely direction among the four directions shown in FIG. For example, the feature amount extraction unit 104 obtains c (p) = dy / dx based on the edge strengths (dx, dy) in the x and y directions of the luminance value of the pixel p (corresponding to obtaining tangent) There is a method for determining which of the four directions corresponds to the inclination.

Next, the feature amount extraction unit 104 similarly obtains c (pp) and c (pn) for the pixels pp and pn adjacent to the pixel p, and averages these values c (p ′) = [c ( pp) + c (pn)] / 2. The feature amount extraction unit 104 obtains c (p) and c (p ′) for all the pixels on the contour δΩ, and the first edge direction specified by c (p) and c (p ′) The second edge direction specified by is acquired. The feature amount extraction unit 104 obtains a correlation value indicating the degree of coincidence between the first edge direction and the second edge direction, and uses this correlation value as the edge continuity feature amount F _c . Note that the positions of the pixels pp and pn adjacent to the pixel p are, for example, the pixel adjacent to the lower side of the pixel p is the pixel pp and the pixel adjacent to the right side of the pixel p is the position of the pixel pn. It is not limited to this.

  The learning model DB 105 stores a learning model indicating the relationship between the image feature amount acquired by the learning model acquisition device 20 and the image quality evaluation of the image. Specifically, the learning model, when at least two or more feature quantities are input, estimates and outputs a ranking indicating whether the image quality of each feature quantity is excellent. Information indicating the relationship.

The image selection unit 106 performs M restorations output from the restoration processing unit 103 based on the feature amounts of the images I _{out — 1 to} I out — _M extracted by the feature amount extraction unit 104 and the learning model stored in the learning model DB 105. An evaluation rank is _assigned to each of the subsequent images I _{out — 1 to} I out — _M , and an image I _best ranked first with an estimated rank that is estimated to have the best image quality is output.

<About the structure of the learning model acquisition apparatus 20>
The configuration of the learning model acquisition device 20 will be described.
The learning model acquisition apparatus 20 includes a learning data input unit 201, a feature amount extraction unit 202, a learning list DB (database) 203, and an evaluation rank learning unit 204.

  The learning data input unit 201 is an input unit to which learning data 24 as shown in FIG. 5 for generating a learning model is input. FIG. 5 is a diagram illustrating a specific example of the learning data 24. As shown in FIG. 5, the learning data 24 is used to obtain a group ID 241 indicating a group of images to which a series of evaluation ranks are assigned, an evaluation rank 242 for the restored image data, and the restored image data 244. The list format data includes the mask information 243 and the restored image data 244. The learning data 24 is a variable length table in which the length of the list varies depending on how many image data 244 after restoration are compared.

  As shown in FIG. 5, the learning data 24 associates the mask information 243 with the group ID, the restored image data 244, and the evaluation rank 242 in the restored image data 244 of the same group. . Therefore, the number of restored image data 244 in the same group needs to be two or more. Further, when the number of image data 244 after restoration of the same group ID is K, the number of the same rank among them is K−1 or less so that at least one different evaluation rank can be assigned. Must.

The feature quantity extraction unit 202 has the same configuration as that of the feature quantity extraction unit 104 described above, and pays attention to the outline of the missing region D, and provides the color consistency feature quantity F _u , edge consistency feature quantity F _v, and edge continuity. Three feature amounts of the feature amount F _c are extracted. The feature amount extraction unit 202 is the target of processing of the restored image data 244 input from the learning data input unit 201. The feature quantity extraction unit 202 applies the color consistency feature quantity F _u , the edge consistency feature quantity F _v, and the edge consistency feature quantity to all the restored image data 244 based on the pixel values corresponding to the outline of the missing area D. processing to extract three features of edge continuity feature quantity F _c.

  The learning list DB 203 is a database that stores evaluation ranks 242 and three feature amounts in association with group IDs.

  The evaluation rank learning unit 204 refers to the learning list from the learning list DB 203 and generates a learning model by performing ranking learning on the correspondence relationship between the image feature amount and the evaluation rank. Ranking learning is a process of generating a learning model by learning in advance the correspondence between image feature amounts and evaluation ranks in a machine learning framework. If this learning model is used, the evaluation order for these images can be calculated based on the feature amounts of two or more images. Such ranking learning is a known technique, and for example, a method such as SVM (Support Vector Machine) -Rank can be used. Moreover, you may use SVM of the following well-known literature 1. FIG.

  Known Document 1: T. Joachims, “Training Linear SVMs in Linear Time”, Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), 2006

The evaluation rank learning unit 204 generates a learning model indicating a correspondence relationship between the image feature amount and the image evaluation rank, and outputs the learning model to the image evaluation apparatus 10. The evaluation rank learning unit 204 associates the three feature quantities of the color consistency feature quantity F _u , the edge consistency feature quantity F _v, and the edge continuity feature quantity F _c extracted from the image with the evaluation rank of the image. Generate a learning model that shows the relationship.

  Note that the learning model generated in the evaluation rank learning unit 204 is not limited to the configuration for estimating the evaluation rank as long as the data is used for classifying data in multiple stages. For example, a learning model that performs binary classification indicating, for example, image quality evaluation OK and image quality evaluation NG based on the feature amount of the image may be generated by processing the learning list using SVM. The value output by the learning model is not limited to binary, and for example, a multi-level classification of 5 levels may be output.

<Operation of Image Evaluation Apparatus 10>
Next, the operation of the image evaluation apparatus 10 in the first embodiment will be described. FIG. 6 is a flowchart showing the operation of the image evaluation apparatus 10 in the first embodiment. In the input unit 101, an image I _in to be subjected to completion processing and a mask image I _mask for designating a missing area are input (step S101). The input unit 101 identifies the missing area D _in the image I _in based on the mask image I _mask (step S102).

The repair processing unit 103 performs a repair process corresponding to all of the M types of repair information (parameter list) acquired from the repair information storage unit 102 based on the image I _in and the defect area D input from the input unit 101. performed on _in, and outputs the image _I out_1 _{~I OUT_M} after M pieces of repair (step S103). Feature quantity extracting unit 104 inputs the image _I out_1 _{~I OUT_M} the repaired inputted from the recovery processing unit 103, and outputs the extracted feature amount of each image _I out_1 _{~I out_M} (step S104). Specifically, the feature quantity extraction unit 104 determines the color consistency feature quantity F _u and the edge consistency feature quantity F _v from each of the images I _{out — 1} to I out — _M based on the value of the pixel corresponding to the outline of the missing region D. and extracted three characteristic amounts of edge continuity feature quantity F _c.

The image selection unit 106 performs M restorations output from the restoration processing unit 103 based on the feature amounts of the images I _{out — 1 to} I out — _M extracted by the feature amount extraction unit 104 and the learning model stored in the learning model DB 105. Evaluation ranks are _assigned to the _subsequent images I _{out — 1 to} I out — _M (step S105). Image selecting unit 106 selects the image imparted with evaluation rank = 1 of the most image quality is estimated to be from the images _I out_1 _{~I OUT_M} outputs as an image _{I best} (step S106).

<Operation of Learning Model Acquisition Device 20>
Next, operation | movement of the learning model acquisition apparatus 20 in 1st Embodiment is demonstrated. FIG. 7 is a flowchart showing the operation of the learning model acquisition apparatus 20 in the first embodiment. In the learning data input unit 201, learning data 24 as shown in FIG. 5 for generating a learning model is input (step S201).

The feature amount extraction unit 202 starts a loop process for extracting feature amounts from all of the learning data 24 input from the learning data input unit 201 (step S202). The feature amount extraction unit 202 acquires the restored image data 244 from the learning data 24 to be processed (step S203). The feature quantity extraction unit 202 pays attention to the outline of the defect area D with respect to the acquired image data 244 after restoration, and features the color consistency feature quantity F _u , the edge consistency feature quantity F _v, and the edge continuity feature quantity. extracting the three feature amounts of F _c. The feature amount extraction unit 202 stores the extracted feature amount and the evaluation rank 242 of the image data 244 in the learning list DB 203 in association with the group ID (step S204).

  The feature quantity extraction unit 202 returns to step S202 if the learning data 24 to be processed is not the last data, ends the loop processing if the learning data 24 to be processed is the last data, and proceeds to step S206 ( Step S205). Through the above loop processing, a database that stores the evaluation rank 242 and the three feature quantities in association with the group ID is constructed in the learning list DB 203.

  The evaluation rank learning unit 204 refers to the learning list from the learning list DB 203 and generates a learning model by performing learning processing on the correspondence relationship between the image feature amount and the evaluation rank (step S206). The learning model generated by the evaluation rank learning unit 204 is output to the image evaluation device 10 and stored in the learning model DB 105.

  As described above, the image processing apparatus 30 according to the first embodiment extracts and extracts feature amounts from a plurality of restored image data that has been subjected to completion processing using various methods and parameters. An evaluation rank can be given to the restored image data based on the feature amount. Thus, by selecting the restored image data having the best evaluation order, it is possible to obtain image data that has been automatically subjected to the optimum completion processing. In addition, unlike existing image evaluation methods, a feature amount focused on the boundary of the missing region D is extracted, and an evaluation order based on the feature amount is obtained with reference to the learning model, thereby achieving higher accuracy. Realizes image quality evaluation. In addition, the learning model acquisition device 20 can improve the accuracy of the learning model by generating a learning model based on a learning list in which feature amounts extracted from learning data are associated with evaluation ranks.

(Second Embodiment)
The feature amount extraction unit 104 included in the image evaluation device 10 of the first embodiment or the feature amount extraction unit 202 included in the learning model acquisition device 20 may be configured as an independent feature amount extraction device.
Hereinafter, an image evaluation system 10A according to the second embodiment in which the feature amount extraction unit 104 included in the image evaluation device 10 according to the first embodiment is configured as an independent feature amount extraction device 11 will be described.

  FIG. 8 is a diagram illustrating a configuration example of an image evaluation system 10A according to the second embodiment. In FIG. 8, the same components as those of the image processing system 1 of the first embodiment shown in FIG. 1 are given the same reference numerals, and the description thereof is omitted or simplified.

  As illustrated in FIG. 8, the image evaluation system 10 </ b> A includes an image evaluation device 10 a and a feature amount extraction device 11. An image evaluation system 10A illustrated in FIG. 8 has a function equivalent to that of the image evaluation apparatus 10 illustrated in FIG.

<About the feature quantity extraction device 11>
As shown in FIG. 8, the feature amount extraction apparatus 11 of the second embodiment includes an image input unit 111, a feature amount extraction unit 112, and a feature amount output unit 113. The image input unit 111 receives the repaired images I _{out — 1 to} I _{out_M} from the repair processing unit 103, and outputs the input repaired images I _{out — 1 to} I _{out_M} to the feature amount extraction unit 112. The feature amount extraction unit 112 has the same function as the feature amount extraction unit 104 illustrated in FIG. 1, and extracts and extracts feature amounts from the restored images I _{out — 1} to I out — _M input from the image input unit 111. The obtained feature amount is output to the feature amount output unit 113. The feature amount output unit 113 outputs the feature amount input from the feature amount extraction unit 112 to the image selection unit 106 of the image evaluation apparatus 10a.

(Third embodiment)
The feature quantity extraction unit 202 provided in the learning model acquisition apparatus 20 of the first embodiment can be configured as an independent feature quantity extraction apparatus.
Hereinafter, a learning model acquisition system 20A of the third embodiment in which the feature amount extraction unit 202 included in the learning model acquisition device 20 of the first embodiment is configured as an independent feature amount extraction device 21 will be described.

  FIG. 9 is a diagram illustrating a configuration example of a learning model acquisition system 20A according to the third embodiment. In FIG. 9, the same components as those in the image processing system 1 of the first embodiment shown in FIG. 1 are given the same reference numerals, and description thereof is omitted or simplified.

  As illustrated in FIG. 9, the learning model acquisition system 20 </ b> A includes a learning model acquisition device 20 a and a feature amount extraction device 21. The learning model acquisition system 20A shown in FIG. 9 has the same function as the learning model acquisition device 20 shown in FIG.

<Regarding Feature Extraction Device 21>
As shown in FIG. 9, the feature quantity extraction device 21 of the third embodiment includes an image input unit 211, a feature quantity extraction unit 212, and a feature quantity output unit 213. The image input unit 211 receives the restored image data 244 included in the learning data 24 from the learning data input unit 201 of the learning model acquisition apparatus 20a, and inputs the restored image data 244 to the feature amount extraction unit 212. Output. The feature amount extraction unit 212 has the same function as the feature amount extraction unit 202 illustrated in FIG. 1, extracts feature amounts from the restored image data 244 input from the image input unit 211, and extracts the feature amounts Is output to the feature amount output unit 213. The feature amount output unit 213 outputs the feature amount input from the feature amount extraction unit 212 to the learning list DB 203 of the learning model acquisition device 20a.

  Even if the learning model output by the evaluation rank learning unit 204 included in the learning model acquisition device 20a is output to the image evaluation device 10 of the first embodiment, the learning model is output to the image evaluation device 10a of the second embodiment. An output configuration may be used.

  The functional units included in the image evaluation device 10 and the learning model acquisition device 20 in the first embodiment described above, the image evaluation system 10A in the second embodiment, and the learning model acquisition system 20A in the third embodiment are, for example, a computer Can be realized. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. You may implement | achieve using programmable logic devices, such as FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The image information acquisition method, image evaluation method, image information acquisition apparatus, image evaluation apparatus, and image processing program of the present invention can be used in an image processing apparatus that performs a completion process.

DESCRIPTION OF SYMBOLS 1 ... Image processing system 10, 10a ... Image evaluation apparatus 20, 20a ... Learning model acquisition apparatus, 11, 21 ... Feature-value extraction apparatus, 102 ... Repair information storage part, 103 ... Repair process part, 104, 112, 202 , 212 ... feature quantity extraction unit, 105 ... learning model DB, 106 ... image selection unit, 203 ... learning list DB, 204 ... evaluation rank learning unit

Claims (9)

A first input step of inputting an image in which the defect area is repaired;
A feature amount extraction step of extracting a feature amount from pixels corresponding to the outline of the missing region with respect to the image input in the first input step;
Have
The feature amount includes a first feature amount obtained based on a color of a pixel on the contour and a color of a pixel not included in the defect area located around the pixel corresponding to the contour, and the contour A second feature amount obtained based on an edge strength of an upper pixel and an edge strength of a pixel not included in the defective region located around the pixel corresponding to the contour; and a pixel on the contour Image information including at least one of an edge direction and a third feature amount obtained based on an edge direction of a pixel located in the periphery of the pixel corresponding to the contour and not included in the defective region Acquisition method. In a small area centered on the pixel on the contour,
The first feature amount is obtained based on the following (Expression 1) to (Expression 3):

The second feature amount is obtained based on the following (formula 4) to (formula 6):

The third feature amount is obtained based on the specified edge direction by specifying the edge direction in the pixel based on the horizontal edge strength and the vertical edge strength in the pixel corresponding to the contour. The image information acquisition method according to claim 1. A second input to which a plurality of second images restored by a plurality of restoration methods with respect to the first image having a defective area and an evaluation rank that is an image quality evaluation rank for the plurality of second images are input. Steps,
An extraction step of extracting the feature amount by applying the image information acquisition method according to claim 1 or 2 to a plurality of the second images input in the second input step;
A learning step of learning a correspondence relationship between the feature amount extracted in the extraction step and the evaluation rank input in the second input step;
An image evaluation method comprising: A third input step in which a plurality of third images repaired by a plurality of repair methods are input to the first image having a defect region;
An extraction step of extracting the feature amount by applying the image information acquisition method according to claim 1 or 2 to a plurality of the third images input in the third input step;
An image selection step of selecting a fourth image having the highest image quality evaluation from among the plurality of third images based on the feature amount extracted in the extraction step;
An image evaluation method comprising: A first input unit for inputting an image in which a defect area is repaired;
A feature quantity extraction unit that extracts a feature quantity from pixels corresponding to the outline of the missing area for the image including the missing area input to the first input unit;
With
The feature amount includes a first feature amount obtained based on a color of a pixel on the contour and a color of a pixel not included in the defect area located around the pixel corresponding to the contour, and the contour A second feature amount obtained based on an edge strength of an upper pixel and an edge strength of a pixel not included in the defective region located around the pixel corresponding to the contour; and a pixel on the contour Image information including at least one of an edge direction and a third feature amount obtained based on an edge direction of a pixel located in the periphery of the pixel corresponding to the contour and not included in the defective region Acquisition device. In a small area centered on the pixel on the contour,
The first feature amount is obtained based on the following (Expression 7) to (Expression 9),

The second feature amount is obtained based on the following (Expression 10) to (Expression 12),

The third feature amount is obtained based on the specified edge direction by specifying the edge direction in the pixel based on the horizontal edge strength and the vertical edge strength in the pixel corresponding to the contour. The image information acquisition apparatus according to claim 5. A second input to which a plurality of second images restored by a plurality of restoration methods with respect to the first image having a defective area and an evaluation rank that is an image quality evaluation rank for the plurality of second images are input. And
The image information acquisition device according to claim 5 or 6, wherein the feature amount is extracted by inputting the plurality of second images input to the second input unit;
A learning processing unit for learning a correspondence relationship between the extracted feature amount and the evaluation rank input to the second input unit;
An image evaluation apparatus comprising: A third input unit for inputting a plurality of third images repaired by a plurality of repair methods on the first image having a defect region;
The image information acquisition device according to claim 5 or 6, wherein the feature information is extracted by inputting the plurality of third images input to the third input unit;
An image selection unit that selects a fourth image having the highest image quality evaluation from among the plurality of third images based on the extracted feature amount;
An image evaluation apparatus comprising:   An image processing program for causing each functional unit in the image information acquisition device according to claim 5 or 6 or the image evaluation device according to claim 7 or 8 to be realized by a computer.

Images