JP2020024612A - Image processing device, image processing method, processing device, processing method and program - Google Patents

Abstract

To solve the problem that an appropriate super-resolution process cannot be executed by utilizing ambiguous image identification information.SOLUTION: An image processing device includes: first obtaining means for obtaining first image data; second obtaining means for obtaining a classification score; and processing means for creating, from the first image data on the basis of the classification score, second image data that has a higher resolution than that of the first image data. Parameters that form the processing means are decided on the basis of learning data formed by a set of multiple pieces of data that contain learning image data and an identifier which identifies the classification of the learning image data. The classification score indicates a probability that the first image data is each of the identifiers.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing technique for increasing the resolution of an image using machine learning.

  When a low-resolution image with a small number of pixels is enlarged to a high-resolution image with a large number of pixels, if the image is enlarged using bilinear interpolation or the like, the image after the enlargement processing is blurred. Super-resolution processing is known as a method for solving such a problem. As the super-resolution processing, a learning-type super-resolution technique is known (Patent Document 1). Hereinafter, in order to clarify the meaning of super-resolution, super-resolution is referred to as high resolution. Patent Literature 1 describes performing high resolution processing using a “correct ID” that uniquely identifies an image type.

International Publication No. 2014/162690

  However, as described in Patent Document 1, it is not always possible to always acquire a “correct answer ID” that uniquely identifies an image type. For example, when an image of a suspicious person in a security camera is sharply enlarged by increasing the resolution, a “correct ID” for identifying the suspicious person is generally not obtainable. Rather than being able to uniquely identify the type of image, such as `` masculine '', `` feeling like middle-aged '', or `` maybe youth '' as sighting information, obscure image identification information can be obtained Is common. In the technique of Patent Literature 1, when a “correct ID” that uniquely identifies an image type cannot be obtained, it is not possible to appropriately perform the resolution increasing process.

  An image processing apparatus according to an aspect of the present invention includes a first obtaining unit that obtains first image data, a second obtaining unit that obtains a classification score, and the first image data based on the classification score. Processing means for generating second image data having a higher resolution than the first image data, wherein the parameters constituting the processing means identify learning image data and classification of the learning image data. The classification score is determined based on learning data composed of a plurality of sets of data including an identifier, and the classification score indicates a probability that the first image data is each of the identifiers.

  According to the present invention, it is possible to appropriately increase the resolution by using ambiguous image identification information that cannot always uniquely identify an image type.

The conceptual diagram which shows the basic framework of machine learning. FIG. 2 is a block diagram illustrating an example of a configuration of an image processing device. FIG. 2 is a functional block diagram of the image processing apparatus. 9 is a flowchart illustrating an example of the flow of a process performed by the image processing apparatus. The figure which shows an example of a structure of a high resolution neural network. The figure for demonstrating parameter optimization of a high resolution neural network. The figure which shows an example of a structure of a determination part neural network. 6 is an example of a UI for setting a classification score. The figure which shows the structure of a high resolution neural network. 6 is an example of a UI for setting a classification score.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all combinations of features described in the present embodiments are not necessarily essential to the solution of the present invention. The same components will be described with the same reference numerals.

<< First Embodiment >>
<Basic framework of machine learning>
FIG. 1 is a diagram illustrating a basic framework of machine learning. First, a basic framework of machine learning will be described in order to smoothly describe the embodiment. A method of configuring a model for predicting unknown y from known x with respect to data x and data y whose correspondence is not clear will be described with reference to FIG.

  The learning data 101 in FIG. 1 is data composed of a plurality of sets of pairs of x and y. In FIG. 1, data of N sets of pairs are included in the learning data 101. The model 102 can be represented by a function y = f (x; θ) that associates x with y. In machine learning, if the function f and its parameter θ are appropriately set, it is assumed that the function f can be used to estimate y well from x. Since the causal relationship is not clear in the first place, it is general to leave room for realizing any correspondence by taking an extremely large number of parameters θ of the function f. For example, it is assumed that the model 102 is a convolutional neural network (hereinafter, referred to as “CNN”), which is a kind of a neural network (hereinafter, referred to as “NN”). In this case, increasing the number of convolution layers and the coefficient associated with the layer corresponds to adopting more parameters θ. In a CNN that performs image processing, the number of parameters is usually tens of thousands or more. The model parameter optimization 103 is a process for optimizing the parameter θ so that the model 102 can satisfactorily predict the unknown y from the known x. A typical optimization method is a method in which the square of the L2 norm given by the equation (1) is set as an objective function L (θ), and a parameter θ is set so as to minimize the objective function L (θ). .

Various objective functions have been proposed in addition to equation (1). The setting of the objective function is important because the characteristics of estimation by machine learning change according to the objective function. When the model parameter optimization 103 is completed so as to minimize the objective function, θ ₀ is determined as the optimal model parameter θ. The prediction model 104 is a model in which the optimal model parameter θ ₀ is applied to the model 102. Using the prediction model 104, the unknown y can be predicted from the known x. That is, unknown y can be predicted from known x by y = f (x; θ ₀ ). The above is the basic framework of machine learning.

<Configuration of image processing device>
FIG. 2 is a diagram illustrating an example of a configuration of the image processing apparatus according to the first embodiment. The image processing apparatus 200 includes a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, and a HDD (Hard Disk Drive) 203. Further, the image processing apparatus 200 includes a bus 204 and an interface 205. The CPU 201 executes various processes using computer programs and data stored in the RAM 202 or the HDD 203. Accordingly, the CPU 201 controls the operation of the entire computer device, and executes or controls each process related to the machine learning described above. Further, the CPU 201 may cause a GPU (Graphics Processing Unit) (not shown) connected to the bus 204 to perform a part of the processing.

  The RAM 202 has an area for storing computer programs and data loaded from the HDD 203. Further, the RAM 202 has a work area used when the CPU 201 executes various processes. As described above, the RAM 202 can appropriately provide various areas. The HDD 203 is a large-capacity information storage device represented by a hard disk drive. The HDD 203 stores an OS (Operating System), and computer programs and data for causing the CPU 201 to execute each process related to the machine learning described above. The data stored in the HDD 203 includes data of an image or a moving image to be processed. The computer programs and data stored in the HDD 203 are appropriately loaded into the RAM 202 under the control of the CPU 201, and are processed by the CPU 201. The HDD 203 may be a memory device such as an SSD (Solid State Drive), a flash memory, or a USB (Universal Serial Bus) memory other than the hard disk drive device. Further, a storage device (not shown) existing on the network 208 connected via the interface 205 may be virtually used as the HDD 203.

  The CPU 201, the RAM 202, the HDD 203, and the interface 205 are all connected to the bus 204. The input device 206, the output device 207, the network 208, and the bus 204 are connected to the interface 205. The input device 206 includes a keyboard, a mouse, a touch panel, a microphone, a camera, a gyro sensor, and the like, and can instruct the image processing device 200 to change settings and start processing in various forms. The output device 207 is configured by a display, a projector, a printer, or the like, and can display, project, and print the processing results of the image processing device 200 as images and characters. Note that the input device 206 and the output device 207 may be integrated by using a touch panel display or the like like a tablet terminal and a smartphone. A plurality of devices are connected to the network 208, and the information is provided to the image processing device 200 in the form of SaaS (Software as a Service) such as an information storage function, an arithmetic function, and an input / output function in accordance with an instruction from the CPU 201. Can be. Note that the configuration illustrated in FIG. 2 is merely an example of a configuration of a computer device that can be applied to the image processing device 200, and another configuration may be employed.

<Functional block diagram>
FIG. 3 is a diagram illustrating an example of a functional block diagram of the image processing apparatus 200. The image processing apparatus 200 includes a low-resolution image acquisition unit 301, a classification score acquisition unit 302, a high-resolution processing unit 303, a learning data acquisition unit 304, a parameter optimization unit 305, and a parameter holding unit 306. In the image processing apparatus 200, the functions of the above-described units are realized by the CPU 201 reading and executing a control program stored in the HDD 203. Note that the image processing apparatus 200 may be configured to include a dedicated processing circuit corresponding to each component. Further, a dedicated processing circuit and a control program may be used together.

  The low resolution image acquisition unit 301 acquires image data (first image data) of a low resolution image. The low-resolution image acquisition unit 301 may acquire image data via the network 208 or may acquire image data stored in the HDD 203. When the acquired image data does not have the predetermined resolution, the enlargement process may be performed so that the image data has a predetermined resolution (low resolution).

  The classification score obtaining unit 302 obtains a classification score corresponding to the image data of the low-resolution image obtained by the low-resolution image obtaining unit 301. In the present embodiment, classifications categorized into a plurality of predetermined classifications are used. An identifier for identifying each classification is used. The classification score is a score value indicating the probability that the low-resolution image is an identifier for identifying each classification. Details of the classification score will be described later.

  The high resolution processing unit 303 performs high resolution processing using the parameters stored in the parameter storage unit 306. The high resolution processing unit 303 is an NN configured with the parameters held in the parameter holding unit 306. When the image data of the low-resolution image and the classification score are input, the high-resolution processing unit 303 generates and outputs image data (second image data) of a high-resolution image having a higher resolution than the low-resolution image. Details will be described later.

  The learning data acquisition unit 304 acquires learning data. As described above, the learning data is a plurality of sets of sets of classification scores s in addition to the learning image data x and y. The acquired learning data is sent to the parameter optimizing unit 305.

  The parameter optimizing unit 305 optimizes parameters used in the high resolution processing unit 303 using the learning data. Details will be described later. The parameter holding unit 306 holds the parameters optimized by the parameter optimizing unit 305.

  Although FIG. 3 has been described as an example of the configuration of the single image processing apparatus 200, each unit illustrated in FIG. 3 may be distributed among a plurality of apparatuses and processing using a plurality of apparatuses may be performed. For example, the low-resolution image acquisition unit 301, the classification score acquisition unit 302, and the high-resolution processing unit 303 are implemented in the first image processing device, and the learning data acquisition unit 304 and the parameter optimization unit 305 perform the second image processing. It may be implemented in a device. The parameter holding unit 306 may be implemented in a parameter holding device. An image processing system in which the first image processing device, the parameter holding device, and the second image processing device are connected via a network may be used. Further, for example, the parameter optimization unit 305 and the high-resolution processing unit 303 may be configured by a plurality of devices (servers), and the processing load may be distributed among the servers.

<Flow chart>
FIG. 4 is a flowchart illustrating processing performed by the image processing apparatus 200 according to the first embodiment. Hereinafter, description will be made with reference to a functional block diagram shown in FIG. 3 and a flowchart shown in FIG. A series of processes shown in the flowchart of FIG. 4 is performed by the CPU 201 expanding the program code stored in the HDD 203 to the RAM 202 and executing the program code. Alternatively, some or all of the functions of the steps in FIG. 4 may be realized by hardware such as an ASIC or an electronic circuit. The symbol “S” in the description of each process means that it is a step in the flowchart.

  In step S401, if the learned parameter is stored in the parameter storage unit 306, the process proceeds to a higher resolution phase of steps S402 to S405, and if not, the process proceeds to a learning phase of steps S406 to S409. In S401, an example has been described in which the processing is divided depending on whether or not the learning parameter is stored in the parameter storage unit 306, that is, whether or not learning has been completed. However, the present invention is not limited to this. Even if the learning has already been performed, when learning is to be performed using further learning data, the process may proceed to S406.

<Learning phase>
First, the flow of processing in the learning phase will be described. In S406, the learning data acquisition unit 304 acquires the learning data. The learning data is a set of data including, for example, a {low-resolution image x _i , a high-resolution image y _i , and an identifier ID _i }, and includes a plurality of sets of data. Note subscript i is the i-th training data set _{{x i, y i, ID} i} indicates a. Low resolution image x _i used in the training data may be generated by down-sampling the high resolution image y _i. An image captured by a camera having a small number of pixels may be referred to as a low-resolution image x _i , and an image captured of the same subject by a camera having a large number of pixels may be referred to as a high-resolution image y _i . Alternatively, even with a camera having the same number of pixels, an image of the first subject area of an image captured at a wide angle is a low-resolution image x _i , and an image corresponding to the same first subject area of an image captured at a telephoto is of a high resolution. The image y _i may be used. The identifier ID is data for identifying the learning image indicated by {x, y}. For example, if the ID is a scalar value and is identified by age, it can be set as a child (ID = 1), an adult (ID = 2), an elderly person (ID = 3), or the like.

In S407, the learning data acquisition unit 304 converts the identifier ID in the learning data obtained in S406 into a classification score s. In the present embodiment, the classification score is a value indicating the probability that the image is each of the identifiers. Specifically, the classification score s _i for the i-th identifier ID _i, a vector j th element s _ij is defined by equation (2).

For example, when the identifier of the old man (ID = 3) is set in the image, the corresponding classification score s _i has the third element as 1 and the other elements as 0 as in [0 0 1]. Vector.

In the above example, since the learning data is used, an identifier provided with a correct identifier is used. For example, if an identifier of an old man (ID = 3) is set in the image, the corresponding classification score s _i is 1 in the third element as in [0 0 1], that is, the probability that the person is an old man is 100%. Will be used. However, it is not limited to this example. Also in the learning data, data indicating the probability of the identifier of the predetermined class among the plurality of classes may be used as the learning data. For example, a value such as [0.5 0.50] is obtained in S407 as the classification score of the learning data such that the probability of being a child is 50% and the probability of being an adult is 50%. May be. That is, a classification score that includes two or more elements whose values are not “0” may be used.

  In this embodiment, an example has been described in which the identifier is set as a scalar value in the learning data. Since the data to be input to the parameter optimization unit 305, which is the NN, is required to be data expressed in a vector, the example in which the process of converting the learning data is performed has been described in S407. However, if the learning data includes a classification score already expressed in a vector, the process of S407 may be skipped.

In step S <b> 408, the parameter optimization unit 305 optimizes the parameters of the high-resolution NN that performs the high-resolution processing using the learning data input from the learning data acquisition unit 304. The resolution enhancement NN can be mathematically defined by a function f of Expression (3).
y = f (x, s; θ) Equation (3)

  In Expression (3), x is a low-resolution image, s is a classification score, θ is a model parameter, and y is a high-resolution image estimated by a function f. The parameter optimizing unit 305 performs a process of optimizing the model parameter θ in the above equation (3). The function f is realized using NN. Prior to describing the optimization processing in the parameter optimization unit 305, first, the high resolution NN constituting the high resolution processing unit 303 of the present embodiment will be described.

  FIG. 5 is a diagram illustrating an example of the network architecture of the high resolution NN according to the present embodiment. The NN in FIG. 5 is based on a U-shaped NN called U-net. The image data of the input image 501 is data of a low-resolution image that is input data. The input image 501 is a monochrome image composed of one channel of 32 × 32 (pixels) obtained by previously enlarging a low-resolution image to a desired resolution by bicubic interpolation or the like. Here, for the sake of explanation, the size of the input image 501 is described as a monochrome image composed of one channel of 32 × 32 (pixels), but is not limited to this. For example, a 512 × 512 (pixel) image or a 1024 × 1024 (pixel) image may be used as input image data.

  The input image 501 is reduced to half in the spatial direction by a strided convolution, and becomes intermediate data D1 in which the number of channels is increased to 64 instead. Further, the process of reducing the space size and increasing the number of channels is repeated to obtain intermediate data D2, D3, and D4. Finally, intermediate data 502 having an image size of 1 × 1 and 1014 channels is obtained. Various processes are performed until the intermediate data 502 is obtained, and many calculations such as convolution are performed until the final image data of the output image 504 is calculated from the intermediate data 502. It should be noted that all of these calculation parameters are collectively called a model parameter θ.

  Now, since the intermediate data 502 has an image size of 1 × 1, the local information of the input image 501 has been lost, and it is considered that the intermediate data 502 is data representing the characteristics or meaning of the entire image. Therefore, in the present embodiment, integrated intermediate data is obtained by adding the classification score 503, which is another input data to the high resolution NN, to the intermediate data 502. In FIG. 5, it is assumed that the classification score is a vector of 10 elements. As a result of adding the 10-element vector to the intermediate data 502, the integrated intermediate data is intermediate data having an image size of 1 × 1 and 1024 channels.

  Subsequently, the integrated intermediate data becomes intermediate data U4 in which the image size increases and the number of channels decreases due to transposed convolution or deconvolution. It is assumed that U-net transposed convolution or deconvolution parameters are set so that the intermediate data U4 is intermediate data having an image size of 2 × 2 and 512 channels. The intermediate data U4 is added to the intermediate data D4, and becomes the intermediate data U3 in which the image size is increased and the number of channels is reduced by the same processing. Such a bypass structure is one of the features of U-NET. By repeating the same processing, the image data of the output image 504 is finally output. Here, similarly to the input image 501, the output image 504 is an image having an image size of 32 × 32 and including one channel.

  One characteristic configuration of the present embodiment will be described with respect to the network architecture of FIG. In the present embodiment, inputs are made to the NN in two systems. The two-system input itself is generally performed. However, it is unusual in the process of the NN that the process of the NN is started by the input of one system and the data of the other system is additionally input during the process of the NN. In general, input is provided collectively and output is collectively received.

  Note that in the NN architecture of the present embodiment, it is also possible to perform a process of giving inputs collectively as in a general process. For example, in the example of FIG. 5 described above, the elements of the classification score are ten elements. Therefore, in the case of providing input collectively, input data obtained by adding the data of these 10 elements may be supplied to the NN. Specifically, the number of channels of the input image may be changed from 1 to 11, and a classification score may be defined for each pixel although it is redundant. In this embodiment, dividing the supply path of the input data into two without performing general processing has the following advantages. First, an increase in input data size can be prevented. In the example of FIG. 5, when the classification score is defined for each pixel of the input image, the number of elements of the input is 32 × 32 × 11 = 111264. On the other hand, when the processing of the present embodiment is adopted, the number of input elements is 32 × 32 × 1 + 10 = 1,034, which is 1/10 or less as compared with the case of general processing. Therefore, when the processing of the present embodiment is adopted, the memory can be used effectively, and as a result, the learning speed can be increased. For example, the batch size for mini-batch learning can be increased. Further, when the processing of the present embodiment is adopted, an increase in parameters can be prevented. Generally, in an NN, when the number of input channels increases, intermediate data of the entire network and parameters for calculating the intermediate data increase. By doing so, there is room for properly processing the increased input data. To put it simply, if the input is 10 times, the intermediate data also needs to be 10 times, and the parameter for calculating it also needs to be 10 times. If there is channel cross-correlation, it can be set smaller, but in any case, the intermediate data and the model parameters increase. And the calculation load for calculating them naturally increases. It goes without saying that these problems become more serious as the number of elements in the classification score increases. In the present embodiment, the classification score that has already become the semantic information is not intentionally shaped into an image format. Adding is done.

  In FIG. 5, the local information is missing, the input image is semantic information (that is, it represents the characteristics of the entire image), and the intermediate data of the image size of 1 × 1 and the number of channels of 1024 is used. The example in which the classification score 503 is added to the above has been described. By adding the classification score indicating the probability of each identifier to the intermediate data representing the features of the entire image, it is possible to determine which identifier feature the entire image should have. This is because an output image can be obtained. However, it is not limited to this example. For example, when it is possible to divide an image into four parts and each of the divided images has no correlation, the classification score is added to the intermediate data U4 having an image size of 2 × 2 and 512 channels. It may be in a form. Alternatively, even in the case of an image in which the entire image does not need to be considered so much, the classification score may be added to the intermediate data U4 having an image size of 2 × 2 and 512 channels. Even in these forms, some effects can be observed. When the classification score is added to the intermediate data U4 having an image size of 2 × 2 and 512 channels in FIG. 5, the data size of the classification score is also 2 × 2 and 10 channels (10 elements). It will be added in data format. In this case, the data of the classification score 503 may be simply copied to a size of 2 × 2 and used.

  Although FIG. 5 illustrates the U-net as an example, the present invention is not limited to this. Any NN may be applied as long as it is a NN that performs processing for losing local information and reducing the image size to the image size representing the characteristics of the entire image. For example, a Convolutional Encoder-Decoder architecture may be used. In this NN, the spatial resolution is reduced in the encoder section until the local information is lost.

  In FIG. 5, for the sake of explanation, the input image 501 is a case where the image size is one channel of 32 × 32, and a 512 × 512 (pixel) image or a 1024 × 1024 (pixel) image is input. That the image data may be used. Here, when the image size of the actual input image is, for example, an image of one channel of 512 × 512 (pixels), the image (whole image) is divided into a plurality of partial images, and the divided partial images are input. It may be image data. That is, the input image 501 in FIG. 5 may be a partial image. Then, the final output image may be obtained by combining the partial images output as the output image. When combining the partial images, predetermined image processing may be performed to obtain a final output image.

  The above is the description of the architecture of the high resolution NN which is the high resolution processing unit 303 of the present embodiment. Next, returning to the description, the optimization processing of the parameters of the high resolution NN performed by the parameter optimization unit 305 in S408 will be described.

  In the present embodiment, a hostile generation network (GAN: Generative Adversarial Network) is used for parameter optimization processing. In the hostile generation network, processing is generally performed using two networks, a generator and a discriminator. The generator is trained to generate a "fake" that is as close as possible to the original so as not to be overlooked by the discriminator. The Discriminator is learned so as to determine whether the input is “fake” generated by the generator or the original (“real”), and to find out “fake” generated by the generator. The learning accuracy of the generator is improved by learning such two networks so as to compete with each other. In the present embodiment, parameter optimization is performed using such a GAN.

  FIG. 6 is a diagram illustrating a parameter optimization method. For the sake of simplicity, a basic optimization method for a high resolution NN that does not use a classification score will be described first. FIG. 6A is a diagram illustrating a method for optimizing parameters when a classification score is not used.

  In FIG. 6A, an additional determination unit (Discriminator) is provided for optimizing the high-resolution processing unit (Generator). The role of the high-resolution processing section is to increase the resolution of input data (low-resolution image). The high-resolution processing unit can be expressed functionally as Expression (4).

  Here, x is a low-resolution image acquired from the learning data, and θ is a model parameter of the high-resolution processing unit.

Is an estimated high-resolution image generated by the high-resolution processing unit.

On the other hand, the role of the determination unit is 0 (false) if the input image is an estimated high-resolution image generated by the high-resolution processing unit, and 1 (true) if the input image is a high-resolution image obtained from learning data. Output. Expression (5) represents the determination unit as a function.
b = g (y; φ) Equation (5)

  Here, b is a value indicating true / false, y is an input image (high-resolution image), and φ is a model parameter of the determination unit. Of course, it should be noted that, at the start of learning, neither the high-resolution processing unit nor the determination unit can perform processing according to roles. For example, the output b of the determination unit g (y; φ) usually takes a continuous value of 0 to 1. As the learning progresses, the output b is aggregated into values of 0 or 1.

In the case where such a high-resolution processing unit and the determination unit are provided, the objective function L _f (θ) of the high-resolution processing unit in FIG. 6A can be expressed by Expression (6).

The first term of the equation (6) is a square error between the estimated high-resolution image generated by the high-resolution processing unit and the correct high-resolution image y _i acquired from the learning data, and has an effect of bringing the two close to each other. is there. The second term in equation (6) has an effect of outputting true (1) when the estimated high-resolution image generated by the high-resolution processing unit is input to the determination unit.

On the other hand, the objective function L _g (φ) of the determination unit in FIG. 6A can be expressed by equation (7).

The first term of the equation (7) has an effect of outputting true (1) when the correct high-resolution image y _i acquired from the learning data is input to the determination unit. The second term has an effect of outputting false (0) when the estimated high-resolution image generated by the high-resolution processing unit is input to the determination unit.

  As described above, the parameters are optimized so that the determination unit determines that the output of the high-resolution processing unit is false. On the other hand, the parameters of the high resolution processing unit are optimized so as to generate an image that makes the output of the determination unit true. To put it simply, the determination unit determines whether the high-resolution image is genuine (derived from learning data) or a fake, and the high-resolution processing unit optimizes both parameters so as to trick the determination unit. This is why it is called a hostile generation network.

  The above is the description of the general parameter optimization processing when the classification score is not used. Based on the above description, a method of optimizing a high-resolution NN using a classification score in the present embodiment will be described below.

  FIG. 6B is a diagram illustrating a parameter optimization process using a classification score in the present embodiment. In the present embodiment, the role of the determination unit includes estimating the classification score of the input image in addition to determining the authenticity of the image. The reason is as follows. The high-resolution processing unit of the present embodiment outputs a high-resolution image (fake) with the low-resolution image and the classification score as inputs. For example, an image obtained by inputting the classification score of “flower” and the low-resolution image to the high-resolution processing unit and performing high-resolution processing becomes a high-resolution flower image (oriented to be “flower”). The image will be a higher resolution image). That is, in this example, the high-resolution processing unit needs to learn a high-resolution method for determining a flower. In order to learn a high-resolution processing method in which the high-resolution processing unit is determined to be a flower, the GAN determines whether the image generated by the high-resolution processing unit is a flower (can be seen). ) A judgment unit is required. Therefore, in the present embodiment, the determination unit plays a role of estimating the classification score of the input image in addition to determining the authenticity of the image.

  When the determination unit in FIG. 6B of the present embodiment is represented by a function, Expression (8) is obtained.

  here,

Is the estimated classification score of the input image. That is, in the above example of “flower”, the estimated classification score includes a value indicating the probability that the input image is a flower. Of course, it will include the probability of being another identifier.

  On the other hand, the role of the high-resolution processing unit of the present embodiment is to increase the resolution of the input low-resolution image according to the input classification score. Expression (9) is obtained by expressing the high-resolution processing unit in FIG. 6B by a function.

  Here, s is a classification score obtained from the learning data.

In the case where the high-resolution processing unit and the determination unit are provided as described above, the objective function L _f (θ) of the high-resolution processing unit in FIG. 6B of the present embodiment can be expressed by Expression (10).

The first term of the equation (10) is a square error between the estimated high-resolution image generated by the high-resolution processing unit and the correct high-resolution image y _i obtained from the learning data, and the action of bringing the two close together. is there. The second term, if you enter the estimated high resolution image high resolution processing section is generated from the low-resolution image x _i based on the classification score s _i to the determination unit, is true (1), and the estimated classification score

And the classification score s _i of the learning data. That is, if you enter the paragraph, the determination unit estimates high-resolution image generated from a low-resolution image x _i based high-resolution processing unit to the classification score s _i of equation (10), true (1) Yes, and has the effect of making the classification score s _i .

On the other hand, the objective function L _g (φ) of the determination unit in FIG. 6B of the present embodiment can be expressed by Expression (11).

The first term of the above equation has an effect of outputting true (1) and the classification score s _i when the correct high-resolution image y _i acquired from the learning data is input to the determination unit. The second term g _b is the two outputs of the judgment unit g.

Means a value b indicating true / false. The second term of equation (11) has an effect of outputting false (0) when the estimated high-resolution image generated by the high-resolution processing unit is input to the determination unit.

  By setting the above objective function, the determining unit learns the relationship between the high-resolution image and the classification score from the learning data and estimates the classification score from the input image, in addition to the ability to determine the authenticity of the image. Gain the ability to In the present embodiment, the images used for learning the classification score are only the high-resolution images acquired from the learning data, and do not use the estimated high-resolution images generated by the high-resolution processing unit. This is because during the learning, the correspondence between the image generated by the high-resolution processing unit and the classification score is not reliable. For this reason, the high-resolution processing unit and the determination unit have a conflicting relationship regarding the authenticity of the image as in FIG. 6A, but have no conflicting relationship regarding the classification score.

  The example of the NN constituting the high-resolution processing unit has already been described with reference to FIG. The NN constituting the determination unit may be any suitable NN that inputs an image and outputs a vector obtained by connecting a boolean value and a classification score. For example, the NN in FIG. 5 may be such that the intermediate data 502 is the final output. At this time, the size of the intermediate data 502 is not 1014 but 1 + the number of elements of the classification score. It goes without saying that the parameters of the NN constituting the determination unit are not shared with the parameters of the high resolution NN.

  FIG. 7 is an example of an Encoder type NN. The determination unit may be configured by an NN as shown in FIG. What is necessary is to improve the final output of the NN having the Discriminator type architecture of the Deep Convolutional GAN so as to output a vector in which the classification score is added in addition to the original true / false value.

  The above is the parameter optimization method of the high resolution NN performed by the parameter optimization unit 305 in S408. Returning to the flowchart of FIG. 4, the description will be continued. In step S409, the parameter optimization unit 305 stores the optimized parameters in the parameter storage unit 306. Thus, the learning phases of S406 to S409 are completed, and the process proceeds to the resolution increasing phase starting from S402.

  The high-resolution processing unit shown in FIG. 6B may be the same as or different from the high-resolution processing unit 303. In any case, in the parameter optimizing unit 305, the parameters used in the high resolution processing unit 303 are generated and stored in the parameter storing unit 306. The high resolution processing unit 303 is configured with the NN using the parameters held in the parameter holding unit 306. The determination unit shown in FIG. 6B is included in the parameter optimization unit 305.

<High resolution phase>
In step S <b> 402, the high-resolution processing unit 303 acquires the learned parameters from the parameter storage unit 306. In the learning phase, learning (parameter optimization) of the two NNs of the high-resolution processing unit and the determination unit was performed. get.

  In step S403, the low-resolution image acquisition unit 301 acquires low-resolution image data from the HDD 203.

  In S404, the classification score obtaining unit 302 obtains a classification score from the HDD 203. Alternatively, the classification score may be obtained interactively from the input device 206 such as a display keyboard.

  FIG. 8 is a diagram illustrating an example of a UI used when interactively acquiring a classification score. When acquiring a classification score interactively, a UI as shown in FIG. 8 can be displayed on the output device 207 such as a display. FIG. 8 includes a classification score setting unit 801 and an image display area 802. The image display area 802 is an area for displaying an image that has been subjected to high resolution processing using the classification score set by the classification score setting unit 801. Note that any one of the classification scores acquired in S404, such as [0 0 1], does not have to be “1” and all others have to be “0”. As in the setting illustrated in FIG. 8, the classification score may be ambiguous information such as [0.2 0.1 0.7]. That is, in the present embodiment, it is possible to increase the resolution using ambiguous image identification information.

  In step S405, the high resolution processing unit 303 performs a high resolution process. Specifically, an input image 501 obtained by enlarging a low-resolution image to a desired size by bicubic interpolation or the like and a classification score 503 are input to the high-resolution NN shown in FIG. 504 are obtained.

  Through the above steps, the low-resolution image can be clearly enlarged according to the classification score. In the example of FIG. 8, since the classification score is [0.2 0.1 0.7], the image that has been sharply enlarged is subjected to the high-resolution processing such that the image is almost an old man with a little child element. You. If the subject is actually an old man and if the resolution is increased without image identification information, the classification score will be a value of a value obtained by equally assigning probabilities of prescribed types. In the example of FIG. 8, the resolution is increased when the classification score is [0.33 0.33 0.33]. Compared with such processing, the accuracy of the high resolution processing of the present embodiment is improved.

  As described above, in the present embodiment, it is possible to appropriately perform the resolution increasing process using the ambiguous image identification information that cannot always uniquely identify the type of the image. That is, in the present embodiment, the resolution is increased using the classification score for classifying the type of the image. The classification score is a score indicating the probability that the image is each of the identifiers for classifying the type of the image. By using the classification score, even in the case of ambiguous information that is not always the correct answer, it is possible to give the direction of higher resolution when the higher resolution processing is performed in the image processing apparatus. Therefore, if the image identification information (classification score) is information that is correct on average, the expected value of the accuracy of high resolution can be improved.

<< Embodiment 2 >>
In the present embodiment, a form in which the concept of the classification score is extended will be described. For example, a description will be given of a form in which different classification scores are defined according to the location of an image, and the resolution is increased, such as "the child is like a child above the nose, but like an adult from the nose". This embodiment is different from the first embodiment in S404, S407, and S408. If there is no special description, the other processes will be described as being the same as in the first embodiment. Hereinafter, the operation will be described.

  In S407, the identifier ID obtained in S406 is converted into a classification score s. The classification score of the present embodiment is a value indicating the probability that each region in the image (in the present embodiment, the upper left, upper right, lower left, and lower right when the image is divided into four) is each of the identifiers. The region in the image is represented by an index (x, y), and the upper left corresponds to (x = 0, y = 0) and the lower right corresponds to (x = 1, y = 1). Since the value of the classification score s differs depending on the region, it is described as s (x, y).

for the i-th identifier ID _i, classification score s (x, y) _i of the image area (x, y) is a vector j-th element s (x, y) _ij is defined by the following equation.

For example, if the identifier of the old man (ID = 3) is set in the image, the corresponding classification score s (x, y) _ij is “0 0 1” for all x and y, and the third element is “0 0 1”. 1 ", and the other elements are vectors of" 0 ". When different identifiers are set according to the image areas, such as the eyes are young but the mouth is old, the image area (x, y) corresponding to the eyes becomes [0 10], The image area (x, y) corresponding to the lip becomes [0 0 1].

  In S408, the parameter optimizing unit 305 optimizes the parameters of the NN for performing the high resolution processing. The architecture of the high resolution NN of this embodiment is different from the example described in the first embodiment. This is because the classification score has a spatial dimension of (x, y), so that the classification score 503 cannot be added to the intermediate data 502 in FIG. The reason why the classification score 503 cannot be added to the intermediate data is that the dimensions do not match.

FIG. 9 is a diagram illustrating the network architecture of the high resolution NN according to the present embodiment. Most of FIG. 9 is the same as FIG. 5 except that the place where the classification score is combined with the intermediate data is different. Since the spatial resolution of the classification score is 2 × 2, the classification score s (x, y) _i is combined with the intermediate data 902 having the spatial resolution of 2 × 2. In FIG. 9, the intermediate data 902 has a size of 2 × 2 × 502, but the intermediate data U4 becomes 2 × 2 × 512 due to the concatenation. Subsequent processing is the same as in the first embodiment.

  In S404, the classification score obtaining unit 302 obtains a classification score from the HDD 203. Alternatively, the classification score may be obtained interactively from the input device 206 such as a display keyboard.

  FIG. 10 is a diagram illustrating an example of a UI used when interactively acquiring a classification score. When acquiring the classification score interactively, a UI as shown in FIG. 10 can be displayed on the output device 207 such as a display. FIG. 10 includes an image area selection frame 1003, a classification score setting unit 1001, and an image display area 1002. The classification score setting unit 1001 is used to set a classification score of the image region selected in the image region selection frame 1003. The image display area 1002 is an area for displaying an image that has been subjected to the high-resolution processing using the classification score. Naturally, a different classification score may be set for each region.

  Through the above steps, a different classification score can be set for each region of the image, and such a region can be sharply enlarged according to a predetermined classification score. For example, in the manner of creating a montage photograph for criminal investigation, identification information can be set for each area of an image, and a clearly enlarged image can be obtained accordingly. It goes without saying that the identification information need not be reliable.

In the first and second embodiments, the learning data is a plurality of sets including (low-resolution image x _i , high-resolution image y _i , and identifier ID _i ), and the identifier is converted into a classification score in S407. However, if necessary, learning data may be stored in the form of (low-resolution image x _i , high-resolution image y _i , classification score s _i ) from the beginning. In this case, since the scalar value ID _i is held in the form of the vector value s _i , the data amount increases. However, in a case where it is appropriate for the identifier ID _i itself to be represented by a vector, it is preferable to hold the learning data in the form of a classification score from the beginning. The case is, for example, a case where a boy between a child and an adult is set to [0.5 0.50] as a learning classification score.

<< Other embodiments >>
In the embodiment described above, an image processing apparatus that processes image data has been described as an example. However, the present invention is not limited to this. The processing device may include a first acquisition unit that acquires the first data, a second acquisition unit that acquires the second data having a smaller space size than the first data, and a processing unit. The processing unit is configured by a neural network, and is configured to generate third data based on the first data and the second data. In such a processing device, the processing unit uses the intermediate data at the stage where the space size of the intermediate data obtained by processing the first data is the same as the space size of the second data, and the second data. Integrated intermediate data may be generated. Then, the processing unit may generate third data based on the integrated intermediate data. More preferably, the second data does not have spatial information, and the integrated intermediate data may be generated using the intermediate data and the second data at the stage where the spatial information has disappeared. .

  The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

301 Low-resolution image acquisition unit 302 Classification score acquisition unit 303 High-resolution processing unit 304 Learning data acquisition unit 305 Parameter optimization unit

Claims (18)

First acquisition means for acquiring first image data,
Second acquisition means for acquiring a classification score;
Processing means for generating second image data of higher resolution than the first image data from the first image data based on the classification score,
Parameters constituting the processing means are determined based on learning data composed of a plurality of sets of data including learning image data and an identifier for identifying the classification of the learning image data,
The image processing device according to claim 1, wherein the classification score indicates a probability that the first image data is each of the identifiers. First acquisition means for acquiring first image data,
Second acquisition means for acquiring a classification score;
Processing means for generating second image data of higher resolution than the first image data from the first image data based on the classification score,
The parameters constituting the processing means are determined based on learning data composed of a plurality of sets of data including learning image data and learning classification scores,
The image processing apparatus according to claim 1, wherein the classification score indicates a probability that the first image data is each of the identifiers for classifying an image.   The image processing apparatus according to claim 1, wherein the classification score includes two or more elements that are not zero. The processing means includes:
A neural network,
Intermediate data in which the space size of the first image data has been reduced, and generating integrated intermediate data based on the classification score,
The image processing apparatus according to claim 1, wherein the second image data is generated using the integrated intermediate data.   The method according to claim 4, wherein the processing unit generates the integrated intermediate data using the intermediate data and the classification score at a stage where the space size of the intermediate data is the same as the space size of the classification score. The image processing apparatus according to any one of the preceding claims.   The image processing device according to claim 1, wherein the classification score is set for each region of an image corresponding to the first image data. Parameters constituting the processing means are as follows:
An objective function for optimizing parameters constituting the processing means,
A parameter constituting a determination unit that outputs a value indicating whether the input image input is an image generated by the processing unit or not, and an estimated classification score obtained by estimating the classification of the input image, An objective function to optimize,
The image processing apparatus according to any one of claims 1 to 6, wherein the image processing apparatus is determined using: The objective function for optimizing the parameters constituting the processing means is:
When the processing unit inputs the second image data generated from the first image data based on the classification score to the determination unit, the estimated classification score output from the determination unit, The image processing apparatus according to claim 7, further comprising a term set based on a difference between the classification score and the classification score. The objective function for optimizing the parameters constituting the determination means is:
The second image data generated from the first image data based on the classification score by the processing means, when input to the determination means, a value indicating the truth output from the determination means and The image processing apparatus according to claim 7, further comprising a term that does not include the estimated classification score among the estimated classification scores.   The image processing apparatus according to claim 1, wherein the second acquisition unit acquires the classification score in accordance with an instruction through a UI. First acquisition means for acquiring first data;
Second acquisition means for acquiring second data having a smaller space size than the first data,
Based on the first data and the second data, to generate third data, processing means configured by a neural network,
With
The processing means uses the intermediate data at the stage where the spatial size of the intermediate data obtained by processing the first data is the same as the spatial size of the second data, and the integrated intermediate data using the second data. , And generating the third data based on the integrated intermediate data. The second data has no spatial information,
12. The processing apparatus according to claim 11, wherein the integrated intermediate data is generated using intermediate data at a stage where spatial information has disappeared and the second data.   13. The processing apparatus according to claim 12, wherein the second data has a space size of 1 × 1. An image processing method using a processing unit that generates second image data of higher resolution than the first image data from the first image data based on the classification score,
Obtaining the first image data;
Obtaining the classification score;
Inputting the first image data and the classification score to the processing unit,
With
Parameters constituting the processing means are determined based on learning data composed of a plurality of sets of data including learning image data and an identifier for identifying the classification of the learning image data,
The image processing method according to claim 1, wherein the classification score indicates a probability that the first image data is each of the identifiers. An image processing method using a processing unit that generates second image data of higher resolution than the first image data from the first image data based on the classification score,
Obtaining the first image data;
Obtaining the classification score;
Inputting the first image data and the classification score to the processing unit,
With
The parameters constituting the processing means are determined based on learning data composed of a plurality of sets of data including learning image data and learning classification scores,
The image processing method according to claim 1, wherein the classification score indicates a probability that the first image data is each of identifiers for identifying a classification of an image. Based on the first data and the second data, to generate third data, a processing method using a processing means configured by a neural network,
Obtaining the first data;
Acquiring the second data having a smaller spatial size than the first data,
Inputting the first data and the second data to the processing means,
The processing means, the intermediate data at the stage where the spatial size of the intermediate data processed the first data is the same as the spatial size of the second data, and the integrated intermediate data using the second data Generating
The processing means generating the third data based on the integrated intermediate data.   A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.   A program for causing a computer to function as each unit of the processing device according to any one of claims 11 to 13.

Images