JP2021039559A - Image processing device - Google Patents

Abstract

To provide an image processing device capable of achieving more suitable image quality.SOLUTION: An image processing device (1) includes a neural network (21) for receiving first input image data and outputting output image data, and a learning part (22) for learning the neural network (21). The learning part (22) uses first loss and second loss to learn the neural network (21).SELECTED DRAWING: Figure 1

Description

Embodiments of the present invention relate to an image processing apparatus.

Techniques for performing image processing such as super-resolution processing and filtering using a neural network are known. Of these, in supervised learning, an input image and a teacher image are used as inputs, and network parameters (NN parameters) are learned so as to minimize the loss between the two images.

For example, Patent Document 1 discloses a super-resolution method using a neural network. Patent Document 1 uses a conditional hostile generation network (GAN Network) composed of two neural networks, a generator and a discriminator.

JP-A-2019-79436

However, in the prior art as in Patent Document 1, there is room for improvement in achieving a desired image quality.

One aspect of the present invention is to realize an image processing apparatus capable of achieving more suitable image quality.

The image processing apparatus according to one aspect of the present invention includes a neural network in which the first input image data is input and outputs the output image data, and a learning unit for learning the neural network. The first loss using the output image data and the first teacher image data corresponding to the output image data, and the second teacher image corresponding to the output image data and the first teacher image data. The feature is that the neural network is trained by using the second loss using the second teacher image data obtained by performing image processing on the first teacher image data. To do.

In the image processing apparatus according to one aspect of the present invention, a first input image data and a third input image data having a processing intensity parameter indicating the degree of image processing as a pixel value are input, and a first output image is obtained. It is characterized by including a neural network that outputs data.

According to one aspect of the present invention, it is possible to provide an image processing apparatus capable of achieving more suitable image quality.

It is the schematic which shows the structure of the image processing apparatus which concerns on Embodiment 1 of this invention. It is a conceptual diagram which shows an example of the input / output of the CNN filter which concerns on Embodiment 1 of this invention. It is a conceptual diagram which shows an example of the input / output of the CNN filter which concerns on the modification of Embodiment 1 of this invention. It is a conceptual diagram which shows an example of the input / output of the CNN filter which concerns on Embodiment 2 of this invention. It is a conceptual diagram which shows an example of the input / output of the CNN filter which concerns on the modification of Embodiment 2 of this invention. It is a conceptual diagram which shows an example of the input / output of the CNN filter which concerns on Embodiment 3 of this invention.

[Embodiment 1]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a schematic view showing the configuration of the image processing device 1 according to the present embodiment.

The image processing device 1 functions as, for example, an image filter device that filters images. The application destination of the image processing device 1 is not limited to this embodiment, but as an example, as an image filter device, super-resolution processing, resolution conversion processing, definition enhancement processing, noise reduction processing, loop filter and post. Examples include filters.

Here, the loop filter is a filter that acts on the decoded image in the decoding loop in the image decoding process or the image coding process, and the filtered image is referred to as a reference picture and used to generate a predicted image. The post filter is a filter that acts on the output image after the decoding loop process in the image decoding process, and is used to improve the image quality of the output image.

The image processing device 1 includes an input / output interface 10, a control unit 20, and a memory 30 as an example. The input / output interface 10 acquires the first input image data A1 from the external input device and outputs the output image data B to the external output device. As the control unit 20, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) is used. The memory 30 stores various data. When the image processing device 1 is provided in the image decoding device or the image coding device, the input / output interface 10 and the memory 30 can be replaced with an interface or a picture buffer in the image decoding device or the image coding device.

The control unit 20 includes a CNN (Convolutional Neural Network) filter 21 (neural network) composed of a network calculated by NN parameters including weights and biases. Further, the control unit 20 may further include a learning unit 22 for learning the filter 21.

Here, CNN is a general term for neural networks having at least a convolutional layer (a layer in which the weighting coefficient and bias / offset in the product-sum operation do not depend on the position in the picture). The weighting factor is also called the kernel. In addition to the convolutional layer, the CNN filter 21 can include a layer called a full connection layer (FCN) whose weight calculation depends on the position in the picture. Further, the CNN filter 21 has a configuration in which neurons belonging to a layer are connected only to some inputs of the layer (in other words, neurons have a spatial position and are connected only to inputs close to the spatial position. ) LCN (Locally Connected Networks) layer can also be included. In the CNN filter 21, the input size to the convolutional layer and the output size may be different. That is, the CNN filter 21 can include a layer in which the output size is smaller than the input size by making the movement amount (step size) when moving the position to which the convolution filter is applied larger than 1. It can also include a deconvolution layer in which the output size is larger than the input size. The deconvolution layer is sometimes called a transposed convolution. In addition, a plurality of channels called DepthToSpace (Pixel shuffler) may be expanded to convert the resolution. For example, 4 channels may be converted into double vertical and double horizontal resolution by expanding each value to the upper left, upper right, lower left, and lower right. Further, the CNN filter 21 can include a pooling layer (Pooling), a dropout (DropOut) layer and the like. The pooling layer is a layer that divides a large image into small windows and obtains representative values such as the maximum value and the average value according to each divided window, and the dropout layer has a fixed output value (for example, depending on the probability). It is a layer that adds randomness by setting it to 0). The number of layers included in the CNN filter 21 does not limit the present embodiment.

The image of the input layer of the CNN filter 21, the feature map of the intermediate layer, and the image of the output layer may be a tensor of width W × height H × number of channels C. Further, a four-dimensional tensor may be used instead of the three-dimensional tensor. The channels of the input layer may be 3 channels of R, G, B and Y, U, V, and 3 + with coded data (for example, NF channel) and processing intensity parameter (for example, 1 channel) added. It may be NF + 1 channel. Further, the channels of the output layer may be three channels of R, G, B and Y, U, V. When processing R, G, B or Y, U, V color components one by one, the input layer may be 1 + NF + 1 channel and the output layer may be 1 channel. Further, the output layer may include an image of the segmentation result and the like in addition to the processed image. The resolutions (width and / or height) of the first input image data A1 and the output image data B of the CNN filter 21 may be different. That is, resolution conversion may be performed. Of the resolution conversions, the conversion in the direction of increasing the resolution is generally called super-resolution.

Subsequently, the specific processing of the CNN filter 21 according to the present embodiment will be described with reference to FIG. FIG. 2 is a conceptual diagram showing an example of input / output of the CNN filter 21 according to the present embodiment.

As shown in FIG. 2, the first input image data A1 is input to the CNN filter 21 via the input / output interface 10. The CNN filter 21 filters the first input image data A1 and outputs the output image data B.

(Learning phase)
The learning unit 22 trains the CNN filter 21. The learning unit 22 includes a loss derivation unit and an NN parameter update unit. The loss derivation unit calculates the combined Loss from the teacher image and the output image data B. The NN parameter update unit derives the gradient of the NN parameter in the direction of decreasing Loss, and updates the NN parameter by adding the product of the learning rate and the gradient to the NN parameter. When the loss GAN Loss of the network (discriminator) from which the loss is derived is further used as a part of the coupling loss, the gradient of the discriminator's NN parameter is derived, and the product of the learning rate and the gradient is used as the NN parameter. Update the NN parameter of the discriminator by adding. When using GAN, it is advisable to simultaneously learn the generator, which is the CNN filter 21, and the discriminator (update the NN parameters alternately).

The learning unit 22 calculates the combined Loss by the loss deriving unit using the first loss Loss1 and the second loss Loss2, and updates the NN parameter in the NN parameter updating unit to learn the CNN filter 21. The first loss is a loss using the output image data B and the first teacher image data C1 corresponding to the output image data B. The second loss is a loss using the output image data B and the second teacher image data C2 corresponding to the first teacher image data C1.

Here, the loss (error, cost) using a certain image data and another image data is a value that quantitatively indicates the difference between the certain image and the other image. The function for calculating the loss from the two images A and B is called a loss function. For example, as the first loss and the second loss, there are MSE (Mean Squared Error) which is the L2 norm and MAE (Mean Absolute Error) which is the L1 norm, respectively. In addition, there is a difference between the feature PA (feature map) due to the neural network with the image A and the feature PB due to the same neural network of the image B (for example, MSE or MAE), which are called Perceptual Loss or Content Loss. Perceptual Loss may use a network such as VGG. The formula for MSE is as follows:
PerceptualLoss = Σ | PA --PB | ^ 2
In addition, there is a loss called GAN loss (hostile generation network loss), which is calculated from two images using a neural network called a discriminator, and a loss called gram loss or style loss. In Gram Loss, the Gram matrix G, which is the inner product between the two feature maps, is derived.
G_ij = ΣPik * Pjk
The above Σ may lose a loss from the difference between the Gram matrix GA of the total image A and the Gram matrix GB of the image B with respect to k.

GramLoss = Σ | GA --GB | ^ 2
Losses can be derived using these loss functions. For example, an example of loss of teacher image data C and output image data B is shown below.

MSE = Σ | C (x, y) --B (x, y) | ^ 2 / N
MAE = Σ | C (x, y) --B (x, y) | / N
Where N is the total number of pixels.

The learning unit 22 sequentially updates various NN parameters (weights, biases) in the CNN filter 21 so that the first loss and the second loss become smaller for various input image data, so that the CNN filter 21 To learn. As an example, the learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 so that the combined Loss of the sum of the first loss and the second loss becomes smaller. Further, a loss other than the first loss and the second loss may be included. For example, the various NN parameters in the CNN filter 21 are sequentially updated so that the combined Loss of the sum of the first loss and the second loss, the third loss Loss3, ..., The Nth loss LossN becomes smaller. May train the CNN filter 21.

As a result, the CNN filter 21 can output output image data B similar to both the first teacher image data C1 and the second teacher image data C2.

(Explanation of image data)
The first input image data A1 used in the learning phase is, for example, image data obtained by subjecting the first teacher image data C1 to a predetermined process. For example, the first input image data A1 is image data in which the first teacher image data C1 is encoded, image data obtained by processing the first teacher image data C1 to have a low resolution, and the like.

The second teacher image data C2 is teacher image data obtained by performing image processing on the first teacher image data C1. The image processing is at least one processing of sharpness, fineness, tone, blurring, noise removal amount, and touch.

(Other examples of loss combinations)
In the above description, as an example, the learning unit 22 has described the case where the CNN filter 21 is trained so that the coupling loss, which is the sum of the first loss and the second loss, becomes small. The form is not limited.

In the following, another example is shown in which the learning unit 22 trains the CNN filter 21 using the first loss and the second loss.

The learning unit 22 may train the CNN filter 21 by using the coupling Loss in which at least one of the first loss and the second loss is weighted. That is, only the first loss may be weighted, only the second loss may be weighted, or both the first loss and the second loss may be weighted.

Specifically, the learning unit 22 uses a weighted linear sum in which at least one of the first loss and the second loss is weighted. Then, the CNN filter 21 is trained by sequentially updating various NN parameters in the CNN filter 21 so that the first loss and the second loss become smaller. As a result, the CNN filter 21 can output appropriate output image data B according to the weight.

For example, when weighted so that the effect of the first loss is greater than the effect of the second loss, the CNN filter 21 is more similar to the first teacher image data C1 than to the second teacher image data C2. Output image data B can be output. On the other hand, when weighting is performed so that the effect of the second loss is larger than the effect of the first loss, the CNN filter 21 is more similar to the second teacher image data C2 than the first teacher image data C1. Output image data B can be output.

As a specific example of weighting, when the learning unit 22 trains the CNN filter 21 by using the weighted linear sum obtained by weighting the second loss, the following may be used as the weighted linear sum.

Loss = Loss1 + w * Loss2
Here, w is a weighting coefficient for the second loss.
Of course, the combined Loss may be calculated using a plurality of weights as follows.

Loss = w1 * Loss1 + w2 * Loss2
In addition, Loss may include GAN Loss, Perceptual Loss, and the like. Further, Loss1 / Loss2 may use a plurality of losses (Loss1L1, Loss1L2 / Loss1L1, Loss1L2). For example
Loss = w1 * Loss1 + w2 * Loss2 + w3 * GANLoss + w4 * Perceptual Loss
It may be. In general, a combination Loss consists of a weighted linear combination of a plurality of Loss identified by the subscript i as shown in the following equation.

Loss = Σwi * Loss_i
Here, Loss_i includes a first loss and a second loss. For example, Loss_0 = first loss, Loss_1 = second loss.

If the effect of the first loss is greater than the effect of the second loss, the value of w may be less than 1 (w1> w2). Further, when the influence of the second loss is made larger than the influence of the first loss, the value of w may be made larger than 1 (w1 <w2).

(Operation phase)
In the operation phase, the CNN filter 21 is made to input a desired image to be actually subjected to image processing. Then, the image processing device 1 performs filter processing using the CNN filter learned as described above. As a result, the image processing device 1 can output output image data that has been appropriately subjected to image processing.

Although the learning phase and the operation phase have been described above as the operations of the image processing device 1, these devices may be independent. That is, the image processing device 1 may be a device that includes a learning unit 22 (loss derivation unit and NN parameter update unit) and performs only the learning phase, or may include only the CNN filter 21 without the learning unit 22 and is in the operation phase. It may be a device that performs only.

(Modified Example of Embodiment 1)
A modified example of the first embodiment will be described below. The configuration of the image processing device 1 is the same as that of the first embodiment. The learning phase and the operation phase are the same as those in the first embodiment except for the differences from the first embodiment.

FIG. 3 is a conceptual diagram showing an example of input / output of the CNN filter 21 according to this modification.

As shown in FIG. 3, in addition to the first input image data A1, a second input image data A2 having a coding parameter related to the first input image data A1 as a pixel value is further input to the CNN filter 21. May be done. The second input image data A2 having the coding parameter as the pixel value is an image in which the value of the coding parameter (for example, the quantization parameter qP (xy)) at each position (x, y) of the image size is used as a pixel. There may be. For example, when a coded image quantized with qP = 22 is used as the first input image data A1, A2 may be an image having 22 as a pixel value.
It is a set. That is, the set of A1 and A2 may be a set of a coded image encoded by the quantization parameter of qP = 27 and an image consisting of qP (x, y) = 27. Prepare and input these sets for a large number of images. Further, as an example, the first input image data A1 and the second input image data A2 may be concatenate and input to the CNN filter 21. For example, when the resolutions of the first input image data A1 and the second input image data A2 are different, the resolution of the second input image data A2 is expanded according to the first input image data A1 and then combined ( You may enter by concatenate).

Further, as is called delayed input, the second input image data A2 may be concatenate and input not only in the input layer of the CNN filter 21 but also in the feature amount of the intermediate layer of the CNN filter 21.

The second input image data A2 may be concatenate and input in both the feature map of the input layer of the CNN filter 21 and the feature map of the intermediate layer of the CNN filter 21. Further, the second input image data A2 may be concatenate and input in the feature amount of the intermediate layer of the plurality of CNN filters 21.

By inputting the second input image data A2 having the coding parameter related to the first input image data A1 as the pixel value to the CNN filter 21, the output image data B corresponding to the coding parameter can be generated. ..

(Explanation of coding parameters)
Coding parameters are quantization parameters (QP: quantization parameters), partitioning parameters (multi-tree partitioning parameters, Multi Type Tree parameters, MTT parameters), prediction parameters, and coding targets generated in connection with coding. It is one or a set of parameters that becomes.

The quantization parameter is a parameter that controls the compression ratio and the image quality of the image. The quantization parameter has a characteristic that the larger the value, the lower the image quality and the decrease in the code amount, and the smaller the value, the higher the image quality and the increase in the code amount. As the quantization parameter, for example, a parameter for deriving the quantization width of the predicted residual can be used. The quantization parameter is not limited to the value itself transmitted on the coded data, and a quantization step derived from the quantization parameter transmitted on the coded data may be used.

As the quantization parameter for each picture, one typical quantization parameter of the frame to be processed can be input. For example, the quantization parameters can be specified by the parameter set applied to the target picture. Also, the quantization parameters can be calculated based on the quantization parameters applied to the components of the picture. Specifically, the quantization parameter can be calculated based on the average value of the quantization parameter applied to the slice.
Further, as the quantization parameter of the unit in which the picture is divided, the quantization parameter for each unit in which the picture is divided can be input based on a predetermined reference. For example, the quantization parameters can be applied slice by slice. Also, the quantization parameters can be applied to the blocks in the slice. In addition, the quantization parameter can be specified in a region unit independent of the existing coding unit (for example, each region obtained by dividing the picture into 16x9 pieces). In this case, since the quantization parameter depends on the number of slices and the number of conversion units, the value of the quantization parameter corresponding to the region becomes undefined, and the CNN filter cannot be constructed. Therefore, the average value of the quantization parameters in the region is calculated. There is a method used as a representative value. There is also a method of using the quantization parameter of one position in the region as a representative value. There is also a method of using the median or mode of the quantization parameters at a plurality of positions in the region as representative values.

When inputting a specific number of quantization parameters, a list of quantization parameters is generated so that the number of quantization parameters is constant, and a predetermined number of quantization parameters are sent to the CNN filter from the list. You may enter it. For example, a method of creating a list of quantization parameters for each slice and creating and inputting a list of three quantization parameters of maximum value, minimum value, and median value can be considered.

Further, as the quantization parameter for each component, the quantization parameter applied to the component to be processed can be input. Examples of this quantization parameter include a luminance quantization parameter (QPL) and a color difference quantization parameter (QPC).

When applying the CNN filter on a block-by-block basis, the quantization parameter of the target block and the quantization parameter (QPN) around the block may be input.

The partitioning parameter (MTT dividing parameter) is divided into quad trees as a division method from the coding tree unit (CTU: Coding Tree Unit) constituting the slice to the coding unit (CU: Coding Unit). In addition to QT division, it is a parameter that performs BT division that divides into a binary tree or TT division that divides into a ternary tree. It may be an image whose value is the division depth of each position. Partitioning increases the size and shape of the CU that can be selected, allowing adaptive partitioning to match the texture of the image. Partitioning makes it easier to select a region with homogeneous properties as one coding unit (CU), compared to HEVC with only quadtree (square) partitioning. The prediction parameters are a reference picture index, a motion vector, and the like.

Prediction parameters are intra prediction mode, Pred Mode that distinguishes between inter prediction modes, Intra Pred Mode that indicates the prediction mode and prediction direction of intra prediction, merge flags and affine flags that indicate the method of motion prediction, values that indicate the accuracy of motion vectors, and motion vectors. It may be a value classified by the size of, a value classified by the size of the motion vector difference, or a motion vector difference between the target pixel and the adjacent pixel.

The coding parameters include, for example, a reference value of the quantization width used for decoding the picture, a flag indicating the application of the weighted prediction, and a scaling list (quantization matrix).

(Learning phase)
The learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 for a large number of input images so that the first loss and the second loss become small. As an example, the learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 so that the coupling Loss, which is the sum of the first loss and the second loss, becomes small. ..

As a result, the CNN filter 21 can output output image data B similar to both the first teacher image data C1 and the second teacher image data C2.

In this example as well, the learning unit 22 applies the CNN filter 21 by using a weighted linear sum in which at least one of the first loss and the second loss is weighted, as in the above-described embodiment. You may let them learn. That is, only the first loss may be weighted, only the second loss may be weighted, or both the first loss and the second loss may be weighted.

(Operation phase)
In the operation phase, the CNN filter 21 is made to input a desired image to be actually subjected to image processing and an image whose pixel value is a coding parameter related to the desired image. Then, the image processing device 1 performs filter processing using the CNN filter learned as described above. As a result, the image processing apparatus 1 can output output image data that has been appropriately subjected to image processing according to the coding parameters.

[Embodiment 2]
Other embodiments of the present invention will be described below. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the above embodiment, and the description will not be repeated.

The configuration of the image processing device 1 according to the present embodiment is the same as that of the first embodiment.

FIG. 4 is a conceptual diagram showing an example of input / output of the CNN filter 21 according to the present embodiment.

As shown in FIG. 4, in the CNN filter 21, in addition to the first input image data A1, a third input image having a processing intensity parameter indicating the degree of image processing (for example, image fineness) as a pixel value. Further data A3 is entered. As an example, the first input image data A1 and the third input image data A3 may be combined and input to the CNN filter 21. The processing intensity parameter is, for example, an image quality parameter indicating the fineness of an image.

(Explanation of image processing)
The processing intensity parameter is a parameter used in at least one processing of sharpness, fineness, tone, blur, noise removal amount, and touch. The processing intensity parameter indicating the fineness of the image may be one parameter for the first input image data A1 or may be as many parameters as the number of pixels (resolution). In this case, each parameter can be adjusted on a pixel-by-pixel basis.

(Learning phase)
The learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 so that the first loss becomes small. As an example, the learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 so that the first loss becomes small.

As a result, the CNN filter 21 can output output image data B similar to the first teacher image data C1.

(Operation phase)
In the operation phase, the CNN filter 21 is made to input a desired image to be actually subjected to image processing and an image having a processing intensity parameter indicating the degree of image processing as a pixel value. Then, the image processing device 1 performs filter processing using the CNN filter learned as described above. As a result, the image processing apparatus 1 can output output image data that has been appropriately subjected to image processing according to the processing intensity parameter.

As an example, the user input regarding the processing intensity parameter is received via the input / output interface 10, and the third input image using the processing intensity parameter corresponding to the user input is input to the CNN filter 21. As a result, it is possible to output the output image data to which the image processing desired by the user has been performed.

(Modified Example of Embodiment 2)
A modified example of the second embodiment will be described below. The configuration of the image processing device 1 is the same as that of the second embodiment. The learning phase and the operation phase are the same as those in the second embodiment except that the differences from the second embodiment will be described below.

FIG. 5 is a conceptual diagram showing an example of input / output of the CNN filter 21 according to this modification.

As shown in FIG. 5, in the CNN filter 21, in addition to the first input image data A1 and the third input image data A3, the second input image data A1 has a second coding parameter as a pixel value. Input image data A2 may be further input. As an example, the first input image data A1, the second input image data A2, and the third input image data A3 may be combined and input to the CNN filter 21. Further, as described in the previous embodiment, the second input image data A2 and the third input image data A3 may be combined and input to the feature map of the intermediate layer. Further, the input layer and the intermediate layer, or a plurality of intermediate layers may be combined for input.

The second input image data A2 having the coding parameter related to the first input image data A1 as the pixel value and the third input image data A3 having the processing intensity parameter as the pixel value are input to the CNN filter 21. As a result, the output image data B can be generated according to the coding parameter and the processing intensity parameter.

The coding parameter is a quantization parameter, a partitioning parameter, a prediction parameter, or a parameter to be coded generated in connection with the prediction parameter. The details of the coding parameters are as described above in the first embodiment.

(Learning phase)
The learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 so that the first loss becomes small. As an example, the learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 so that the first loss becomes small.

As a result, the CNN filter 21 can output output data B similar to the first teacher image data C1.

(Operation phase)
In the operation phase, the CNN filter 21 is made to input a desired image to be actually subjected to image processing, an image having a coding parameter related to the desired image as a pixel value, and an image having a processing intensity parameter as a pixel value. Then, the image processing device 1 performs filter processing using the CNN filter learned as described above. As a result, the image processing apparatus 1 can output output image data that has been appropriately subjected to image processing according to the processing intensity parameter and the coding parameter.

As an example, the user input regarding the processing intensity parameter is received via the input / output interface 10, and the third input image using the processing intensity parameter corresponding to the user input is input to the CNN filter 21. As a result, it is possible to output the output image data which is the image processing desired by the user and has been subjected to the image processing according to the coding parameter.

[Embodiment 3]
Other embodiments of the present invention will be described below. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the above embodiment, and the description will not be repeated.

The configuration of the image processing device 1 according to the present embodiment is the same as that of the first embodiment.

FIG. 6 is a conceptual diagram showing an example of input / output of the CNN filter 21 according to the present embodiment.

As shown in FIG. 6, in the CNN filter 21, in addition to the first input image data A1, a second input image data A2 having a coding parameter related to the first input image data A1 as a pixel value and processing intensity The third input image data A3 having the parameter as the pixel value is further input. As an example, the first input image data A1, the second input image data A2, and the third input image data A3 may be combined and input to the CNN filter 21. Further, as described in the previous embodiment, the second input image data A2 and the third input image data A3 may be combined and input to the feature map of the intermediate layer. Further, the input layer and the intermediate layer, or a plurality of intermediate layers may be combined for input.

Further, in the CNN filter 21, in addition to the first input image data A1 and the second input image data A2, a third input in which the processing intensity parameter is a pixel value is weighted corresponding to the weighted linear sum. Image data A3 may be further input.

The CNN filter 21 is input with the second input image data A2 having the coding parameter related to the first input image data A1 as the pixel value and the third input image data A3 having the processing intensity parameter as the pixel value. To. As a result, the output image data B can be generated according to the coding parameter and the processing intensity parameter.

The coding parameter is a quantization parameter, a partitioning parameter, a prediction parameter, or a parameter to be coded generated in connection with the prediction parameter. The details of the coding parameters are as described above in the first embodiment.

(Learning phase)
The learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 so that the first loss and the second loss become smaller. As an example, the learning unit 22 trains the CNN filter 21 by sequentially updating various NN parameters in the CNN filter 21 so that the combined Loss of the sum of the first loss and the second loss becomes smaller.

As a result, the CNN filter 21 can output output image data B similar to both the first teacher image data C1 and the second teacher image data C2.

In this embodiment as well, the learning unit 22 learns the CNN filter 21 by using a weighted linear sum in which at least one of the first loss and the second loss is weighted, as in the first embodiment. You may let me. That is, only the first loss may be weighted, only the second loss may be weighted, or both the first loss and the second loss may be weighted.

As a specific example of weighting, when the learning unit 22 trains the CNN filter 21 by using the weighted linear sum obtained by weighting the second loss, the following may be used as the weighted linear sum.

Loss = Loss1 + w * Loss2
Here, w is a weighting coefficient for the second loss.
Of course, the combined Loss may be calculated using a plurality of weights as follows.

Loss = w1 * Loss1 + w2 * Loss2
If the effect of the first loss is greater than the effect of the second loss, the value of w may be less than 1. Further, when the influence of the second loss is made larger than the influence of the first loss, the value of w may be made larger than 1.

Further, the learning unit 22 may train the CNN filter so as to link the cost weight of the processing intensity parameter with w.

If the cost weight value of the processing intensity parameter is small, the value of w should be small. Further, when the value of the cost weight of the processing intensity parameter is large, the value of w may be increased.

As a specific example, when the cost weight of the processing intensity parameter is 0.1, the value of w is 0.25. When the cost weight of the processing intensity parameter is 0.2, the value of w is 0.5. When the cost weight of the processing intensity parameter is 0.3, the value of w is 0.75. When the cost weight of the processing intensity parameter is 0.4, the value of w is 1. The cost weight value and the w value of the processing strength parameter do not limit the present embodiment.

(Operation phase)
In the operation phase, the CNN filter 21 is made to input a desired image to be actually subjected to image processing, an image having a coding parameter related to the desired image as a pixel value, and an image having a processing intensity parameter as a pixel value. Then, the image processing device 1 performs filter processing using the CNN filter learned as described above. As a result, the image processing apparatus 1 can output output image data that has been appropriately subjected to image processing according to the processing intensity parameter and the coding parameter.

As an example, the user input regarding the processing intensity parameter is received via the input / output interface 10, and the third input image using the processing intensity parameter corresponding to the user input is input to the CNN filter 21. As a result, it is possible to output the output image data which is the image processing desired by the user and has been subjected to more suitable image processing according to the coding parameter.

(Hardware realization and software realization)
Further, each block of the image processing device 1 described above may be realized by hardware by a logic circuit formed on an integrated circuit (IC chip), or by software by using a CPU (Central Processing Unit). It may be realized.

In the latter case, each of the above devices includes a CPU that executes instructions of a program that realizes each function, a ROM (Read Only Memory) that stores the above program, a RAM (Random Access Memory) that expands the above program, the above program, and various types. It is equipped with a storage device (recording medium) such as a memory for storing data. Then, an object of the embodiment of the present invention is a record in which the program code (execution format program, intermediate code program, source program) of the control program of each of the above devices, which is software for realizing the above-mentioned functions, is recorded readable by a computer. It can also be achieved by supplying the medium to each of the above devices and having the computer (or CPU or MPU) read and execute the program code recorded on the recording medium.

Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic discs such as floppy (registered trademark) discs / hard disks, and CD-ROMs (Compact Disc Read-Only Memory) / MO discs (Magneto-Optical discs). ) / MD (Mini Disc) / DVD (Digital Versatile Disc: registered trademark) / CD-R (CD Recordable) / Blu-ray Disc (registered trademark) and other discs including optical discs, IC cards (memory cards) (Including) / Cards such as optical cards, mask ROM / EPROM (Erasable Programmable Read-Only Memory) / EEPROM (Electrically Erasable and Programmable Read-Only Memory: registered trademark) / Semiconductor memories such as flash ROM, or PLD ( Logic circuits such as Programmable logic device) and FPGA (Field Programmable Gate Array) can be used.

Further, each of the above devices may be configured to be connectable to a communication network, and the above program code may be supplied via the communication network. This communication network is not particularly limited as long as it can transmit the program code. For example, Internet, Intranet, Extranet, LAN (Local Area Network), ISDN (Integrated Services Digital Network), VAN (Value-Added Network), CATV (Community Antenna television / Cable Television) communication network, Virtual Private network (Virtual Private) Network), telephone line network, mobile communication network, satellite communication network, etc. can be used. Further, the transmission medium constituting this communication network may be any medium as long as it can transmit the program code, and is not limited to a specific configuration or type. For example, even wired such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, infrared data such as IrDA (Infrared Data Association) and remote control , BlueTooth (registered trademark), IEEE802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone network, satellite line, terrestrial digital broadcasting network, etc. It is also available wirelessly. The embodiment of the present invention can also be realized in the form of a computer data signal embedded in a carrier wave, in which the program code is embodied by electronic transmission.

The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the claims. That is, an embodiment obtained by combining technical means appropriately modified within the scope of the claims is also included in the technical scope of the present invention.

1 Image processing device
10 I / O interface
20 Control unit
21 CNN filter (neural network)
22 Learning Department
30 memory

Claims (9)

A neural network in which the first input image data is input and the output image data is output,
The learning unit that trains the above neural network and
With
The above learning department
The first loss using the output image data and the first teacher image data corresponding to the output image data, and the second teacher image corresponding to the output image data and the first teacher image data. The feature is that the neural network is trained by using the second loss using the second teacher image data obtained by performing image processing on the first teacher image data. Image processing device. The first aspect of claim 1, wherein the learning unit trains the neural network by using a weighted linear sum in which at least one of the first loss and the second loss is weighted. Image processing device. The image processing apparatus according to claim 1 or 2, wherein the image processing is at least one processing of sharpness, fineness, tone, blurring, noise removal amount, and touch. The invention according to any one of claims 1 to 3, wherein the second input image data having the coding parameter related to the first input image data as a pixel value is further input to the neural network. Image processing equipment. The invention according to any one of claims 1 to 4, wherein a third input image data having a processing intensity parameter indicating the degree of image processing as a pixel value is further input to the neural network. Image processing device. The learning unit trains the neural network by using a weighted linear sum in which at least one of the first loss and the second loss is weighted.
A third input image data having a processing intensity parameter indicating the degree of image processing, which is weighted corresponding to the weighted linear sum, is further input to the neural network. Item 6. The image processing apparatus according to any one of Items 1 to 4. It is characterized by including a neural network in which a first input image data and a third input image data having a processing intensity parameter indicating the degree of image processing as a pixel value are input and the first output image data is output. Image processing device. The image processing apparatus according to claim 7, wherein a second input image data having a coding parameter related to the first input image data as a pixel value is further input to the neural network. The image processing apparatus according to claim 7 or 8, wherein the image processing is at least one processing of sharpness, fineness, tone, blurring, noise removal amount, and touch.

Images