JP2022143211A - Image processing device and learning method - Google Patents

Abstract

To make it possible to output a more suitable demosaic image.SOLUTION: An image processing device that performs image processing using a neural network (NN) includes: acquisition means for acquiring teacher image data; generation means for generating student image data obtained by mosaicking the teacher image data; weight acquisition means for acquiring a loss weight map indicating a weight of a loss value at each pixel position of the teacher image data; and learning means for performing learning of demosaic processing based on a difference between result image data obtained by inputting the student image data to the NN and the teacher image data, and the loss weight map. In the loss weight map, the weight of the loss value at a pixel position of a specific phase in the teacher image data is different from the weight of the loss value at a pixel position other than the specific phase.SELECTED DRAWING: Figure 4

Description

The present invention relates to image processing technology using machine learning.

2. Description of the Related Art A digital imaging apparatus such as a digital camera uses an imaging device equipped with a color filter having an RGB arrangement, for example. As a result, light of a specific wavelength is incident on each pixel of the imaging device. The Bayer array is often used as the RGB array of such color filters.

A captured image obtained by an imaging device having a Bayer array color filter is a so-called mosaic image in which only pixel values corresponding to any one of RGB colors are set for each pixel of the imaging device. The development processing unit of the camera performs various signal processing such as demosaic processing for interpolating the remaining two color components on each pixel of this mosaic image to generate a color image. As a conventional method of demosaic processing, a linear filter is applied to sparse RGB color data to perform linear interpolation of surrounding pixel values of the same color to calculate pixel values (RGB components) corresponding to each pixel. There is a method.

Since this method has low interpolation accuracy, many non-linear interpolation methods have been proposed so far, but all the methods have the problem of generating false colors and artifacts in specific image regions.

On the other hand, in recent years, a data-driven interpolation method using deep learning technology has been proposed. Non-Patent Document 1 discloses a technique for training a CNN-based demosaicing network. When the learning is completed, the learning result is used to perform inference (regression task on the input data) to input the mosaic image to the CNN and convert it to an RGB image.

M Gharbi, G Chaurasia, S Paris, F Durand, "Deep Joint Demosaicking and Denoising", Siggraph Asia 2016, ACM Transactions on Graphics (TOG), November 2016

By the way, the pixels in the above-described mosaic image are obtained by actual observation. Therefore, it is desirable that the pixel values (observed values) obtained by actual observation are included in the RGB image as a result of the demosaicing with the same values as those in the mosaic image. In other words, when a mosaic image is input to a CNN and a demosaic result image is obtained as an output, it is desirable that the observed values be output without being changed by the processing of the CNN. However, the output value of CNN can be different from the input value. That is, there is a problem that the observed value may change between the mosaic image and the demosaiced image.

The present invention has been made in view of such problems, and an object of the present invention is to provide a technique capable of outputting a more suitable demosaiced image.

In order to solve the above problems, an image processing apparatus according to the present invention has the following configuration. That is, in an image processing device that performs image processing using a neural network (NN),
Acquisition means for acquiring teacher image data;
generating means for generating student image data obtained by mosaicking the teacher image data;
weight acquisition means for acquiring a loss weight map indicating the weight of the loss value at each pixel position of the teacher image data;
learning means for learning demosaic processing based on the difference between the result image data obtained by inputting the student image data to the NN and the teacher image data and the loss weight map;
with
In the loss weight map, weights of loss values at pixel positions of a specific phase in the teacher image data are different from weights of loss values at pixel positions other than the specific phase.

ADVANTAGE OF THE INVENTION According to this invention, the technique which can output a more suitable demosaicing image can be provided.

2 is a block diagram showing the hardware configuration of the image processing device; FIG. FIG. 4 is a diagram for explaining the process of CNN generating an inference result image; It is a figure explaining the process in which CNN calculates a loss value. 2 is a block diagram showing the functional configuration of the image processing apparatus according to the first embodiment; FIG. 4 is a flowchart showing the flow of image conversion processing in the first embodiment; FIG. 11 is a block diagram showing the functional configuration of an image processing apparatus according to a second embodiment; FIG. FIG. 11 is a flowchart showing the flow of image conversion processing in the second embodiment; FIG. 5 is a diagram for explaining the flow of processing in a teacher image generation unit; It is a figure explaining a loss weight map. It is a figure explaining the flow which produces|generates a mosaic image. FIG. 2 is a diagram illustrating the flow of processing in a demosaicing network (prior art); It is a figure explaining the flow of the learning process in 2nd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
As a first embodiment of an image processing apparatus according to the present invention, a learning process in an image processing apparatus using a convolutional neural network (CNN) will be described below as an example. Inference (demosaicing) based on learning results (network parameters of CNN) will also be described. In particular, in the first embodiment, a form in which changes in observed values before and after demosaicing are suppressed by loss function calculation with loss weights introduced will be described.

<Device configuration>
FIG. 1 is a block diagram showing the hardware configuration of an image processing apparatus. The information processing apparatus 100 includes a CPU 101, a RAM 102, a ROM 103, a secondary storage device 104, an input interface 105, an output interface 106, an imaging device 111, and a GPU 112. Each component of the information processing apparatus 100 is interconnected by a system bus 107 . The information processing apparatus 100 is also connected to an external storage device 108 and an operation unit 110 via an input interface 105 . The information processing apparatus 100 is also connected to an external storage device 108 and a display device 109 via an output interface 106 .

The CPU 101 executes programs stored in the ROM 103 using the RAM 102 as a work memory, and comprehensively controls each component of the information processing apparatus 100 via the system bus 107 . As a result, various processes to be described later are executed. The GPU 112 performs operations on data received from the CPU 101 and outputs the results to the CPU 101 .

The secondary storage device 104 is a storage device that stores various data handled by the information processing apparatus 100, and an HDD is used in the first embodiment. The CPU 101 writes data to the secondary storage device 104 and reads data stored in the secondary storage device 104 via the system bus 107 . In addition to the HDD, various storage devices such as an optical disk drive and a flash memory can be used as the secondary storage device 104 .

The input interface 105 is, for example, a serial bus interface such as USB or IEEE1394. The information processing apparatus 100 receives data, instructions, and the like from an external device via the input interface 105 . In the first embodiment, the information processing apparatus 100 acquires data from an external storage device 108 (for example, a storage medium such as a hard disk, memory card, CF card, SD card, USB memory, etc.) via the input interface 105 . Further, in the first embodiment, the information processing apparatus 100 acquires a user's instruction input to the operation unit 110 via the input interface 105 . An operation unit 110 is an input device such as a mouse and a keyboard, and inputs user's instructions.

The output interface 106, like the input interface 105, is a serial bus interface such as USB or IEEE1394. Note that the output interface 106 may be a video output terminal such as DVI or HDMI (registered trademark). The information processing apparatus 100 outputs data and the like to an external device via the output interface 106 . In the first embodiment, the information processing apparatus 100 outputs data (eg, image data) processed by the CPU 101 to the display device 109 (various image display devices such as a liquid crystal display) via the output interface 106 . Although there are components of the information processing apparatus 100 other than those described above, they are not the focus of the present invention, so description thereof will be omitted.

The imaging device 111 captures an input image to be processed by the information processing device 100 . In the first embodiment, in the image processing apparatus 100, based on a command from the CPU 101, Bayer data (mosaic image) is input to an image processing application and demosaiced image data (demosaiced image) is output. shall be

<CNN>
First, CNN, which is used in general image processing technology applying deep learning technology, will be described. Then, demosaic processing using CNN and the principle of changes in observed values before and after demosaic processing will be described.

CNN is a learning-type image processing technology that repeats non-linear operations after convolution of an image with a filter generated by training or learning. A filter is also called a local receptive field (LRF). After convolving the filter with respect to the image, the image obtained by non-linear operation is called a feature map. Also, learning is performed using training data (training images or data sets) consisting of pairs of input images and output images. Simply put, “learning” is to generate, from training data, filter values that can be converted from an input image to a corresponding output image with high accuracy. The details of learning will be described later.

If the image has RGB color channels, or if the feature map consists of multiple images, the filter used for convolution will accordingly have multiple channels. That is, the convolution filter is represented by a four-dimensional array that includes the number of channels in addition to the vertical and horizontal size and the number of sheets. After convolving the image (or feature map) with the filter, the process of non-linear calculation is expressed in units called layers. For example, it is called an n-th layer feature map or an n-th layer filter. Also, for example, a CNN that repeats filter convolution and nonlinear operation three times is said to have a three-layer network structure. This process can be formulated as in the following formula (1).

In Equation (1), W _n is the n-th layer filter and b _n is the n-th layer bias. Also, G is a non-linear operator, X _n is an n-th layer feature map, and * is a convolution operator. Note that (l) in the right shoulder represents the l-th filter or feature map. Filters and biases are generated by learning, which will be described later, and are collectively called network parameters. A sigmoid function or ReLU (Rectified Linear Unit), for example, is used as the nonlinear operation. ReLU is given by Equation (2) below.

That is, ReLU is a non-linear process in which negative elements of the input vector X elements are zeroed and positive elements are left unchanged.

Next, learning of CNN will be described. CNN learning is to minimize an objective function generally represented by the following formula (3) for learning data consisting of a set of input learning images (student images) and corresponding output learning images (teacher images). is done in

where L is a loss function that measures the error between the correct answer and its estimate. Also, Y _i is the i-th output learning image, and X _i is the i-th input learning image. Also, F is a function collectively representing the calculations (formula 1) performed in each layer of the CNN. Also, θ is the network parameters (filter and bias). Furthermore, ||Z _||

In addition, n is the total number of learning data used for learning, but since the total number of learning data is generally large, in Stochastic Gradient Descent (SGD), a part of the learning image is randomly selected for learning used for This reduces the computational load in learning using a large amount of learning data.

Also, various methods such as the momentum method, the AdaGrad method, the AdaDelta method, and the Adam method are known as methods for minimizing (=optimizing) the objective function. Adam's method is given by Equation (4) below.

(4), θ _i ^t is the ith network parameter at the t th iteration and g is the slope of the loss function L with respect to θ _i ^t . Also, m and v are moment vectors, α is a base learning rate, β ₁ and β ₂ are hyperparameters, and ε is a small constant. In addition, since there is no selection guideline for optimization methods in learning, basically any method can be used. .

As networks using CNN, ResNet in the field of image recognition and RED-Net, which is applied in the field of super-resolution, are well known. In both cases, CNN is multi-layered and filter convolution is performed many times to improve the accuracy of processing. For example, ResNet is characterized by a network structure in which paths are provided to shortcut convolutional layers, thereby realizing a multi-layered network of 152 layers and realizing highly accurate recognition approaching the recognition rate of humans. The reason why the multi-layer CNN improves the accuracy of the processing is simply that the nonlinear relationship between the input and output can be expressed by repeating the nonlinear calculation many times.

Next, the principle that observed values change before and after demosaic processing in the prior art (Non-Patent Document 1) will be described. After that, we will discuss measures to suppress this change in observed values. FIG. 2 is a diagram for explaining the process by which the CNN generates an inference result image, and FIG. 3A is a diagram for explaining the process by which the CNN calculates a loss value.

During inference, the CNN receives a mosaic image 201 as input and outputs a demosaic result image 202 in RGB format. Here, the mosaic image conforms to the Bayer array color filter, but may be a mosaic image of another array such as X-trans.

Pixel values R ₀₀ , G ₀₁ , G ₁₀ , B ₁₁ , . . . are lined up. Observations R' ₀₀ , G' ₀₁ , G' ₁₀ , B' ₁₁ , . . . is shown with a white background. Other pixel values shown with a gray background are values (unobserved values) calculated by interpolation based on observed values. Here, R represents a red pixel (component), G represents a green pixel (component), and B represents a blue pixel (component). In each of the R-plane 203, G-plane 204, and B-plane 205 that constitute the demosaiced image, the observed values are arranged in a specific phase.

FIG. 3(a) shows processing at the time of learning necessary for performing such inference. Inference is also performed on the mosaic image during learning, and a demosaic result 202 is obtained. This is compared with the teacher image 302, which is demosaic correct data, to calculate an inference error according to a loss function, and learning proceeds so as to reduce the error (called a loss value).

The loss value l _i between the i-th teacher image Y _i and the i-th demosaic result image Y′ _i (=F(X _i ; θ)) is generally calculated by the L2 norm expressed by the following formula: be done.

Here, Y _i (x, y, c) represents the c-th channel value at the position (x, y) on the teacher image Y _i . This expression rewrites the terms in the summation symbol Σ of Equation (3) in a different form.

In this way, in the loss value calculation of the prior art, each phase is treated equivalently as in Equation (4), despite the difference in properties such as observed values and non-observed values depending on the phase in the image. . As a result, it is thought that the CNN could not demonstrate its performance and the observed value changed. In addition to the L2 norm shown in Equation (4), other loss functions such as the L1 norm, PSNR, and SSIM are also used, but each phase is treated equally.

Therefore, in the first embodiment, a weight w(x, y, c) for each phase is introduced, and the loss value is calculated as in Equation (6) below.

This weight is called a loss weight. In the first embodiment, changes in observed values are suppressed by setting loss weights so that observed values are emphasized more than unobserved values.

<Device operation>
FIG. 4 is a block diagram showing the functional configuration of the image processing apparatus according to the first embodiment. More specifically, it shows the functional configuration of the image processing apparatus when performing the learning process. Note that the configuration shown in FIG. 4A can be appropriately modified/changed. For example, one functional unit may be divided into a plurality of functional units by function, or two or more functional units may be integrated into one functional unit. Also, the configuration of FIG. 4(a) may be configured by two or more devices. In that case, each device is connected via a circuit or a wired or wireless network, performs data communication with each other, and performs cooperative operations, thereby realizing each process described below as what is performed by the image processing device.

Each functional unit shown in FIG. 4A may be implemented as hardware such as an application specific integrated circuit (ASIC), or may be implemented as software by the CPU 101 executing a computer program. . Alternatively, it may be implemented by a combination of hardware and software.

FIG. 5 is a flow chart showing the flow of image conversion processing in the first embodiment. In step S501, the teacher image acquisition unit 401 acquires a teacher image (teacher image data) in RGB format. The acquired teacher image is output to the learning data generation unit 402 . A teacher image is generated according to the method described in Non-Patent Document 1, for example.

FIG. 8 is a diagram for explaining the flow of processing in the teacher image generation unit. Specifically, the imaging device 111 acquires a mosaic image 801 (color mosaic image data) and applies simple demosaicing to generate an RGB image 802 . Finally, image reduction is applied to generate a teacher image 803 .

Bilinear interpolation is used for simple demosaicing, but other demosaicing techniques may be used. Also, here, the Bayer array is assumed as the color array of the color filters, but other arrays such as X-Trans may be used.

The training images are in the form of small images (patches) of constant size. When the size of the teacher image is not constant, or when the size is larger than the size that can be input to the CNN, the image is divided into patches in advance.

Note that the teacher image in RGB format may be acquired by a method other than Non-Patent Document 1. For example, an object captured and stored in advance may be read out, or a teacher image in RGB format may be obtained by capturing images while shifting the position of the image sensor.

In step S502, the learning data generation unit 402 performs sub-sampling from the received teacher image according to the color filter array pattern to generate a mosaic image (student image data). A student image is generated by sub-sampling the teacher image according to the color filter array pattern.

FIG. 10 is a diagram explaining the flow of generating a mosaic image. Specifically, a student image 1004 is obtained by sub-sampling the R component 1001, G component 1002, and B component 1003 of the teacher image based on the color filter array 1005 . The generated student image and teacher image are paired and output to the demosaicing learning unit 405 .

In step S503, the loss weight calculator 403 generates a loss weight map w(x, y, c) based on the color filter array. First, the loss weight calculation unit acquires information on the color filter array of the imaging device that generated the teacher image from the database, and calculates the pixel position (specific phase) where the observed value exists. Next, a loss weight map is created based on the weight w _{a for the observed value that is the pixel position of the specific phase and the weight w b for} the non-observed value that is the pixel position other than the specific phase. If the c-th channel at pixel position (x, y) contains an observed value, then w(x, y, c) = w _a , and if an unobserved value contains w(x, y, c) = w _b . For example, when w _a =5 and w _b =1, the loss weight map is as shown in FIG. 3(b).

Although the weights w _a and w _b are given values, they may be given by the user's input, or the values may be updated according to the progress of learning. Alternatively, w _b =0 may be set so that non-observed values do not contribute to loss value calculation. Unobserved values may be removed/masked when calculating the loss value, which is equivalent to setting w _b =0. Also, the loss weight of the observed value does not have to be _uniform and may be different for each pixel. The loss weight map is output to demosaicing learning section 405 .

In step S504, the network parameter acquisition unit 404 acquires CNN network parameters used for demosaic learning. A network parameter is a coefficient of each filter that constitutes the CNN. The network parameters are set as random numbers following the He normal distribution. The normal distribution of He is a normal distribution that has a mean of 0 and a variance of σ _h shown in Equation (7) below.

where m _N is the number of neurons for that filter in the CNN. Note that network parameters may be determined by other methods. The acquired network parameters are output to the demosaicing learning unit 405 .

In step S505, the demosaicing learning unit 405 uses the received image pair to learn CNN. For learning, the method disclosed in Non-Patent Document 1 may be used.

FIG. 11 is a diagram illustrating the flow of processing in the demosaicing network. The structure of the CNN is a stack of a plurality of filters 1102 that perform the computation of Equation (1). FIG. 4B shows the detailed functional configuration of the demosaicing learning unit in the learning process.

First, network parameter storage section 410 receives network parameter θ from network parameter acquisition section 404 . The network parameters are used to initialize the weights of each layer of the CNN.

Subsequently, the demosaicing inference unit 411 inputs the student image 1004 to this CNN and performs demosaicing inference. On input, the student image is converted to a 3-channel missing image 1101 . The R channel of the missing image contains only the R component pixels of the student image, and the pixel values of the other pixels are set to the missing value (0). Similarly, only the pixel values of G and B are recorded for the G channel and the B channel, respectively, and the remaining pixel values are zero. Missing values may be interpolated by a method such as bilinear interpolation.

Next, a filter 1102 is sequentially applied to this missing image to calculate a feature map. Subsequently, a connection layer 1103 connects the calculated feature map and the missing image 1101 in the channel direction. If the number of channels of the feature map and the missing image is n ₁ and n ₂ respectively, the number of channels of the concatenation result is (n ₁ +n ₂ ). Subsequently, a filter is applied to this connection result, and the final filter outputs three channels to obtain a demosaic result image 1104 (result image data).

Subsequently, the loss value calculator 412 calculates the error (difference) between the obtained demosaic result image and the teacher image, and calculates the loss value by taking the weighted average of the entire image. Specifically, based on the loss weight map acquired from the loss weight calculator 403, the loss value l _i of the i-th demosaic result image is calculated according to Equation (5). Then, the final loss value L(θ) is calculated according to the following formula (8).

This formula is equivalent to formula (3). In addition to the loss value obtained by Equation (8), one or more other loss values may be calculated by a method such as Non-Patent Document 1, and a linear combination thereof may be determined as the final loss value. The obtained loss value is output to network parameter updating section 413 .

Subsequently, the network parameter updating unit 413 updates the network parameters from the calculated loss value by error back propagation or the like. This is a technique of changing the network parameters so that the loss value becomes smaller by calculating the derivative of the loss value. The obtained updated network parameters are stored in network parameter holding section 410 and output to inspection section 406 .

In step S506, the inspection unit 406 determines whether or not learning has been completed. In order to perform determination, a test chart, which is test image data, is prepared in a group of image data (chart images) such as landscape photographs and portrait photographs, which are not used for learning. A test chart is a mosaic image that includes areas where changes in observed values are likely to occur, such as high brightness or high saturation. This test chart is demosaiced using the learning result CNN to generate test result image data, and the degree of change ε of the observed value is evaluated according to the following equation (8).

Here, X _i is the i-th chart image, and F-1 is a function for mosaicking the RGB image as shown in FIG. ||Z|| _p is the p-norm and p=1, but p can be other values. If the degree of change ε is less than a given threshold, it is determined that learning has been completed. When selecting a test chart, the degree of change ε may be calculated from a plurality of candidate images according to Equation (9), and a candidate image having a large ε may be determined as the test chart.

Note that the learning completion criteria are not limited to this. For example, it is also possible to use a criterion of whether or not the amount of change in the network parameter at the time of updating is smaller than a specified value, or a criterion of whether or not the residual between the inference result and the teacher image is smaller than a specified value. Also, learning may be completed when the number of iterations of learning (update of network parameters) reaches a specified value. When learning is completed, the updated network parameters are output to the learning result storage unit 407 .

If the learning has not been completed, the process returns to step S505, and learning processing is performed again using the acquired learning data. It is not necessary to use all the learning data for this learning, and each time the process returns to step S505, an image group, which is a subset, may be randomly extracted from the learning data group and used for learning. Also, a new teacher image group may be acquired from the teacher image acquisition unit 401 .

In step S507, the learning result storage unit 407 stores the received network parameters.

The learning process is as described above, but when inference of demosaicing (demosaicing process) is performed using the learning result, the process proceeds to the following steps. In that case, the learning result storage unit 407 outputs the network parameters to the demosaicing inference unit 409 . FIG. 4C shows the functional configuration of the image processing device when performing inference processing (demosaicing processing).

In step S508, the input image acquisition unit 408 captures an image with the imaging device 111 and acquires a mosaic image (input image data) to be demosaiced. It should be noted that the input image may be read from an image that has been captured and stored in advance. The acquired input image is output to the demosaicing inference unit 409 .

In step S<b>509 , the demosaicing inference unit 409 constructs the same CNN as used in the learning in the demosaicing learning unit 405 . This network parameter is initialized with the network parameter received from the learning result storage unit 407 . The received input image is input to this CNN, and an output image is generated by the same method as that performed by the demosaicing learning unit 405 to obtain an inference result (output image data).

As described above, according to the first embodiment, learning is performed so as to reduce the error (loss value) in the loss function calculation to which the loss weight is introduced. This enables CNN learning that suppresses changes in observed values before and after demosaic processing. That is, it is possible to output a suitable demosaic image by performing inference (demosaicing) using the CNN obtained by learning.

(Second embodiment)
In the second embodiment, another form of CNN learning will be described. In the first embodiment, learning is performed by setting the loss weight w _{a of the observed value to a value greater than the loss weight w b of} the unobserved value. That is, when the amplification factor r was defined as r ₌ wa/ _wb , learning was performed with r>1.

However, learning with a large r may result in a CNN that causes image quality defects such as moire, especially in high-frequency areas. In order to suppress such image quality deterioration, it is desirable to perform learning with r=1. Therefore, in the second embodiment, an example of performing both learning with r=1 and learning with r>1 is shown. The description of the parts common to the first embodiment will be omitted, and the differences will be mainly described below.

<Device operation>
FIG. 12 is a diagram illustrating the flow of learning processing in the second embodiment. As shown in the figure, first, pre-learning with r=1 is performed to suppress image quality deterioration in high-frequency areas. After that, main learning with r>1 is performed to suppress changes in observed values.

FIG. 6A is a block diagram showing the functional configuration of the image processing apparatus according to the second embodiment. FIG. 7 is a flow chart showing the flow of image conversion processing in the second embodiment.

In step S701, the loss weight calculator 403 generates a loss weight map under the condition r=1. That is, all values in the loss weight map will be 1 (or a given constant). Note that r may be substantially equal to 1, and may be set to a value such as r=1.1, for example. The loss weight map is output to demosaicing learning section 405 .

In step S702, the demosaicing learning unit 405 trains the CNN based on the received loss weight map. The training method is the same as in step S505. This training is CNN pre-learning.

In step S703, the learning result storage unit 407 stores the network parameters obtained by pre-learning.

In step S704, loss weight calculator 403 generates a loss weight map under the condition of r>1. For example, with r=5, the loss weight map of FIG. 3(b) is obtained. The loss weight map is output to demosaicing learning section 405 .

In step S<b>705 , the network parameter acquisition unit 404 reads the pre-learned network parameters from the learning result storage unit 407 and outputs them to the demosaicing learning unit 405 .

In step S706, the demosaicing learning unit 405 initializes the weights of the CNN using the received pre-learned network parameters as initial values, and then performs the main learning of the CNN based on the received loss weight map of r>1. The network parameters obtained as learning results are output to the learning result storage unit 407 after the end determination.

In the above description, it was explained that learning is performed by first setting the amplification factor to r=1, and then learning is performed by setting r>1. However, the order in which both settings are used is not limited to this, and for example, learning may be performed in the reverse order, or learning may be performed alternately. Also, the number of stages of learning is not limited to two stages, and multiple stages of learning may be performed while increasing the value of r.

Furthermore, the value of r may be increased according to the progress of learning. For example, the value of r may be increased in conjunction with an increase in the number of epochs (the number of repetitions of learning) or a decrease in the loss value. At this time, every time the value of r is updated, the loss weight calculator 403 regenerates the loss weight map, and the loss value calculator 412 reloads the loss weight map.

As described above, according to the second embodiment, learning is performed for each of r=1 and r>1 with respect to the amplification factor r. As a result, it is possible to perform CNN learning with reduced image quality impairments such as high-frequency portions compared to the first embodiment.

(Modification A)
In the first and second embodiments, as shown in FIG. 3B, the same loss weight is set for the observed value for each channel. However, in the Bayer array, the number of G pixels is twice the number of R and B pixels, so the contribution of the G channel to the loss value in the loss weight map is about twice as large as that of the R and B channels. . Therefore, in this modified example, an example in which correction is performed so that the contributions are equal will be described with reference to FIG. 6(a).

FIG. 6(a) shows the functional configuration of the image processing apparatus when learning processing is performed, and is the same as FIG. 4(a). However, the internal configuration of loss weight calculation section 403 is different from the above-described embodiment.

FIG. 6(b) shows the detailed functional configuration of the loss weight calculator in the learning process. FIG. 9(a) is a diagram for explaining a loss weight map.

The existing pixel number calculation unit 601 calculates the ratio of existing pixel numbers for each color channel of the color filter array. In the Bayer array, the number of existing pixels is R:G:B=1:2:1. The ratio obtained by dividing the number of existing colors by the number of existing colors is defined as color amplification factors r _R , r _G and r _B . This calculation method can be expressed as the following formula (10).

Here, _nc is the number of existing pixels in that channel c. This is used to compute the observation loss weight as r _c w _a (cε{R,G,B}). Loss weight map calculator 603 generates a loss weight map based on this calculation result. In the example 902 of the generated result, since r _R =r _B =2, the loss values of R and B are twice that of G.

Note that the color filter array to which the above-described embodiments and modifications can be applied is not limited to the Bayer array. For example, for X-trans, the number of existing pixels is 2:5:2, and the loss weight may be calculated based on this. Also, the loss weight of the non-observed value may be corrected based on the color amplification factor.

Also, the method of calculating the color amplification factor r _C is not limited to Equation (10). For example, the color amplification factor may be set so that the average value and variance value of the loss weights in each channel are uniform. Alternatively, a random number for each pixel may be added to the color amplification factor. Furthermore, in equation (10), the color amplification factor r _C is proportional to the -1 power of the number of existing pixels n _c , but it may be proportional to the k power (k is an arbitrary real number).

By using the loss weight map generated in this way, it becomes possible to output a more suitable demosaiced image in which the difference in the degree of contribution of each color channel is compensated.

(Modification B)
In the first and second embodiments, the loss weights of in-phase pixels are set to be uniform regardless of their positions. However, in CNN with padding, the pixels near the edges of the demosaic result image are affected by the padding. Since this effect can lead to incorrect inference results, it is desirable to make the contribution of these pixels to the loss value small. Therefore, in this modified example, an example of attenuation correction of the loss weight according to the influence of padding will be described with reference to the block diagram 6(b).

The architecture information acquisition unit 602 acquires information on the configuration of the CNN architecture. Based on this information, a value (padding width) indicating how many pixels counted from the image edge is affected by padding is calculated.

For example, in the case of a CNN composed of two layers of 3×3 convolution filters, one pixel at the end of each filter layer is affected by padding, so the total padding width is two pixels. Therefore, pixels within a distance of 2 from the end of the loss weight map are multiplied by _a given attenuation coefficient _rd , and the loss weight of the observed value is _calculated as _rdrCwa .

Loss weight map calculator 603 generates a loss weight map based on this calculation result. FIG. 9(b) shows an example of an R channel map when a loss weight map is calculated for an 8×8 image. Considering that the effect of padding increases closer to the image edge, in the loss weight map, r _d =0.2 for pixels at a distance of 1 from the image edge and r _d =0.2 for pixels at a distance of 2 from the image edge. It gives a damping factor of 0.6. Similar attenuation correction is applied to non-observed pixels.

By using the loss weight map generated in this way, it is possible to output a more suitable demosaiced image in which the influence of padding is compensated.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

400 image processing device; 401 teacher image acquisition unit; 402 learning data generation unit; 403 loss weight calculation unit; 404 network parameter acquisition unit;

Claims (14)

An image processing device that performs image processing using a neural network (NN),
Acquisition means for acquiring teacher image data;
generating means for generating student image data obtained by mosaicking the teacher image data;
weight acquisition means for acquiring a loss weight map indicating the weight of the loss value at each pixel position of the teacher image data;
learning means for learning demosaic processing based on the difference between the result image data obtained by inputting the student image data to the NN and the teacher image data and the loss weight map;
with
The image processing apparatus according to claim 1, wherein in the loss weight map, a weight of a loss value at a pixel position of a specific phase in the teacher image data is different from a weight of a loss value at a pixel position other than the specific phase. 2. The image processing apparatus according to claim 1, wherein the specific phase is a phase in which pixel values obtained by actual observation exist in the teacher image data. The teacher image data is color mosaic image data obtained by an imaging device having a color filter,
3. The image processing apparatus according to claim 2, wherein said specific phase is determined by a color arrangement in said color filter. 4. The image processing according to claim 2, wherein in the loss weight map, the weight of the loss value at the pixel position of the specific phase is greater than the weight of the loss value at the pixel position other than the specific phase. Device. 5. The image processing apparatus according to claim 4, wherein in said loss weight map, the weight of the loss value at pixel positions other than said specific phase is zero. The learning means is
a parameter acquiring means for acquiring network parameters of the NN;
inference means for inputting the student image data to the NN in which the network parameters are set, applying inference of demosaic processing, and acquiring the result image data;
loss value calculation means for calculating a loss value based on the difference between the result image data and the teacher image data and the loss weight map;
updating means for updating the network parameters based on the loss value;
6. The image processing apparatus according to any one of claims 1 to 5, comprising: 7. The image processing apparatus according to claim 6, wherein said updating means updates said network parameters by error backpropagation using the loss value calculated by said loss value calculating means. input image acquisition means for acquiring input image data;
output image generating means for inputting the input image data to the NN set with network parameters obtained by learning by the learning means and applying inference of demosaic processing to generate output image data;
8. The image processing apparatus according to any one of claims 1 to 7, further comprising: Based on test result image data obtained by inputting given test image data to the NN set with network parameters obtained by learning by the learning means and applying inference of demosaic processing, the network parameters are set. 9. The image processing apparatus according to claim 1, further comprising inspection means for inspecting. The learning means is
a first learning means for performing learning by setting an amplification factor, which is a ratio of the weight of the loss value at the pixel position of the specific phase and the weight of the loss value at the pixel position other than the specific phase, to 1;
a second learning means for performing learning by setting the amplification factor to be greater than 1 using the network parameters obtained by the learning by the first learning means as initial values;
10. The image processing apparatus according to any one of claims 1 to 9, comprising: The teacher image data is color mosaic image data obtained by an imaging device having a color filter,
11. The method according to any one of claims 1 to 10, wherein in said loss weight map, the weight of the loss value at each pixel position of said teacher image data is determined based on the ratio of the number of pixels for each color channel in said color filter. The image processing apparatus according to any one of items 1 to 3. 3. In the loss weight map, the weight of a loss value at a pixel location affected by padding in the neural net is less than the weight of a loss value at a pixel location not affected by padding in the neural network. 12. The image processing device according to any one of 11. A learning method in an image processing device that performs image processing using a neural network (NN),
an acquisition step of acquiring teacher image data;
a generating step of generating student image data obtained by mosaicking the teacher image data;
a weight acquisition step of acquiring a loss weight map indicating the weight of the loss value at each pixel position of the teacher image data;
a learning step of learning demosaic processing based on the difference between the result image data obtained by inputting the student image data to the NN and the teacher image data and the loss weight map;
including
A learning method, wherein, in the loss weight map, a weight of a loss value at a pixel position of a specific phase in the teacher image data is different from a weight of a loss value at a pixel position other than the specific phase. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 12.

Images