JP7504629B2 - IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, IMAGE PROCESSING PROGRAM, AND STORAGE MEDIUM - Google Patents

Description

The present invention relates to an image processing method using deep learning.

Patent Document 1 discloses a method for suppressing undershoot and ringing that accompanies high resolution and high contrast (sharpening) by taking into account the effects of gamma correction when training a multi-layer neural network that uses raw images as input. Non-Patent Document 1 discloses a network configuration that can be generally applied to various regression problems. Non-Patent Document 1 also discloses using a network to perform upsampling of input images, JPEG deblocking (removal of compression noise), denoising, non-blind deblurring, or inpainting.

JP 2019-121252 A

X. Mao, C. Shen, Y. Yang, "Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections", https://arxiv.org/. org/abs/1606.08921.

However, the method disclosed in Non-Patent Document 1 cannot perform appropriate estimation when the input image is a RAW image. In addition, Patent Document 1 realizes a neural network whose estimation accuracy is not easily affected by the magnitude of the luminance of the developed image by learning using an error that takes into account the effect of gamma correction. However, when a user actually views a developed output image, not only gamma correction but also white balance processing is performed, so if a RAW image is learned without considering white balance processing, the color balance may be significantly different from when it was developed. For example, even if an achromatic subject is photographed with white light that has no wavelength dependency as a light source, the luminance value of the acquired RAW image varies for each color due to the sensitivity characteristics of the image sensor. If learning is performed using such a RAW image without adjusting the luminance for each color, the estimation accuracy may also vary for each color.

The present invention therefore aims to provide an image processing method capable of obtaining a neural network that reduces the variation in estimation accuracy for each color.

An image processing method as one aspect of the present invention includes an acquisition step of acquiring a training image, a reference image, and information related to white balance ; a generation step of generating a color component image consisting only of color components corresponding to a plurality of color components of the training image; and an update step of inputting the color component image to a neural network to generate an output image consisting only of color components corresponding to the plurality of color components, acquiring a first error for each color component between the output image and the reference image, acquiring a second error based on the first error and information related to white balance, and updating parameters of the neural network based on the second error .

Other objects and features of the present invention are described in the following examples.

The present invention provides an image processing method that can obtain a neural network that reduces the variation in estimation accuracy for each color.

FIG. 1 is a diagram illustrating a convolutional neural network in a first embodiment. FIG. 4 is an explanatory diagram regarding white balance in each embodiment. 1 is a block diagram of an image processing system according to a first embodiment. 1 is an external view of an image processing system according to a first embodiment. 4 is a flowchart of a learning process in the first embodiment. FIG. 4 is an explanatory diagram regarding color components of an image in each embodiment. FIG. 4 is an explanatory diagram regarding gamma correction in each embodiment. 1 is a flowchart of an estimation process in each embodiment. FIG. 11 is a block diagram of an image processing system according to a second embodiment. FIG. 11 is an external view of an image processing system according to a second embodiment. FIG. 13 is a diagram showing a convolutional neural network in Example 2. 13 is a flowchart of a learning process in the second embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings. In each drawing, the same components are given the same reference symbols, and duplicate descriptions are omitted.

First, the terms used in each embodiment are defined below. Each embodiment relates to a method for solving a regression problem using deep learning and estimating various output images from an input image. Deep learning is machine learning using a multi-layered neural network. By learning network parameters (weights and biases) from a large number of pairs of training images and their corresponding ground-truth images (desired output), highly accurate estimation is possible even for unknown input images.

Image processing using a multi-layer neural network involves two processes: a process for updating the network parameters (weights and biases), and a process for making inferences about unknown inputs using the updated parameters. Below, the former is called the learning process, and the latter is called the estimation process.

Next, the names of the images in the learning process and the estimation process are determined. The image input to the network is the input image, and the input image used in the learning process, in which the correct answer image is known, is called the training image. The image output from the network is the output image, and the output image in the estimation process is called the estimated image. The input image to the network and the correct answer image are RAW images. Here, the RAW image is undeveloped image data output from the imaging element, and the light amount and signal value of each pixel have a roughly linear relationship. The RAW image is developed before the user views the image, and gamma correction is performed at that time. Gamma correction is, for example, a process of raising the input signal value to a power, and 1/2.2 is used as the exponent. In each embodiment, the image equivalent to losslessness that is the basis for generating the correct answer image or training image is called the original image.

The output image also has the properties of a raw image, since it is generated as an image similar to the ground truth image through estimation. The estimation process includes a variety of processes, such as upsampling, denoising, removal of compression noise, deblurring (blur correction), inpainting, demosaicing, dehaze, high gradation, and relighting (changing the lighting environment).

Before proceeding to a detailed description of the embodiments, the gist of the present invention will be described. The present invention considers the influence of white balance in the learning process of a multi-layered neural network that inputs RAW images. In general, imaging devices using an image sensor, such as digital cameras, are equipped with a white balance control function that adjusts the color tone of an image obtained by imaging. White balance processing is a process that performs gain processing for each color component on the RGB components output by the image sensor and adjusts the brightness level so that achromatic parts of the subject become achromatic in the output image. If white balance processing is not performed, the color of the subject will not be reproduced correctly due to the color characteristics of the image sensor, and an image with a color different from the actual subject will be generated.

Figure 2 is an explanatory diagram regarding white balance, where Figure 2(A) shows the spectral distribution characteristics of a light source, and Figure 2(B) shows the spectral sensitivity characteristics of an image sensor having a color filter and an IR cut filter. In Figure 2(A), the horizontal axis shows wavelength, and the vertical axis shows light intensity, with the solid line in Figure 2(A) showing the spectral distribution of a white LED and the dashed line showing the spectral distribution of an incandescent light bulb. In Figure 2(B), the horizontal axis shows wavelength, and the vertical axis shows spectral sensitivity, with the solid line in Figure 2(B) showing the spectral sensitivity of the B component, the dashed line showing the G component, and the dashed line showing the R component.

The signal output from the image sensor reflects the characteristics of the light source and image sensor as shown in Figures 2(A) and (B), and in fact also includes the effects of the transmittance of the optical system in addition to these characteristics. For example, the solid and dashed lines in Figure 2(A) represent a wavelength-dependent light source, but even if an achromatic subject is photographed in an environment with no wavelength dependency and completely flat light intensity, it will be affected by the spectral sensitivity characteristics of the image sensor as shown in Figure 2(B). In this case, since the subject is achromatic, the luminance values of the captured image should ideally be the same for the RGB components, but due to the influence of the spectral characteristics shown in Figure 2(B), the R and B components are lower than the G component. As a result, the output image will be colored green.

When such raw images are collected and training is performed, only green subjects are trained, and the output network parameters have high estimation accuracy for greener subjects, but low estimation accuracy for red and blue subjects. In reality, light sources are wavelength-dependent, and as shown in Figure 2(A), the color tone changes depending on the type of light source used at the time of shooting, which also affects the estimation accuracy. The present invention aims to reduce this variation in estimation accuracy for each color, and methods for achieving this are described in detail in the following embodiments.

First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, a multi-layer neural network learns and executes blur correction. However, this embodiment is not limited to blur correction and can also be applied to other types of image processing.

Figure 3 is a block diagram of the image processing system 100 in this embodiment. Figure 4 is an external view of the image processing system 100. The image processing system 100 has a learning device (image processing device) 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107.

The learning device 101 is an image processing device that executes a learning process, and includes a memory unit 101a, an acquisition unit 101b, a calculation unit 101c, an update unit 101d, and a generation unit 101e. The acquisition unit 101b acquires training images, correct images, and information related to white balance. The generation unit 101e inputs the training images into a multi-layer neural network to generate an output image. The update unit 101d updates the network parameters of the neural network based on the difference (error) between the output image and the correct image calculated by the calculation unit 101c. Details of the learning process will be described later using a flowchart. The learned network parameters are stored in the memory unit 101a.

The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives (photoelectrically converts) the optical image (subject image) formed via the optical system 102a to obtain an image. The imaging element 102b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The image obtained by the imaging device 102 contains blur due to aberration and diffraction of the optical system 102a and noise due to the imaging element 102b.

The image estimation device 103 is a device that executes the estimation process, and has a memory unit 103a, an acquisition unit 103b, and an estimation unit 103c. The image estimation device 103 performs blur correction on the acquired captured image to generate an estimated image. A multi-layer neural network is used for blur correction, and information on the network parameters is read from the memory unit 103a. The network parameters are learned by the learning device 101, and the image estimation device 103 reads the network parameters from the memory unit 101a via the network 107 in advance and stores them in the memory unit 103a. The stored network parameters may be their numerical values or may be in an encoded format. Details regarding the learning of the network parameters and the blur correction process using the network parameters will be described later.

The output image is output to at least one of the display device 104, the recording medium 105, and the output device 106. The display device 104 is, for example, a liquid crystal display or a projector. The user can perform editing work and the like while checking the image being processed via the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The output device 106 is, for example, a printer. The image estimation device 103 has a function of performing development processing and other image processing as necessary.

Next, a method for learning network parameters (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to FIG. 5. FIG. 5 is a flowchart related to learning network parameters. Each step in FIG. 5 is executed mainly by the acquisition unit 101b, the calculation unit 101c, the update unit 101d, and the generation unit 101e of the learning device 101.

First, in step S101 of FIG. 5, the acquisition unit 101b acquires a correct answer patch (correct answer image) and a training patch (training image). The correct answer patch is an image with relatively little blur, and the training patch is an image with relatively more blur. Note that a patch refers to an image having a predetermined number of pixels (e.g., 64×64 pixels). The number of pixels of the correct answer patch and the training patch do not necessarily need to be the same. In this embodiment, mini-batch learning is used to learn the network parameters of the multi-layered neural network. For this reason, in step S101, multiple pairs of correct answer patches and training patches are acquired. However, this embodiment is not limited to this, and online learning or batch learning may be used.

In this embodiment, the correct patch and the training patch are obtained by the following method, but the present invention is not limited thereto. In this embodiment, a plurality of original images stored in the storage unit 101a are used as subjects, and imaging simulation is performed to generate a plurality of high-resolution images substantially free of aberration and diffraction and a plurality of low-resolution images with aberration and diffraction. Then, a plurality of correct patches and training patches are obtained by extracting partial areas at the same position from each of the plurality of high-resolution images and the plurality of low-resolution images. In this embodiment, the original image is an undeveloped RAW image, and the correct patch and the training patch are also RAW images, but the present invention is not limited thereto and may be images after development. In addition, the position of the partial area refers to the center of the partial area. The plurality of original images are images having various subjects, that is, edges of various strengths and directions, textures, gradations, flat parts, etc. The original images may be real images or images generated by CG (Computer Graphics).

Preferably, the original image has a luminance value higher than the luminance saturation value of the image sensor 102b. This is because, even in actual subjects, there are subjects that do not fall within the luminance saturation value when photographed by the image sensor 102 under specific exposure conditions. The high-resolution image is generated by reducing the original image and clipping it with the luminance saturation value of the image sensor 102b. In particular, when a real image is used as the original image, blurring has already occurred due to aberration and diffraction, so by reducing the image, the effect of blurring can be reduced and a high-resolution (high-quality) image can be obtained. Note that if the original image contains sufficient high-frequency components, reduction is not necessary. The low-resolution image is generated by reducing the image in the same way as the high-resolution image, adding blurring due to the aberration and diffraction of the optical system 102a, and then clipping it with the luminance saturation value. The optical system 102a has different aberrations and diffractions depending on a plurality of lens states (zoom, aperture, and focal distance states), image height, and azimuth. For this reason, multiple low-resolution captured images are generated by adding different lens states, image heights, azimuth aberrations, and diffraction blurs to each original image.

The order of shrinking and blurring may be reversed. If blurring is performed first, the sampling rate of the blur must be finer, taking shrinking into consideration. In the case of a PSF (point spread function), the spatial sampling points should be finer, and in the case of an OTF (optical transfer function), the maximum frequency should be increased. If necessary, a component such as an optical low-pass filter included in the imaging device 102 may be added to the blur to be added. Note that distortion is not included in the blur to be added when generating a low-resolution captured image. This is because if distortion is large, the position of the subject changes, and the subject may differ between the correct patch and the training patch. For this reason, the neural network learned in this embodiment does not correct distortion. Distortion is corrected individually after blur correction using bilinear interpolation, bicubic interpolation, etc.

Next, a partial area of a specified pixel size is extracted from the generated high-resolution captured image and used as a correct patch. A partial area is extracted from the low-resolution captured image at the same position as the extraction position and used as a training patch. In this embodiment, mini-batch learning is used, so multiple correct patches and training patches are obtained from the multiple high-resolution captured images and low-resolution captured images generated. Note that the original image may contain noise components. In this case, the correct patch and training patch can be generated by considering the noise contained in the original image as the subject, so noise in the original image does not pose a particular problem.

For processes other than blur correction due to aberration and diffraction, a pair of training images and correct images can be prepared in a similar simulation to execute the learning process. For denoising, training images can be generated by adding expected noise to a low-noise correct image. For upsampling, training images can be prepared by downsampling the correct image. For removing compression noise, training images can be generated by compressing a correct image that is uncompressed or has a low compression rate. For deblurring other than aberration and diffraction (such as defocus blur), training images can be generated by convolving an expected blur into a correct image with little blur. In the case of defocus blur, since it depends on the distance, defocus blur of different distances is convolved into multiple training images and the correct image. For inpainting, training images can be generated by adding defects to a correct image without defects. For demosaicing, training images can be generated by resampling a correct image captured by a three-chip image sensor or the like in a Bayer array or the like. Regarding dehaze, training images can be generated by adding scattered light to a ground truth image without fog or mist. Since the intensity of scattered light in fog or mist varies depending on the density and distance, multiple training images are generated for scattered light of different densities and distances. Regarding high gradation, training images can be obtained by reducing the gradation of a high-gradation ground truth image. Regarding relighting, if the normal, shape, and reflectance distribution of the subject in the ground truth image are known, training images in different light source environments can be generated by simulation. However, since this involves a large measurement load, pairs of ground truth and training images can also be generated by actually photographing the subject in different lighting environments.

Next, in step S102, the acquisition unit 101b acquires information (learning condition information, white balance coefficient) related to the white balance used in the learning process. In this embodiment, the learning condition information is, for example, information related to settings such as "white balance setting at the time of shooting" and "auto white balance setting", or color temperature information of the light source. Typically, digital cameras are equipped with a function called auto white balance, which automatically identifies and corrects the type of light source. However, if the subject does not contain white, it is not easy to identify the light source. For this reason, digital cameras are generally equipped with a preset white balance function that allows the user to select the type of light source from a menu, and a manual white balance function that allows the user to directly specify the color temperature of the light source, etc.

The preset white balance function provides white balance coefficients (gain values for each color) suitable for each shooting condition, such as incandescent light bulbs, sunny days, cloudy days, and fluorescent lights. These white balance coefficients correspond to color temperatures, for example, 3000K for incandescent light bulbs and 6000K for cloudy days. If the color temperature is 3000K, the shooting environment is assumed to be redder than the subject's actual color, so the white balance coefficient is larger for the B component than the R component. Conversely, if the color temperature is 6000K, the shooting environment is assumed to be bluish than the subject's actual color, so the white balance coefficient is larger for the R component than the B component. In other words, if learning is performed using a white balance coefficient with a color temperature of 3000K, the coefficient value of the B component will be larger than the actual color, and the estimation accuracy will be higher for the B component than the R component. Conversely, if learning is performed using a white balance coefficient with a color temperature of 6000K, the coefficient value of the R component will be larger than the actual color, and the estimation accuracy will be higher for the R component than the B component.

In this way, by allowing the learning condition information to be selected as a color temperature and using network parameters corresponding to each color temperature, the user can select which RGB color estimation accuracy is to be prioritized. In this embodiment, the learning condition information is set to "white balance setting at the time of shooting", and in step S102, the white balance coefficient set at the time of shooting the RAW image that is the source of the correct patch or training patch is obtained. The white balance coefficient may be obtained from the header information of the RAW image, or from the imaging device 102. In the following description, when "header information" is mentioned, it represents additional information of the image, and may be footer information. In this embodiment, the white balance coefficient set at the time of shooting is obtained, but the learning condition information may be set to "auto white balance setting" and the white balance coefficient automatically determined and calculated by the imaging device may be used.

Next, in step S103, the generation unit 101e selects at least one training image from the multiple training images acquired in step S101, inputs the selected training image to the network, and generates an output image. The case where all of the multiple training images are selected (all of the training images are input to the network, and the network parameters are updated using all of the outputs) is called batch learning. This method imposes a huge computational load as the number of training images increases. The case where only one training image is selected (only one training image is used to update the network parameters, and a different training image is used for each update) is called online learning. This method does not increase the amount of computation even if the total number of training images increases, but is susceptible to the influence of noise present in one training image. For this reason, it is preferable to use a mini-batch method in which a small number (mini-batch) of multiple training images is selected and used to update the network parameters. In the next update, a small number of different training images are selected and used. By repeating this process, the weaknesses of batch learning and online learning can be reduced.

Now, referring to FIG. 1, the processing performed in a multi-layered neural network will be described. FIG. 1 is a diagram showing a convolutional neural network (CNN). However, this embodiment is not limited to this, and for example, a residual network can be adopted for the CNN, or a GAN (generative adversarial network) or the like may be used. Note that in FIG. 1, for simplicity, only one training image 201 is drawn to be input, but in reality, an output image is generated for each of the multiple selected training images. The training image 201 is an image in which RAW images are arranged in a three-dimensional direction for each color component.

Figure 6 is an explanatory diagram of the color components of an image. In this embodiment, the training image is a Bayer array image (RAW image) as shown in Figure 6 (A). Here, RGB represents red, green, and blue, respectively. Figure 6 (B) shows a configuration in which only the components of each color are rearranged from the Bayer array of Figure 6 (A). Since there are two types of G, G1 and G2, each is extracted and arranged. The four-channel image in which the four images of Figure 6 (B) are arranged in a three-dimensional direction is the training image 201 in Figure 1. This work is not necessarily necessary, but since aberration and diffraction change depending on the wavelength, it is easier to correct by arranging color components with the same blur. In addition, if RGB are arranged in the same dimension, pixels with locally different brightness are mixed, so the estimation accuracy is likely to decrease. For this reason, it is preferable to separate the training image for each color component. Note that although the case of a Bayer array is shown here, the same applies to other arrays (such as a honeycomb structure). In FIG. 1, to simplify the drawing, the training image 201 is a 4x4 four-channel image, but the horizontal and vertical image sizes are not limited to this.

In this embodiment, the training images and the correct answer image each have a plurality of color components arranged periodically, and a generation step may be provided for generating color component images consisting of only each color component of the training images or the correct answer image. Here, the step of generating color component images is performed on the training images before inputting them into the neural network, and on the correct answer image before calculating the error.

CNN has a multi-layer structure, and linear and nonlinear transformations are performed in each layer. Linear transformations are expressed as the sum of the input image (or feature map), the convolution of the filter, and the bias (bias in Figure 1). The network parameters (filter weights and biases) in each layer are updated through a learning process. Nonlinear transformations are transformations using a nonlinear function called an activation function (AF in Figure 1). Examples of activation functions include the sigmoid function and the hyperbolic tangent function, and in this embodiment, ReLU (Rectified Linear Unit) expressed by the following formula (1) is used.

In formula (1), max represents the MAX function that outputs the maximum value among the arguments.

The training image 201 input to the input layer is convolved with each of the multiple filters 202 in the first convolution layer, and summed with the bias. The number of channels of each filter 202 matches the training image 201, and if the number of channels of the training image 201 is two or more, it becomes a three-dimensional filter (the third dimension represents the number of channels). The length and width of the filter are arbitrary. The result of the convolution and sum is nonlinearly transformed by an activation function, and the first feature map 203 is output to the first hidden layer. Here, the number of channels (the number of arrays in the three-dimensional direction) of the first feature map 203 is the same as the number of filters 202. Next, the first feature map 203 is input to the second convolution layer, and convolution with each of the multiple filters 204 and summed with the bias are taken as described above. The result is nonlinearly transformed, and the same process is repeated for the number of convolution layers. In general, a CNN with three or more convolution layers corresponds to deep learning. The result output from the final convolutional layer is the CNN output image 211. Note that in the final convolutional layer, it is not necessary to perform nonlinear transformation using an activation function.

Next, in step S104, the generation unit 101e corrects the output image 211 and the correct image 221 using the information on white balance (white balance coefficients) acquired in step S102. Here, the white balance coefficients of R, G, and B are _Wr , _Wg , and _Wb , respectively, the images before adjustment are _Ir0 , _Ig0 , and _Ib0 , respectively, and the images after adjustment are _Ir , _Ig , and _Ib , respectively. At this time, the images _Ir , _Ig , and _Ib after adjustment using the white balance coefficients are expressed by equations (2) to (4), respectively.

Note that instead of directly multiplying each color by the white balance coefficient as in equations (2) to (4), it is also possible to standardize and then multiply the coefficients. In that case, for example, if standardizing by the G coefficient, the R and B coefficients can be divided by the G coefficient and multiplied by the R and B images respectively. Also, if G is divided into two, G1 and G2, as in Figure 6, it is possible to multiply each by its own white balance coefficient, or to calculate the average value of G1 and G2 and multiply the G image by the average white balance coefficient. Note that optical black contained in a RAW image does not depend on the color components, so when optical black is taken into consideration, the optical black value can be subtracted from each image before carrying out the calculations of equations (2) to (4) and then added after the calculations.

Next, clipping is performed as necessary on the output image 211 and correct image 221 to which the white balance coefficient has been applied. In this embodiment, clipping is a process of replacing a luminance value equal to or greater than a specified upper limit value with the upper limit value. If there is a pixel that has reached a luminance saturation value (the upper limit value that a pixel can take) in the output image 211 or correct image 221 before adjustment using the white balance coefficient, clipping is performed. For example, when all colors are saturated with luminance, pixels that had the same luminance value in RGB before adjustment will be colored in the opposite direction by the amount of adjustment when the white balance coefficient is applied. Therefore, as a countermeasure, clipping is performed at the luminance saturation value on the output image 211 and correct image 221 after adjustment using the white balance coefficient. Note that this clipping does not affect pixels other than the luminance saturation part, so it may be performed on all pixels regardless of the presence or absence of luminance saturation, or the process may be branched depending on the presence or absence of luminance saturation. In addition, taking into account the variation in luminance saturation for each pixel, clipping may be performed by setting a value slightly lower than the luminance saturation value. Also, this process is unnecessary if brightness saturation has not been reached, so it does not necessarily have to be performed.

Next, in step S105, the calculation unit 101c calculates the difference (error) between the output image 211 after adjustment using the white balance coefficient and the correct image 221. In this embodiment, gamma correction is performed on the output image 211 and the correct image 221, and then the error is calculated. Gamma correction is, for example, a process of raising an input luminance value to a power, and 1/2.2 or the like is used as the exponent. The correct image 221 is arranged for each color component and stacked in the channel direction, similar to the training image 201. In this embodiment, the calculation unit 101c calculates the error L using the following formula (5).

In formula (5), t is the luminance value of the correct image 221, y is the luminance value of the output image 211, j is the pixel number, N is the total number of pixels, and g is gamma correction. Formula (5) uses the Euclidean norm, but other indices may be used as long as they are values that represent the difference between the correct image and the output image. Note that in this embodiment, gamma correction is performed on the output image 211 and the correct image 221 before the error is calculated, but this process is not essential, and the error may be calculated without performing gamma correction.

Next, in step S106, the update unit 101d calculates the update amount of the network parameters from the error calculated in step S105, and updates the network parameters. Here, the backpropagation method is used. In the backpropagation method, the update amount is calculated based on the derivative of the error. However, this embodiment is not limited to this.

Next, in step S107, the update unit 101d determines whether a predetermined termination condition is met, i.e., whether optimization of the network parameters is completed. Here, the predetermined termination condition is, for example, when the learning process reaches a predetermined time, when the number of parameter updates reaches a predetermined number, when training images and correct images that are not used for parameter update are prepared and the error between the output image and the correct image becomes equal to or less than a predetermined value, etc. Alternatively, the user may instruct the end of optimization. If the predetermined termination condition is not met, the process returns to step S103, and the update unit 101d obtains a new mini-batch and updates the network parameters. On the other hand, if the predetermined termination condition is met, the process proceeds to step S108.

In step S108, the update unit 101d outputs the updated network parameters to the storage unit 101a for storage. In this embodiment, in order to learn the network parameters for each different learning condition information (information related to white balance), the network parameters and the corresponding learning condition information are stored together in the storage unit 101a. Through the above learning process, it is possible to obtain a multi-layered neural network that reduces the variation in estimation accuracy for each color.

In this embodiment, as shown in FIG. 1, the output image 211 and the correct image 221 are adjusted using a white balance coefficient, and gamma correction is then performed on each adjusted image, but this order is not essential. For example, adjustment using a white balance coefficient can be performed after gamma correction. In this case, adjustment processing using a white balance coefficient is performed on the output image 211 and the correct image 221 after nonlinear conversion using gamma correction. Note that gamma correction is processing in which a curve (gamma curve) showing the relationship between luminance values before and after correction is at a position equal to or greater than a straight line with a slope of 1 (the dashed line in FIG. 7), as shown in FIG. 7. FIG. 7 is an explanatory diagram of gamma correction. In FIG. 7, the horizontal axis indicates the luminance value before gamma correction, and the vertical axis indicates the luminance value after gamma correction.

When gamma correction is performed first and then adjustment using the white balance coefficient is performed, a network parameter different from that generated when gamma correction is performed later can be generated. When performing after gamma correction, adjustment may be performed on the image as shown in formulas (2) to (4), or adjustment may be performed on the error between the output image 211 and the correct image 221. When adjusting for the error L, the error L is expressed by the following formula (6), where the white balance coefficients of _R , G, and _B are Wr, _Wg , and Wb, the correct image 221 is _tr , _tg , and _tb , and the output image 211 is _yr , _yg , and _yb .

In this way, the error can be calculated for each color, and the calculated errors can be adjusted using the white balance coefficient and then summed up. This type of processing can also perform learning that reduces the variation in estimation accuracy for each color. Also, when G is divided into two, G1 and G2, as in Figure 6, the white balance coefficients of each can be multiplied, or the average value of G1 and G2 can be calculated and the image of G can be multiplied by the average white balance coefficient.

Next, the estimation process executed by the image estimation device 103 will be described with reference to FIG. 8. FIG. 8 is a flowchart of the estimation process.

First, in step S201, the acquisition unit 103b acquires a captured image from the imaging device 102 or the recording medium 105. The captured image is an undeveloped RAW image. If the luminance value of the RAW image is encoded, the estimation unit 103c executes a decoding process. The acquisition unit 103b also acquires learning condition information from the imaging device 102 or the recording medium 105. Note that the learning condition information in step S201 is a parameter used to select network parameters during learning, and therefore does not necessarily have to be "auto white balance setting" even if the captured image is set to auto white balance. In addition, the learning condition information may be freely selected by the user, or the imaging device 102 may automatically determine the learning condition information according to the shooting scene.

Next, in step S202, the estimation unit 103c acquires network parameters corresponding to the learning condition information acquired in step S201. The network parameters are read from the memory unit 101a of the learning device 101. Alternatively, multiple network parameters may be stored in the memory unit 103a of the image estimation device 103 and read from the memory unit 103a. The acquired network parameters are those in which the learning condition information acquired in step S301 matches or is closest to the learning condition information used in the learning process.

Next, in step S203, the estimation unit 103c acquires an input image to be input to the CNN from the captured image. The input image is arranged by color component and stacked in three dimensions, similar to the training image. Note that the size of the input image in the estimation process does not necessarily have to match the size of the training image in the learning process.

Next, in step S204, the estimation unit 103c generates an estimated image based on the input image and the network parameters. To generate the estimated image, the CNN shown in FIG. 1 is used, as in the learning process. However, the output image 211 in FIG. 1 becomes the estimated image, and no further processing such as error calculation with the correct image is performed.

Next, in step S205, the estimation unit 103c determines whether estimation has been completed for a predetermined region of the captured image. If estimation has not been completed, the process returns to step S203, where the estimation unit 103c acquires a new input image from the predetermined region of the captured image. In the CNN used for estimation, if the size of the output image is smaller than the input image, it is necessary to acquire the input image by overlapping from the predetermined region. The predetermined region is the entire captured image or a part of it. Since the captured image is a RAW image, in addition to the image obtained by receiving light, header information (information such as the number of pixels of the image and the shooting time) and optical black information of the image sensor may be included. Since the header information and optical black are unrelated to aberration and diffraction blur, they may be excluded from the predetermined region.

Next, in step S206, the estimation unit 103c combines the generated estimated images to output a captured image in which blurring due to aberration and diffraction has been corrected. If necessary, the estimation unit 103c outputs the captured image including header information and optical black information.

The above estimation process allows estimation to be performed using network parameters with little variation in estimation accuracy for each color. As a result, the correction effect of blur due to aberration and diffraction does not vary in estimation accuracy depending on the color, and more accurate correction can be achieved. In addition, after the estimation process, the user can arbitrarily perform editing such as exposure correction, and the final developed image is obtained by development processing. In this embodiment, a method of performing correction by switching network parameters according to learning condition information has been described, but multiple network parameters may be obtained and multiple output images may be generated by inputting an input image to each network. In this way, multiple output images with different learning condition information can be generated, and for example, an output image of intermediate learning condition information can be generated by interpolating them. For example, when the learning condition information is a color temperature K3000 and a color temperature K6000, estimated images can be generated using network parameters corresponding to each, and an estimated image equivalent to a color temperature K5000 can be output by interpolating these. Conversely, only one learning condition information may be used, and only specific network parameters may be stored in the imaging device 102 or the recording medium 105.

In this embodiment, we have described the correction of blur caused by aberration and diffraction, but the same effect can be obtained with other methods such as upsampling and denoising by adjusting the white balance coefficient using the corresponding training images and ground truth images.

Next, an image processing system according to a second embodiment of the present invention will be described.
Fig. 9 is a block diagram of an image processing system 300 in this embodiment. Fig. 10 is an external view of the image processing system 300. The image processing system 300 includes a learning device 301 and an imaging device 302 connected via a network 303.

The learning device 301 has a memory unit 311, an acquisition unit 312, a calculation unit 313, an update unit 314, and a generation unit 315, and learns network parameters for correcting blurring due to aberration and diffraction using a neural network.

The imaging device 302 captures an image of a subject space, and corrects blurring due to aberration and diffraction in the captured image using the read network parameters. The imaging device 302 has an optical system 321 and an imaging element 322. The image estimation unit 323 has an acquisition unit 323a and an estimation unit 323b, and performs correction of the captured image using the network parameters stored in the storage unit 324. The network parameters are learned in advance by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads the network parameters from the storage unit 311 via the network 303 and stores them in the storage unit 324. The captured image (output image) in which blurring due to aberration and diffraction has been corrected is stored in the recording medium 325. When an instruction regarding display of the output image is issued by the user, the stored output image is read and displayed on the display unit 326. Note that the captured image already stored in the recording medium 325 may be read, and blurring correction may be performed by the image estimation unit 323. The above series of controls are performed by the system controller 327.

Next, referring to FIG. 11, the processing performed by the multi-layered neural network in this embodiment will be described. FIG. 11 is a diagram showing a convolutional neural network in this embodiment. FIG. 11 differs from FIG. 1 in the embodiment 1 in the adjustment method using the white balance coefficient for the training image 401. In this embodiment, as shown in FIG. 11, the training image 401 is first adjusted using the white balance coefficient, and then input to the neural network. Then, gamma correction is performed on the output image 411. The correct image 421 is the same as in the embodiment 1, and the correct image 421 is adjusted using the white balance coefficient, and then gamma correction is performed. Then, the difference (error) between the output image 411 after gamma correction and the correct image 421 is calculated. Note that clipping processing after white balance is performed as necessary. Also, the filter 402, the first feature map 403, and the filter 404 are the same as the filter 202, the first feature map 203, and the filter 204 in FIG. 1, respectively, and therefore their description will be omitted.

In this embodiment, the learning process is executed by the learning device 301, and the estimation process is executed by the image estimation unit 323. Note that the estimation process in this embodiment is the same process as that shown in the flowchart in FIG. 8 of the first embodiment, and therefore will not be described here.

Next, with reference to FIG. 12, a method for learning network parameters (learning process, method for manufacturing a trained model) executed by the learning device 301 in this embodiment will be described. FIG. 12 is a flowchart related to learning network parameters (learning process). Each step in FIG. 12 is mainly executed by the acquisition unit 312, calculation unit 313, update unit 314, and generation unit 315 of the learning device 301. Note that steps S301 and S302 in FIG. 12 are similar processes to steps S101 and S102 in FIG. 5, respectively, and therefore descriptions thereof will be omitted.

Next, in step S303, the generation unit 315 corrects the training image 401 and the correct answer image 421 using the white balance coefficient acquired in step S302. The adjustment using the white balance coefficient in this embodiment is performed using equations (2) to (4) as in the first embodiment. Next, the generation unit 315 performs clipping processing as necessary on the training image 401 and the correct answer image 421 to which the white balance coefficient has been applied. In the clipping processing, if there is a pixel that has reached the brightness saturation value in the training image 401 or the correct answer image 421 before the adjustment using the white balance coefficient is performed, it is replaced with the brightness saturation value. The brightness saturation value may be different for each color, or may be the same value.

Next, in step S304, the generation unit 315 inputs the training image 401 adjusted in step S303 to the neural network to generate an output image 411. In this embodiment, mini-batch learning is performed as in the first embodiment, but batch learning or online learning may also be performed. In addition, ReLU is used as the activation function in this embodiment, but a sigmoid function or a hyperbolic tangent function may also be used.

The steps from step S305 onwards are the same as those from step S105 onwards in the first embodiment, and so a description thereof will be omitted. The above is the learning process executed in this embodiment. In this way, a white balance adjustment may be performed on the training image 401 before it is input to the neural network, and network parameters that can reduce the variation in estimation accuracy for each color can be generated, as in the first embodiment.

In this embodiment, the adjustment process using the white balance coefficients is performed on the training image 401 using equations (2) to (4), but other methods may be used. For example, instead of using equations (2) to (4), information on the white balance coefficients may be input to the neural network together with the training image 401. In this case, the white balance coefficients are mapped so that they can be input to the neural network. For example, if the training image 401 has four channels of RG1G2B, the map of the white balance coefficients (WB map) is also four channels of RG1G2B, and the number of elements (number of pixels) per channel is set to be equal to that of the training image 401. Then, the training image 401 and the WB map are linked in a specified order in the channel direction. The linked training image 401 and the WB map are input to the neural network as input data to obtain the output image 411. In this case, since the white balance coefficients are input to the neural network, adjustment processes using the white balance coefficients are not required for the output image 411 and the correct answer image 421. Then, gamma correction is performed on the output image 411 and the correct image 421, and an error is calculated using the gamma-corrected output image 411 and the correct image 421. At this time, if the estimation accuracy of high luminance areas is prioritized over low luminance areas, gamma correction does not need to be performed.

In this way, the white balance coefficient can be used by inputting it into the neural network together with the training image 401, and this method can also generate network parameters that suppress the variation in estimation accuracy for each color. When performing estimation processing using this network parameter, the captured image and the white balance coefficient can be similarly linked in the channel direction and input to the neural network to generate an estimated image with less variation in estimation accuracy for each color. In addition, although the case where the training image 401 and the WB map are linked and input has been described, the input method to the neural network is not limited to this. Only one of the training image 401 or the WB map may be input to the first layer of the neural network, and the feature map that is the output after passing through the first layer or several layers and the other one that was not input to the first layer may be linked in the channel direction and input to the subsequent layer of the neural network. Also, the input part of the neural network may be branched, and the training image 401 and the WB map may be converted into feature maps in different layers, and these feature maps may be linked and input to the subsequent layer. This method can also generate network parameters that suppress variation in estimation accuracy for each color.

As described above, the image processing method of each embodiment includes an acquisition step of acquiring information related to white balance, and an update step of updating the parameters of the neural network. In the update step, a training image is input to the neural network to generate an output image so that the learning result is adjusted based on the information related to white balance, and the parameters of the neural network are updated based on the difference between the output image and the correct image.

Preferably, in the update process, white balance adjustment is performed on the master image and the output image using information related to white balance, and the difference is calculated using the master image and the output image after the white balance adjustment. More preferably, in the update process, gamma correction is performed on the master image and the output image after the white balance adjustment.

Preferably, in the update process, the difference between the training image or output image and the correct image for each color component is calculated, and the difference between the output image and the correct image is calculated by weighting the difference for each color component based on information related to white balance and adding them up.

Preferably, in the update process, white balance adjustment is performed on the training image and the correct answer image using information related to white balance, and the difference is calculated using the training image and the correct answer image after the white balance adjustment. More preferably, in the update process, the training image after the white balance adjustment is input to a neural network to generate an output image, and gamma correction is performed on the correct answer image and the output image after the white balance adjustment.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing program, a storage medium, and a method for manufacturing a trained model that can obtain a neural network that reduces the variation in estimation accuracy for each color.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

101: Learning device (image processing device)
101b: Acquisition unit 101d: Update unit

Claims (15)

An acquisition step of acquiring a training image, a ground truth image, and information regarding white balance;
A generating step of generating a color component image composed only of color components corresponding to the plurality of color components of the training image;
an update step of inputting the color component image into a neural network to generate an output image composed only of color components corresponding to the plurality of color components, obtaining a first error for each color component between the output image and the correct image, obtaining a second error based on the first error and information related to the white balance, and updating parameters of the neural network based on the second error . An acquisition step of acquiring a training image, a ground truth image, and information regarding white balance;
A generating step of generating a color component image composed only of color components corresponding to the plurality of color components of the training image;
an update step of inputting the color component image into a neural network to generate a first output image composed only of each color component corresponding to the plurality of color components, generating a second output image for each color component based on the first output image and information related to the white balance, and updating parameters of the neural network based on an error between the second output image and the correct image. The image processing method according to claim 1, characterized in that the training images and the ground truth images each have a plurality of color components arranged periodically. 3. The image processing method according to claim 2, wherein in the updating step , gamma correction is performed on the correct image and the second output image. 5. The image processing method according to claim 2, wherein in the updating step , a clipping process is performed within a predetermined range on the luminance values of at least one of the original image and the second output image. 6. The image processing method according to claim 1, wherein the information regarding the white balance is acquired from additional information of the training image or the correct image. 7. The image processing method according to claim 1, wherein the information regarding white balance is a white balance coefficient. 8. An image processing method comprising the step of generating an estimated image by inputting an input image to a neural network having the parameters updated using the image processing method according to claim 1 . 9. The image processing method according to claim 8, wherein the estimated image is an image generated by subjecting the input image to at least one of upsampling, denoising, removal of compression noise, deblurring, inpainting, demosaicing, dehazing, high gradation, and relighting. an acquisition unit that acquires training images, a correct answer image, and information regarding white balance;
a generating unit for generating a color component image composed only of color components corresponding to the plurality of color components of the training image;
an update unit that generates an output image composed only of color components corresponding to the plurality of color components by inputting the color component image into a neural network, obtains a first error for each color component between the output image and the correct image, obtains a second error based on the first error and information related to white balance, and updates parameters of the neural network based on the second error . an acquisition unit that acquires training images, a correct answer image, and information regarding white balance;
a generating unit for generating a color component image composed only of color components corresponding to the plurality of color components of the training image;
an update unit that generates a first output image composed only of each color component corresponding to the plurality of color components by inputting the color component image into a neural network, generates a second output image of each color component based on the first output image and information related to the white balance, and updates parameters of the neural network based on an error between the second output image and the correct image. 10. An image processing program for causing a computer to execute the image processing method according to claim 1. A storage medium storing the image processing program according to claim 12 . An acquisition step of acquiring a training image, a ground truth image, and information regarding white balance;
A generating step of generating a color component image composed only of color components corresponding to the plurality of color components of the training image;
a first error for each color component between the output image and the correct image, a second error based on the first error and information related to the white balance, and an updating step for updating parameters of the neural network based on the second error . An acquisition step of acquiring a training image, a ground truth image, and information regarding white balance;
A generating step of generating a color component image composed only of color components corresponding to the plurality of color components of the training image;
a first output image composed only of each color component corresponding to the plurality of color components by inputting the color component image into a neural network, a second output image for each color component based on the first output image and information relating to the white balance, and an update step of updating parameters of the neural network based on an error between the second output image and the correct image.