JP2023179838A - Image processing method, image processor, image processing system, and image processing program - Google Patents

Abstract

To provide an image processing method that highly accurately increases image resolution and reduces noise while suppressing a training load of a machine learning model.SOLUTION: An image processing method comprises steps of: generating a first noise reduction output; generating a high-resolution output based on first input data according to the first noise reduction output using a first machine learning model; generating a second noise reduction output based on second input data according to the high-resolution output using a second machine learning model; and generating an output image based on the second noise reduction output. The second input data is an image that has a larger difference from a second addition image than difference from a first addition image. The first addition image is an image obtained by adding the image and a high-resolution residual map based on the high-resolution output. The second addition image is an image obtained by adding the high-resolution residual map and the noise-reduced image based on the first noise reduction output.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing method for performing high resolution and noise reduction on a captured image.

Patent Document 1 discloses a method of performing noise reduction and upscaling of an image using one neural network by using weights corresponding to a noise reduction level specified by a user.

US Patent No. 10552944

However, in the method of Patent Document 1, the number of data sets becomes enormous, and the training load of the neural network increases.

When high-resolution and noise reduction are performed using separate machine learning models, the image quality of the final image deteriorates due to the adverse effects of both processes. When high resolution is performed first, noise is amplified, and even if noise reduction is performed, the amplified noise remains. If noise reduction is performed first, the blurred image changes, resulting in over-correction or under-correction due to higher resolution.

An object of the present invention is to provide an image processing method that highly accurately increases the resolution of a captured image and reduces noise while suppressing the training load of a machine learning model.

An image processing method as one aspect of the present invention includes the steps of: generating an output related to first noise reduction using a captured image; and generating an output related to first noise reduction using a first machine learning model. generating an output related to high resolution based on the first input data; and a second noise reduction based on the second input data according to the output related to high resolution using a second machine learning model. and generating an output image based on the output regarding the second noise reduction, wherein the second input data is larger than the difference between the second summation image and the first summation image. The first addition image is an image that has a large difference from the image, and the first addition image is an image obtained by adding the high-resolution residual map based on the output regarding high-resolution and the captured image. The second added image is characterized in that it is an image obtained by adding the high-resolution residual map and the captured image with noise reduced based on the output related to the first noise reduction.

According to the present invention, it is possible to provide an image processing method that can highly accurately improve the resolution and reduce noise of a captured image while suppressing the training load of a machine learning model.

1 is an external view of an image processing system of Example 1. FIG. 1 is a block diagram of an image processing system according to a first embodiment; FIG. 3 is a flowchart showing training of a machine learning model. 5 is a flowchart illustrating generation of an output image in Example 1. FIG. FIG. 2 is an external view of an image processing system according to a second embodiment. FIG. 2 is a block diagram of an image processing system according to a second embodiment. 7 is a flowchart showing generation of an output image in Example 2. FIG.

Embodiments of the present invention will be described in detail below with reference to the drawings. In each figure, the same reference numerals are given to the same members, and overlapping explanations will be omitted.

Before explaining this embodiment in detail, the gist of the present invention will be briefly explained. In the present invention, high resolution and noise reduction are performed on captured images. High resolution includes blur correction or upscaling. Upscaling includes enlarging the entire captured image (increasing the number of pixels) and enlarging a portion of the captured image (digital zoom, etc.). Noise reduction includes reduction of noise generated in an image sensor and reduction of compression noise when an image is irreversibly compressed (JPEG compression, etc.).

When training a neural network that only performs noise reduction, the dataset only needs to cover noise variations. The same applies when only upscaling is performed. However, if one neural network performs both noise reduction and upscaling, the data set must include both noise and resolution degradation, and the variation increases with the combination. As a result, the number of data sets becomes enormous, increasing the training load.

Therefore, in the present invention, high resolution and noise reduction of a captured image are performed using individual machine learning models. This makes it possible to reduce the number of datasets used for training, thereby reducing the training load of the machine learning model.

Further, in the present invention, high resolution is performed on the captured image after noise reduction, and a high resolution image without noise reduction is generated. For example, a high-resolution image without noise reduction is generated by adding the obtained high-resolution residual map to a captured image without noise reduction. Noise reduction is performed again on the high-resolution image that has not been subjected to noise reduction, and a final output image is generated. With this configuration, it is possible to prevent the image quality of the final output image from deteriorating due to the adverse effects of either noise reduction or high resolution. The above processing will be explained below.

The key point is that the estimation of noise reduction largely depends on the signal distribution of the input image, and the estimation of high resolution has little dependence on the signal distribution of the input image. To simplify the explanation, noise reduction targets Gaussian noise with a specific variance, and resolution enhancement targets blur expressed by a Gaussian distribution function with a specific variance. At this time, in the case of high resolution, since the same blur always acts on the input image, similar corrections are performed on various signal distributions. However, in noise reduction, only the variance of the noise in the input image is the same, and the distribution of the noise is completely unknown. Therefore, in noise reduction, it is necessary to distinguish between a subject and noise based on the signal distribution of the input image, and the estimation result is greatly influenced by the signal distribution of the input image.

If noise reduction is performed on a captured image after increasing the resolution, some strong noises (or all noises) will be emphasized due to the adverse effect of increasing the resolution. Since the emphasized noise is larger than the specific variance that noise reduction is based on, it is regarded as an object rather than noise, and remains without noise reduction.

Conversely, when performing high resolution on a captured image after noise reduction, a part of the blurred image is deformed due to the adverse effects of noise reduction. However, as described above, the high-resolution estimation has little dependence on the signal distribution of the input image, so even if the blurred image changes, the high-resolution correction does not change significantly. Therefore, if the blurred image becomes wider due to noise reduction, the correction will be insufficient, and if the blurred image becomes smaller, the correction will be overcorrected.

Therefore, in the present invention, for example, by adding a high-resolution residual map obtained for a captured image after noise reduction to a captured image that has not undergone noise reduction, Generate an image. The residual map is a component obtained by subtracting the captured image for which only noise reduction has been performed from the captured image for which noise reduction and resolution enhancement have been performed. Since high resolution is performed on the captured image after noise reduction, the noise amplification component included in the high resolution residual map becomes small. In addition, since high-resolution estimation has little dependence on the signal distribution of the input image, the high-resolution residual map obtained for the captured image after noise reduction is added to the captured image. However, the decrease in correction accuracy is small. Thereby, it is possible to suppress excessive or insufficient resolution and noise amplification, and to generate a high-resolution captured image. In a high-resolution captured image, the resolution performance and contrast of the subject are improved by increasing the resolution of the blurred image, so it is easier to distinguish between the subject and noise. Therefore, by performing noise reduction again on the high-resolution captured image, it is possible to suppress deformation of the subject and reduce noise.

For the above reasons, it is possible to highly accurately improve the resolution of a captured image and reduce noise while suppressing the training load of a machine learning model.

FIG. 1 is an external view of an image processing system 100 of this embodiment. FIG. 2 is a block diagram of the image processing system 100 of this embodiment. The image processing system 100 includes a training device 101, an image processing device 102, and an imaging device 103, which are connected to each other via a wired or wireless network.

The imaging device 103 includes an imaging optical system 131, an image sensor 132, a storage section 133, and a communication section 134. The imaging optical system 131 forms an image of the subject from light in the subject space. The image sensor 132 includes a plurality of pixels and converts an image of a subject into a captured image by photoelectric conversion. Since the image of the subject is blurred due to aberrations and diffraction caused by the imaging optical system 131, the captured image is also degraded due to the blurring. In this embodiment, the above-mentioned blur is corrected to improve the resolution. Further, shot noise, dark current noise, read noise, and the like occur in the image sensor 132, so that noise is present in the captured image. In this embodiment, the above-mentioned noise is reduced as noise reduction. The storage unit 133 stores captured images. In this embodiment, blur correction is performed to increase resolution, but upscaling or the like may also be performed. Further, in this embodiment, the target of noise reduction is the noise generated in the image sensor, but it may be compressed noise or the like.

The image processing device 102 includes a storage section 121, a communication section 122, an acquisition section 123, a noise reduction section (first generation section, third generation section) 124, a high resolution section (second generation section) 125, and a calculation section ( output section) 126, and a display section 127. The communication unit 122 acquires a captured image via the communication unit 134 of the imaging device 103. Note that the image processing device 102 may acquire the captured image by recording the captured image on a recording medium and connecting the recording medium to the image processing device 102. The image processing device 102 generates an output image by performing high resolution and noise reduction on the captured image using the first machine learning model and the second machine learning model. The first and second machine learning models use a set of weights generated in advance by the training device 101. The image processing device 102 acquires a set of weights from the training device 101 via the communication unit 122 and stores it in the storage unit 121. The generated output image is presented to the user via the display unit 127 or stored in the storage unit 121.

The training device 101 includes a storage section 111, an acquisition section 112, a calculation section 113, and an update section 114. The training device 101 uses the data set to train a first machine learning model that performs high resolution, and generates a trained first set of weights. The training device 101 also uses the data set to train a second machine learning model that performs noise reduction, and generates a trained second set of weights.

Hereinafter, with reference to FIG. 3, training of a machine learning model (determination of a set of weights) executed by the training device 101 will be described. FIG. 3 is a flowchart illustrating training of a machine learning model. In this embodiment, a CNN (Convolutional Neural Network) is used as a machine learning model. However, the present invention is not limited to this. Machine learning models include neural networks, genetic programming, Bayesian networks, and the like. Neural networks include CNNs, GANs (Generative Adversarial Networks), RNNs (Recurrent Neural Networks), and the like.

First, training of the second machine learning model that performs noise reduction will be explained.

In step S101, the acquisition unit 112 acquires one or more sets of correct images and training images from the storage unit 111. The second machine learning model wants to obtain the effect of noise reduction through training, so the training image is an image with noise, the correct image is an image of the same scene as the training image, and the noise is smaller than the training image (or (no) image. The storage unit 111 stores a data set including a plurality of correct images and training images. The noise present in the training images is similar to the noise generated by the image sensor 132. In order for the first machine learning model to be able to handle captured images of various subjects, the multiple ground truth images and training images used for training are of various subjects (edges with different orientations and strengths, textures, gradations, and flat areas). etc.) is desirable.

The correct image and training image used in this embodiment are images generated by adding noise to the original image. However, the same scene may be actually photographed using the image sensor 132 and a condition with a better S/N ratio, and may be used as the training image and the correct image. The original image is an undeveloped RAW image (light intensity and signal value have a linear relationship), and a training image is generated by adding noise generated by the image sensor 132 to this image. If the image sensor 132 can have multiple types and ISO sensitivities, the dataset includes training images of noises of various intensities generated by the image sensor 132. The correct image may be the original image as it is, or may be an image added with noise smaller than the noise added to the training image. It is desirable that the added noise be noise that is correlated with the training images. When uncorrelated noise is added, training with multiple images in the dataset averages out the influence of noise on the correct image, resulting in almost no difference from the case where no noise is added. The original image may be a real RAW image or CG (Computer Graphics). Noise already exists in real RAW images, but in training a machine learning model, that noise is treated as an object, so there is no particular problem. In this embodiment, the second machine learning model performs noise reduction on the RAW image. Therefore, the correct image and the training image are RAW images. When it is desired to perform noise reduction on a developed image, it is sufficient to develop the correct image and the training image.

In step S102, the calculation unit 113 uses the second machine learning model to generate output (information) regarding second noise reduction based on the training image. Specifically, the training image is input to the second machine learning model to generate an output related to the second noise reduction. The output related to the second noise reduction includes a noise-reduced training image, a noise-reduction residual map for the training image (second noise-reduction residual map), and the like. The training image becomes a noise-reduced image when the noise-reduction residual map is summed. In this embodiment, the output related to the second noise reduction is a noise-reduced training image (estimated image). The second machine learning model is CNN and has multiple weights (second set of weights). The initial value of each weight may be determined using random numbers or the like.

In step S103, the updating unit 114 updates each weight of the second machine learning model based on the difference between the correct image and the estimated image. In this embodiment, the mean squared error between the correct image and the estimated image is used as a loss function, and the weights are updated using error backpropagation. However, the present invention is not limited to this. By performing optimization so that the difference between the correct image and the estimated image becomes small, the second machine learning model acquires the effect of noise reduction on the input image. When the output related to the second noise reduction is a residual map, the difference between the training image and the correct image may be minimized.

In step S104, the update unit 114 determines whether training of the second machine learning model is completed. If it is determined that the training is not completed (incomplete), the process returns to step S101, and a new set of one or more correct images and training images is acquired. If it is determined that the training has been completed, information on the second set of trained weights is stored in the storage unit 111.

Next, training of the first machine learning model that performs high resolution will be explained. Descriptions of parts similar to those for training the second machine learning model will be omitted.

In step S101, the training image is an image in which blur has occurred due to aberrations and diffraction caused by the imaging optical system 131. The correct image is an image of the same scene as the training image, and is an image with less blur (or no blur) than the training image. In this embodiment, a training image and a correct image are generated by adding blur to the original image. A training image is generated by adding blur due to aberrations and diffraction generated in the imaging optical system 131 to the original image. If necessary, blurring may be provided by an optical low-pass filter, a pixel aperture of the image sensor 132, or the like. If there are multiple types and conditions (focal length, aperture value, focus distance, etc.) of the imaging optical system 131, and different blurs may be applied to the captured image depending on the types and conditions, the plurality of blurs may be added to the data set. Include training images. Blur can vary depending on the position of each pixel of the image sensor 132 (image height and azimuth with respect to the optical axis of the imaging optical system 131), and can also vary depending on various states of the imaging optical system 131 (focal length, aperture value, and focus). (distance, etc.), it also changes depending on the state. Furthermore, when the imaging optical system 131 can take on a plurality of types, such as an interchangeable lens, the blur changes depending on the type. Note that the blur imparted to the original image may be the blur itself generated in the imaging optical system 131, or may be a blur that approximates the blur. The correct image is generated by adding no blur to the original image or adding a smaller blur than the blur added to the training image.

In this embodiment, noise is not added to the training images and correct images used for training the first machine learning model. In the present invention, the purpose is to increase the resolution of a captured image with reduced noise. However, correlated noise may be added to both. When correlated noise is added, the first machine learning model can correct blur and simultaneously obtain the effect of suppressing noise changes through training.

In this embodiment, the first machine learning model performs blur correction on the RAW image. Therefore, the correct image and the training image are RAW images. If you want to perform blur correction on a developed image, you can develop the correct image and the training image.

In step S102, the calculation unit 113 uses the first machine learning model to generate output (information) regarding high resolution based on the training image. Specifically, by inputting the training image to the first machine learning model, an output related to high resolution is generated. The output related to high resolution includes a high resolution (blur corrected) training image, a high resolution residual map for the training image, and the like. A training image becomes a high-resolution image when a high-resolution residual map is added to the training image. In this embodiment, the output related to high resolution is a high-resolution training image (estimated image). The first machine learning model is CNN and has multiple weights (first set of weights).

In step S103, the updating unit 114 updates each weight of the first machine learning model based on the difference between the correct image and the estimated image.

In step S104, the update unit 114 determines whether training of the first machine learning model is completed. If it is determined that the training has not been completed (incomplete), the process returns to step S101, and if it is determined that the training has been completed, information on the trained first set of weights is stored in the storage unit 111. do.

Note that in this embodiment, the first machine learning model and the second machine learning model are trained independently. When processing a captured image, the first machine learning model is used to increase the resolution, and then the second machine learning model is used to perform noise reduction. Another possibility is to train a machine learning model. However, when training with a correlation between the two, it becomes necessary to perform calculations using the first machine learning model to generate training images, and when a change occurs in the first machine learning model, the second machine learning model Machine learning models also have to be retrained, which increases the training load.

Hereinafter, with reference to the flowchart of FIG. 4, noise reduction and resolution enhancement of a captured image executed by the image processing apparatus 102 will be described. FIG. 4 is a flowchart showing the generation of an output image in this embodiment.

In step S201, the acquisition unit 123 acquires a captured image and a set of machine learning model weights. In this embodiment, three sets of weights are acquired. Specifically, a first set of weights used in the first machine learning model, a second set of weights used in the second machine learning model, and a third set of weights used in the third machine learning model. be. The third machine learning model performs noise reduction before increasing the resolution using the first machine learning model. In this example, the third machine learning model is the same as the second machine learning model. Therefore, the third set of weights is also the same as the second set of weights. However, the present invention is not limited to this.

In step S202, the noise reduction unit 124 uses the third machine learning model to generate output (information) regarding the first noise reduction based on the captured image. Specifically, the captured image is input to the third machine learning model to generate an output related to the first noise reduction. In this embodiment, the output related to the first noise reduction is a captured image (first image) in which noise has been reduced. Note that instead of the first image, a noise reduction residual map (first noise reduction residual map) for the captured image may be generated. The captured image becomes the first image when the noise reduction residual map is added. Therefore, the output related to the first noise reduction is a noise-reduced captured image or a noise reduction residual map for the captured image. Note that in this step, a noise reduction method that does not use a machine learning model, such as NLM (Non-local means filter) or BM3D (Block-matching and 3D filtering), may be used.

In step S203, the high resolution unit 125 uses the first machine learning model to generate an output (information) regarding high resolution based on the first input data corresponding to the output regarding the first noise reduction. do. In this embodiment, the first input data is the first image. However, the present invention is not limited to this. For example, data obtained by adding the residual map of the first noise reduction (output related to the first noise reduction) and the captured image or combining them in the channel direction may be used as the first input data. In this case, the first machine learning model is trained using data in the same format as the first input data. By inputting the first input data to the first machine learning model, an output related to high resolution is generated. In this embodiment, the output related to high resolution is a first image (second image) that has been subjected to blur correction (high resolution). Note that a high-resolution residual map for the first image may be generated instead of the second image. The first image becomes the second image when the high-resolution residual map is added. Therefore, the output related to high resolution is the high resolution first image (noise reduction and high resolution captured image) or the high resolution residual map for the first image. be.

In step S204, the calculation unit 126 uses the captured image and the first image to generate a noise reduction residual map (first noise reduction residual map) for the captured image. A first noise reduction residual map is generated by subtracting the captured image from the first image. Note that if the residual map is generated in step S202, step S204 may not be executed. Further, step S204 may be executed before step S203.

In step S205, the calculation unit 126 generates a third image using the second image and the first noise reduction residual map. The third image is an image obtained by subtracting the first noise reduction residual map from the second image, and is an image obtained by canceling the noise reduction in step S202 and performing only high resolution on the captured image. . The third image becomes the second input data for the second machine learning model. Thereby, as described before the description of this embodiment, it is possible to suppress noise amplification and generate a high-resolution captured image. In the third image, since the noise has hardly changed from the captured image, the noise can be favorably reduced by the machine learning model. Furthermore, since the blurred image has already been made to have a high resolution, it is easy to distinguish between noise and the subject, and deformation of the subject due to noise reduction can also be suppressed. The third image may be generated by adding a high-resolution residual map to the captured image. Alternatively, the third image may be generated by adding the high-resolution residual map to the first image and subtracting the first noise reduction residual map. Therefore, the second input data (third image) in this embodiment is generated based on the output related to high resolution and the output related to the captured image or first noise reduction.

Note that when generating the third image, it is not necessary to completely cancel out the effect of noise reduction in step S202, and a part of the effect may be left. For this reason, the third image is different from the captured image (first addition image) in which the effect of noise reduction is canceled and only high resolution is performed, so that high resolution is performed after noise reduction. This is an image that has a large difference from the captured image (second added image). The first added image is an image obtained by adding a high-resolution residual map (a map obtained by subtracting the first image from the second image) based on the output related to high-resolution and the captured image. It is equivalent to the image obtained by subtracting the noise reduction residual map from the image No. 2. The second added image is an image (second image) obtained by adding the high-resolution residual map and the captured image (first image) whose noise has been reduced using the output related to the first noise reduction. be. The difference between two images can be evaluated by MSE (mean square error), MAE (mean absolute error), or the like. The characteristic of the third image can also be expressed as that the noise in the third image is closer to the noise in the captured image than the noise in the first image.

The third image is generated, for example, by multiplying the first noise reduction residual map by a coefficient greater than 0.5 and less than or equal to 1, and subtracting it from the second image. Note that in step S203, if the output related to high resolution is a high resolution residual map for the first image, by adding the high resolution residual map to the captured image, the third Images can be generated.

In step S206, the noise reduction unit 124 uses the second machine learning model to generate an output regarding second noise reduction based on the second input data. Specifically, by inputting the second input data to the second machine learning model, an output related to the second noise reduction is generated. In this embodiment, the output related to the second noise reduction is a fourth image that is a captured image with noise reduction and high resolution. However, the output related to the second noise reduction may be a noise reduction residual map (second noise reduction residual map) for the third image (second input data).

In step S207, the calculation unit 126 generates an output image based on the captured image, the output related to high resolution, and the output related to second noise reduction. In this embodiment, the output related to high resolution is the third image, and the output related to the second noise reduction is the fourth image. By subtracting the fourth image from the third image, a second noise reduction residual map is generated, and the second noise reduction residual map is multiplied by a coefficient representing the intensity of noise reduction to obtain the captured image. The strength of noise reduction can be adjusted by adding . Similarly, by subtracting the captured image from the third image, a high-resolution residual map is generated, and the high-resolution residual map is multiplied by a coefficient representing the strength of the high-resolution image. By adding it to the image, the intensity of high resolution can be adjusted. The output image that has been adjusted in intensity for noise reduction and high resolution is displayed on the display unit 127, and the user can adjust the intensity for noise reduction and high resolution while checking the displayed output image. can be adjusted freely. The intensity can be adjusted by changing the coefficients representing the respective intensities of noise reduction and resolution enhancement. With the above configuration, there is no need to recalculate the machine learning model each time the intensity of noise reduction and resolution enhancement is changed, and intensity adjustment can be performed quickly with lightweight calculations. Note that if intensity adjustment is not necessary, the fourth image may be output as is as an output image.

As described above, according to the configuration of this embodiment, it is possible to highly accurately improve the resolution and reduce noise of a captured image while suppressing the training load of a machine learning model.

Desirable configurations that enhance the effects of the present invention will be described below.

It is desirable that the output related to high resolution generated in step S203 be generated based on information regarding the optical system used when acquiring the captured image. When training the first machine learning model that performs high resolution, if various types of resolution degradation are mixed in the dataset, the first machine learning model is trained to obtain an average high resolution for the degradation. Acquire image. Therefore, in order to increase the resolution in response to deterioration, the first machine learning model needs to generate an output related to the increase in resolution based on information about the optical system used to capture the captured image. There is. Information regarding the optical system includes the type of optical system used to acquire the captured image, the focal length at the time of imaging, the aperture value, the focus distance, the position of the pixel of the captured image with respect to the optical axis, and the optical system at the pixel of the captured image. This is information regarding any of the resolution performance. The types of optical systems include the type of imaging optical system 131, the type of optical low-pass filter, and the like. Once the type of optical system, focal length, aperture value, focus distance, and position of a pixel of the captured image with respect to the optical axis are specified, blurring due to aberrations and diffraction occurring in the optical system can be uniquely determined at that pixel position. It's decided. When training and executing the first machine learning model, by inputting not only the image but also information regarding the optical system, high resolution suitable for each pixel of the image can be achieved. Maps (with the same number of two-dimensional pixels as the image) each having a value representing information about the optical system in its channel component may be connected in the channel direction of the image and input into the first machine learning model. Since the type of optical system, focal length, aperture value, and focus distance do not change depending on the position of the pixel, a flat map having the same values is obtained. The position of a pixel of a captured image with respect to the optical axis becomes a map having gradations in two directions (horizontal, vertical, etc.) in different channel components. Instead of these pieces of information, information regarding the resolution performance of the optical system at each pixel of the image may be used. It is possible to generate a map representing the resolution performance at frequencies, etc., at which the blur acting on each pixel has an MTF (modulation transfer function) greater than a predetermined value. Further, information regarding resolution performance in two or more different directions (horizontal/vertical, meridional/sagittal) may be included in different channel components. Note that the information regarding the optical system may be directly acquired from the metadata of the captured image, or may be generated by calculation to obtain a desired data format based on the information of the metadata.

Further, in step S201, it is desirable to select a first weight set to be used in the first machine learning model from a plurality of first weight sets based on information regarding high resolution. Furthermore, it is desirable to select a second set of weights to be used in the second machine learning model from a plurality of second weight sets based on information regarding noise in the captured image. Rather than using a single machine learning model to reduce noise from all ISO sensitivities that can be taken by the image sensor 132 (for example, ISO 100 to 25600), machine learning that divides the ISO sensitivity range into multiple ranges and trains each range separately Reducing noise in the model will result in higher accuracy. The same applies to high resolution. When the imaging device 103 is an interchangeable lens camera, the imaging optical system 131 can be of a plurality of types. For example, accuracy can be improved by training a second machine learning model for each type of imaging optical system 131. For example, a second machine learning model can be trained separately depending on the noise strength, such as ISO100-400, 400-1600, 1600-6400, 6400-25600, and a set of four second weights. Suppose that it is generated. Similarly, it is assumed that the first machine learning model is trained individually according to the type (for example, eight types) of the imaging optical system 131, and a set of eight first weights are generated. In this case, the number of times the machine learning model is trained and the number of weight sets to be retained are each 12. However, when noise reduction and resolution enhancement are performed using one machine learning model, the number of training executions and the number of weight sets are respectively 32, which is the product of 4 and 8. Therefore, by separately executing noise reduction and resolution enhancement using separate machine learning models, it is possible to reduce the training load and the amount of data to be retained. The second set of weights is selected based on information regarding noise in the captured image. The information regarding the noise of the captured image includes the type of the image sensor 132, the ISO sensitivity at the time of imaging, the signal distribution (dispersion, etc.) of the optical black area of the captured image, the compression quality of the captured image (Q value of JEPG compression, etc.), etc. Contains information. Similarly, the first set of weights is selected based on information regarding high resolution. The information regarding high resolution is information related to the resolution performance of the captured image, including the type of the imaging optical system 131 and the state of the imaging optical system 131 at the time of imaging (focal length, aperture value, focus distance, etc.). ), and information regarding upscaling magnification, etc.

Further, in step S205, the first noise reduction residual map is modified based on the captured image and the output related to high resolution, and the first noise reduction residual map is modified based on the modified first noise reduction residual map. It is desirable to generate two input data. Since the noise generated by the image sensor 132 includes shot noise and the like, the intensity of the noise changes depending on the amount of light. When the flare of a blurred image is corrected by increasing the resolution, a region where the brightness locally changes is created within the captured image. If the residual map of the first noise reduction is not corrected, noise corresponding to the amount of light in the blurred state will exist in the area where the amount of light has changed after high resolution. Discrepancies occur in strength. Since this discrepancy may reduce the noise reduction effect of the second machine learning model, the noise reduction residual map is modified according to local brightness changes in the captured image due to higher resolution. It is desirable to resolve the mismatch between brightness and noise intensity.

In this embodiment, only the configurations that are different from the first embodiment will be explained, and the explanation of the same configurations as the first embodiment will be omitted. In this embodiment, the object of high resolution is the blur generated in the optical system, and the object of noise reduction is the noise generated in the image sensor.

FIG. 5 is an external view of the image processing system 300 of this embodiment. FIG. 6 is a block diagram of the image processing system 300 of this embodiment. The image processing system 300 includes a training device 301, an image processing device (second device) 302, a control device (first device) 303, and an imaging device 304.

The imaging device 304 includes an imaging optical system 341, an imaging element 342, a storage section 343, and a communication section 344. A captured image captured by the imaging device 304 is transmitted to the control device 303.

The control device 303 includes a storage section 331, a communication section (transmission section) 332, a calculation section (obtainment section) 333, and a display section 334. The control device 303 transmits the captured image and a request to perform noise reduction and resolution enhancement on the captured image to the image processing device 302. Further, the control device 303 receives the output of the image processing device 302 (a high-resolution captured image and a second noise reduction residual map, which will be described later), and generates an output image according to a user's instruction.

The image processing device 302 includes a storage section 321, a communication section 322, an acquisition section 323, a noise reduction section 324, a high resolution section 325, and a calculation section 326. The image processing device 302 performs noise reduction and resolution enhancement on the captured image using a machine learning model using a set of weights trained by the training device 301.

The training device 301 includes a storage section 311, an acquisition section 312, a calculation section 313, and an update section 314. The training device 301 executes training of the machine learning model according to the flowchart in FIG. In this embodiment, the training images and correct images used for training the first machine learning model are different from those in the first embodiment. The training image is an image obtained by adding blur and noise to the original image. However, data obtained by combining the training image and a map (corresponding to the residual map of the first noise reduction) obtained by multiplying the added noise by -1 in the channel direction is input to the first machine learning model. The correct image is an image in which a smaller blur than that of the training image is added to the original image (or no blur is added), and noise of a similar strength that is correlated with the noise added to the training image is added. By exposing the noise present in the training image and inputting it to the first machine learning model, the first machine learning model can easily separate the subject and noise in the training image, thereby suppressing noise changes. It is possible to obtain the effects of high resolution. Note that the noise map does not necessarily need to be combined in the channel direction of the training images. A training image and a noise map may each be input to a convolution layer, and the generated feature maps may be combined in the channel direction.

Hereinafter, with reference to FIG. 7, noise reduction and resolution enhancement for the captured image, which are executed by the control device 303 and the image processing device 302, will be described. FIG. 7 is a flowchart showing the generation of an output image in this embodiment.

In step S301, the communication unit 332 transmits the captured image and a request to perform noise reduction and resolution enhancement on the captured image to the image processing device 302.

In step S302, the communication unit 322 acquires the captured image and a request to perform noise reduction and resolution enhancement on the captured image. Note that the captured image may be stored in advance in the storage unit 321, or may be read from another recording medium.

In step S303, the acquisition unit 323 acquires a set of weights of the machine learning model. Specifically, the acquisition unit 323 acquires sets of first to third weights used in each of the first to third machine learning models.

In step S304, the noise reduction unit 324 uses the third machine learning model to generate an output regarding the first noise reduction based on the captured image. In this example, the output for the first noise reduction is a residual map of the first noise reduction. The first noise reduction residual map is a component that cancels noise in the captured image.

In step S305, the calculation unit 326 combines the captured image and the first noise reduction residual map in the channel direction to generate first input data.

In step S306, the high-resolution unit 325 uses the first machine learning model to generate an output related to high-resolution based on the first input data. In this embodiment, the output related to high resolution is a third image that is a captured image that has been high resolved without noise reduction. Unlike Example 1, in this example, the first machine learning model is directly trained by the first input data including the captured image and the first noise reduction residual map, and by training the first machine learning model. , a third image can be generated. Note that the first input data is an image (first image) obtained by adding the captured image and the residual map of the first noise reduction, and the residual map of the first noise reduction, which are combined in the channel direction. It can also be data. Alternatively, data may be obtained by combining the captured image and the first image in the channel direction. In these cases, a similar type of input is used when training the first machine learning model. In this embodiment, the output related to high resolution is a captured image, a captured image with noise reduced based on the output related to the first noise reduction, and a residual of the first noise reduction based on the output related to the first noise reduction. The map is generated using at least two of the maps.

In step S307, the noise reduction unit 324 uses the second machine learning model to generate an output related to second noise reduction based on the second input data. In this example, the second input data is the third image, and the output for the second noise reduction is the residual map of the second noise reduction. By adding the third image and the second noise reduction residual map, a captured image in which noise reduction and resolution enhancement have been performed is obtained.

In step S308, the communication unit 322 transmits to the control device 303 an output related to high resolution (third image) and an output related to second noise reduction (residual map of second noise reduction).

In step S309, the communication unit 332 obtains an output related to high resolution (third image) and an output related to second noise reduction (residual map of second noise reduction).

In step S310, the calculation unit 333 calculates an output image based on the captured image, the output related to high resolution (third image), and the output related to second noise reduction (residual map of second noise reduction). generate. By changing the weight of the weighted average of the captured image and the third image, the intensity of high resolution can be adjusted. The intensity of noise reduction can be adjusted by adding a second noise reduction residual map obtained by multiplying the weighted average of the captured image and the third image by a coefficient representing the intensity of noise reduction. While checking the output image displayed on the display unit 334, the user can quickly adjust the intensity of high resolution and noise reduction.

As described above, according to the configuration of this embodiment, it is possible to highly accurately improve the resolution and reduce noise of a captured image while suppressing the training load of a machine learning model.

The disclosure of this embodiment includes the following methods and configurations.
(Method 1)
generating an output related to first noise reduction using the captured image;
using a first machine learning model to generate an output related to high resolution based on first input data corresponding to the output related to the first noise reduction;
using a second machine learning model to generate an output related to second noise reduction based on second input data corresponding to the output related to high resolution;
generating an output image based on the output related to the second noise reduction;
The second input data is an image that has a larger difference from the second addition image than from the first addition image,
The first addition image is an image obtained by adding the high-resolution residual map based on the output regarding the high-resolution and the captured image,
The second added image is an image obtained by adding the high-resolution residual map and the captured image whose noise has been reduced based on the output regarding the first noise reduction. Image processing method.
(Method 2)
The image processing method according to method 1, wherein the output regarding the high resolution is generated based on information regarding an optical system used when acquiring the captured image.
(Method 3)
The information regarding the optical system includes the type of the optical system, the focal length of the optical system, the aperture value of the optical system, the focal distance of the optical system, the position of the pixel of the captured image with respect to the optical axis of the optical system, and The image processing method according to method 2, characterized in that the information is information regarding any of the resolution performance of the optical system in pixels of the captured image.
(Method 4)
4. The image processing method according to any one of configurations 1 to 3, wherein the increase in resolution includes at least one of blur correction and upscaling.
(Method 5)
5. The image processing method according to any one of methods 1 to 4, wherein the increase in resolution includes correction of blur caused in an optical system used when acquiring the captured image.
(Method 6)
6. The image processing method according to any one of methods 1 to 5, wherein the output image is generated based on the captured image and the output related to high resolution.
(Method 7)
selecting a first set of weights to be used in the first machine learning model based on information regarding high resolution;
The method according to any one of methods 1 to 6, further comprising the step of selecting a second set of weights to be used in the second machine learning model based on information regarding noise in the captured image. Image processing method described.
(Method 8)
8. The image processing method according to any one of methods 1 to 7, wherein the output related to the first noise reduction is generated using a machine learning model.
(Method 9)
9. The image processing method according to method 8, wherein the machine learning model is the same as the second machine learning model.
(Method 10)
The second input data is generated using a residual map of the first noise reduction based on the output related to the first noise reduction, which is corrected based on the captured image and the output related to high resolution. 10. The image processing method according to any one of methods 1 to 9.
(Method 11)
Any one of methods 1 to 10, characterized in that the second input data is generated based on the output related to the high resolution and the captured image or the output related to the first noise reduction. The image processing method described in the above method.
(Method 12)
The output related to the high resolution includes the captured image, the captured image with noise reduced based on the output related to the first noise reduction, and the residual of the first noise reduction based on the output related to the first noise reduction. The image processing method according to any one of methods 1 to 11, characterized in that the image processing method is generated using at least two of the maps.
(Configuration 1)
a first generation unit that generates an output related to first noise reduction using the captured image;
A first generation unit that uses a first machine learning model that performs high resolution to generate an output regarding high resolution based on first input data corresponding to the output regarding the first noise reduction;
a third generation unit that uses a second machine learning model that performs noise reduction to generate an output related to second noise reduction based on second input data corresponding to the output related to high resolution;
an output unit that outputs an output image based on the output related to the second noise reduction,
The second input data is an image that has a larger difference from the second addition image than from the first addition image,
The first addition image is an image obtained by adding a high-resolution residual map based on the output regarding the high-resolution and the captured image,
Image processing characterized in that the second added image is an image obtained by adding the high-resolution residual map and the captured image whose noise has been reduced based on the output related to the first noise reduction. Device.
(Configuration 2)
An image processing system comprising a first device and a second device,
The first device includes:
a transmitter that transmits a request regarding execution of processing to the second device;
an acquisition unit that acquires an output image using the output acquired from the second device,
The second device includes:
a first generation unit that generates an output related to first noise reduction using the captured image;
A first generation unit that uses a first machine learning model that performs high resolution to generate an output regarding high resolution based on first input data corresponding to the output regarding the first noise reduction;
a third generation unit that uses a second machine learning model that performs noise reduction to generate an output related to second noise reduction based on second input data corresponding to the output related to high resolution;
an output unit that outputs the output based on the output related to the second noise reduction,
The second input data is an image that has a larger difference from the second addition image than from the first addition image,
The first addition image is an image obtained by adding a high-resolution residual map based on the output regarding the high-resolution and the captured image,
Image processing characterized in that the second added image is an image obtained by adding the high-resolution residual map and the captured image whose noise has been reduced based on the output related to the first noise reduction. system.
(Configuration 3)
A program that causes a computer to execute the image processing method described in any one of methods 1 to 12.
[Other Examples]
The present invention provides a system or device with a program that implements one or more of the functions of the above-described embodiments via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the invention.

102 Image processing device 124 Noise reduction section 125 High resolution section 126 Arithmetic section

Claims (15)

generating an output related to first noise reduction using the captured image;
using a first machine learning model to generate an output related to high resolution based on first input data corresponding to the output related to the first noise reduction;
using a second machine learning model to generate an output related to second noise reduction based on second input data corresponding to the output related to high resolution;
generating an output image based on the output related to the second noise reduction;
The second input data is an image that has a larger difference from the second addition image than from the first addition image,
The first addition image is an image obtained by adding the high-resolution residual map based on the output regarding the high-resolution and the captured image,
The second added image is an image obtained by adding the high-resolution residual map and the captured image whose noise has been reduced based on the output regarding the first noise reduction. Image processing method. 2. The image processing method according to claim 1, wherein the output regarding high resolution is generated based on information regarding an optical system used when acquiring the captured image. The information regarding the optical system includes the type of the optical system, the focal length of the optical system, the aperture value of the optical system, the focal distance of the optical system, the position of the pixel of the captured image with respect to the optical axis of the optical system, and 3. The image processing method according to claim 2, wherein the information is information regarding any one of the resolution performance of the optical system in a pixel of the captured image. 3. The image processing method according to claim 1, wherein the high resolution includes at least one of blur correction and upscaling. 3. The image processing method according to claim 1, wherein the increase in resolution includes correction of blur caused in an optical system used when acquiring the captured image. 3. The image processing method according to claim 1, wherein the output image is generated based on the captured image and the output related to high resolution. selecting a first set of weights to be used in the first machine learning model based on information regarding high resolution;
The image processing method according to claim 1 or 2, further comprising the step of selecting a second set of weights to be used in the second machine learning model based on information regarding noise in the captured image. . 3. The image processing method according to claim 1, wherein the output related to the first noise reduction is generated using a machine learning model. The image processing method according to claim 8, wherein the machine learning model is the same as the second machine learning model. The second input data is generated using a residual map of the first noise reduction based on the output related to the first noise reduction, which is corrected based on the captured image and the output related to high resolution. The image processing method according to claim 1 or 2, characterized in that: 3. The second input data is generated based on the output related to the high resolution and the output related to the captured image or the first noise reduction. Image processing method. The output related to the high resolution includes the captured image, the captured image with noise reduced based on the output related to the first noise reduction, and the residual of the first noise reduction based on the output related to the first noise reduction. The image processing method according to claim 1 or 2, wherein the image processing method is generated using at least two of the maps. a first generation unit that generates an output related to first noise reduction using the captured image;
a second generation unit that uses a first machine learning model that performs high resolution to generate an output regarding high resolution based on first input data that corresponds to the output regarding the first noise reduction;
a third generation unit that uses a second machine learning model that performs noise reduction to generate an output related to second noise reduction based on second input data corresponding to the output related to high resolution;
an output unit that outputs an output image based on the output related to the second noise reduction,
The second input data is an image that has a larger difference from the second addition image than from the first addition image,
The first addition image is an image obtained by adding a high-resolution residual map based on the output regarding the high-resolution and the captured image,
Image processing characterized in that the second added image is an image obtained by adding the high-resolution residual map and the captured image whose noise has been reduced based on the output related to the first noise reduction. Device. An image processing system having a first device and a second device,
The first device includes:
a transmitter that transmits a request regarding execution of processing to the second device;
an acquisition unit that acquires an output image using the output acquired from the second device,
The second device includes:
a first generation unit that generates an output related to first noise reduction using the captured image;
a second generation unit that uses a first machine learning model that performs high resolution to generate an output regarding high resolution based on first input data that corresponds to the output regarding the first noise reduction;
a third generation unit that uses a second machine learning model that performs noise reduction to generate an output related to second noise reduction based on second input data corresponding to the output related to high resolution;
an output unit that outputs the output based on the output related to the second noise reduction,
The second input data is an image that has a larger difference from the second addition image than from the first addition image,
The first addition image is an image obtained by adding a high-resolution residual map based on the output regarding the high-resolution and the captured image,
Image processing characterized in that the second added image is an image obtained by adding the high-resolution residual map and the captured image whose noise has been reduced based on the output related to the first noise reduction. system. A program for causing a computer to execute the image processing method according to claim 1 or 2.