JP7284688B2 - Image processing device, image processing method, image processing program and recording medium - Google Patents

Description

The present invention relates to an image processing device, an image processing method, an image processing program, and a recording medium.

In many cases, noise is superimposed on an image acquired by imaging or the like. For example, the shorter the exposure time, the more noise is superimposed on an image captured by an optical device such as a fluorescence microscope. A lot of noise is also superimposed on a tomographic image created by reconstruction processing in a PET device, an X-ray CT device, an MRI device, or the like. Various techniques are known for reducing noise superimposed on a target image.

The noise reduction technique based on the median method prevents pixels of a target image from having a large luminance difference by rearranging the luminance of surrounding pixels. However, this noise reduction technique has problems such as lowering the resolution of the image and deforming the image.

The noise reduction technique based on the smoothing method has a problem of lowering the spatial resolution of the image due to its principle. However, the proposal of edge-preserving noise reduction techniques such as the Non-Local Means filter has made it possible to smooth non-edge portions while leaving edges in the target image.

Also, in recent years, attention has been focused on Deep Image Prior technology using a convolutional neural network (CNN), which is a type of deep neural network. This noise reduction technique utilizes the property of CNN that meaningful structures in target images are learned faster than random noise (that is, random noise is difficult to learn), and reduces noise in target images. (Non-Patent Documents 1 to 3).

Dmitry Ulyanov, et al, “DeepImage Prior”, [online], [searched on October 8, 2019], Internet <URL: https://dmitryulyanov.github.io/deep_image_prior> Dmitry Ulyanov, et al, “DeepImage Prior”, [online], [searched on October 8, 2019], Internet <URL: https://sites.skoltech.ru/app/data/uploads/sites/25 /2018/04/deep_image_prior.pdf＞ Dmitry Ulyanov, et al, “DeepImage Prior”, [online], [searched on October 8, 2019], Internet <URL: https://box.skoltech.ru/index.php/s/ib52BOoV58ztuPM#pdfviewer ＞ Jaakko Lehtinen, et al, “Noise2Noise: Learning Image Restoration without Clean Data,” [online], [searched on October 8, 2019], Internet <https://arxiv.org/abs/1803.04189>

Conventional noise reduction technology is used when there is only one target image for which noise is to be reduced, or when the SN ratio of the target image is low (when noise is superimposed on the signal component in the target image at a ratio that cannot be ignored). However, it is difficult to reduce noise effectively.

The present invention has been made to solve the above problems, and effectively reduces noise in a target image even when there is only one target image or when the SN ratio of the target image is low. It is an object of the present invention to provide an image processing apparatus and an image processing method that can

The image processing apparatus of the present invention includes: (1) For each of a plurality of random noise images, the convolutional neural network is repeatedly trained using the random noise image as an input image and the target image as a teacher image, and after the repeated learning, the convolutional neural network (2) a first processing unit that acquires an image to be output as an intermediate image; and a processing unit.

In one aspect of the image processing apparatus of the present invention, the first processing unit performs iterative learning when the difference between the image output from the convolutional neural network and the target image falls within a target range in the process of iterative learning. It is preferable to obtain the image that is finished and output from the convolutional neural network at the time of its finish as the intermediate image. The first processing unit preferably uses a non-encoder/decoder type convolutional neural network. It is preferable that the first processing unit performs in parallel any two or more of the processes of obtaining intermediate images for each of the plurality of random noise images.

In another aspect of the image processing apparatus of the present invention, the second processing unit uses one of the plurality of intermediate images as an input image and one of the others as a teacher image to make the convolutional neural network learn by the Noise2Noise method. is preferably used to create the output image. Preferably, the second processing unit creates an output image by averaging a plurality of intermediate images.

In the image processing method of the present invention, (1) for each of a plurality of random noise images, the convolutional neural network is repeatedly trained using the random noise image as an input image and the target image as a teacher image, and after the iterative learning, the convolutional neural network (2) a first processing step of obtaining an image to be output as an intermediate image; and a processing step.

In one aspect of the image processing method of the present invention, in the first processing step, iterative learning is performed when the difference between the image output from the convolutional neural network and the target image in the process of iterative learning falls within a target range. It is preferable to obtain the image that is finished and output from the convolutional neural network at the time of its finish as the intermediate image. Preferably, in the first processing step, the convolutional neural network is of the non-encoder-decoder type. In the first processing step, it is preferable to perform in parallel any two or more of the processes of acquiring intermediate images for each of the plurality of random noise images.

In another aspect of the image processing method of the present invention, in the second processing step, one of the plurality of intermediate images is used as an input image and any other of the intermediate images is used as a teacher image to train a convolutional neural network by a Noise2Noise method. is preferably used to create the output image. Preferably, in the second processing step, an output image is created by averaging a plurality of intermediate images.

An image processing program of the present invention is for causing a computer to execute the first processing step and the second processing step of the image processing method of the present invention. A recording medium of the present invention is a computer-readable medium recording the image processing program of the present invention.

According to the present invention, even when there is only one target image or when the SN ratio of the target image is low, the noise of the target image can be effectively reduced.

FIG. 1 is a diagram showing the configuration of an image processing apparatus 1 of this embodiment. FIG. 2 is a flow chart of the image processing method of this embodiment. FIG. 3 is a diagram showing the configuration of the first processing unit 10. As shown in FIG. FIG. 4 is a diagram showing the configuration of the second processing section 20. As shown in FIG. FIG. 5(a) is a diagram showing an original image. 5B is a diagram showing the target image A. FIG. FIG. 6(a) is a diagram showing an example of the random noise image _Bn . FIG. 6B is a diagram showing an example of the intermediate image _Cn . FIG. 7 is a histogram showing the distribution of luminance values in difference images between two intermediate images randomly selected from the intermediate images C ₁ to C ₁₀₀ . FIG. 8 is a graph showing the relationship between the number of iterative learnings of the CNN in the first processing step and the loss function value representing the difference between the image output from the CNN and the teacher image. FIG. 9 is a graph showing an enlarged part of FIG. FIG. 10 is a graph showing the relationship between the number of iterative learnings of the CNN in the first processing step and the loss function value representing the difference between the image output from the CNN and the teacher image. FIG. 11 is a graph showing an enlarged part of FIG. FIG. 12(a) is a diagram showing an image output from the CNN when the number of times of learning is 32 in the first processing step. FIG. 12(b) is a diagram showing an image output from the CNN when the number of times of learning is 64 in the first processing step. FIG. 13(a) is a diagram showing an image output from the CNN when the number of times of learning is 96 in the first processing step. FIG. 13(b) is a diagram showing an image output from the CNN when the number of times of learning is 128 in the first processing step. 14 is a diagram showing the target image A. FIG. FIG. 15(a) is a diagram showing an output image D obtained by the example. 15B is a diagram showing an image obtained by Comparative Example 1. FIG. FIG. 16A is a diagram showing the distribution of fixed noise added to the original image in the region of interest R1. FIG. 16(b) is a diagram showing the distribution of values obtained by subtracting the target image A from the output image D obtained in the example. 16C is a diagram showing the distribution of values obtained by subtracting the target image A from the image obtained in Comparative Example 1. FIG. FIG. 17A is a diagram showing the distribution of fixed noise added to the original image in the region of interest R2. FIG. 17(b) is a diagram showing the distribution of values obtained by subtracting the target image A from the output image D obtained in the example. 17C is a diagram showing the distribution of values obtained by subtracting the target image A from the image obtained in Comparative Example 1. FIG. FIG. 18(a) is a diagram showing the target image A in the attention area R1. FIG. 18(b) is a diagram showing an output image D obtained by the example. 18C is a diagram showing an image obtained by Comparative Example 1. FIG. FIG. 19A is a diagram showing the target image A in the attention area R2. FIG. 19(b) is a diagram showing an output image D obtained by the example. FIG. 19C is a diagram showing an image obtained by Comparative Example 1. FIG. FIG. 20 is a diagram showing an original image. FIG. 21(a) is a diagram showing an original image. FIG. 21(b) is a diagram showing the target image A. FIG. FIG. 22(a) is a diagram showing an output image D obtained by the example. 22(b) is a diagram showing an image obtained by Comparative Example 1. FIG. FIG. 23(a) is a diagram showing an image obtained in Comparative Example 2. FIG. FIG. 23(b) is a diagram showing an image obtained in Comparative Example 3. FIG. FIG. 24(a) is a diagram showing an original image. FIG. 24(b) is a diagram showing the target image A. FIG. FIG. 25(a) is a diagram showing an output image D obtained by the example. FIG. 25(b) is a diagram showing an image obtained by Comparative Example 1. FIG. FIG. 26(a) is a diagram showing an image obtained in Comparative Example 2. FIG. FIG. 26(b) is a diagram showing an image obtained in Comparative Example 3. FIG. FIG. 27 is a diagram showing an original image. FIG.27(b) is a figure which expands and shows a part of Fig.27 (a). 28 is a diagram showing a target image A. FIG. FIG.28(b) is a figure which expands and shows a part of Fig.28 (a). FIG. 29 is a diagram showing an output image D obtained by the example. FIG.29(b) is a figure which expands and shows a part of FIG.29(a). 30 is a diagram showing an image obtained by Comparative Example 1. FIG. FIG.30(b) is a figure which expands and shows a part of Fig.30 (a). 31 is a diagram showing an image obtained by Comparative Example 3. FIG. FIG.31(b) is a figure which expands and shows a part of Fig.31 (a). FIG. 32(a) is a diagram showing an original image. FIG. 32(b) is a diagram showing the target image A. FIG. FIG. 33(a) is a diagram showing an output image D obtained by the example. 33(b) is a diagram showing an image obtained by Comparative Example 1. FIG. FIG. 33(c) is a diagram showing an image obtained by Comparative Example 3. FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and overlapping descriptions are omitted. The present invention is not limited to these exemplifications, but is indicated by the scope of the claims, and is intended to include all modifications within the meaning and scope of equivalents of the scope of the claims.

FIG. 1 is a diagram showing the configuration of an image processing apparatus 1 of this embodiment. The image processing apparatus 1 is an apparatus for reducing noise superimposed on a target image A to create an output image D, and includes a first processing section 10 and a second processing section 20 . The first processing unit 10 repeatedly learns a convolutional neural network (CNN) using the random noise image B _n as an input image and the target image A as a teacher image for each of a plurality (N) of random noise images B ₁ to B _N. , and an image output from the CNN after the iterative learning is obtained as an intermediate image _Cn . The second processing unit 20 creates an output image D in which noise is reduced from the target image A based on the plurality (N) of intermediate images C ₁ to C _N acquired by the first processing unit 10 .

The N random noise images B ₁ to B _N and the N intermediate images C ₁ to C _N are of the same size as the target image A. The random noise images B ₁ -B _N need not have any meaningful structure and may be images consisting only of noise. N is preferably an integer of 16 or more, and more preferably an integer of 50 or more. n is each integer of 1 or more and N or less. It is expected that the larger the value of N, the greater the noise reduction effect. is preferred. The CNN used in the first processing unit 10 is preferably of a format (non-encoder/decoder format) that does not have the characteristics of the encoder/decoder format, and is preferably of the ResNet format, for example.

The image processing apparatus 1 can be configured to include a computer. The image processing apparatus 1 includes an arithmetic unit including a CPU, a GPU, etc. for performing various processes, a display unit including a liquid crystal display for displaying images, etc., and an input unit including a keyboard, mouse, etc. for receiving input of processing conditions and the like. and a storage unit including a hard disk drive for storing image data and the like, a RAM and a ROM.

The storage unit of the image processing apparatus 1 also stores an image processing program for causing a computer to execute the first processing step S1 and the second processing step S2 of the image processing method of this embodiment, which will be described below. The calculation unit executes the first processing step S1 and the second processing step S2 based on the image processing program. The image processing program may be stored in the storage unit when the image processing apparatus 1 is shipped, may be acquired via a communication line after shipment, and may be stored in the storage unit. What has been recorded on the recording medium 30 may be stored in the storage unit. The recording medium 30 is arbitrary, such as a flexible disk, CD-ROM, DVD-ROM, BD-ROM, and USB memory.

FIG. 2 is a flow chart of the image processing method of this embodiment. This image processing method is a method of reducing noise superimposed on the target image A to create an output image D, and includes a first processing step S1 and a second processing step S2. The first processing step S1 is performed by the first processing unit 10, and the second processing step S2 is performed by the second processing unit 20. FIG. In the first processing step S1, for each of a plurality (N) of random noise images B ₁ to B _N , the random noise image B _n is used as an input image, and the target image A is used as a teacher image. An image output from the CNN after is acquired as an intermediate image _Cn . In the second processing step S2, a noise-reduced output image D is created from the target image A based on the plurality (N) of intermediate images C ₁ to C _N acquired in the first processing step S1.

FIG. 3 is a diagram showing the configuration of the first processing unit 10. As shown in FIG. The first processing unit 10 creates an intermediate image C _n by learning a CNN based on a teacher image (target image A) and an input image (random noise image B _n ). A first evaluation unit 12 is included.

The first CNN unit 11 inputs an input image (random noise image B _n ) to the CNN, and outputs an image corresponding to the learning state of the CNN at the time of input from the CNN. The first evaluation unit 12 obtains a loss function value representing the difference between the image output from the CNN and the teacher image (target image A). A loss function is, for example, the L2 norm. The first CNN unit 11 learns the CNN so that this loss function value becomes small. In this way, the first processing unit 10 repeatedly learns the CNN by the first CNN unit 11 and the first evaluation unit 12 using a combination of the teacher image (target image A) and the input image (random noise image B _n ). .

In the process of iterative learning of the CNN, the image output from the CNN is close to the input image (random noise image B _n ) at first, but gradually the original fixed noise in the teacher image (target image A) It is replaced by the input image (random noise image B _n ), and eventually becomes closer to the teacher image (target image A). This is based on the property of CNNs that meaningful structures are learned faster than random noise (that is, random noise is less learned).

Therefore, by ending the learning in the middle of the repeated learning of the CNN, the original fixed noise in the teacher image (target image A) is replaced with the input image (random noise image B _n ). Output from the CNN be able to. An image output from the CNN at the end of learning is acquired as an intermediate image _Cn . The iterative learning of the CNN may end when the number of times of learning reaches the target value, but the loss function value representing the difference between the image output from the CNN and the teacher image (target image A) is within the target range. It is preferable to terminate when the The target range of loss function values used to determine the end of learning is determined based on the loss function value representing the difference between the original input image (random noise image B _n ) and the teacher image (target image A). is preferred.

Similar processing is performed using each of the N random noise images B ₁ to _BN as input images to acquire N intermediate images C ₁ to C _N . Looking at the common pixels of the N intermediate images C ₁ to C _N acquired in this way, the luminance value of the pixel is the original luminance value of the same pixel of the target image A (luminance excluding fixed noise value) and has a substantially symmetrical distribution (normal distribution). This luminance value distribution corresponds to the noise distribution of the N random noise images B ₁ to _BN .

Note that the first processing unit 10 performs any two or more of the processes of acquiring N intermediate images C ₁ to C _N from N random noise images B ₁ to B _N in parallel. good too. For example, the first processing unit 10 is configured to include N partial processing units that can operate in parallel, and the n-th partial processing unit among them is the target image A and the n-th random noise image B _n to the n-th intermediate image C by CNN. By creating _n , the processing time can be reduced to 1/N.

FIG. 4 is a diagram showing the configuration of the second processing section 20. As shown in FIG. The second processing unit 20 creates an output image D by learning the CNN by the Noise2Noise method (Non-Patent Document 4) using the N intermediate images C ₁ to C _N . , a second evaluator 22 and an image selector 23 .

The image selection unit 23 selects any intermediate image C _n1 from among the N intermediate images C ₁ to C _N as an input image for the CNN, and selects any other intermediate image C _n2 as the input image for the CNN. Let it be a CNN teacher image. There are N(N−1) possible combinations of the input image (intermediate image C _n1 ) and the teacher image (intermediate image C _n2 ), but it is not necessary to employ all N(N−1) combinations. .

The second CNN unit 21 inputs the input image (intermediate image C _n1 ) to the CNN, and outputs from the CNN an image corresponding to the learning state of the CNN at the time of the input. The second evaluation unit 22 obtains a loss function value representing the difference between the image output from the CNN and the teacher image (intermediate image C _n2 ). A loss function is, for example, the L2 norm. The second CNN unit 21 learns the CNN so that this loss function value becomes small. In this way, the second processing unit 20 uses various combinations of the teacher image (intermediate image C _n2 ) and the input image (intermediate image C _n1 ) selected by the image selection unit 23 to select the second CNN unit 21 and the 2 The evaluation unit 22 repeatedly learns the CNN.

By repeatedly learning the CNN using various combinations of teacher images (intermediate images C _n2 ) and input images (intermediate images C _n1 ) having different noise patterns, the image output from the CNN is the noise from the target image A. is reduced as an output image D.

The second processing unit 20 preferably creates the output image D by learning the CNN by the Noise2Noise method using the N intermediate images C ₁ to C _N , but the N intermediate images C ₁ to C _N can also be used to create an output image D. For example, by averaging N intermediate images C ₁ to C _N , an output image D in which noise is reduced from the target image A can be created.

According to this embodiment, even when there is only one target image A or when the SN ratio of the target image A is low, the noise of the target image A can be effectively reduced. Below, the noise reduction effect of this embodiment will be described by showing simulation results.

Next, simulation results will be described. In the following, the output image D is created by CNN learning by the Noise2Noise method in the second processing step.

FIG. 5A is a diagram showing the original image used in the simulation. FIG. 5B is a diagram showing the target image A used in the simulation. The target image A (FIG. 5(b)) was created by superimposing fixed noise having a normal distribution with a standard deviation of 5 on the original image (FIG. 5(a)). FIG. 6(a) is a diagram showing an example of the random noise image _Bn used in the simulation. FIG. 6B is a diagram showing an example of an intermediate image _Cn created by simulation. In the first processing step, intermediate images C _{1 to C 100 were created from the random noise images B 1 to} B ₁₀₀ and the target image A with N= ₁₀₀ .

FIG. 7 is a histogram showing the distribution of luminance values in difference images between two intermediate images randomly selected from the intermediate images C ₁ to C ₁₀₀ . The horizontal axis is the luminance value, and the vertical axis is the frequency. As shown in this figure, the luminance values of each pixel of the intermediate image _Cn have a substantially symmetrical distribution (normal distribution). The standard deviation of this distribution was 14.256.

The standard deviation of the distribution of noise contained in the luminance value of each pixel in a certain common area of each image (the area above the hat in the image) is 5.894 for the target image A and 5.894 for the intermediate image C. 11.137 for one intermediate image out of C ₁ to C ₁₀₀ , 5.675 for the average image of intermediate images C ₁ to C ₁₀₀ , and 5.675 for the output image D obtained by CNN learning by the Noise2Noise method. was 2.893. Compared to the standard deviation of the noise distribution of the target image A, although the standard deviation of the noise distribution of each intermediate image C _n is larger, the standard deviation of the noise distribution of the average image of the intermediate images C ₁ to C ₁₀₀ is smaller, and the output The standard deviation of the noise distribution for image D was even smaller.

8 to 11 are graphs showing the relationship between the number of CNN learning times in the first processing step and the loss function value representing the difference between the image output from the CNN and the teacher image. FIG. 9 is an enlarged view of a part of FIG. FIG. 11 is an enlarged view of a part of FIG. The horizontal axis is the number of CNN learning times, and the vertical axis is the loss function value.

8 and 9 show the case where the random noise pattern image (fixed noise image) added to the original image (FIG. 5(a)) when preparing the target image A (FIG. 5(b)) is used as the teacher image. show. As shown in these figures, there is a tendency for the loss function value to decrease as the number of times of CNN learning increases in the first processing step. By repeating learning, the fixed noise image is learned from the random noise image, and the loss function value becomes smaller. The final image output from the CNN is close to a fixed noise image. In the first processing step, learning is terminated when an image output from the CNN contains a random noise image before learning up to a fixed noise image. In the examples of FIGS. 8 and 9, it is preferable to finish learning after 60 to 80 times. As described above, the iterative learning of the CNN may end when the number of times of learning reaches the target value. It is preferred to terminate when in range.

10 and 11 show the case where the original image (FIG. 5(a)) is used as the teacher image (SignalOnly), and the target image A (FIG. 5(b)) superimposed with fixed noise is used as the teacher image. (Signal + Noise). As shown in these figures, learning is slower when the target image A superimposed with fixed noise is used as the teacher image than when the original image is used as the teacher image. This indicates that, like the Deep Image Prior technology (Non-Patent Documents 1 to 3), superimposition of noise on the teacher image slows down learning. Therefore, by ending the CNN learning at an appropriate number of times, the image output from the CNN can be assumed to be the original image (original image) superimposed with random noise without learning fixed noise. can.

12 and 13 are diagrams showing examples of images output from the CNN in the process of iterative learning of the CNN in the first processing step. Here, the original image (FIG. 5(a)) is used as the teacher image. FIG. 12(a) is a diagram showing an image output from the CNN when the number of times of learning is 32 in the first processing step. FIG. 12(b) is a diagram showing an image output from the CNN when the number of times of learning is 64 in the first processing step. FIG. 13(a) is a diagram showing an image output from the CNN when the number of times of learning is 96 in the first processing step. FIG. 13(b) is a diagram showing an image output from the CNN when the number of times of learning is 128 in the first processing step. As shown in these figures, as the number of times the CNN learns in the first processing step increases, the image output from the CNN approaches the teacher image.

Next, the method of this embodiment (example ) and another method (comparative example) in terms of noise reduction. 14 is a diagram showing the target image A. FIG. Fixed noise was added to the original image by extending it to 32-bit single-precision decimals so that noise below 0 or above 255 is not rounded. In this figure, two regions of interest R1 and R2 are indicated by rectangular frames. FIG. 15(a) is a diagram showing an output image D obtained by the example. 15B is a diagram showing an image obtained by Comparative Example 1. FIG. Comparative Example 1 is based on Deep Image Prior technology.

As shown in these figures, the image obtained in Comparative Example 1 looks sharp at first glance, but the noise appears to contain lumpy low-frequency components. On the other hand, in the output image D obtained by the embodiment, although it seems that the noise is not totally eliminated, the noise is reduced in a well-balanced manner, and the original image is reproduced more naturally.

FIG. 16 is an enlarged view showing a difference image between each image and the target image A in the attention area R1. FIG. 16(a) is a diagram showing the distribution of fixed noise added to the original image. The standard deviation of the noise distribution for this image was 48.6. FIG. 16(b) is a diagram showing the distribution of values obtained by subtracting the target image A from the output image D obtained in the example. The standard deviation of the noise distribution for this image was 16.2. 16C is a diagram showing the distribution of values obtained by subtracting the target image A from the image obtained in Comparative Example 1. FIG. The standard deviation of the noise distribution for this image was 16.3.

FIG. 17 is an enlarged view showing a difference image between each image and the target image A in the attention area R2. FIG. 17A is a diagram showing the distribution of fixed noise added to the original image. The standard deviation of the noise distribution for this image was 50.0. FIG. 17(b) is a diagram showing the distribution of values obtained by subtracting the target image A from the output image D obtained in the example. The standard deviation of the noise distribution for this image was 13.8. 17C is a diagram showing the distribution of values obtained by subtracting the target image A from the image obtained in Comparative Example 1. FIG. The standard deviation of the noise distribution for this image was 16.9.

Ideally, the difference between the output image and the target image A is fixed noise added to the original image. However, as shown in FIGS. 16 and 17, in Comparative Example 1, low-frequency component noise is more conspicuous than the fixed noise added to the original image. On the other hand, in the embodiment, although the standard deviation is small, the difference between the output image and the target image A has the same characteristics as fixed noise.

FIG. 18 is an enlarged view of each image in the region of interest R1. FIG. 18A is a diagram showing the target image A. FIG. The standard deviation of the noise distribution for this image was 49.8. FIG. 18(b) is a diagram showing an output image D obtained by the example. The standard deviation of the noise distribution for this image was 18.9. 18C is a diagram showing an image obtained by Comparative Example 1. FIG. The standard deviation of the noise distribution for this image was 19.3.

FIG. 19 is an enlarged view of each image in the attention area R2. FIG. 19A is a diagram showing the target image A. FIG. The standard deviation of the noise distribution for this image was 50.2. FIG. 19(b) is a diagram showing an output image D obtained by the example. The standard deviation of the noise distribution for this image was 14.6. FIG. 19C is a diagram showing an image obtained by Comparative Example 1. FIG. The standard deviation of the noise distribution for this image was 17.9.

As shown in FIGS. 18 and 19, in Comparative Example 1, compared to the target image A, low-frequency component noise that is not present in either the original image or the fixed noise image is conspicuous. On the other hand, in the embodiment, it seems that the noise is reduced in a pattern similar to that of the target image A.

Next, the degree of noise reduction was compared between the example and comparative examples 1 to 3 by simulation. Comparative Example 1 is based on Deep Image Prior technology. Comparative Example 2 is based on Non-LocalMeans (template size 3×3). Comparative Example 3 is based on Non-Local Means (template size 5×5). The original image used here is shown in FIG. 5(a), and the target image A is shown in FIG. 5(b). FIG. 20 is a diagram showing an original image, in which two regions of interest R3 and R4 are indicated by rectangular frames.

21 to 23 are diagrams showing enlarged images of the attention area R3. FIG. 21(a) is a diagram showing an original image. FIG. 21(b) is a diagram showing the target image A. FIG. FIG. 22(a) is a diagram showing an output image D obtained by the example. 22(b) is a diagram showing an image obtained by Comparative Example 1. FIG. FIG. 23(a) is a diagram showing an image obtained in Comparative Example 2. FIG. FIG. 23(b) is a diagram showing an image obtained in Comparative Example 3. FIG. As shown in FIGS. 21 to 23, among the example and comparative examples 1 to 3, the output image obtained in the example is the closest to the original image, and the reproducibility of the thin and disheveled hair is good. In the images obtained in Comparative Examples 1 to 3, thin hair portions are cut off, and straight hair cannot be reproduced.

24 to 26 are enlarged views showing respective images of the attention area R4. FIG. 24(a) is a diagram showing an original image. FIG. 24(b) is a diagram showing the target image A. FIG. FIG. 25(a) is a diagram showing an output image D obtained by the example. FIG. 25(b) is a diagram showing an image obtained by Comparative Example 1. FIG. FIG. 26(a) is a diagram showing an image obtained in Comparative Example 2. FIG. FIG. 26(b) is a diagram showing an image obtained in Comparative Example 3. FIG. As shown in FIGS. 24 to 26, among the example and comparative examples 1 to 3, the output image obtained in the example is probably the texture of the cloth of the hat ribbon portion that is most faithful to the original image. The vertical stripes are reproduced, and the horizontal stripes of the cloth are also reproduced. The image obtained in Comparative Example 1 has almost no vertical stripes, and the reproducibility of horizontal stripes on the cloth is poor.

Next, the degree of noise reduction was compared between the example and comparative examples 1 and 3 by simulation using another target image. Comparative Example 1 is based on Deep Image Prior technology. Comparative Example 3 is based on Non-LocalMeans (template size 5×5). The images used as the target image and the original image are fluorescence images captured by a digital CMOS camera (ORCA-Fusion) manufactured by Hamamatsu Photonics K.K.

FIG. 27 is a diagram showing an original image. FIG.27(b) is a figure which expands and shows a part of Fig.27 (a). This figure shows the attention area R5 with a rectangular frame. This image was obtained by imaging with a sufficiently long exposure time of 200 ms, and can be treated as an original image with a good SN ratio.

28 is a diagram showing a target image A. FIG. FIG.28(b) is a figure which expands and shows a part of Fig.28 (a). This image was obtained by imaging with a sufficiently short exposure time of 6 ms, and can be treated as the target image A with a poor SN ratio.

FIG. 29 is a diagram showing an output image D obtained by the example. FIG.29(b) is a figure which expands and shows a part of FIG.29(a). 30 is a diagram showing an image obtained by Comparative Example 1. FIG. FIG.30(b) is a figure which expands and shows a part of Fig.30 (a). 31 is a diagram showing an image obtained by Comparative Example 3. FIG. FIG.31(b) is a figure which expands and shows a part of Fig.31 (a).

32 and 33 are diagrams showing further enlarged images of the attention area R5. FIG. 32(a) is a diagram showing an original image. The standard deviation of the noise distribution for this image was 6.467. FIG. 32(b) is a diagram showing the target image A. FIG. The standard deviation of the noise distribution for this image was 3.979. FIG. 33(a) is a diagram showing an output image D obtained by the example. The standard deviation of the noise distribution for this image was 0.029. 33(b) is a diagram showing an image obtained by Comparative Example 1. FIG. The standard deviation of the noise distribution for this image was 0.214. FIG. 33(c) is a diagram showing an image obtained by Comparative Example 3. FIG. The standard deviation of the noise distribution for this image was 0.901.

As shown in FIGS. 27 to 33, among the example and comparative examples 1 and 3, the output image D obtained by the example has the smallest standard deviation of noise distribution and is closest to the original image. .

REFERENCE SIGNS LIST 1 image processing device 10 first processing unit 11 first CNN unit 12 first evaluation unit 20 second processing unit 21 second CNN unit 22 second evaluation unit 23 image selection Part 30... Recording medium.

Claims (14)

For each of a plurality of random noise images, a convolutional neural network is repeatedly trained using the random noise image as an input image and the target image as a teacher image, and after the iterative learning, the image output from the convolutional neural network is acquired as an intermediate image. a first processing unit;
a second processing unit that creates an output image in which noise is reduced from the target image based on the plurality of intermediate images acquired by the first processing unit;
An image processing device comprising: The first processing unit terminates the iterative learning when a loss function value representing a difference between the image output from the convolutional neural network and the target image in the process of the iterative learning falls within a target range. and acquiring an image output from the convolutional neural network at the end thereof as the intermediate image;
The image processing apparatus according to claim 1. The first processing unit uses a non-encoder/decoder type as the convolutional neural network,
The image processing apparatus according to claim 1 or 2. The first processing unit performs any two or more of the processes of acquiring the intermediate image for each of the plurality of random noise images in parallel,
The image processing apparatus according to any one of claims 1 to 3. The second processing unit uses one of the plurality of intermediate images as an input image and one of the others as a teacher image, and trains a convolutional neural network by a Noise2Noise method to create the output image.
The image processing device according to any one of claims 1 to 4. The second processing unit creates the output image by averaging the plurality of intermediate images.
The image processing device according to any one of claims 1 to 4. For each of a plurality of random noise images, a convolutional neural network is repeatedly trained using the random noise image as an input image and the target image as a teacher image, and after the iterative learning, the image output from the convolutional neural network is acquired as an intermediate image. a first processing step;
a second processing step of creating an output image in which noise is reduced from the target image based on the plurality of intermediate images obtained in the first processing step;
An image processing method comprising: In the first processing step, the iterative learning is terminated when a loss function value representing a difference between the image output from the convolutional neural network and the target image in the process of the iterative learning falls within a target range. and acquiring an image output from the convolutional neural network at the end thereof as the intermediate image;
The image processing method according to claim 7. In the first processing step, using a non-encoder-decoder type as the convolutional neural network;
9. The image processing method according to claim 7 or 8. In the first processing step, any two or more of the processing of obtaining the intermediate image for each of the plurality of random noise images are performed in parallel;
The image processing method according to any one of claims 7 to 9. In the second processing step, one of the plurality of intermediate images is used as an input image and one of the other images is used as a teacher image, and a convolutional neural network is trained by a Noise2Noise method to create the output image.
The image processing method according to any one of claims 7 to 10. creating the output image by averaging the plurality of intermediate images in the second processing step;
The image processing method according to any one of claims 7 to 10. An image processing program for causing a computer to execute the first processing step and the second processing step of the image processing method according to any one of claims 7 to 12. 14. A computer-readable recording medium recording the image processing program according to claim 13.

Images