WO2019163244A1

WO2019163244A1 - Image processing method, image processing device, and image processing program

Info

Publication number: WO2019163244A1
Application number: PCT/JP2018/044067
Authority: WO
Inventors: 智親竹嶋
Original assignee: 浜松ホトニクス株式会社
Priority date: 2018-02-20
Filing date: 2018-11-29
Publication date: 2019-08-29
Also published as: JP2019144808A

Abstract

This image processing method is provided with: a first step for performing a convolution operation on a preset ideal image I with a PSF to generate a prediction image P; a second step for differential-processing the original image O from the prediction image P to generate an error image E; a third step for performing a deconvolution operation on the error image E with the PSF multiplied with a coefficient α (0<α<1), and differential-processing, from the ideal image I, an image obtained after the deconvolution operation, thereby generating a differential image D; and a fourth step for obtaining a noise image N by taking the differential image D as a new ideal image I and repeating the above-described steps multiple times, and then differential-processing the noise image N from the original image O to thereby obtain a corrected original image (noise-removed image) OC.

Description

Image processing method, image processing apparatus, and image processing program

The present disclosure relates to an image processing method, an image processing apparatus, and an image processing program.

Super-resolution technology is an image processing technology that increases the resolution of images such as moving images and still images, and is expected to be applied to various fields such as digital cameras, semiconductor exposure devices, and optical microscopes in addition to fields such as televisions. (For example, refer

nonpatent literatures

1 and 2.). Examples of the super-resolution technique include a learning method and a reconstruction method. The learning-type super-resolution technique is a method of identifying a deteriorated image estimated from an analysis image as a database and comparing an acquired image with a database to identify an image before deterioration. The reconstruction method is a method for estimating a high resolution image based on a plurality of low resolution images.

As the image processing technology, there is a deconvolution technology that removes image blur (point spread function) caused by the apparatus from the acquired image. In general deconvolution, an original image is obtained by performing a Fourier transform on the acquired luminance value of the acquired image and dividing the result by a Fourier transform value of a point spread function. However, since the acquired image that is actually acquired includes noise, a technique for removing the noise from the original image is desired.

The present disclosure has been made to solve the above-described problem, and an object thereof is to provide an image processing method, an image processing apparatus, and an image processing program capable of removing noise from an original image.

An image processing method according to an aspect of the present disclosure is an image processing method for removing noise from an original image, and includes a first step of generating a predicted image by performing a convolution operation on a preset ideal image using a point spread function. And a second step of generating an error image by performing a differential process on the original image from the predicted image, and performing a deconvolution operation using a point spread function obtained by multiplying the error image by a coefficient α (0 <α <1) to obtain an error image from the ideal image. A noise image is obtained by repeating the third step of generating a difference image by performing a difference process on the image after the deconvolution operation and the first to third steps a plurality of times with the difference image as a new ideal image. And a fourth step of performing a differential process on the noise image from the original image to obtain a noise-removed image.

In this image processing method, the difference image obtained in the loop of the first step to the third step is used as a new ideal image, and the loop after the first step to the third step is executed. By repeatedly executing this loop, the ideal image is gradually corrected so that the error is minimized, and the ideal image can be brought close to a true value. In this process, since the noise of the original image is not related to the ideal image, the error image is brought closer to the noise image representing the noise in the original image by repeating the first to third steps a plurality of times. be able to. Therefore, a noise-removed image can be obtained by performing differential processing on the noise image from the original image in the fourth step.

In the first step, an initial ideal image may be generated based on the original image. Thereby, it is possible to prevent the first ideal image from greatly deviating from the original image.

In the first step, the average value of the pixel values of each pixel of the original image may be used as the pixel value of each pixel of the initial ideal image. As a result, it is possible to more reliably prevent the first ideal image from greatly deviating from the original image.

In this image processing method, the first to third steps may be repeated 10 times or more. Thereby, the error image can be made sufficiently close to a noise image representing noise in the original image.

Further, the coefficient α used in the third step may be gradually reduced every time the first to third steps are repeated. As a result, the ideal image can be corrected appropriately, and the error image can be made more sufficiently closer to a noise image representing noise in the original image.

An image processing device according to an aspect of the present disclosure is an image processing device including an image processing unit that removes noise from an original image, and the image processing unit converts a preset ideal image by a point spread function. A first processing unit that generates a predicted image by performing a convolution operation, a second processing unit that generates an error image by performing difference processing on the original image from the predicted image, and an error image with a coefficient α (0 <α <1). ) And a third processing unit that generates a difference image by performing a difference process on the image after the deconvolution operation from the ideal image and a first image as a new ideal image. And a fourth processing unit that obtains a noise-removed image by performing differential processing on the noise image from the original image after obtaining the noise image by repeating each of the processing units to the third processing unit a plurality of times.

In this image processing apparatus, the difference image obtained in each processing loop of the first processing unit to the third processing unit is used as a new ideal image, and the next processing of each of the first processing unit to the third processing unit. This loop is executed. By repeatedly executing this loop, the ideal image is gradually corrected so as to minimize the error, and the ideal image can be brought close to a true value. In this process, since the noise of the original image is not related to the ideal image, the error image represents the noise in the original image by repeating each process in the first processing unit to the third processing unit a plurality of times. It can be close to a noise image. Therefore, a noise-removed image can be obtained by performing differential processing on the noise image from the original image in the fourth processing unit.

Further, the first processing unit may generate an initial ideal image based on the original image. Thereby, it is possible to prevent the first ideal image from greatly deviating from the original image.

Further, the first processing unit may use an average value of pixel values of each pixel of the original image as a pixel value of each pixel of the first ideal image. As a result, it is possible to more reliably prevent the first ideal image from greatly deviating from the original image.

Further, the image processing unit may repeat each process of the first processing unit to the third processing unit 10 times or more. Thereby, the error image can be made sufficiently close to a noise image representing noise in the original image.

Further, the image processing unit may gradually reduce the coefficient α used in the third processing unit every time the processes of the first processing unit to the third processing unit are repeated. As a result, the ideal image can be corrected appropriately, and the error image can be made more sufficiently closer to a noise image representing noise in the original image.

An image processing program according to an aspect of the present disclosure is an image processing program for removing noise from an original image, and generates a predicted image by performing a convolution operation on a preset ideal image using a point spread function. And a second process for generating an error image by performing a difference process on the original image from the predicted image, and a deconvolution operation using a point spread function obtained by multiplying the error image by a coefficient α (0 <α <1) Each of the third process for generating a difference image by performing a difference process on the image after the deconvolution operation from the image and the first process to the third process using the difference image as a new ideal image are repeated a plurality of times. After obtaining the noise image, the computer is caused to execute a fourth process for obtaining a noise-removed image by performing differential processing on the noise image from the original image.

In the computer that has executed the image processing program, the difference image obtained in each processing loop of the first processing unit to the third processing unit is used as a new ideal image, and the first processing unit to the third processing unit. The next loop of each process is executed. By repeatedly executing this loop, the ideal image is gradually corrected so as to minimize the error, and the ideal image can be brought close to a true value. In this process, since the noise of the original image is not related to the ideal image, the error image represents the noise in the original image by repeating each process in the first processing unit to the third processing unit a plurality of times. It can be close to a noise image. Therefore, a noise-removed image can be obtained by performing differential processing on the noise image from the original image in the fourth processing unit.

According to the present disclosure, noise can be removed from the original image.

It is a schematic block diagram which shows one Embodiment of an image processing apparatus. 3 is a flowchart illustrating an example of an operation of the image processing apparatus illustrated in FIG. 1. It is a flowchart which shows an example of a deconvolution process. It is a figure which shows the example of an image in 1st loop of a deconvolution process. It is a figure which shows an example of the deconvolution calculation of an error image. It is a figure which shows the example of an image in 2nd loop-29th loop of a deconvolution process. It is a figure which shows the example of an image in the 30th loop of a deconvolution process. It is a flowchart which shows an example of a noise removal process. It is a figure which shows the example of an image in a noise removal process. It is a figure which shows an example of the correction calculation of an original image. It is a block diagram which shows an example of an image processing program and its recording medium. It is a schematic block diagram which shows the modification of an image processing apparatus.

Hereinafter, preferred embodiments of an image processing method, an image processing apparatus, and an image processing program according to an aspect of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a schematic configuration diagram showing an embodiment of an image processing apparatus. An image processing apparatus 1 shown in FIG. 1 is configured as an apparatus that generates a noise-removed image from which noise has been removed based on an original image acquired for a sample S. The sample S is not particularly limited, and examples thereof include biological tissues such as humans and animals, and various devices such as optical devices and electronic devices.

The image processing apparatus 1 includes a camera 2 and a computer 3 as shown in FIG. The camera 2 has an image sensor 4 and an image control unit 5. The image sensor 4 is a one-dimensional or two-dimensional sensor such as a CCD image sensor or a CMOS image sensor. The image sensor 4 images the light L from the sample S imaged by the imaging optical system 7 through the objective lens 6, and outputs a digital signal based on the imaging result to the image control unit 5. The light L from the sample S varies depending on the type of the sample S, and is, for example, light emission such as fluorescence generated from the sample S, reflected light or transmitted light from the sample S, and the like.

The image control unit 5 executes image processing based on the digital signal from the image sensor 4. The image control unit 5 is configured by, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (field-programmable Gate Array). The image control unit 5 generates a digital image based on the digital signal received from the image sensor 4, performs predetermined image processing on the generated digital image, and outputs the digital image to the computer 3. In the following description, a digital image output from the camera 2 to the computer 3 is referred to as an “original image”.

The computer 3 is a device that performs image processing on the digital image output from the camera 2 and generates a noise-removed image from which noise has been removed. The computer 3 is connected to the camera 2 so as to be communicable with each other by wire or wireless. The computer 3 includes, for example, a memory such as a RAM and a ROM, a processor such as a CPU, a communication interface, and a storage unit such as a hard disk. Examples of such computers include personal computers, microcomputers, cloud servers, smart devices (smartphones, tablet terminals, etc.), and the like.

A display device 8 such as a monitor and an input device 9 such as a keyboard, a mouse, and a touch panel are connected to the computer 3. Various images and the like processed by the computer 3 are displayed on the display device 8. In addition, various input information such as operation start and condition setting is transmitted from the input device 9 to the computer 3 based on a user operation.

The computer 3 has an image processing unit 11 that generates a noise-removed image from which noise has been removed based on the original image input from the camera 2. The image processing unit 11 may be configured by an integrated circuit such as an FPGA. The image processing unit 11 includes a first processing unit 12, a second processing unit 13, a third processing unit 14, and a fourth processing unit 15. The first processing unit 12, the second processing unit 13, and the third processing unit 14 execute deconvolution processing as the first stage of image processing. The fourth processing unit 15 executes noise removal processing after the deconvolution processing as the second stage of image processing.

Hereinafter, the operation of the image processing apparatus 1 including processing by the image processing unit 11 will be described.

FIG. 2 is a flowchart showing an example of the operation of the image processing apparatus 1. As shown in the figure, in the image processing apparatus 1, the sample S is first imaged by the camera 2 (step S01). In the camera 2, a digital image of the sample S is generated as an original image O (see FIG. 4), and the original image is output to the computer 3 (step S02).

In the computer 3 to which the original image O is input, deconvolution processing is executed as the first stage of image processing (step S03). In the deconvolution process, each process in the first processing unit 12, the second processing unit 13, and the third processing unit 14 is repeatedly executed in this order a plurality of times. The number of repetitions of the deconvolution process is preferably 10 times or more, and is 30 times in the present embodiment. After execution of the deconvolution processing, noise removal processing by the fourth processing unit 15 is executed as the second stage of image processing (step S04). After the noise removal process is executed, the deconvolution process is executed again (step S05).

In the 1st loop of the deconvolution process, as shown in FIGS. 3 and 4, first, the first processing unit 12 generates an ideal image I based on the original image O (step S11: first step). The number of pixels of the ideal image I is not particularly limited, and may be equal to the number of pixels of the original image O. When the number of pixels of the ideal image I is larger than the number of pixels of the original image O, an image (super-resolution image) with higher resolution than the original image O can be obtained. In the 1st loop of the deconvolution process, for example, the average value of the pixel values (luminance values) of each pixel constituting the original image O is set as the pixel value of each pixel of the first ideal image I. In this case, the first ideal image I is a uniform image in which the pixel value of each pixel is constant.

The initial ideal image I is preferably generated based on the pixel value of each pixel of the original image O, but is not based on the original image O, and is a uniform image with pixel values set in advance. It may be. The number of pixels of the ideal image I is a parameter for determining the number of pixels of the finally obtained image (difference image D). The number of pixels of the ideal image I is preferably an odd number of times (for example, about 3 times or 5 times) of the original image in consideration of the processing speed in the image processing unit 11 and the like.

After the generation of the ideal image I, the first processing unit 12 performs a convolution operation on the ideal image I with a point spread function (hereinafter referred to as “PSF”) to generate a predicted image P (step S12). : First step). PSF is a parameter possessed by an optical element such as a lens. The PSF used in this process is calculated by simulation based on the PSF of the objective lens 6 to be used.

After the predicted image P is generated, the second processing unit 13 performs a difference process for subtracting the original image O from the predicted image P, and generates an error image E (step S13: second step). Next, the third processing unit 14 generates a difference image D based on the error image E (step S14: third step). Here, first, the deconvolution operation of the error image E is performed by the PSF multiplied by the coefficient α (0 <α <1). α gradually decreases as the number of deconvolution loops increases.

The calculation for generating the difference image D is expressed by, for example, the following formulas (1) and (2) (see FIG. 5). In equations (1) and (2), i is the entire pixel, k is the entire domain of the PSF (= W), In _i is the input pixel value (pixel value of the ideal image I), and Out _i is the output pixel value ( (Pixel value of difference image D), α is a coefficient, σ _k is PSF, Er _{i + k} is a pixel value of error image E, and Loop is the number of loops of deconvolution processing.

... (1)

... (2)

After the generation of the difference image D, the image processing unit 11 determines whether or not the number of repetitions of the deconvolution process has reached a predetermined number of times (step S15). Here, when the number of deconvolution process loops reaches 30, the process ends. When the number of deconvolution process loops is less than 30, each process of step S11 to step S15 is re-executed. . In the 2nd loop to the 29th loop, as shown in FIG. 6, the processing of steps S11 to S15 is re-executed with the difference image D obtained in the previous loop as the new ideal image I. In the 30th loop, a difference image D is finally obtained as shown in FIG. The difference image D is an image having the same resolution as the original image O when the number of pixels of the ideal image I is equal to the number of pixels of the original image O. Further, the difference image D is an image (super-resolution image) having a resolution higher than that of the original image O when the number of pixels of the ideal image I is larger than the number of pixels of the original image O.

In the noise removal process following the deconvolution process, as shown in FIGS. 8 and 9, the fourth processing unit 15 performs a difference process of the original image O by the noise image N obtained by the deconvolution process described above. (Step S21: Fourth step). In the deconvolution process, noise is not related to the ideal image I. Therefore, even if the deconvolution process is repeated with a sufficient number of loops, noise may remain in the error image E. Here, for example, the error image E obtained in the 30th loop of the deconvolution process is used as the noise image N, and the noise image N is subjected to differential processing from the original image O, thereby correcting the original image (noise removal). (Image) OC is generated (step S22: fourth step).

This modified original image OC is replaced with the original image O when the deconvolution process is re-executed (see FIG. 2) in step S05. That is, in step S05, each process of step S11 to step S15 is repeatedly executed a plurality of times using the modified original image OC as a new original image. Thereby, when the number of pixels of the ideal image I is larger than the number of pixels of the original image O, noise is removed from the difference image D obtained in step S03, and a difference image D with higher resolution is obtained. Can do. Note that the noise removal process and the deconvolution process after the noise removal process may be repeatedly executed a plurality of times.

The calculation for correcting the original image O is expressed by, for example, the following formulas (3) and (4) (see FIG. 10). In equations (3) and (4), i is the entire pixel, In _i is the input pixel value (pixel value of the original image O), Out _i is the output pixel value (pixel value of the modified original image OC), and Er _i is The pixel value of the error image E, Loop is the number of noise removal loops, abs (Er) is the absolute value of the error intensity, S.P. D. Is the standard deviation of the error intensity, and Level is the correction level. In this example, Level gradually decreases as the number of noise removal processing loops increases.

... (3)

(4)

FIG. 11 is a block diagram showing an example of an image processing program and its recording medium. In the example of FIG. 2, the image processing program 21 includes a main module 22 and an image processing module 23. The image processing module 23 includes a first processing module 24, a second processing module 25, a third processing module 26, and a fourth processing module 27. The functions realized by the computer by executing the image processing program 21 are the same as the functions of the image processing unit 11 described above. The image processing program 21 is provided by a computer-readable recording medium 28 such as a CD-ROM, DVD or ROM. The image processing program 21 may be provided by a semiconductor memory, or may be provided as a computer data signal superimposed on a carrier wave via a network.

As described above, in this image processing method, the next loop of the deconvolution process is executed with the difference image D obtained in the deconvolution process loop as the new ideal image I. By repeatedly executing this loop, the ideal image I is gradually corrected so as to minimize the error, and the ideal image I can be brought close to a true value. In this process, since the noise of the original image O is not related to the ideal image I, the error image E is changed to a noise image N representing the noise in the original image O by repeating the deconvolution loop a plurality of times. You can get closer. Therefore, by performing a differential process on the noise image N from the original image O, it is possible to obtain a modified original image OC from which noise has been removed.

In this embodiment, the first ideal image I is generated based on the original image O in the first step of the deconvolution process. Specifically, the average value of the pixel values of each pixel of the original image O is used as the pixel value of each pixel of the initial ideal image I. Thereby, it is possible to prevent the first ideal image I from greatly deviating from the original image O.

In the present embodiment, the first to third steps of the deconvolution process are repeated ten times or more, and the third step is repeated each time the first to third steps of the deconvolution process are repeated. The coefficient α used in the step is gradually reduced. Thereby, the correction of the ideal image I that minimizes the error can be suitably executed. Therefore, the error image E can be made sufficiently closer to the noise image N representing the noise in the original image O, so that noise removal from the original image O can be performed with high accuracy.

The present disclosure is not limited to the above embodiment. For example, in the above embodiment, the image processing apparatus 1 provided with the image processing unit 11 on the computer 3 side is illustrated, but the image processing unit 11 is provided on the camera 2 side like the image processing apparatus 31 illustrated in FIG. Other configurations may be adopted. The image processing apparatus may be configured as a microscope having a camera or computer having the image processing unit 11.

DESCRIPTION OF

SYMBOLS

1,31 ... Image processing apparatus, 11 ... Image processing part, 12 ... 1st processing part, 13 ... 2nd processing part, 14 ... 3rd processing part, 15 ... 4th processing part, O ... Original image , I ... ideal image, P ... predicted image, E ... error image, D ... difference image, N ... noise image, OC ... modified original image (noise-removed image).

Claims

An image processing method for removing noise from an original image,
A first step of generating a predicted image by convolving a preset ideal image with a point spread function;
A second step of differentially processing the original image from the predicted image to generate an error image;
A third step of performing a deconvolution operation by a point spread function obtained by multiplying the error image by a coefficient α (0 <α <1) and performing a difference process on the image after the deconvolution operation from the ideal image to generate a difference image. When,
After obtaining the noise image by repeating the first step to the third step a plurality of times using the difference image as a new ideal image, the noise image is subjected to difference processing from the original image to obtain a noise-removed image. And an image processing method.
The image processing method according to claim 1, wherein in the first step, an initial ideal image is generated based on the original image.
3. The image processing method according to claim 2, wherein in the first step, an average value of pixel values of each pixel of the original image is used as a pixel value of each pixel of the initial ideal image.
4. The image processing method according to claim 1, wherein the first step to the third step are repeated 10 times or more.
5. The image processing method according to claim 1, wherein the coefficient α used in the third step is gradually reduced every time the first step to the third step are repeated.
An image processing apparatus including an image processing unit that removes noise from an original image,
The image processing unit
A first processing unit that generates a predicted image by performing a convolution operation on a preset ideal image using a point spread function;
A second processing unit that generates an error image by differentially processing the original image from the predicted image;
Third processing for generating a difference image by performing a deconvolution operation by a point spread function obtained by multiplying the error image by a coefficient α (0 <α <1) and performing a difference process on the image after the deconvolution operation from the ideal image. And
The difference image is used as a new ideal image, and each process of the first processing unit to the third processing unit is repeated a plurality of times to obtain a noise image, and then the noise image is subjected to difference processing from the original image to obtain a noise image. An image processing apparatus comprising: a fourth processing unit that obtains a removed image.
The image processing apparatus according to claim 6, wherein the first processing unit generates an initial ideal image based on the original image.
The image processing device according to claim 7, wherein the first processing unit uses an average value of pixel values of each pixel of the original image as a pixel value of each pixel of the initial ideal image.
The image processing apparatus according to any one of claims 6 to 8, wherein the image processing unit repeats each process of the first processing unit to the third processing unit 10 times or more.
The image processing unit gradually decreases the coefficient α used in the third processing unit every time the processes of the first processing unit to the third processing unit are repeated. The image processing device according to any one of the above.
An image processing program for removing noise from an original image,
A first process for generating a predicted image by convolving a preset ideal image with a point spread function;
A second process for generating an error image by differentially processing the original image from the predicted image;
Third processing for generating a difference image by performing a deconvolution operation by a point spread function obtained by multiplying the error image by a coefficient α (0 <α <1) and performing a difference process on the image after the deconvolution operation from the ideal image. When,
After obtaining the noise image by repeating each of the first to third processes a plurality of times using the difference image as a new ideal image, the noise image is subjected to difference processing from the original image to obtain a noise-removed image. An image processing program for causing a computer to execute a fourth process for obtaining