KR102319643B1 - Method for processing microscopy image using artificial neural network with point spread function layer and apparatus therefore - Google Patents

Abstract

Disclosed are a microscope image processing method and apparatus using a neural network having a point spread function layer. An image processing method according to an embodiment of the present invention includes receiving a microscope image; and deblurring the microscope image corresponding to the microscope image by deconvolution processing the microscope image using a neural network based on unsupervised learning learned using non-matching data ( and restoring a deblurring image, wherein the neural network learns based on predefined cyclic loss and adversarial loss with respect to a microscope image and a high-resolution image corresponding to the deblurring image. can be

Description

Microscopic image processing method and apparatus using a neural network having a point spread function layer

The present invention relates to an image processing method and apparatus using a neural network having a point spread function (PSF) layer, and more particularly, to a microscope image processed by a point spread function using mismatching data. A method for reconstructing a high-resolution deblurring image corresponding to a microscope image by deconvolution using a learned unsupervised learning-based neural network, and an apparatus for the same will be.

In fluorescence microscopy, diffraction of light from a given optic reduces the resolution of the image. To improve the resolution, many optimization-based deconvolution algorithms have been developed. The conventional technique of an embodiment proposes a blind deconvolution method by solving a joint minimization problem in order to estimate an image with an unknown blur kernel when a point spread function (PSF) measurement cannot be used. In the related art of another embodiment, an improved version of blind deconvolution using total variance (TV) normalization is proposed.

Recently, convolutional neural networks (CNNs) have been widely used to increase the suitability of optical microscopes without hardware changes. The prior art of an embodiment uses a deep neural network to improve an optical microscope, and improves spatial resolution in a wide field of view and depth. Conventionally, the technique of another embodiment uses a deep convolutional neural network that can be trained on simulation data or experimental measurements to obtain a super-resolution image in a localization microscopy. In another embodiment of the related art, a CNN method capable of recovering isotope resolution from anisotropic data has been proposed. Furthermore, generative adversarial networks (GANs) have attracted much attention to the inverse problem by providing a method using unlabeled data to train deep neural networks. In another embodiment of the related art, a conditional GAN and a DeblurGAN for motion blurring using content loss have been proposed. However, since this GAN approach often introduces artificial features due to mode collapse, a cycle-continuous adversarial network (Cycle-GAN) that imposes one-to-one correspondence also affects image reconstruction.

However, this Cycle-GAN approach usually requires two large generators, which are often difficult to train with a small number of training data.

In embodiments of the present invention, a high-resolution deblur corresponding to a microscope image by deconvolution processing a microscope image processed by a point spread function using an unsupervised learning-based neural network learned using non-matching data. A method and apparatus for restoring a ring image are provided.

An image processing method according to an embodiment of the present invention includes receiving a microscope image; and deblurring the microscope image corresponding to the microscope image by deconvolution processing the microscope image using a neural network based on unsupervised learning learned using non-matching data ( deblurring) and restoring the image.

The neural network may be trained based on predefined cyclic loss and adversarial loss with respect to a microscope image and a high-resolution image corresponding to a deblurring image.

The neural network may include: a first neural network for outputting a first deblurring image corresponding to the first microscope image as an input; a second neural network for outputting a second microscope image corresponding to the first deblurred image as an input of the first deblurred image; a third neural network for outputting a third microscope image corresponding to the first high-resolution image by inputting a first high-resolution image corresponding to the deblurring image; a fourth neural network for outputting a second high-resolution image corresponding to the third microscope image by inputting the third microscope image; a first discriminator for discriminating the first microscope image and the third microscope image; and a second discriminator for discriminating the first high-resolution image and the first deblurring image.

Furthermore, in the image processing method according to an embodiment of the present invention, a cyclic loss between the first microscope image and the second microscope image, a cycle loss between the first high-resolution image and the second high-resolution image, and the second Adversarial loss of the first discriminator for discriminating the first microscopic image from the third microscope image and adversarial loss of the second discriminator for discriminating the first high-resolution image from the first deblurred image The method further comprises the step of learning the first neural network, the fourth neural network, the first discriminator, and the second discriminator based on the loss, wherein the restoring step is performed using the learned first neural network. The microscope image may be restored to a deblurring image corresponding to the microscope image.

The second neural network and the fourth neural network may include a single convolutional layer to model a point spread function.

The neural network may include any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.

An image processing method according to another embodiment of the present invention includes receiving a microscope image; and reconstructing the microscope image into a deblurring image corresponding to the microscope image by using a neural network that performs deconvolution based on unsupervised learning.

An image processing method according to an embodiment of the present invention includes a receiving unit for receiving a microscope image; and deblurring the microscope image corresponding to the microscope image by deconvolution processing the microscope image using a neural network based on unsupervised learning learned using non-matching data ( deblurring) includes a restoration unit that restores the image.

The neural network may be trained based on predefined cyclic loss and adversarial loss with respect to a microscope image and a high-resolution image corresponding to a deblurring image.

The neural network may include: a first neural network for outputting a first deblurring image corresponding to the first microscope image as an input; a second neural network for outputting a second microscope image corresponding to the first deblurred image as an input of the first deblurred image; a third neural network for outputting a third microscope image corresponding to the first high-resolution image by inputting a first high-resolution image corresponding to the deblurring image; a fourth neural network for outputting a second high-resolution image corresponding to the third microscope image by inputting the third microscope image; a first discriminator for discriminating the first microscope image and the third microscope image; and a second discriminator for discriminating the first high-resolution image and the first deblurring image.

The image processing apparatus may determine a cyclic loss between the first microscope image and the second microscope image, a cycle loss between the first high-resolution image and the second high-resolution image, and the first microscope image and the third microscope image. The first neural network based on the adversarial loss of the first discriminator for discriminating between to the fourth neural network, the first discriminator, and the second discriminator, and the restoration unit restores the microscope image to a deblurred image corresponding to the microscope image by using the learned first neural network. can do.

The second neural network and the fourth neural network may include a single convolutional layer to model a point spread function.

The neural network may include any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.

According to embodiments of the present invention, by deconvolution processing a microscope image processed by a point spread function using an unsupervised learning-based neural network learned using non-matching data, a high-resolution image corresponding to the microscope image is obtained. It can be restored as a deblurred image.

1 is a flowchart illustrating an image processing method according to an embodiment of the present invention.
2 shows an exemplary diagram of the overall network structure according to the method of the present invention.
FIG. 3 shows an exemplary diagram of a network structure of the _{generator G AB shown in FIG. 2 .}
4 shows an exemplary diagram of a network structure of a 3D discriminator.
FIG. 5 shows an exemplary diagram of a network structure of the multiple discriminator shown in FIG. 2 .
6 shows an exemplary view comparing the method of the present invention and the existing methods.
7 shows another exemplary diagram comparing the method of the present invention and the existing methods.
8 shows the configuration of an image processing apparatus according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, operations and/or elements mentioned. or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted.

Deconvolution microscopy has been used extensively to improve the resolution of large-area fluorescence microscopy. Conventional approaches that typically require point spread function (PSF) measurements or blind estimation are computationally expensive. Recently, an approach based on CNN is being sought as a fast and high-performance alternative.

In embodiments of the present invention, a microscope image processed by a point spread function (PSF) is deconvolved using an unsupervised learning-based neural network learned using non-matching data, thereby providing a high resolution corresponding to a microscope image. Its gist is to restore the deblurring image of

Here, the present invention deconvolves a microscope image by learning a microscope image using a clear high-resolution image, and deblurs the blur included in the microscope image through this to restore a high-resolution image, a deblurred image. can do.

와

를 트레이닝한다. 여기서, G_AB는 A를 입력으로 B를 출력하는 네트워크를 의미하고, G_BA는 B를 입력으로 A를 출력하는 네트워크를 의미할 수 있다. 그런 다음, 트레이닝 목표는 G_AB와 G_BA가 서로 역방향이라는 것, 즉

와

이다. 역방향 경로가 존재하기 때문에 생성적 적대 네트워크(GAN)에서 맵핑의 퇴화(degeneracy) 문제를 피할 수 있다. In the present invention, two networks A and B correspond to two different domains (microscopic image and clear high-resolution image).

Wow

to train Here, G _AB may mean a network that outputs B with A as an input, and G _BA may mean a network that outputs A with B as an input. Then, the training goal is that G _AB and G _BA are inverses of each other, i.e.

Wow

am. Because the reverse path exists, the problem of degeneracy of mapping in generative adversarial networks (GANs) can be avoided.

Here, the neural network used in the present invention may include a convolution framelet-based neural network, a neural network including a pooling layer and an unpooling layer, for example, U-Net. can

A convolutional framelet refers to a method of representing an input signal through a local basis and a non-local basis. A study on a new mathematical theory of deep convolutional framelets to reveal the black box characteristics of deep convolutional neural networks (Ye , JC., Han, Y., Cha, E.: Deep convolutional framelets: a general deep learning framework for inverse problems. .

1 is a flowchart illustrating an image processing method according to an embodiment of the present invention.

Referring to FIG. 1 , in the method according to an embodiment of the present invention, a microscope image is deconvolved using a neural network based on unsupervised learning learned using data that is not matched with the step of receiving a microscope image ( S110 ). (deconvolution), and reconstructing the microscope image into a deblurring image corresponding to the microscope image (S120).

Here, the microscope image received in step S110 may mean a point spread function (PSF)-processed image.

Here, the neural network used in step S120 may include a convolutional framelet-based neural network and a neural network including a pooling layer and an unpooling layer, for example, U-Net, as well as the method of the present invention. It can include all kinds of neural networks that can perform In addition, the neural network used in step S120 may be trained based on predefined cyclic loss and adversarial loss with respect to a microscope image and a high-resolution image corresponding to a deblurring image.

Furthermore, the neural network used in the method according to the present invention receives a first microscope image as an input and outputs a first deblurring image corresponding to the first microscope image as an input, and a first deblurring image as an input. A second neural network for outputting a second microscope image corresponding to the first deblurring image, a second neural network for outputting a third microscope image corresponding to the first high-resolution image by inputting a first high-resolution image corresponding to the deblurring image 3 neural networks, a fourth neural network outputting a second high-resolution image corresponding to the third microscope image as an input, a first discriminator for discriminating the first microscope image from the third microscope image, and a first high-resolution image A second discriminator for discriminating the image from the first deblurred image may be included.

Furthermore, the method according to the present invention distinguishes between a cyclic loss between a first microscope image and a second microscope image, a cyclic loss between the first high-resolution image and a second high-resolution image, and a first microscope image and a third microscope image. First neural networks to fourth neural networks, based on the adversarial loss of the first discriminator for The first discriminator and the second discriminator may be learned, and in step S120, the microscope image may be restored to a deblurring image corresponding to the microscope image using the learned first neural network.

The method according to the present invention will be described in detail below.

2 shows an exemplary diagram of the overall network structure according to the method of the present invention.

A shown in FIG. 2 denotes a blurred image domain, that is, a microscope image, B denotes a deblurred image domain from which blur is removed, and the generator G _AB receives the blurred image of A as an input and a deblurred image of B. It means a network that outputs , and the generator G _BA means a network that outputs a blur image of A by inputting a deblurring image. Here, the generator G _AB may mean mapping _{from A to B, and G BA} may mean mapping from B to A. In contrast to the existing cycle-GAN architecture for blind deconvolution, the present invention can use the PSF layer for _{the map G BA in which the actual PSF values are estimated from the training data.}

The generators (G _AB , G _BA ) are a neural network that reconstructs a microscope image like a deblurred image, respectively, and a neural network that processes a high-resolution image such as a deblurred image with a point spread function like a microscope image.

Regarding the generator, an optimized network is available for deconvolution of the microscopic image, including two adversarial discriminators multi-D _A and multi-D _B that distinguish the input image and the synthesized image from the generator. do.

That is, the framework shown in FIG. 2 includes a first generator (G _AB ), a second generator (G _BA ), a first discriminator (multi-D _A ), and a second discriminator (multi-D _B ), , because there is no exact matching image, two networks (G _AB , G _BA ) are trained simultaneously to match the two domain distributions. Here, the neural network shown in FIG. 2 may be trained based on predefined cyclic loss and adversarial loss with respect to a microscope image and a high-resolution image corresponding to a deblurring image.

The first generator (G _{AB ) outputs a deblurring image (X AB} ), which is a high-resolution image obtained by deconvolution of the blur from the microscope image (X _A ) by inputting the photographed microscope image (X _A ), and the second _{A deblurring image (X BAB} ), which is a high-resolution image obtained by deconvolution of blur _{from the microscope image (X BA} ) processed with the point spread function output by the generator (G _{BA ), is output.}

The second generator (G _{BA ) receives the high-resolution image (X B} ) corresponding to the photographed high-resolution image, that is, the deblurring image, and processes the high-resolution image (X _B ) with a point spread function, so that the high-resolution image (X _B) ) microscope, and output the image (X _BA) corresponding to the first and to the high-resolution image is de-blurred input image (X _AB) output by the first generator (G _AB) that the de-blurred image (X _AB) By processing with a diffusion function, _{a microscope image (X ABA} ) corresponding to the deblurring image (X _AB ) is output. That is, the second generator G _BA may be a neural network having a point spread function convolution function.

The first discriminator (multi-D _A ) is a neural network that discriminates the photographed microscope image (X _A ) and the microscope image (X _BA ) corresponding to the high-resolution image (X _B ), and the second discriminator (multi-D) _B ) is a neural network that distinguishes a deblurring image (X _AB ) which is a high-resolution image and a high-resolution image (X _{B ) corresponding to the deblurring image.}

Here, the first discriminator multi-D _A and the second discriminator multi-D _B may include three independent discriminators receiving patches having three different scales.

That is, in the present invention, cycle continuity is _{inputted to the G AB} network with deconvolution function starting from the _{damaged microscope image X A} , and the deconvolution-processed X _{AB result is inputted again to G BA} with PSF convolution function, so that X Back to _ABA. The convolutional neural network is trained in a recursive manner so that _{X A} and X _{ABA are equal to each other.} The unsupervised learning method trains multi-DA and multi-DBs using a least squares generative adversarial neural network (LS-GAN). fit

Although the use of a PSF layer may run the risk of reducing the generality of the PSF, the present invention does not require much sample-dependent PSF adaptation as the PSF is usually fixed in a typical microscope setup using predetermined optics. . Instead, the stability of the algorithm can be greatly improved by using an explicit PSF layer.

In addition, the discriminator D _A may be designed to discriminate (or discriminate) the input image, that is, the microscope image and the synthetically generated blur image, and similarly the discriminator D _B is the deblurring generated by the generator. It can be designed to distinguish between images and sharp image distributions. Here, for sharp image distribution, the present invention can use unmatched high-resolution images, which can be acquired in a super-resolution microscope or in commercially available deconvolution software. And, the present invention can train the generators and the discriminators at the same time in an alternating manner by solving the optimization problem as shown in Equation 1 below.

[Equation 1]

The loss function may be defined as in Equation 2 below.

[Equation 2]

Here, λ ₁ and λ ₂ denote _{hyperparameters} , LGAN , L _Cyclic denote adversarial loss and cyclic loss, and ||G _BA || ₁ means the L1-norm for normalization of the blur kernel.

Each of the hostile loss and the cycle loss will be described in detail as follows.

adversarial loss

The present invention can use a modified GAN loss using a log-likelihood function as in a least squares GAN (LSGAN). Specifically, the minimum and maximum optimization problem for GAN training may be composed of two separate minimization problems as shown in Equation 3 and Equation 4 below.

[Equation 3]

[Equation 4]

Here, P _A and P _B may mean distributions for domains A and B.

By optimizing the adversarial loss, the present invention can regulate the generators so that the generated sharp image volume is as realistic as possible, while at the same time the discriminators are optimized to distinguish the generated deconvolutional image volume from the actual high resolution image. Also the same hostility loss can be imposed on _{G BA} to deceive the generation of synthesized blur data.

cyclic loss

Although the mapping between (A) and (B) can be estimated by a well-trained adversarial network, it is still vulnerable to the mode failure problem, which brings many input images into fixed output images. Also, because of the large capacity of deep neural networks, the network can map (A) to random permutations of the output of domains (B) where the target distribution is likely to match. That is, hostile losses alone cannot guarantee the reversal of the two domains. Cycle coherence loss can be used to solve this problem. In the case of the present invention, the loss of cycle coherence supports a one-to-one correspondence between the blur image volume and the deconvolution volume. A specific cycle coherence loss can be expressed as in Equation 5 below.

[Equation 5]

Multi-patch GAN in cycle GAN (patchGAN)

로 정의한다. 여기서, f_i는 i번째 스케일 패치를 의미할 수 있다. 그런 다음 멀티스케일 패치들을 가지는 적대 손실은 아래 <수학식 6>과 같이 공식화될 수 있다.The present invention provides an improved model from the original cycle GAN using a multi-patch GAN with input patches of different sizes for the discriminator. Patch GANs typically focus on high-frequency structures by including local patches for the entire image. Since patches with different scales can include different high-frequency structures, the present invention can use multiple discriminators that take patches with different scales. Specifically, the present invention provides a multiple discriminator

to be defined as Here, f _i may mean an i-th scale patch. Then, the adversarial loss with multiscale patches can be formulated as Equation 6 below.

[Equation 6]

Here, N may mean the total number of scales.

L _GAN (G _AB , D _B , A, B) can be defined similarly.

network architecture

For the network architecture of the generator G _AB , the present invention may use a 3D-Unet structure as shown in FIG. 3 . The U-net structure for the generator G _AB can consist of a path that contracts and expands. The contraction path consists of repetitions of 3D convolutional layer-instance normalization layer-rectified linear unit (ReLU) layer blocks. Here, the 3D convolution layer may perform a 3D linear transform operation, the instance normalization layer may perform a normalization operation, and the ReLU layer may perform a nonlinear function operation. Here, the convolution layer may use a 3 Х 3 Х 3 convolution kernel, and the channel of the feature map in the first layer may be 64. In addition, U-Net includes a pooling layer and an unpooling layer, and may include a skip connection (or bypass connection) between an encoder and a decoder. Here, the skip connection can compensate for the high frequency lost during pooling.

의 네트워크 아키텍처는 도 4에 도시된 바와 같이, 3D 컨볼루션 레이어(Conv), 인스턴스 정규화 레이어(Instance Norm)와 ReLU 레이어로 구성된 4개의 모듈들로 구성될 수 있으며, 모든 컨볼루션 레이어는 스트라이드 2를 가지고 입력 볼륨을 다운 샘플링하고, 마지막 레이어에서의 출력 채널 수는 1이다. 멀티 판별기는 패치 GAN이며, 도 5에 도시된 바와 같이, 3 개의 다른 스케일로 패치를 처리하는 3개의 판별기들을 사용할 수 있다. 각 판별기는 다른 스케일들에서 패치들을 가져온다. 특히, D_full은 오리지널 크기로 패치들을 가지고 오고, D_half은 오리지널 패치 크기의 절반 크기로 무작위로 자른 패치(crop 0.5x)를 가지고 오며, D_quarter은 오리지널 패치 크기의 1/4 크기로 무작위로 자른 패치(crop 0.25x)를 가지고 온다.discriminator

As shown in FIG. 4, the network architecture of can be composed of four modules consisting of a 3D convolution layer (Conv), an instance normalization layer (Instance Norm), and a ReLU layer, and all convolutional layers are stride 2 downsamples the input volume, and the number of output channels in the last layer is 1. The multi discriminator is a patch GAN, and as shown in FIG. 5 , three discriminators that process patches at three different scales can be used. Each discriminator takes patches at different scales. Specifically, D _full brings the patches in their original size, D _half brings the patches randomly cropped to half the original patch size (crop 0.5x), and D _quarter brings them randomly to 1/4 the original patch size. Comes with a cropped patch (crop 0.25x).

On the other hand, the generator G _BA uses a single 3D convolutional layer to model the 3D blurring kernel. The size of the 3D PSF modeling layer may be selected according to the training set.

Way

The present invention may use epifluorescence (EPF) samples of tubulin having a size of 512 Х 512 Х 30 for training. For the unmatched sharp image volume, the present invention may use a deblurred image, that is, a deblurred image generated using commercial software, for example, AutoQuant X3 (Media Cybernetics, Rockville). Volume depth can be increased to 64 with reflective padding. Due to memory limitations, the volume can be partitioned into 64 Х 64 Х 64 patches. For data augmentation, rotations, flips, transforms and scales can be imposed on the input patches. An Adam optimizer having β ₁ =0.9 and β ₂ =0.999 may be used to optimize Equation 1 above, and the learning rate may be 0.0001. The learning rate decreases linearly after 40 epochs, and the total number of epochs may be 200. To reduce model oscillations, discriminators can use a history of volumes created from a frame buffer containing 50 previously created volumes.

In the present invention, the patch can be standardized and set to [0,1], and the PSF size can be set to 20. The present invention can be implemented in Python with Tensorflow, and the GeForce GTX 1080 Ti GPU can be used for both network training and testing. In order to validate the performance of the method according to the present invention, we use AutoQuant X3, supervised learning and other CNNs, e.g., the _{original cycle GAN with both multi-PatchGAN and G BA} of non-PSF layers, and compared with commercial deconvolution methods. do. In contrast to the method using a regular CNN, the model of the present invention can make the training process much easier by using only a single PSF modeling layer in _{G BA.} For supervised learning networks, the present invention trains a 3D U-Net with matched label data in AutoQuant X3, as L1-loss makes less blurring. All reconstruction results can be post-processed for better visualization than adaptive histogram normalization.

6 shows an exemplary diagram comparing the method of the present invention and the existing methods, and shows the result of a transverse view.

As shown in FIG. 6 , the results of comparing the methods of the present invention with AutoQuantX3 (commercial deconvolution software), Supervised (supervised learning), and Non-PSF layer ( _{using other convolutional neural networks in general for G BA) methods} As can be seen through, it can be seen that the method of the present invention restores the results more clearly and structurally more clearly than other methods.

7 shows another exemplary view comparing the method of the present invention and the existing methods, and shows the results for the sagittal view of the sample 3 shown in FIG. Here, the method shown in FIG. 7 has the same order as the method shown in FIG. 6 .

As shown in FIG. 7 , it can be seen that the method according to the present invention restores the results clearly and structurally more clearly than other existing methods even in the sagittal view.

As such, the method according to an embodiment of the present invention deconvolutionally processes a microscope image using an unsupervised learning-based neural network learned using non-matching data, thereby converting a microscope image to a high-resolution image corresponding to the microscope image. It can be restored as a deblurred image.

A high-quality image can be restored by removing noise from a low-dose X-ray computed tomography image.

In addition, the method according to an embodiment of the present invention improves restoration performance by constructing a neural network using a local basis and a non-local basis according to a convolution framelet. In this case, the amount of computation required for restoration can be reduced and restoration speed can be improved.

8 shows the configuration of an image processing apparatus according to an embodiment of the present invention, and shows the configuration of the apparatus performing the above-described FIGS. 1 to 7 .

Referring to FIG. 8 , an image processing apparatus 800 according to an embodiment of the present invention includes a receiving unit 810 and a restoration unit 820 .

The receiver 810 receives a microscope image.

The restoration unit 820 reconstructs the microscope image into a deblurring image corresponding to the microscope image by deconvolutionally processing the microscope image using an unsupervised learning-based neural network learned using the non-matching data. do.

Here, the neural network may include a convolutional framelet-based neural network and a neural network including a pooling layer and an unpooling layer, for example, U-Net.

Furthermore, the neural network used in the device according to the present invention receives a first microscope image as an input and outputs a first deblurring image corresponding to the first microscope image as an input, and a first deblurring image as an input. A second neural network for outputting a second microscope image corresponding to the first deblurring image, a second neural network for outputting a third microscope image corresponding to the first high-resolution image by inputting a first high-resolution image corresponding to the deblurring image 3 neural networks, a fourth neural network outputting a second high-resolution image corresponding to the third microscope image as an input, a first discriminator for discriminating the first microscope image from the third microscope image, and a first high-resolution image A second discriminator for discriminating the image from the first deblurred image may be included.

Furthermore, the neural network used in the apparatus according to the present invention is to be learned based on predefined cyclic loss and adversarial loss for a high-resolution image corresponding to a microscope image and a deblurring image. and may be learned from the network structure shown in FIG. 2 .

Although the description of the device of FIG. 8 is omitted, each component constituting FIG. 8 may include all the contents described in FIGS. 1 to 7 , which is apparent to those skilled in the art.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (13)

Receiving a microscope image in the receiving unit; and
By deconvolution processing the microscope image using an unsupervised learning-based neural network learned using non-matching data in the restoration unit, the microscope image is deblurred corresponding to the microscope image. Restoring to a ring (deblurring) image
including,
The receiving step
Receive a microscope image processed by a point spread function,
The neural network is
An image processing method learned through unsupervised learning using the non-matching data with another neural network that processes a microscope image deconvolved by the neural network with the point spread function.
According to claim 1,
The neural network is
An image processing method, characterized in that the microscopic image and the high-resolution image corresponding to the deblurring image are learned based on predefined cyclic loss and adversarial loss.
According to claim 1,
The neural network is
a first neural network for outputting a first deblurring image corresponding to the first microscope image by inputting a first microscope image;
a second neural network for outputting a second microscope image corresponding to the first deblurred image as an input of the first deblurred image;
a third neural network for outputting a third microscope image corresponding to the first high-resolution image by inputting a first high-resolution image corresponding to the deblurring image;
a fourth neural network for outputting a second high-resolution image corresponding to the third microscope image by inputting the third microscope image;
a first discriminator for discriminating the first microscope image and the third microscope image; and
A second discriminator for discriminating the first high-resolution image and the first deblurring image
An image processing method comprising a.
4. The method of claim 3,
In an image processing apparatus, a cyclic loss between the first microscope image and the second microscope image, a cycle loss between the first high-resolution image and the second high-resolution image, and the first microscope image and the third microscope image Based on the adversarial loss of the first discriminator for discriminating and the adversarial loss of the second discriminator for discriminating the first high-resolution image and the first deblurring image, the first neural network to learning the fourth neural network, the first discriminator, and the second discriminator
further comprising,
The restoration step
and reconstructing the microscope image into a deblurring image corresponding to the microscope image by using the learned first neural network.
4. The method of claim 3,
The second neural network and the fourth neural network are
and a single convolutional layer to model the point spread function.
According to claim 1,
The neural network is
An image processing method comprising any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.
Receiving a microscope image in the receiving unit; and
Restoring the microscope image to a deblurring image corresponding to the microscope image by using a neural network that performs deconvolution processing based on unsupervised learning in a restoration unit
including,
The receiving step
Receive a microscope image processed by a point spread function,
The neural network is
An image processing method learned through unsupervised learning with another neural network that processes a microscope image deconvolved by the neural network with the point spread function.
a receiver for receiving a microscope image; and
Deblurring the microscope image corresponding to the microscope image by deconvolution processing the microscope image using a neural network based on unsupervised learning learned using non-matching data. ) Restoration unit that restores images
including,
the receiving unit
Receive a microscope image processed by a point spread function,
The neural network is
An image processing apparatus that is learned through unsupervised learning using the non-matching data with another neural network that processes a microscope image deconvolved by the neural network with the point spread function.
9. The method of claim 8,
The neural network is
An image processing apparatus, characterized in that the microscopic image and the high-resolution image corresponding to the deblurring image are learned based on predefined cyclic loss and adversarial loss.
9. The method of claim 8,
The neural network is
a first neural network for outputting a first deblurring image corresponding to the first microscope image by inputting a first microscope image;
a second neural network for outputting a second microscope image corresponding to the first deblurred image as an input of the first deblurred image;
a third neural network for outputting a third microscope image corresponding to the first high-resolution image by inputting a first high-resolution image corresponding to the deblurring image;
a fourth neural network for outputting a second high-resolution image corresponding to the third microscope image by inputting the third microscope image;
a first discriminator for discriminating the first microscope image and the third microscope image; and
A second discriminator for discriminating the first high-resolution image and the first deblurring image
An image processing apparatus comprising a.
11. The method of claim 10,
The image processing device
a cyclic loss between the first microscope image and the second microscope image, a cyclic loss between the first high-resolution image and the second high-resolution image, and the first microscope image and the third microscope image The first neural network to the fourth neural network based on an adversarial loss of a first discriminator and an adversarial loss of the second discriminator for discriminating the first high-resolution image and the first deblurring image learning the network, the first discriminator and the second discriminator;
the restoration unit
and reconstructing the microscope image into a deblurred image corresponding to the microscope image by using the learned first neural network.
11. The method of claim 10,
The second neural network and the fourth neural network are
and a single convolutional layer to model the point spread function.
9. The method of claim 8,
The neural network is
An image processing apparatus comprising any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.

Images