KR20200126881A - Method for processing microscopy image using artificial neural network with point spread function layer and apparatus therefore - Google Patents

Abstract

Disclosed are a microscope image processing method using a neural network having a point spread function layer and a device therefor. According to an embodiment of the present invention, the image processing method comprises the following steps of: receiving a microscope image; and restoring the microscope image to a deblurring image corresponding to the microscope image by deconvolution-processing the microscope image by using an unsupervised learning based neural network trained by using mismatched data, wherein the neural network can be trained based on predefined cyclic loss and adversarial loss for high-resolution images corresponding to the microscope image and the deblurring image.

Description

A method and apparatus for processing microscopic images using a neural network with a point spread function layer {METHOD FOR PROCESSING MICROSCOPY IMAGE USING ARTIFICIAL NEURAL NETWORK WITH POINT SPREAD FUNCTION LAYER AND APPARATUS THEREFORE}

The present invention relates to an image processing method and apparatus using a neural network having a point spread function (PSF) layer, and more specifically, a microscope image processed by a point spread function is processed using mismatched data. A method for reconstructing a high-resolution deblurring image corresponding to a microscope image by processing a deconvolution using a neural network based on learned unsupervised learning and its apparatus will be.

In a fluorescence microscope, the diffraction of light from a given optic degrades the resolution of the image. Many optimization-based deconvolution algorithms have been developed to improve resolution. A conventional technique of an embodiment proposes a blind deconvolution method by solving a joint minimization problem in order to estimate an unknown blur kernel and image when a point spread function (PSF) measurement cannot be used. Another embodiment of the related art has proposed an improved version of blind deconvolution using total variance (TV) normalization.

Recently, convolutional neural networks (CNNs) have been widely used to increase the suitability of optical microscopes without changing hardware. The conventional technique of an embodiment uses a deep neural network to improve an optical microscope, and improves spatial resolution in a wide field of view and depth. The technique of another embodiment of the related art used a deep convolutional neural network capable of training on simulation data or experimental measurements in order to obtain super-resolution images in a localization microscopy. Another embodiment of the related art has proposed a CNN method capable of recovering isotope resolution from anisotropic data. In addition, generative adversarial networks (GANs) have attracted a lot of attention to the inverse problem by providing a method using unlabeled data to train deep neural networks. In another embodiment of the related art, a conditional GAN and DeblurGAN for motion blurring using content loss have been proposed. However, since this GAN approach often generates artificial features due to mode collapse, a cycle-sustaining hostile network (Cycle-GAN) that imposes a one-to-one response also affects image reconstruction.

However, this Cycle-GAN approach usually requires two large-capacity generators, which are often difficult to train with a small number of training data.

Embodiments of the present invention deconvolution processing a microscope image processed by a point spread function using an unsupervised learning-based neural network learned using unmatched data, thereby providing a high-resolution deblurrer corresponding to the microscope image. A method and apparatus for reconstructing a ring image are provided.

An image processing method according to an embodiment of the present invention includes receiving a microscope image; And deconvolution processing the microscope image using a neural network based on unsupervised learning learned using unmatched data, thereby deblurring the microscope image corresponding to the microscope image ( deblurring) image restoration.

The neural network may be learned based on a predefined cyclic loss and an adversarial loss for a high-resolution image corresponding to a microscope image and a deblurring image.

The neural network may include: a first neural network configured to output a first deblurred image corresponding to the first microscope image as an input of a first microscope image; A second neural network for outputting a second microscope image corresponding to the first deblurred image by receiving the first deblurred image; A third neural network configured to output a third microscope image corresponding to the first high-resolution image by receiving a first high-resolution image corresponding to the deblurring image; A fourth neural network for outputting a second high-resolution image corresponding to the third microscope image as an input of the third microscope image; A first discriminator for distinguishing the first microscope image from the third microscope image; And a second discriminator for distinguishing the first high-resolution image from the first deblurring image.

Further, the image processing method according to an embodiment of the present invention includes a cyclic loss between the first microscope image and the second microscope image, a cycle loss between the first high-resolution image and the second high-resolution image, and the second 1 Adversarial loss of the first discriminator for discriminating between the microscopic image and the third microscopic image and hostility of the second discriminator for discriminating the first high-resolution image and the first deblurring image Further comprising learning the first neural network to the fourth neural network, the first discriminator, and the second discriminator based on the loss, and the restoring step is performed by using the learned first neural network. The microscope image may be restored to a deblurring image corresponding to the microscope image.

The second neural network and the fourth neural network may include a single convolutional layer to model a point spread function.

The neural network may include any one of a neural network based on a convolution framelet and a neural network including a pooling layer and an unpooling layer.

An image processing method according to another embodiment of the present invention includes the steps of receiving a microscope image; And restoring the microscope image into a deblurring image corresponding to the microscope image by using a neural network that performs deconvolution based on unsupervised learning.

An image processing method according to an embodiment of the present invention includes: a receiving unit for receiving a microscope image; And deconvolution processing the microscope image using a neural network based on unsupervised learning learned using unmatched data, thereby deblurring the microscope image corresponding to the microscope image ( deblurring) includes a reconstruction unit for restoring an image.

The neural network may be learned based on a predefined cyclic loss and an adversarial loss for a high-resolution image corresponding to a microscope image and a deblurring image.

The neural network may include: a first neural network configured to output a first deblurred image corresponding to the first microscope image as an input of a first microscope image; A second neural network for outputting a second microscope image corresponding to the first deblurred image by receiving the first deblurred image; A third neural network configured to output a third microscope image corresponding to the first high-resolution image by receiving a first high-resolution image corresponding to the deblurring image; A fourth neural network for outputting a second high-resolution image corresponding to the third microscope image as an input of the third microscope image; A first discriminator for distinguishing the first microscope image from the third microscope image; And a second discriminator for distinguishing the first high-resolution image from the first deblurring image.

The image processing apparatus includes a cyclic loss between the first microscope image and the second microscope image, a cycle loss between the first high-resolution image and the second high-resolution image, the first microscope image and the third microscope image The first neural network based on an adversarial loss of the first discriminator for discriminating between and an adversarial loss of the second discriminator for discriminating between the first high-resolution image and the first deblurring image. To the fourth neural network, the first discriminator, and the second discriminator, and the restoration unit restores the microscope image to a deblurring image corresponding to the microscope image using the learned first neural network can do.

The second neural network and the fourth neural network may include a single convolutional layer to model a point spread function.

The neural network may include any one of a neural network based on a convolution framelet and a neural network including a pooling layer and an unpooling layer.

According to embodiments of the present invention, by deconvolution processing a microscope image processed by a point spread function using an unsupervised learning-based neural network learned using unmatched data, a high resolution corresponding to the microscope image It can be restored as a deblurred image.

1 is a flowchart illustrating an operation of an image processing method according to an embodiment of the present invention.
2 shows an exemplary diagram of the overall network structure according to the method of the present invention.
FIG. 3 shows an exemplary diagram of the network structure of the generator G _AB shown in FIG. 2.
4 shows an exemplary diagram of a network structure of a 3D discriminator.
5 shows an exemplary diagram of a network structure of the multiple discriminator shown in FIG. 2.
6 shows an exemplary diagram comparing the method of the present invention and the existing method.
7 shows another exemplary diagram comparing the method of the present invention and the existing method.
8 shows a configuration of an image processing apparatus according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, only these embodiments make the disclosure of the present invention complete, and common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have, and the invention is only defined by the scope of the claims.

The terms used in this specification are for describing exemplary embodiments, and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements, and/or elements, steps, actions and/or elements mentioned. Or does not exclude additions.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same elements in the drawings, and duplicate descriptions for the same elements are omitted.

The deconvolution microscope has been used extensively to improve the resolution of a large-area fluorescence microscope. Conventional approaches that typically require point spread function (PSF) measurements or blind estimation are computationally expensive. Recently, the CNN-based approach is being sought as a fast and high-performance alternative.

Embodiments of the present invention, by deconvolution processing a microscope image processed by a point spread function (PSF) using an unsupervised learning-based neural network learned using unmatched data, high resolution corresponding to the microscope image The gist is to restore the deblurred image of.

Here, in the present invention, by learning a microscope image using a clear high-resolution image, the microscope image is deconvoluted, and the blur included in the microscope image is deblurred to restore a high-resolution image, which is a deblurred image. can do.

와

를 트레이닝한다. 여기서, G_AB는 A를 입력으로 B를 출력하는 네트워크를 의미하고, G_BA는 B를 입력으로 A를 출력하는 네트워크를 의미할 수 있다. 그런 다음, 트레이닝 목표는 G_AB와 G_BA가 서로 역방향이라는 것, 즉

와

이다. 역방향 경로가 존재하기 때문에 생성적 적대 네트워크(GAN)에서 맵핑의 퇴화(degeneracy) 문제를 피할 수 있다. In the present invention, A and B correspond to two different domains (microscopic image and clear high-resolution image).

Wow

To train. Here, G _AB may mean a network that outputs B as an input, and G _BA may mean a network that outputs A as an input. Then, the training goal is that G _AB and G _BA are opposite each other, i.e.

Wow

to be. Since there is a reverse path, it is possible to avoid the problem of mapping degeneracy in a generative hostile network (GAN).

Here, the neural network used in the present invention includes a neural network based on a convolution framelet, a neural network including a pooling layer and an unpooling layer, for example, U-Net. I can.

Convolutional framelet refers to a method of expressing an input signal through a local basis and a non-local basis, and a study on a new mathematical theory of the deep convolutional framelet to reveal the black box characteristics of a deep convolutional neural network (Ye , JC., Han, Y., Cha, E.: Deep convolutional framelets: a general deep learning framework for inverse problems.SIAM Journal on Imaging Sciences 11(2), 991-1048(2018)). .

1 is a flowchart illustrating an operation of an image processing method according to an embodiment of the present invention.

Referring to FIG. 1, the method according to an embodiment of the present invention is a method of receiving a microscope image (S110) and deconvolution of a microscope image using an unsupervised learning-based neural network learned using unmatched data. By performing (deconvolution) processing, the microscopic image is restored to a deblurred image corresponding to the microscopic image (S120).

Here, the microscope image received in step S110 may mean an image processed with a point spread function (PSF).

Here, the neural network used in step S120 may include a neural network based on a convolutional framelet and a neural network including a pooling layer and an unpooling layer, for example, U-Net, as well as the method of the present invention. It may include all types of neural networks capable of performing. Also, the neural network used in step S120 may be learned based on predefined cyclic loss and adversarial loss for high-resolution images corresponding to the microscope image and the deblurring image.

Furthermore, the neural network used in the method according to the present invention is a first neural network for outputting a first deblurred image corresponding to the first microscope image as an input, and a first deblurring image as an input. A second neural network that outputs a second microscope image corresponding to the first deblurred image, and a second neural network that outputs a third microscope image corresponding to the first high-resolution image by inputting a first high-resolution image corresponding to the deblurred image. 3 neural network, a fourth neural network that outputs a second high-resolution image corresponding to a third microscope image by inputting a third microscope image, a first discriminator for distinguishing the first microscope image from the third microscope image, and a first high resolution It may include a second discriminator that distinguishes between the image and the first deblurred image.

Further, the method according to the present invention is characterized by a cyclic loss between the first microscope image and the second microscope image, a cycle loss between the first high-resolution image and the second high-resolution image, and distinguish between the first and third microscope images. The first to fourth neural networks, based on the adversarial loss of the first discriminator and the adversarial loss of the second discriminator for discriminating the first high-resolution image from the first deblurred image. The first discriminator and the second discriminator may be learned, and in step S120, the microscope image may be restored to a deblurring image corresponding to the microscope image by using the learned first neural network.

The method according to the present invention will be described in detail below.

2 shows an exemplary diagram of the overall network structure according to the method of the present invention.

A shown in FIG. 2 refers to a blur image domain, that is, a microscope image, B refers to a deblurred image domain from which blur is removed, and a generator G _AB is a deblurred image of B by inputting the blurred image of A. It means a network that outputs A, and the generator G _BA means a network that outputs the blurred image of A as an input of the deblurred image. Here, the generator G _AB may mean mapping from A to B, and G _BA may mean mapping from B to A. In contrast to the existing cycle-GAN architecture for blind deconvolution, the present invention can use the PSF layer for the map G _BA where actual PSF values are estimated from training data.

Generators (G _AB , G _BA ) are neural networks that restore microscope images like deblurring images and neural networks that process high-resolution images such as deblurred images with a point spread function like microscope images.

Regarding the generator, an optimized network can be used to deconvolve the microscope image, and it includes two discriminators, multi-D _A and multi-D _B , that distinguish the input image from the composite image from the generator. do.

That is, the framework shown in FIG. 2 includes a first generator (G _AB ), a second generator (G _BA ), a first discriminator (multi-D _A ) and a second discriminator (multi-D _B ), , Since there is no image that matches exactly, two networks (G _AB , G _BA ) are _trained at the same time to match two domain distributions. Here, the neural network shown in FIG. 2 may be learned based on predefined cyclic loss and adversarial loss for high-resolution images corresponding to the microscope image and the deblurring image.

The first generator (G _AB ) inputs the photographed microscope image (X _A ) and outputs a deblurring image (X _AB ), which is a high-resolution image obtained by deconvolving the blur from the microscope image (X _A ), and the second A deblurring image (X _BAB ), which is a high-resolution image obtained by deconvolving blur from the microscope image (X _BA ) processed with the point spread function output by the generator (G _BA ), is output.

A second generator (G _BA) is by treatment by a high-resolution image (X _B) corresponding to the recorded high resolution image that is, de-blurring image as an input a high-resolution image (X _B) to the point spread function, a high resolution image (X _B ) Outputs the microscope image (X _BA ) corresponding to ), and points the deblurring image (X _AB ) by inputting the deblurring image (X _AB ), which is a high-resolution image output by the first generator (G _AB ). By processing with a diffusion function, a microscope image (X _ABA ) corresponding to the deblurring image (X _AB ) is output. That is, the second generator G _BA may be a neural network having a point spread function convolution function.

The first discriminator (multi-D _A ) is a neural network that distinguishes between the photographed microscope image (X _A ) and the microscope image (X _BA ) corresponding to the high-resolution image (X _B ), and the second discriminator (multi-D _B ) is a neural network that distinguishes between a deblurring image (X _AB ) that is a high-resolution image and a high-resolution image (X _B ) corresponding to the deblurring image.

Here, the first discriminator (multi-D _A ) and the second discriminator (multi-D _B ) may include three independent discriminators that receive patches having three different scales.

That is, in the present invention, the cycle persistence starts from the damaged microscope image X _A and is input to the G _AB network with the deconvolution function, and the deconvoluted X _AB result is again input to the G _BA with the PSF convolution function. Return to _ABA . The convolutional neural network is trained in a circular fashion so that X _A and X _ABA are equal to each other. The unsupervised learning method trains multi-DA and multi-DBs using a least squares generative adversarial neural network (LS-GAN), and LS-GAN calculates the probability distribution of target domain data and input domain data that do not match each other. Match

The use of the PSF layer may have the risk of reducing the generality of the PSF, but the present invention does not require much sample dependent PSF adaptation as the PSF is generally fixed in a typical microscope setup using a predetermined optical device. . Instead, the use of an explicit PSF layer can greatly improve the stability of the algorithm.

In addition, the discriminator D _A can be designed to discriminate (or discriminate) the input image, that is, the microscopic image and the synthetically generated blur image, and similarly, the discriminator D _B is the deblurring generated by the generator. It can be designed to distinguish between images and sharp image distributions. Here, for clear image distribution, the present invention can use unmatched high-resolution images, which can be obtained in super-resolution microscopes or commercially available deconvolution software. In addition, the present invention solves the optimization problem as shown in Equation 1 below, so that generators and discriminators can be simultaneously trained in an alternating manner.

[Equation 1]

The loss function can be defined as in Equation 2 below.

[Equation 2]

Here, λ ₁ and λ ₂ refer to _{hyperparameters} , L _GAN and L _Cyclic refer to adversarial loss and cyclic loss, and ||G _BA || ₁ stands for the L1-norm for normalization of the blur kernel.

Each of the hostile losses and cycle losses will be described in detail as follows.

Adversarial loss

The present invention can use the modified GAN loss by using the log-likelihood function using the least squares loss as in the least squares GAN (LSGAN). Specifically, the minimum and maximum optimization problem for GAN training may consist of two separate minimization problems as shown in Equation 3 and Equation 4 below.

[Equation 3]

[Equation 4]

Here, P _A and P _B may mean distributions for domains A and B.

The present invention can regulate the generators so that the generated clear image volume becomes as realistic as possible by optimizing the hostile loss, and at the same time, the discriminators are optimized to distinguish between the generated deconvolutional image volume and the actual high-resolution image. Also, the same hostile losses can be imposed on the G _BA to deceive the generation of synthesized blur data.

Cyclic loss

The mapping between (A) and (B) can be estimated by a well-trained hostile network, but it is still vulnerable to the problem of mode failure, which brings many input images as fixed output images. In addition, since the capacity of the deep neural network is large, the network can map (A) to an output random permutation of domain (B) in which the target distribution is likely to match. In other words, the reversal of the two domains cannot be guaranteed only by hostile losses. Cycle coherence loss can be used to solve this problem. In the case of the present invention, the cycle coherence loss supports a one-to-one correspondence between a blur image volume and a deconvolution volume. The specific cycle coherence loss can be expressed as in Equation 5 below.

[Equation 5]

Cycle GAN's multi-patch GAN (patchGAN)

로 정의한다. 여기서, f_i는 i번째 스케일 패치를 의미할 수 있다. 그런 다음 멀티스케일 패치들을 가지는 적대 손실은 아래 <수학식 6>과 같이 공식화될 수 있다.The present invention provides an improved model from the original cycle GAN using a multi-patch GAN having input patches of different sizes for the discriminator. Patch GANs typically focus on high-frequency structures by including local patches for the entire image. Since patches having different scales may include different high-frequency structures, the present invention can use multiple discriminators that take patches with different scales. Specifically, the present invention

Is defined as Here, f _i may mean the i-th scale patch. Then, the hostile loss with multiscale patches can be formulated as shown in Equation 6 below.

[Equation 6]

Here, N may mean the total number of scales.

L _GAN (G _AB , D _B , A, B) can be defined similarly.

Network architecture

For the network architecture of the generator G _AB , the present invention can use a 3D-Unet structure as shown in FIG. 3. The U-net structure for the generator G _AB can consist of paths that contract and expand. The contraction path is composed of repetitions of 3D convolutional layer-instance normalized layer-ReLU (rectified linear unit) layer blocks. Here, the 3D convolution layer may perform a 3D linear transform operation, the instance normalization layer may perform a normalization operation, and the ReLU layer may perform a nonlinear function operation. Here, a 3 Х 3 Х 3 convolution kernel may be used as the convolution layer, and a channel of the feature map may be 64 in the first layer. In addition, the U-Net may include a pooling layer and an unpooling layer, and may include a skip connection (or bypass connection) between the encoder and the decoder. Here, the skip connection can compensate for high frequencies lost during pooling.

의 네트워크 아키텍처는 도 4에 도시된 바와 같이, 3D 컨볼루션 레이어(Conv), 인스턴스 정규화 레이어(Instance Norm)와 ReLU 레이어로 구성된 4개의 모듈들로 구성될 수 있으며, 모든 컨볼루션 레이어는 스트라이드 2를 가지고 입력 볼륨을 다운 샘플링하고, 마지막 레이어에서의 출력 채널 수는 1이다. 멀티 판별기는 패치 GAN이며, 도 5에 도시된 바와 같이, 3 개의 다른 스케일로 패치를 처리하는 3개의 판별기들을 사용할 수 있다. 각 판별기는 다른 스케일들에서 패치들을 가져온다. 특히, D_full은 오리지널 크기로 패치들을 가지고 오고, D_half은 오리지널 패치 크기의 절반 크기로 무작위로 자른 패치(crop 0.5x)를 가지고 오며, D_quarter은 오리지널 패치 크기의 1/4 크기로 무작위로 자른 패치(crop 0.25x)를 가지고 온다.Discriminator

The network architecture of FIG. 4 can be composed of 4 modules consisting of a 3D convolution layer (Conv), an instance normalization layer (Instance Norm), and a ReLU layer, and all convolution layers have stride 2 And down-samples the input volume, and the number of output channels in the last layer is 1. The multi discriminator is a patch GAN, and as shown in FIG. 5, three discriminators that process patches in three different scales can be used. Each discriminator fetches patches at different scales. In particular, D _full brings patches in the original size, D _half brings patches that are randomly cut to half the size of the original patch (crop 0.5x), and D _quarter brings patches randomly to 1/4 the size of the original patch. Bring a cropped patch (crop 0.25x).

On the other hand, generator G _BA uses a single 3D convolution layer to model the 3D blurring kernel. The size of the 3D PSF modeling layer may be selected according to the training set.

Way

The present invention may use EPF (epifluorescence) samples of tubulin having a size of 512 Х 512 Х 30 for training. For a non-matching clear image volume, the present invention can use a de-blurred image, that is, a deblurred image, generated using a commercial software, for example, AutoQuant X3 (Media Cybernetics, Rockville). The volume depth can be increased to 64 with reflective padding. Due to memory limitations, the volume can be divided into 64 x 64 x 64 patches. In order to increase the data, rotation, flip, transformation and scale may be imposed on the input patch. The Adam optimizer with β ₁ =0.9 and β ₂ =0.999 may be used to optimize Equation 1, and the learning rate may be 0.0001. The learning rate decreases linearly after 40 epochs, and the total number of epochs may be 200. To reduce model oscillation, the discriminators can use the history of volumes created from a frame buffer containing 50 volumes created previously.

In the present invention, the patch can be standardized and set to [0,1], and the PSF size can be set to 20. The present invention can be implemented in Python with Tensorflow, and the GeForce GTX 1080 Ti GPU can be used for both network training and testing. In order to verify the performance of the process according to the invention, compared with AutoQuant X3, supervised learning and another CNN, for example, the non-PatchGAN the original commercial deconvolution method using the cycle GAN of a combination of G _BA of PSF layer do. In contrast to the method using a regular CNN, the model of the present invention can make the training process much easier by using only a single PSF modeling layer in G _BA . For the supervised learning network, the present invention trains a 3D U-Net with the matched label data of AutoQuant X3, as the L1-loss makes less blurring. All reconstruction results can be post-processed for better visualization than adaptive histogram equalization.

6 shows an exemplary diagram comparing the method of the present invention with the existing method, and shows the result of a transverse view.

As shown in Figure 6, AutoQuantX3 (commercial deconvolution software), Supervised (supervised learning), Non-PSF layer (General different convolutional neural network used in the G _BA) method and the result of comparing the process of the invention As can be seen through, it can be seen that the method of the present invention restores the results more clearly and structurally than other methods, and the connectivity is distinct.

FIG. 7 shows another exemplary diagram comparing the method of the present invention with the conventional method, and shows the result of the sagittal view of Sample 3 shown in FIG. 6. Here, the method illustrated in FIG. 7 has the same order as the method illustrated in FIG. 6.

As shown in FIG. 7, it can be seen that the method according to the present invention restores the result more clearly and structurally than other conventional methods even in a sagittal view, and the connection is distinct.

As described above, the method according to an embodiment of the present invention deconvolves a microscope image using an unsupervised learning-based neural network learned using unmatched data, so that the microscope image is a high-resolution image corresponding to the microscope image. It can be restored as a deblurred image.

Noise can be removed from a low-dose X-ray computed tomography image to restore a high-quality image.

In addition, the method according to an embodiment of the present invention improves reconstruction performance by configuring a neural network using a local basis and a non-local basis according to a convolution framelet. And, it is possible to reduce the amount of computation required for restoration and to improve the restoration speed.

8 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention, and illustrates a configuration of an apparatus performing the above-described FIGS. 1 to 7.

Referring to FIG. 8, an image processing apparatus 800 according to an embodiment of the present invention includes a receiving unit 810 and a reconstructing unit 820.

The receiving unit 810 receives a microscope image.

The reconstruction unit 820 deconvolutions the microscope image using an unsupervised learning-based neural network learned using unmatched data, thereby restoring the microscope image to a deblurring image corresponding to the microscope image. do.

Here, the neural network may include a neural network based on a convolutional framelet, and a neural network including a pooling layer and an unpooling layer, for example, U-Net.

Further, the neural network used in the apparatus according to the present invention is a first neural network that outputs a first deblurred image corresponding to the first microscope image as an input, and a first deblurred image as an input. A second neural network that outputs a second microscope image corresponding to the first deblurred image, and a second neural network that outputs a third microscope image corresponding to the first high-resolution image by inputting a first high-resolution image corresponding to the deblurred image. 3 neural network, a fourth neural network that outputs a second high-resolution image corresponding to a third microscope image by inputting a third microscope image, a first discriminator for distinguishing the first microscope image from the third microscope image, and a first high resolution It may include a second discriminator that distinguishes between the image and the first deblurred image.

Furthermore, the neural network used in the apparatus according to the present invention is to be learned based on predefined cyclic loss and adversarial loss for high-resolution images corresponding to the microscope image and the deblurring image. And can be learned in the network structure shown in FIG. 2.

Even though the description of the device of FIG. 8 is omitted, each component constituting FIG. 8 may include all the contents described in FIGS. 1 to 7, which is obvious to those skilled in the art.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It can be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (13)

Receiving a microscope image; And
Deblurring the microscope image corresponding to the microscope image by deconvolution processing the microscope image using a neural network based on unsupervised learning learned using unmatched data. ) Steps to restore to an image
Image processing method comprising a.
The method of claim 1,
The neural network is
An image processing method, characterized in that the high-resolution images corresponding to the microscope image and the deblurring image are learned based on predefined cyclic loss and adversarial loss.
The method of claim 1,
The neural network is
A first neural network for outputting a first deblurred image corresponding to the first microscope image as an input of a first microscope image;
A second neural network for outputting a second microscope image corresponding to the first deblurred image by receiving the first deblurred image;
A third neural network for outputting a third microscope image corresponding to the first high-resolution image by receiving a first high-resolution image corresponding to the deblurring image;
A fourth neural network for outputting a second high-resolution image corresponding to the third microscope image as an input of the third microscope image;
A first discriminator for distinguishing between the first microscope image and the third microscope image; And
A second discriminator to distinguish between the first high-resolution image and the first deblurring image
Image processing method comprising a.
The method of claim 3,
Cyclic loss between the first microscope image and the second microscope image, a cycle loss between the first high-resolution image and the second high-resolution image, the first microscope image and the third microscope image The first neural network to the fourth neural network based on an adversarial loss of a first discriminator and an adversarial loss of the second discriminator for distinguishing the first high-resolution image from the first deblurring image. Learning a network, the first discriminator and the second discriminator
Including more,
The restoring step
And reconstructing the microscope image into a deblurring image corresponding to the microscope image by using the learned first neural network.
The method of claim 3,
The second neural network and the fourth neural network
An image processing method comprising a single convolutional layer to model a point spread function.
The method of claim 1,
The neural network is
An image processing method comprising a neural network based on a convolution framelet and a neural network including a pooling layer and an unpooling layer.
Receiving a microscope image; And
Restoring the microscope image to a deblurring image corresponding to the microscope image using a neural network that processes deconvolution based on unsupervised learning
Image processing method comprising a.
A receiver for receiving a microscope image; And
Deblurring the microscope image corresponding to the microscope image by deconvolution processing the microscope image using a neural network based on unsupervised learning learned using unmatched data. ) Restoration unit to restore image
Image processing apparatus comprising a.
The method of claim 8,
The neural network is
An image processing apparatus, characterized in that the high-resolution images corresponding to the microscope image and the deblurring image are learned based on predefined cyclic loss and adversarial loss.
The method of claim 8,
The neural network is
A first neural network for outputting a first deblurred image corresponding to the first microscope image as an input of a first microscope image;
A second neural network for outputting a second microscope image corresponding to the first deblurred image by receiving the first deblurred image;
A third neural network configured to output a third microscope image corresponding to the first high-resolution image by receiving a first high-resolution image corresponding to the deblurring image;
A fourth neural network for outputting a second high-resolution image corresponding to the third microscope image as an input of the third microscope image;
A first discriminator for distinguishing between the first microscope image and the third microscope image; And
A second discriminator to distinguish between the first high-resolution image and the first deblurring image
Image processing apparatus comprising a.
The method of claim 10,
The image processing device
Cyclic loss between the first microscope image and the second microscope image, a cycle loss between the first high-resolution image and the second high-resolution image, the first microscope image and the third microscope image The first neural network to the fourth neural network based on an adversarial loss of a first discriminator and an adversarial loss of the second discriminator for distinguishing the first high-resolution image from the first deblurring image. Learning a network, the first discriminator and the second discriminator,
The restoration unit
And reconstructing the microscope image into a deblurring image corresponding to the microscope image by using the learned first neural network.
The method of claim 10,
The second neural network and the fourth neural network
An image processing apparatus comprising a single convolutional layer to model a point spread function.
The method of claim 8,
The neural network is
An image processing apparatus comprising a neural network based on a convolution framelet and a neural network including a pooling layer and an unpooling layer.

Images