KR102410380B1 - Apparatus and method for reconstructing noiseless phase image from garbor hologram based on deep-learning - Google Patents

Abstract

A method of generating a phase image of a computing device according to an embodiment of the present disclosure is a method of generating a phase image in which a processor performs at least a part of each step. and generating a second phase image from which noise has been removed by inputting a phase image into a machine learning-based learning model, wherein the learning model is an off-axis digital hologram. It is possible to include a generative model learned based on a conditional generative adversarial network (cGAN) by using a phase image restored from and noise-removed as a label.

Description

Apparatus and method for reconstructing noise-free phase images from heirlooms based on deep learning {APPARATUS AND METHOD FOR RECONSTRUCTING NOISELESS PHASE IMAGE FROM GARBOR HOLOGRAM BASED ON DEEP-LEARNING}

The present disclosure relates to an apparatus and method for removing noise from a phase image reconstructed from a digital hologram, in particular, a Garbor hologram.

Most cells are transparent or semi-transparent, so it remains a challenge to acquire a two-dimensional high-resolution live cell image based on light microscopy technology. However, quantitative phase imaging techniques, including off-axis quantitative phase digital holographic microscopy (QP-DHM), provide accurate and quantitative visualization and kinetic information of cells in a non-invasive manner. can provide

That is, the non-optical digital holographic microscope is one of the technologies that can provide quantitative marker-free information on the shape and content of the sample to be observed. A user can acquire a quantitative phase image by recording the spatial interference pattern between a coherent or semi-coherent reference wave and an object wave passing through live cells.

The non-optical digital holographic microscope is a method in which the object wave and the reference wave are not located exactly on the same optical axis, and a small angle (tilt) is inserted between the object wave and the reference wave through spatial filtering in the spectral region to form a twin-image (twin-image). ) and the real image. However, the non-optical digital holographic microscope has a difficulty in that it requires the optical path length matching between the object wave and the reference wave and precise control of optical elements in order to accurately obtain a quantitative phase image with respect to the shape and information of the sample. In addition, since the non-optical axis digital holographic microscope records twin images and real images in different frequency domains, it is impossible to use the entire band of the camera and has high price.

Unlike the conventional non-optical digital holographic microscope, there is a Garbor holographic microscopy using the same optical axis. The heirloom holographic microscope does not require precise setup and is economically useful, but as a disadvantage, there is a problem in that an in-focus real image and an unfocused twin image overlap.

As a prior art for solving the problem of such heirloom holograms, there is a mechanical approach (prior art 1) that performs phase shifting by adding specific optical elements, but there is a problem in that several holograms need to be photographed. In addition, as another phase recovery method, a method of recording additional intensity information at different distances (prior art 2) exists, but the result is hindered even if the equipment must be kept stationary or small vibrations occur in the optical setup There is this. Other methods also have various problems, such as being difficult to implement in a real environment.

Therefore, there is a need for a technology capable of easily and accurately resolving noise such as twin images in a phase image reconstructed from an heirloom hologram.

Prior Art 1: J. Barton, "Removing multiple scattering and twin images from holographic images", Phys. Rev. Lett. Vol. 67 No. 22 (published on November 25, 1991) Prior Art 2: Y. Zhang et al., "Reconstruction of in-line digital holograms from two intensity measurements", Opt. Lett. Vol. 29 No. 15, (issued on April 1, 2004)

Another embodiment of the present disclosure provides an apparatus and method for efficiently and accurately removing noise from a phase image reconstructed from a heirloom hologram in real time.

Another embodiment of the present disclosure provides an apparatus and method for efficiently removing noise from a phase image reconstructed from a heirloom hologram with a small amount of computation.

Another embodiment of the present disclosure provides an apparatus and method for removing noise from a phase image reconstructed from an accurate heirloom hologram based on deep learning.

Another embodiment of the present disclosure provides an apparatus and method for training a learning model for removing noise of a phase image reconstructed from an accurate heirloom hologram based on deep learning.

The object of the present invention is not limited to the above-mentioned problems, and other objects and advantages of the present invention that are not mentioned may be understood by the following description, and will be more clearly understood by the embodiments of the present invention. It will also be appreciated that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations thereof indicated in the claims.

A method of generating a phase image of a computing device according to an embodiment of the present disclosure is a method of generating a phase image in which a processor performs at least a part of each step. and generating a second phase image from which noise has been removed by inputting a phase image into a machine learning-based learning model, wherein the learning model is an off-axis digital hologram. It is possible to include a generative model learned based on a conditional generative adversarial network (cGAN) by using a phase image restored from and noise-removed as a label.

A computing device according to an embodiment of the present disclosure includes a processor and a memory electrically connected to the processor and storing at least one code executed by the processor, and the memory is when the processor is executed through the processor Code for receiving a hologram (Garbor hologram) and inputting the first phase image restored from the heirloom hologram into a machine learning-based learning model to generate a second phase image from which noise has been removed , and the learning model is based on a conditional generative adversarial network (cGAN) with a phase image restored from an off-axis digital hologram and removed from noise as a label. It may include a learned generative model.

The learning model training apparatus according to an embodiment of the present disclosure includes a processor and a memory that is electrically connected to the processor, and stores at least one code executed by the processor, and when the memory is executed through the processor, the processor A learning model including a generative model and a discriminant model is trained based on a conditional generative adversarial network (cGAN), and noise from a first phase image reconstructed from a Garbor hologram is code that causes the generative model trained to generate the removed second phase image and the discriminant model trained to discriminate whether the generated second phase image is a real phase image or a generated phase image, competing adversarially with each other to be trained have.

An apparatus and method for removing noise from a phase image according to an embodiment of the present disclosure enable noise removal with a small amount of computation to be performed, thereby reducing hardware specifications required for an apparatus for performing noise removal and performing noise removal in real time can do.

The apparatus and method for removing noise from a phase image according to an embodiment of the present disclosure can improve the quality of a phase image by solving a problem that noise removal of a phase image reconstructed from a conventional heirloom hologram is not properly performed.

The apparatus and method for removing noise from a phase image according to an embodiment of the present disclosure may accurately and efficiently remove noise from a twin image and a zero-order part.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating a configuration in which a non-optical-axis digital hologram acquisition device is set and changed to acquire an heirloom hologram for evaluation of a learning model according to an embodiment of the present disclosure.
2 is a diagram illustrating an environment in which an apparatus for removing noise on a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure is driven.
3 is a diagram illustrating an environment in which an apparatus for removing noise on a phase image reconstructed from a heirloom hologram according to another embodiment of the present disclosure is driven.
4 is a diagram illustrating a configuration of an apparatus for removing noise on a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure.
5 is a view for explaining input and output of a learning model of an apparatus for removing noise on a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure.
6 is a diagram illustrating a configuration of an apparatus for training a learning model for performing noise removal on a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure.
7 is a diagram illustrating a method of training a model for generating a learning model for denoising a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure.
8 is a view for explaining a discrimination model of a learning model for performing noise removal on a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure.
9 is a view for explaining a method of generating training data of a learning model for performing noise removal on a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure.
10 is a view for explaining a training result of a learning model for performing noise removal on a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure.
11 is a flowchart illustrating a method for performing noise removal on a phase image reconstructed from a heirloom hologram according to an embodiment of the present disclosure.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

A configuration of a digital holographic microscope (quantitative phase digital holographic microscopy: QP-DHM) for acquiring heirloom holograms used to evaluate a training result of a learning model in an embodiment of the present disclosure will be described with reference to FIG. 1 . 1 is a configuration in which a heirloom hologram can be obtained by illuminating only the object wave sample by adding a shutter to the reference wave path of the non-optical axis digital holographic microscope.

The digital holographic microscope may acquire phase information of an object such as a cell. A digital holographic microscope can be operated based on a Mach-Zender interferometer. The coherent laser source of the digital holographic microscope can split the light source into an object wave and a reference wave by using a beam splitter having a small inclination angle between two beams. The object wave may illuminate a specimen and create an object wave front. A microscope objective lens (MO) magnifies the object wavefront, and the object and reference wavefronts may be combined by a beam collector at the exit of an interferometer to generate a hologram. The generated hologram may be transmitted to a computing device controlling the digital holographic microscope, and the digital holographic microscope or digital holographic microscope controlling computing device may perform filtering processing in the frequency domain of the hologram to remove unwanted signals. .

The digital holographic microscope or digital holographic microscope control computing device may illuminate the filtered hologram using a replica of the reference wave and reconstruct the hologram by Fresnel approximation.

When a plane wave propagates to an object, such as a cell, and the plane wave passes through the cell, output waves having different phases are output according to the thickness or refractive index of the object, such as a cell, and output wave compared to the phase of the input plane wave An image having the amount of phase change as a pixel value may be referred to as a phase image.

Since those skilled in the art can clearly understand the digital holographic imaging technique, further detailed description will be omitted.

An environment for driving the computing device 100a, which is a device for removing noise on a phase image reconstructed from a heirloom hologram, according to an embodiment of the present disclosure will be described with reference to FIG. 2 .

An environment for driving the computing device 100a that performs noise removal according to an embodiment of the present disclosure may include the digital hologram acquisition device 200 and the learning model training device 300a.

The digital hologram acquisition device 200 may be a heirloom holographic microscope device.

The computing device 100a reconstructs (restores) the hologram received from the digital hologram acquisition device 200 to generate a phase image, and inputs the phase image including noise into a machine learning-based learning model to generate noise. A phase image from which is removed may be generated.

In this specification, a learning model that outputs a phase image from which noise has been removed by inputting a phase image containing noise as an example is described. When the image is restored to a hologram as part of the implementation, the twin image and the zero order part are removed from the phase image restored from the hologram containing noise that overlaps the real image region in the frequency domain as input. Those skilled in the art can clearly see that it can be implemented as a learning model that outputs a phase image.

The phase image from which the noise has been removed is a frequency domain twin image (twin image, twin part, for example, twin part in FIG. 9) and zero order (for example, zero- in FIG. 9) in the phase image reconstructed from the hologram. order part) may be a phase image from which parts have been removed. In the description below, the twin image and the zero-order part mean noise identified in the frequency domain of the phase image.

In another embodiment, the computing device 100a receives the phase image from which the hologram is reconstructed, and inputs the phase image including the twin image and the noise of the zero-order part into the machine learning-based learning model to remove the noise. can create

In this embodiment, it is assumed that the computing device 100a is implemented separately from the digital hologram acquiring device 200 , but the computing device 100a can be implemented as a part of the digital hologram acquiring device 200 .

That is, the digital hologram acquisition device 200 may be a digital holographic microscope or a device for controlling the digital holographic microscope. In another embodiment, the digital hologram acquiring device 200 may be a single device in which a digital holographic microscope and a device for controlling the digital holographic microscope are combined or a system consisting of a plurality of devices. The device for controlling the digital holographic microscope is a computing device including a processor, a memory, a non-transitory storage medium, and the like, and may be a personal computer (PC), a laptop, a tablet computer, or a server device. You can control the imaging of the microscope and the generation of holograms. In addition, a digital hologram can be reconstructed or a phase image can be generated.

After the hologram is generated, reconstruction of the digital hologram or generation of a phase image may be performed in the digital hologram acquisition device 200, the computing device 100a, or the machine learning model training device 300a, and the subject is not particularly limited.

The computing device 100a may receive a learning model for performing noise removal from the learning model training device 300a through the network 400 .

In one embodiment, the learning model training apparatus 300a is a phase image restored from a digital hologram obtained from a non-optical-axis digital holographic microscope (a twin image and a zero-order part of a frequency domain are removed from a phase image to form a real part) Learning with training data using a phase image containing only a region as a label and a phase image containing noise (twin images and zero-order parts) acquired from the same sample from which the image was acquired as input You can train the model. The construction of the training data is described in detail below.

A learning model that generates a phase image from which noise such as twin images and zero-order parts have been removed is a learning model based on a conditional generative adversarial network (cGAN) as shown in FIG. It may include a generative model trained to generate and a discriminant model trained to determine whether the generated phase image is an actual phase image or a generated phase image. The noise removal may be performed based on a generative model trained to generate a phase image among them, and the computing device 100a may receive only the generative model from the learning model training apparatus 300a. Generative and discriminant models can be trained to compete adversarially with each other. The structure of the learning model will be described in detail below with reference to FIG. 7 .

An environment for driving the computing device 100b, which is a device for removing noise on a phase image according to another embodiment of the present disclosure, will be described with reference to FIG. 3 .

The computing device 100b is a learning model trained to generate a phase image from which noise has been removed by inputting a phase image in which noise such as twin images and zero-order parts are overwrapped in the same frequency region as the real image region as input Models can be included. In an embodiment, the computing device 100b includes a learning model based on a conditional generative adversarial neural network including the generation model and a discrimination model trained to determine whether the generated phase image is an actual phase image or a generated phase image. can

In an embodiment, the computing device 100b receives a phase image including noise from the external device 500 through the network 400 , inputs the received phase image to a learning model, and outputs the generated phase image to the corresponding external device. It can be transmitted to the device or to a separate external device.

In an embodiment, the computing device 100b is configured such that the input data of the phase image including the twin image and the zero-order part overlapping the same frequency region as the real image region and the noise-removed phase image are training data composed of labels. A training model for training a learning model or a learning model for performing noise removal of a phase image through the network 400 may be received from a separate external device.

A configuration of the computing device 100 as an apparatus for removing noise on a phase image according to an embodiment of the present disclosure will be described with reference to FIG. 4 .

The computing device 100 may include a communication unit supporting machine to machine (M2M) communication, device to device (D2D) communication, and the like, and the communication unit may include a wireless communication unit or a wired communication unit.

The wireless communication unit may include at least one of a mobile communication module, a wireless Internet module, a short-range communication module, and a location information module.

The mobile communication module includes technical standards or communication methods for mobile communication (eg, GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000)), EV-DO ( Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term (LTE-A) Evolution-Advanced), etc.), transmits and receives radio signals to and from at least one of a base station, an external terminal, and a server on a mobile communication network.

As wireless Internet technology, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), and the like.

The short-range communication module is for short-range communication, and includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field (NFC). Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology may be used to support short-distance communication.

The computing device 100 may be, for example, a computing device that is not fixed to a specific type, such as a personal computer, a smart phone, a tablet, a desktop computer, and a NAS (Network Attached Storage), and is implemented as a fixed device and a movable device. can be

The processor 130 may include all kinds of devices capable of processing data, for example, an MCU, a GPU, and an AI accelerator chip. Here, the 'processor' may refer to, for example, a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or command included in a program.

As an example of the data processing apparatus embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but the scope of the present disclosure is not limited thereto.

The computing device 100 receives a phase image containing noise as an input to generate a twin image and a phase image from which zero-order parts are removed. It may include a trained generative model 150 as part of it. In an embodiment, the computing device 100 includes a learning model based on a conditional generative adversarial neural network including the generation model and a discrimination model trained to determine whether the generated phase image is an actual phase image or a generated phase image. can In another embodiment, the computing device 100 may train the learning model, and may use the learning processor 140 for the learning model.

The processor 130 may perform a process for generating a noise-removed phase image by inputting a phase image including noise into a learning model trained based on a conditional generative adversarial neural network or a generation model of the learning model. .

In the present specification, an artificial neural network whose parameters are estimated by being trained using training data may be referred to as a learning model or a trained model.

The learning model 150 may be implemented in hardware, software, or a combination of hardware and software, and includes one or more instructions (code, code) and parameters constituting the learning model when a part or all of the learning model is implemented in software. data may be stored in the memory 120 . Memory 120 may include one or more transitory or non-transitory storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and the like.

The computing device 100 may display, on the display 170 , the noise-removed phase image generated based on the learning model 150 , or transmit it to an external device through the communication unit 110 .

In addition, the computing device 100 may receive commands from the user including the interface 160 , and may transmit output information to the user. The interface 160 may include various input means such as a keyboard, a mouse, a touch screen, a microphone, and a camera, and various output means such as a monitor, a speaker, and a display. The interface 160 may include at least one of a wired/wireless data port, a memory card port, and an audio or video input/output (I/O) port. In response to the connection of the external device to the interface, the computing device 100 may perform appropriate control related to the connected external device.

An input and output of a learning model of an apparatus for removing noise on a phase image according to an embodiment of the present disclosure will be described with reference to FIG. 5 .

The input of the learning model may be a phase image 510 reconstructed from a heirloom hologram, and the phase image 510 may be a phase image in which noise of a twin image and a zero-order part is overlapped in the same frequency region as the real image region.

The learning model may output the twin image overlapping the real image region in the frequency domain and the phase image 520 in which noise of the zero-order part is removed. .

A configuration of the learning model training apparatus 300, which is an apparatus for training a learning model for noise removal of a phase image, according to an embodiment of the present disclosure will be described with reference to FIG. 6 .

The processor 330 of the training model training apparatus 300 generates a phase image without noise and twin images from a phase image containing noise restored from a heirloom hologram in which a twin image and a zero-order noise part in the frequency domain are superimposed on the real region. In order to obtain , the learning model can be trained based on the conditional generative adversarial neural network using training data composed of labels with the phase image restored after filtering only the real image region in the frequency domain of the non-optical hologram of the same sample. In an embodiment, the learning model may be trained using the machine learning-based learning processor 340 implemented suitable for machine learning. The construction of the training data is described in detail below.

The learning model training apparatus 300 may train the learning model by receiving the training data through the communication unit 310 or by loading the training data from the database of the memory 320 .

The learning model training apparatus 300 may transmit a training model trained based on the conditional generative adversarial neural network to an external device, or may transmit a trained generative model as a part of the learning model to an external device.

The learning model can be based on an artificial neural network, and in the artificial neural network, artificial neurons (nodes) that form a network through the combination of synapses change the strength of synaptic bonds through learning, thereby providing an overall model with problem-solving ability. can mean

The artificial neural network may include a plurality of layers, and each of the layers may include a plurality of neurons. Also, the artificial neural network may include neurons and synapses connecting neurons.

A multi-layered artificial neural network consists of an input layer, one or more hidden layers, and an output layer. The hidden layer may be plural.

The input layer is a layer that receives external data. The number of neurons in the input layer is the same as the number of input variables, and the hidden layer is located between the input layer and the output layer. do. The output layer receives a signal from the hidden layer and outputs an output value based on the received signal. The input signal between neurons is multiplied by each connection strength (weight) and then summed. If the sum is greater than the neuron threshold, the neuron is activated and the output value obtained through the activation function is output.

A generative adversarial neural network is a machine learning method in which two different artificial intelligences, a generator and a discriminator, compete to improve performance.

In this case, the generative model is a model that creates new data, and new data can be created based on the original data, and the discriminant model is a model that recognizes patterns in the data. It can play a role in determining whether or not it is new data.

And the generative model learns by receiving the data that did not deceive the discriminant model, and the discriminant model can learn by receiving the data deceived from the generative model. Accordingly, the generative model can evolve to deceive the discriminant model as best as possible, and the discriminant model can evolve competitively with each other to distinguish the original data and the data generated by the generative model well.

The learning model may be implemented in hardware, software, or a combination of hardware and software, and when a part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in the memory 320 .

A structure of a generative model among conditional generative adversarial neural network-based learning models for performing noise removal on a phase image according to an embodiment of the present disclosure will be described with reference to FIG. 7 .

A generation model having a structure as shown in FIG. 7 of a learning model according to an embodiment of the present disclosure may perform noise removal of a phase image. The generative model receives the phase image 710 restored from the heirloom hologram in which the real image region in the frequency domain overlaps the twin image and the zero-order part, and generates a phase image 720 from which noise has been removed, as shown in FIG. Discriminant models can train generative models accurately. The generated phase image 720 and the actual phase image are fed as inputs to the discriminant model, which is trained to determine whether the input image is a generated phase image or a real phase image. In an embodiment, the generative model and the discriminant model may be composed of multi-layered layers including a convolution layer having a filter map of a certain size, a BatchNorm layer, and a ReLU active layer layer. A generative model may include multiple down-sampling layers and up-sampling layers.

In an embodiment, the generative model may transfer information in the high frequency domain from an input path to an output path through a skip connection.

1 = 0.5 및

2 = 0.999의 파라미터들과 적응적 모멘텀을 가진 최적화 프로세스로서 Adam 솔버가 훈련에 사용될 수 있다.In general, if downsampling is performed before upsampling, a lot of information of the original image may be lost and the output may be blurred. For example, the skip connection between input and output reduces the blurring effect of the image generated by connecting the information of the ith layer of the downsampling process with the information of the nith layer of the upsampling process in the general form of U-net. can In an embodiment, the discrimination model may receive a 256x256 input image, pass through four convolutional layers, and derive a 16x16 sized image. The discrimination model of FIG. 8 may be trained to learn to distinguish a real patch from a fake patch, and may be trained to evaluate an image in the form of a patch, which is a partial form of an input phase image. In one embodiment, the adaptive momentum and

1 = 0.5 and

As an optimization process with parameters of 2 = 0.999 and adaptive momentum, the Adam solver can be used for training.

Training data of a conditionally generative adversarial neural network-based learning model trained to remove noise included in a phase image of a heirloom hologram, which is an embodiment of the present disclosure, will be described with reference to FIG. 9 .

The learning model can be trained in a state where a label for an input image is given, where the label is the correct answer (or result value) that the artificial neural network must infer when the input image is input to the artificial neural network of the learning model. can mean

The present invention uses unique training data to train a learning model.

In order to train a conventional learning model, a phase image reconstructed from a noise-containing hologram of the same sample (which may be a cell or tissue) obtained from an organism and a phase image reconstructed from a noise-removed hologram are required. However, as described above, when training a learning model to remove the noise of the phase image restored from the heirloom hologram, the non-optical axis hologram image is again applied to the same sample from which the heirloom hologram is acquired to obtain the phase image from which the noise has been removed. However, there is a problem in that even the same sample cannot be used as a label due to cell migration due to a difference in hologram acquisition time.

In order to solve this problem, unique training data for training a learning model will be described as an embodiment of the present invention.

The input data is obtained by transforming the non-optical hologram 910 image into the frequency domain (FFT: Fast Fourier Transform) 920, and then frequency shifting (FS: frequency shifting) the twin image and real image parts to zero order parts in the frequency domain. Then, it may be a hologram 940 that is converted into a spatial domain and combined, or a phase image generated from the hologram 940 by a method such as Fresnel propagation and Fresnel approximation.

That is, when the non-optical axis hologram is expressed as an object wave (O) and a reference wave (R), it can be expressed as in Equation 1, where O* and R* are complex conjugates.

In the non-optical hologram, since the real image region and the twin image are separated from the zero-order part and the frequency region due to the angle of the reference wave and the object wave, the heirloom hologram ( 940) can be artificially generated to generate the input data of the learning model.

Here, H1 and H2 are after filtering only the real part and twin images in the frequency domain (IH) of the non-optical axis hologram 910, respectively, and then frequency shifting them to the frequency domain position of the zero order part (FS) , means an image 930 transformed back into the spatial domain. Heirloom holographic image ( GH) 940 can be obtained. In the heirloom holographic image (GH) 940 generated through this process, the twin image, the real part, and the zero-order part are overlapped in the same region in the frequency domain, and this can be used as input data of the learning model.

The label of the training data used in the generating model of the learning model may be a phase image obtained by separating and reconstructing only the real part region from the non-optical hologram 910 used to obtain the heirloom hologram image (GH) 940 .

An experimental result of performing noise removal on a phase image based on a learning model according to an embodiment of the present disclosure will be described with reference to FIG. 10 .

The performance of the present disclosure using a noisey phase image restored from a heirloom hologram obtained by blocking a reference wave with respect to the non-optical-axis hologram and the non-optical-axis hologram acquired in the non-optical digital holographic microscope device as shown in FIG. 1 . Performance of the present disclosure can be evaluated.

10 is a diagram illustrating a phase image output by inputting a heirloom hologram obtained by blocking a reference wave for the same sample from which a non-optical axis hologram is obtained to a learning model according to an embodiment of the present disclosure after filtering the real image region of the non-optical axis hologram; The results of evaluating the performance of the present disclosure in comparison with the reconstructed phase image are shown.

10( e ) shows the mean corpuscular-hemoglobin (MCH) in blood cells calculated from quantitative phase images reconstructed from non-optical-axis holograms for a red blood cell sample and heirloom holograms output from the learning model according to an embodiment of the present disclosure; It shows the correlation of the average hemoglobin amount in blood cells measured from the noise-free phase image reconstructed from

10(f) shows a dry mass calculated from a quantitative phase image reconstructed from a non-optical-axis hologram for a red blood cell sample and a noise-free phase image reconstructed from a heirloom hologram output from a learning model according to an embodiment of the present disclosure. It shows the correlation of the calculated volume.

According to FIGS. 10(e) and 10(f) and the statistical results of Table 1 below (results performed on more than 70 samples), statistical indicators calculated from phase images reconstructed from non-optical holograms for cell analysis are It can be seen that high correlation and similar values to the results obtained from the noise-free phase image restored from the heirloom hologram output from the learning model according to the embodiment of the present disclosure show high correlation and similar values, and thus the output from the learning model according to the embodiment of the present disclosure It can be seen that the noise-free phase image reconstructed from the heirloom hologram can be used instead of the phase image reconstructed from the non-optical axis hologram.

A method of denoising a phase image reconstructed from a heirloom hologram of a computing device according to an embodiment of the present disclosure will be described with reference to FIG. 11 .

The computing device receives a digital holographic image heirloom hologram or a phase image of the heirloom hologram (S110), and based on the conditional generative adversarial neural network, the twin image and the zero-order part are superimposed on the same region as the real image in the frequency domain. A phase image from which noise has been removed may be generated by inputting it to a generative model included in the learned learning model (S120).

The generation model of the learning model used in the method of performing noise removal on the phase image of the computing device is the phase image obtained by moving the real part and twin image of the phase image of the non-optical digital hologram to the same frequency domain as input, and the non-optical axis Twin images of digital holograms and phase images with zero order parts removed can be labeled. The learning model may include a discrimination model trained to determine whether the generated phase image is an actual phase image or a generated phase image. Generative and discriminant models can be trained to compete adversarially with each other.

The present disclosure described above can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is this. In addition, the computer may include a processor of each device.

Meanwhile, the program may be specially designed and configured for the present disclosure, or may be known and used by those skilled in the art of computer software. Examples of the program may include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification of the present disclosure (especially in the claims), the use of the term “above” and similar referential terms may be used in both the singular and the plural. In addition, when a range is described in the present disclosure, each individual value constituting the range is described in the detailed description of the invention as including the invention to which individual values within the range are applied (unless there is a description to the contrary). same as

The steps constituting the method according to the present disclosure may be performed in an appropriate order unless the order is explicitly stated or there is no description to the contrary. The present disclosure is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terminology (eg, etc.) in the present disclosure is merely for the purpose of describing the present disclosure in detail, and the scope of the present disclosure is limited by the examples or exemplary terms unless defined by the claims. it's not going to be In addition, those skilled in the art will appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or their equivalents.

Accordingly, the spirit of the present disclosure should not be limited to the above-described embodiments, and the scope of the spirit of the present disclosure as well as the claims to be described later are equivalent to or equivalently changed therefrom. will be said to belong to

510, 710: phase image restored from heirloom hologram
520, 720: phase image with noise removed
910: non-optical axis hologram
920: non-optical axis hologram converted to frequency domain
930: Real real area, zero order part, spatial area hologram of twin image
940: A hologram in which the real region, zero order part, and twin image are superimposed on the same spatial region

Claims (18)

A method for generating a phase image in which a processor of a computing device performs at least a portion of each step, the method comprising:
receiving a heirloom hologram (Garbor hologram); and
and generating a second phase image from which noise has been removed by inputting a first phase image restored from the heirloom hologram into a machine learning-based learning model,
The learning model is a generation learned based on a conditional generative adversarial network (cGAN) with a phase image restored from an off-axis digital hologram and noise removed as a label. including the model,
A method of generating a phase image of a computing device.
According to claim 1,
The second phase image is a phase image from which twin parts and zero order parts are removed,
A method of generating a phase image of a computing device.
According to claim 1,
The generation model receives a phase image obtained by moving the real part and twin images of the phase image of the non-optical axis digital hologram to the same frequency domain as input,
The label of the generative model is a phase image obtained by removing a twin image and a zero-order part of a phase image restored from the non-optical axis digital hologram,
A method of generating a phase image of a computing device.
According to claim 1,
The generative model is a generative model learned by inputting a phase image obtained by superimposing noise of a phase image restored from the non-optical-axis digital hologram in the same frequency domain as an input,
A method of generating a phase image of a computing device.
According to claim 1,
The generative model is a generative model learned by inputting the phase image restored from the heirloom hologram obtained from the non-optical axis digital hologram as an input,
A method of generating a phase image of a computing device.
According to claim 1,
The learning model is
The generative model trained to generate a noise-removed phase image from the phase image reconstructed from the heirloom hologram and the discrimination model trained to determine whether the phase image generated by the generative model is an actual phase image or a generated phase image are antagonistic to each other A learning model trained by competing with
A method of generating a phase image of a computing device.
processor; and
It is electrically connected to the processor and includes a memory in which at least one code (code) executed by the processor is stored,
The memory, when executed by the processor, causes the processor to:
A Garbor hologram is provided, and a first phase image restored from the heirloom hologram is input to a machine learning-based learning model to generate a second phase image from which noise has been removed. save the code to
The learning model is a generation learned based on a conditional generative adversarial network (cGAN) with a phase image restored from an off-axis digital hologram and noise removed as a label. including the model,
computing device.
8. The method of claim 7,
The second phase image is a phase image from which twin parts and zero order parts are removed,
computing device.
8. The method of claim 7,
The generation model receives a phase image obtained by moving the real part and twin images of the phase image restored from the non-optical digital hologram to the same frequency domain as input,
The label of the generative model is a phase image obtained by removing a twin image and a zero-order part of a phase image restored from the non-optical axis digital hologram,
computing device.
8. The method of claim 7,
The generative model is a generative model learned by inputting a phase image obtained by superimposing noise of a phase image restored from the non-optical-axis digital hologram in the same frequency domain as an input,
computing device.
8. The method of claim 7,
The generative model is a generative model learned by inputting the phase image restored from the heirloom hologram obtained from the non-optical axis digital hologram as an input,
computing device.
8. The method of claim 7,
The learning model is
The generative model trained to generate a noise-removed phase image from the phase image reconstructed from the heirloom hologram and the discrimination model trained to determine whether the phase image generated by the generative model is an actual phase image or a generated phase image are antagonistic to each other A learning model trained by competing with
computing device.
8. The method of claim 7,
the memory stores code that, when executed by the processor, causes the processor to receive the first phase image from a first external device over a network, and transmit the generated second phase image to the first external device; to save more,
computing device.
8. The method of claim 7,
wherein the memory further stores code that, when executed by the processor, causes the processor to receive the generation model from a second external device;
computing device.
processor; and
It is electrically connected to the processor and includes a memory in which at least one code (code) executed by the processor is stored,
When the memory is executed through the processor, the processor trains a learning model including a generative model and a discriminant model based on a conditional generative adversarial network (cGAN), and recovers from a Garbor hologram. The generative model is trained to generate a second phase image in which noise is removed from the first phase image, and the discrimination trained to determine whether the generated second phase image is an actual phase image or a generated phase image. stores the code that causes the models to compete against each other to be trained;
Learning model training device.
16. The method of claim 15,
The first phase image is a phase image restored from a heirloom hologram obtained from an off-axis digital hologram,
Learning model training device.
16. The method of claim 15,
The first phase image is a phase image obtained by moving the real part and twin images of the phase image restored from the non-optical axis digital hologram to the same frequency domain,
Learning model training device.
16. The method of claim 15,
The generation model receives a phase image obtained by moving a real part and a twin part of a phase image restored from an off-axis digital hologram to the same frequency domain as input,
The label of the generative model is a phase image obtained by removing a twin image and a zero-order part of a phase image restored from the non-optical axis digital hologram,
Learning model training device.

Images