KR20220021368A - Tomography image processing method using neural network based on unsupervised learning to remove missing cone artifacts and apparatus therefor - Google Patents

Abstract

Disclosed are a method for processing a tomography image using neural network based on unsupervised learning for removing missing cone artifacts and a device thereof. According to an embodiment of the present invention, the method comprises: a step of receiving a tomography image; and a step of generating projection images from which the missing cone artifact is removed by removing the cone artifact missing from the tomography image using an unsupervised learning based neural network generated based on an optimal transport theory.

Description

TOMOGRAPHY IMAGE PROCESSING METHOD USING NEURAL NETWORK BASED ON UNSUPERVISED LEARNING TO REMOVE MISSING CONE ARTIFACTS AND APPARATUS THEREFOR}

The present invention relates to a tomography image processing technology using an unsupervised learning-based neural network, and more particularly, by removing missing cone artifacts from a tomographic image using an unsupervised learning-based neural network. , to a tomography image processing method capable of generating high-quality tomographic images, and an apparatus therefor.

Optical diffraction tomography (ODT) is a technique for reconstructing a three-dimensional refractive index (RI) distribution. Here, refractive index (RI), n is an optical property related to electro-susceptibility and magnetic-susceptibility (X _e and X _m , respectively) representing light-matter interaction, below It can be expressed as <Equation 1>.

[Equation 1]

Since the mineral interaction depends on the electron distribution and density, the RI value is different for each material. Thus, by reconstructing the RI distribution, ODT provides not only the structure of the cell, but also the state of the cell, such as dry mass or chemical information.

Nevertheless, technical challenges still remain in ODT. A particularly well-known missing cone problem stems from practical ODT instruments that can measure the scattering field only within a limited angle. As a result, the Fourier component is omitted in the cone-shaped angular region, resulting in image distortion that makes it difficult to quantify the RI value. Moreover, recent state-of-the-art systems acquire holograms using extremely sparse view sampling, for example, a circular sampling pattern with 49 views. This is because to use Digital Micromirror Devices (DMDs), the system must memorize rotating reflection patterns, making it easier to implement fewer projected views. However, the lack of projection views causes technical problems such as missing cone artifacts and low-quality images with inaccurate RI values, making it difficult to accurately quantify the original RI values.

Many researchers have studied various methods to solve this problem by using an iterative reconstruction approach that incorporates prior knowledge of the RI distribution. Improperly established inverse problems were normalized using edge conservation or sparsity promoting regularizations, such as l ₁ , total variation (TV), for example. Among them, the Gerchberg-Papoulis (GP) algorithm is one of the representative methods for integrating the non-negativity constraint into the reconstruction. Although this iterative approach improves resolution compared to the original reconstruction, it cannot overcome the inherent missing angle problem. Specifically, the quality of the reconstructed distribution is greatly degraded except for a few slices aligned with the acquisition angle. Moreover, smoothness constraints such as TV normalization often introduce cartoon-like artifacts into the reconstruction.

Inspired by the recent success of deep learning approaches to solve improperly established inverse problems, such as CT reconstruction, MR reconstruction, positron emission tomography, ultrasound imaging, and light microscopy, several authors have applied deep learning to ODT. . Most of these methods focus on supervised learning that enables real-world data collection. Nevertheless, in many interesting real-world situations in ODT, such ground truth data cannot be directly measured. Some tasks use phantoms to train neural networks in a supervised manner, but lack generality in practice. Therefore, there is an increasing need for unsupervised learning without matching actual data.

Recently, generative adversarial networks (GANs) have become the protagonists of various unsupervised learning methods that cannot be supervised. Specifically, there are two components in a GAN: a generator responsible for mapping the input data to a target, and a discriminator, which learns the target distribution to induce the generated samples to be more realistic. GANs have found many applications in image creation, translation, and missing value imputation tasks. In addition, replacing the generator with a stochastic forward model and updating the reconstruction volume directly was found to be effective for single-particle cryo-EM reconstructions.

However, another powerful aspect of GANs is that they are particularly effective at learning probability distributions and performing bidirectional transformations between two probability distributions. In fact, this problem is closely related to the mathematical theory of optimal transport (OT). More specifically, optimal transport considers finding an optimal transport map that transforms one probability measure into another with minimal transport cost. Moreover, when the Wasserstein-1 metric is used as a means to measure the statistical distance in bidirectional optimal transport, the general form of the cycleGAN framework appears as the Kantorovich double formula of the primitive optimal transport problem.

Embodiments of the present invention provide a tomography image processing method capable of generating a high-quality tomography image by removing missing cone artifacts of a tomography image using an unsupervised learning-based neural network, and provide that device.

A tomography image processing method according to an embodiment of the present invention comprises the steps of receiving a tomography image; And by removing the missing cone artifact from the tomography image using an unsupervised learning-based neural network generated based on the optimal transport theory, the projection image from which the missing cone artifact is removed including creating them.

In the receiving step, the projection images for the optical diffraction tomography image are received, and in the generating step, the high-resolution projection images are generated by removing the missing cone artifact included in the projection images using the neural network. can create

Furthermore, the tomography image processing method according to an embodiment of the present invention may further include reconstructing a 3D tomography image through filtered backprojection (FBP) using the generated projection images. .

The neural network may be trained using a training data set including non-matching data.

The neural network may include: a first neural network for converting the first projection image into a second projection image from which the missing cone artifact is removed by receiving a first projection image; a second neural network for converting the second projection image into a third projection image including the missing cone artifact by receiving the second projection image; a third neural network for discriminating the first projection image and the third projection image; and a fourth neural network for discriminating the second projection image from an actual image corresponding to the second projection image.

The neural network may be trained unsupervised based on a cyclic loss between the first projection image and the third projection image and an adversarial loss between the second projection image and the real image.

The neural network may be trained based on predefined cyclic loss and adversarial loss with respect to the projection image and the projection image from which the missing cone artifact is removed.

The neural network may include any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.

A tomography image processing method according to another embodiment of the present invention comprises the steps of receiving a tomography image; By removing the missing cone artifact from the tomography image using an unsupervised learning-based neural network generated based on the optimal transport theory, the projected images from which the missing cone artifact has been removed generating; and reconstructing a 3D tomography image through filtered backprojection (FBP) using the generated projection images.

A tomography image processing apparatus according to an embodiment of the present invention includes: a receiver for receiving a tomography image; And by removing the missing cone artifact from the tomography image using an unsupervised learning-based neural network generated based on the optimal transport theory, the projection image from which the missing cone artifact is removed and a generator for generating them.

The receiving unit may receive projection images for the optical diffraction tomography image, and the generating unit may generate high-resolution projection images by removing the missing cone artifact included in the projection images using the neural network. .

Furthermore, the tomography image processing apparatus according to an embodiment of the present invention may further include a reconstruction unit for reconstructing a 3D tomography image through filtered backprojection (FBP) using the generated projection images. .

The neural network may be trained using a training data set including non-matching data.

The neural network may include: a first neural network for converting the first projection image into a second projection image from which the missing cone artifact is removed by receiving a first projection image; a second neural network for converting the second projection image into a third projection image including the missing cone artifact by receiving the second projection image; a third neural network for discriminating the first projection image and the third projection image; and a fourth neural network for discriminating the second projection image from an actual image corresponding to the second projection image.

The neural network may be trained unsupervised based on a cyclic loss between the first projection image and the third projection image and an adversarial loss between the second projection image and the real image.

The neural network may be trained based on predefined cyclic loss and adversarial loss with respect to the projection image and the projection image from which the missing cone artifact is removed.

The neural network may include any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.

According to embodiments of the present invention, missing cone artifacts of a tomography image are removed by using an unsupervised learning-based neural network generated based on an optimal transport theory. By doing so, a high-quality tomography image can be generated.

Since the present invention is a general algorithm that can be used in tomography imaging in all fields, it can be applied to all fields using tomography imaging technology. can be applied to the field. That is, the present invention relates to optical diffraction tomography and optical coherence tomography used to reveal the physicochemical properties of nanoscale particles or cells, and electron microscope images and X-rays used in medicine. It can be applied to all fields of tomography imaging, up to the improvement of tomography imaging. For example, in an optical diffraction tomography image, a portion that was not visible even with a small number of hologram images can be restored with high resolution through the present invention. It is also possible to reconstruct a 3D image of In this way, it is possible to restore a part that was not seen due to a physical limitation with high quality, and it is also possible to accelerate an image obtained at a low speed.

Therefore, the present invention can observe a previously invisible part with high resolution and can be used to dramatically shorten the measurement time, so that it can be highly useful in terms of time and economy.

1 is a flowchart showing an operation of a tomography image processing method according to an embodiment of the present invention.
Figure 2 shows an exemplary diagram for explaining an approach to the method of the present invention for resolving the missing cone artifact.
3 shows a geometric view of CycleGAN and an exemplary diagram for explaining the structure of CycleGAN.
4 shows an exemplary diagram of a neural network structure used in the TomoGAN framework of the present invention.
5 shows an exemplary diagram for verifying the effectiveness of the present invention.
6 shows an exemplary diagram comparing the method of the present invention and the existing methods for an NIH3T3 cell.
7 shows an exemplary diagram for 3D rendering reconstructed by the existing methods and the method of the present invention.
8 shows the configuration of a tomography image processing apparatus according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be embodied in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, operations and/or elements mentioned. or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted.

Embodiments of the present invention remove missing cone artifacts of a tomography image using an unsupervised learning-based neural network generated based on an optimal transport theory. make it the gist of it.

In this case, in the present invention, a 3D tomography image can be reconstructed using projection images from which the missing cone artifact has been removed, and a 3D tomography image can be reconstructed using filtered backprojection (FBP). .

That is, the present invention removes the missing cone artifact included in the projection image of the tomography image using only a single neural network based on unsupervised learning, thereby generating a high-quality projection image from which the missing cone artifact is removed. High-resolution 3D tomography images can be reconstructed.

An image input to the neural network may be projection images of a tomography image, and the projection images may be projection images of an optical diffraction tomography image, and these projection images may include a missing cone artifact.

Furthermore, the neural network of the present invention can be trained using a training data set including mismatching data, and with respect to the projection image and the projection image from which the missing cone artifact is removed, a predefined cyclic loss and It can be learned based on adversarial loss.

Here, the neural network used in the present invention may include a convolution framelet-based neural network, a neural network including a pooling layer and an unpooling layer, for example, U-Net. In addition to this, various types of neural networks applicable to the present invention may be included.

A convolutional framelet refers to a method of representing an input signal through a local basis and a non-local basis. In order to reveal the black box characteristics of deep convolutional neural networks, a study on a new mathematical theory of deep convolutional framelets (Ye , JC., Han, Y., Cha, E.: Deep convolutional framelets: a general deep learning framework for inverse problems. .

1 is a flowchart showing an operation of a tomography image processing method according to an embodiment of the present invention.

Referring to FIG. 1 , a tomography image processing method according to an embodiment of the present invention receives a tomography image ( S110 ).

Here, step S110 may be projection images for an optical diffraction tomography image, and these projection images may include not only high-quality projection images but also projection images including missing cone artifacts.

When a tomography image, that is, projection images is received through step S110, a cone artifact missing from the received projection images is removed using an unsupervised learning-based neural network generated based on an optimal transport theory. , high-quality projection images from which the missing cone artifact is removed are generated ( S120 ).

Here, the neural network may be trained using a training data set including non-matching data, and the first projection image is input to convert the first projection image into a second projection image from which the missing cone artifact is removed. a neural network, a second neural network for converting a second projection image to a third projection image including a missing cone artifact by inputting the second projection image, a third neural network for discriminating the first projection image from the third projection image The neural network of the present invention may be learned by using a fourth neural network for discriminating the second projection image and the real image corresponding to the second projection image. The neural network of the present invention may be trained based on predefined cyclic loss and adversarial loss with respect to the projection image and the projection image from which the missing cone artifact is removed. For example, the neural network of the present invention may be trained unsupervised based on a cyclic loss between the first projection image and the third projection image and an adversarial loss between the second projection image and the real image. .

In this case, the neural network may include any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.

When projection images from which the missing cone artifact is removed are generated by step S120, a high-quality 3D tomography image is reconstructed through filtered backprojection (FBP) using the generated projection images (S130).

This method of the present invention provides a new reconstruction method for ODT by utilizing both CycleGAN's ability to transform probability distribution and a widely used GP algorithm. The projection data of all angles of the GP reconstruction may also have similar properties. However, due to the limited viewing angle of the original ODT measurements, the synthetic 2D projection data of the GP reconstructions aligned to the acquisition angle when projected at random angles have high resolution, whereas the rest can be characterized by high noise levels and blurring. there is. Accordingly, the method of the present invention may divide the projection data into two sets, one of which may contain the projection image aligned with the acquisition angle and the other may contain the projection measurements of all angles.

The method of the present invention will be described with reference to FIGS. 2 to 7 as follows.

The main idea of the present invention is to use CycleGAN to learn an optimal transport map that can transform the probability distribution of a blurred projection into a probability distribution of a high-resolution projection image at the acquisition angle. After the neural network of the present invention, for example, the TomoGAN network is trained to improve the blurry projection image, as shown in FIG. to achieve the final reconstruction.

Fig. 2 shows an exemplary view for explaining the approach of the present invention for solving the missing cone artifact. As shown in Fig. 2, the ODT is a hologram in which the acquisition angle is greatly limited. Measuring, GP algorithm is applied to the initial reconstruction, reconstructing 3D RI data and then performing parallel beam projection to generate 2D projection data, and then using the enhanced projection data after enhancing the projection image through the TomoGAN projection enhancement process The final reconstruction can be obtained using the FBP algorithm.

Regarding the Rytov reconstruction, three 2D holograms measured at various illumination angles contain amplitude and phase delay information. After Rytov approximation, the Fourier sample of each hologram is mapped into 3D k-space along a specific 3D hemispherical shell according to the Fourier diffraction theorem, and the RI Tomogram can be obtained by inverse Fourier transform.

Regarding the Gerchberg-Papoulis (GP) algorithm, The limited numerical aperture (NA) and the limitation of specific illumination angles prevent the acquisition of side scatter signals. It yields missing information in k-space, resulting in a decrease in RI value and a decrease in xy resolution. To overcome this problem, we use a GP algorithm based on an iterative method of non-voice imposition. Non-negative restriction means that the scattering probability should not have an RI value lower than the background. Specifically, after finding an index lower than the background RI value and replacing it with the background RI value, the actual measurement data is inserted back into the k-space, and this process is repeated repeatedly. At this time, the 3D RI data reconstructed by the iterative GP algorithm may be used as an initialization point of the TomoGAN framework of the present invention.

Regarding TV reconstruction, the TV normalized iterative reconstruction method can be implemented using the same approach in the previous work.

When describing the TomoGAN reconstruction, which is the method of the present invention, first, an initial 3D reconstruction is performed using a GP algorithm representing x, and then parallel beam forward projection is performed as shown in Equation 2 below.

[Equation 2]

Here, y _ω may mean a projection measurement at a solid angle ω.

It should be noted that this projection step uses parallel beam projection for high-speed computation, which is somewhat different from the forward diffraction model of ODT. However, thanks to the relationship between the projection-slice theorem and the projection-diffraction theorem, it provides a high-resolution projection as in Fig. 3 at the solid angle at which the actual hologram is measured.

This forward projection produces two projection image distributions, Y _Ω and Y _Ω ^c , where Ω represents the solid angle with respect to the measured projection angle, and Ω ^c may represent the unmeasured solid angle. The projected image y _ω at ω ∈ Ω usually has very high resolution with no missing cone angle artifacts, whereas the image of ω ∈ Ω ^c suffers from blurring with missing cone angles. suffer Therefore, it is important to find a transformation that improves the probability distribution of Y _Ω ^c by learning the probability distribution of Y _Ω .

In fact, this problem can be addressed rigorously using the mathematical theory of optimal transport (OT). More specifically, optimal transport considers finding the optimal transport map that can transport one probability measure to another with minimal transport cost. In particular, it is assumed that the input projection space Y _Ω ^c has a probability measure μ, as shown in FIG. 3A , whereas the target projection space Y _Ω is equipped with a probability measure ν. It can be seen that the bulk transfer from one measurement space (Y ^c _Ω ,ν) to another probabilistic measurement space (Y _Ω , μ) is performed by the missing cone artifact correction generator G. That is, the generator G pushes forward the probability measurement ν of Y _Ω ^c to the measurement value μ _G of the target space Y _Ω . On the other hand, mass transport from (Y _Ω ,μ) to (Y ^c _Ω ,ν) is performed by the artifact generation operator F such that F drives the probability measurement from μ in space Y _Ω to ν _F in Y _Ω ^c . . Then, by minimizing the statistical distance dist(μ,μ _G ) between μ and μ _G and the statistical distance dist( ν,ν _F ) between ν and ν _F , we can achieve an optimal transport map for unsupervised learning. . In this case, the method of the present invention can use the Wasserstein-1 metric as a means for measuring the statistical distance.

In particular, if the two distances are minimized together using the Kantorovich OT formula, cycleGAN formulas such as Equation 3 and Equation 4 below can be derived.

[Equation 3]

[Equation 4]

Here, λ may mean a preset hyper parameter, and the cycle coherence loss may be defined by the following equation.

And the loss for the discriminator term can be given by the following equation.

Here, ϕ and ψ may mean a discriminator term that distinguishes the spurious projection and the real projection of Y _Ω and Y _Ω ^c , respectively.

The Kantorovich probability must be 1-Lipschitz, and the present invention can impose a finite Lipschitz condition using the LS-GAN formula as the discriminator loss. Once the CycleGAN network shown in Figure 3b is trained, network G can be utilized to generate enhanced projection data at any angle in the inference step.

For the final 3D reconstruction on the improved parallel projection data, there can be several ways to obtain 3D tomography from the projection data set. For example, weighted backprojection (WBP), Fourier reconstruction, and simultaneous iterative reconstruction (SIRT) can be used in the cryp-EM community to describe the projection reconstruction when the projection angle is random. Because the present invention can control the angle of the composite projection, it is possible to use a filter corrected inverse projection (FBP) with the same angular increment. Specifically, 360 isometric projections are generated around each coordinate axis, and thereafter, filter-corrected inverse projection (FBP) is applied to perform final reconstruction. The final result can be obtained by averaging the reconstructed 3D potential set to maintain the balance of the three axes.

As shown in Fig. 3b, the neural network of the present invention can be trained using two generators G and F and two discriminators φ and ψ, and the two generators are the same neural network as shown in Fig. 4a. It can be implemented using an architecture, and the two distinguishers can be implemented using the same neural network architecture as shown in FIG. 4B .

That is, the neural network shown in FIG. 3b receives a first projection image (Y _Ω ^c ) as an input and converts the first projection image into a second projection image (Y _Ω ) from which the missing cone artifact is removed. ), a second generator (F) that converts the second projection image into a third projection image (Y _Ω ^c ) with missing cone artifacts as input, the first projection image and the second projection image (Y _Ω c ). 3 and a first distinguisher ψ for discriminating the projection image and a second discriminator φ for discriminating the second projection image and the real image corresponding to the second projection image. Such a neural network may be trained based on predefined cyclic loss and adversarial loss, as shown in FIG. 3B .

The first generator G receives a projection image including the missing cone artifact as an input and outputs a projection image from which the missing cone artifact is removed, or the projection image output from the second generator F, that is, includes the missing cone artifact A high-quality projection image in which the missing cone artifact included in the corresponding projection image is removed is output by inputting a projection image including the missing cone artifact in the high-quality projection image that is not in the projection image.

The second generator F receives a high-quality projection image that does not include the missing cone artifact as an input and outputs a projection image including the missing cone artifact, or receives the projection image output from the first generator as an input and receives a corresponding projection image Outputs the projection image including the missing cone artifacts.

The first discriminator ψ is a neural network that discriminates the projection image including the missing cone artifact from the projection image output by the second generator F, and the second discriminator φ is the first generator G ) is a neural network that discriminates the projected image from which the missing cone artifacts have been removed from the high-quality real projection image that does not include the missing cone artifacts.

That is, in the present invention, cycle continuity refers to outputting a target projection image in which the missing cone artifact is removed from the projection image including the missing cone artifact, outputting the projection image including the missing cone artifact for the target projection image, and then re-using it. Repeat the process of returning to the input of the generator. Then, the neural network is trained in a cyclic manner so that the input image and the projected image output from the second generator are identical to each other. Of course, this cyclic learning is simultaneously performed for the lower generators of FIG. 3B . The unsupervised learning method uses generative adversarial neural networks (GANs) to train discriminators.

The present invention may use a generator architecture as shown in Fig. 4a, and may be a partially modified encoder-decoder architecture in U-Net.

Encoder and decoder parts contain 3×3 convolution layer, ReLU activation function and group normalization with group size of 8, for pooling layer you can use 3×3 convolution with stride 2, for unpooling layer You can use bilinear upscaling. Skip concatenation is used to connect the features of each level of the encoder part, and a 1×1 convolutional layer is used in the final stage of reconstruction.

And, the discriminator architecture can be adopted in the patchGAN of pix2pix3 as shown in Fig. 4b, and the image size can be efficiently reduced by using a 4×4 convolution with stride 2. LeakyReLU activation instance normalization can be used between convolution blocks, and a 1×1 convolution layer is used in the final step.

Figure 5 shows an exemplary view for verifying the effectiveness of the present invention, comparing the method of the present invention with Rytov reconstruction, GP algorithm, and total variation (TV) reconstruction for the reconstruction of microbeads having an original value of 1.46 . Here, Fig. 5a shows the corresponding visualization of k-space with different slices in different planes, Fig. 5b shows the histogram analysis of the RI distribution, and Fig. 5c shows the line profile along the x-axis. Each slice of FIG. 5A may be taken from the center of the volume, and the intensity of the color bar of FIG. 5B may mean the number of bins of data.

As shown in Fig. 5, it can be seen that the original shape of the microbeads is not preserved through Rytov reconstruction, and moreover, elongation along the optical axis is particularly evident in the yz and xz planes (Fig. 5a). Reconstruction with GP solves the extension problem to some extent, but does not capture the original shape well, and moreover, the RI value is greatly underestimated from the real value due to the missing cone problem in the k-space pointing to the 3D Fourier space. Able to know. TV reconstruction results show better results in that this method produces a more homogenous structure because it clearly requires smoothness of reconstruction, but not only underestimated RI values, but also in the yz or xz plane. It can be seen that the visible cutviews still do not capture the spherical shape properly, and moreover, the missing cone problem is still noticeable in k-space. On the other hand, it can be seen that the reconstitution by the method of the present invention preserves the spherical shape of the microbeads and maintains an underestimated RI value. In addition, the k-space shows the approximate sinc function shown when the homogeneous sphere is reconstructed, and it can be seen that the RI distribution is shifted to the actual RI value compared to the Rytov reconstruction and the GP reconstruction. In addition, as shown in Fig. 5c, it can be seen that the average RI value is improved to the actual value by the method of the present invention.

6 shows an exemplary diagram comparing the method of the present invention and the existing methods for the NIH3T3 cell, and shows an exemplary diagram comparing the projection view according to the rotation angle. Here, the projection view may be generated by rotating it along the y-axis.

As shown in FIG. 6 , it can be seen that the existing methods TV, GP, and Rytov significantly degrade the image quality when the projection angle deviates from a specific angle, that is, when the projection angle deviates from the measurement angle. It can be a decisive influence. On the other hand, the TomoGAN method of the present invention greatly improves the resolution of the projection view, so that the cell can be observed clearly from all angles.

Furthermore, the method of the present invention may perform three-dimensional reconstruction by using the FBP for in-depth analysis of the reconstructed RI distribution using the improved projection data of FIG. 6 .

7 shows an exemplary diagram of a 3D rendering reconstructed by the existing methods and the method of the present invention, and shows a 3D rendering of a reconstructed cell distribution and a cut view of the cell distribution, and each cut view is a yz plane, The xy plane and the xz plane are shown respectively. Here, the cut view shows an image of excellent resolution with a higher RI value.

As can be seen from FIG. 7 , in the case of reconstruction using Rytov and TV, it can be seen that the resolution fell far short of the satisfaction level. In particular, the TV reconstruction shown in Fig. 7c has a cartoon-like appearance that does not capture the actual texture of the cell, and the GP reconstruction shown in Fig. 7b can achieve high resolution while preserving more details. However, this resolution still does not meet the standard, and furthermore, when the cutview RI distribution is visualized, it can be seen that the resolution of the y-z plane and x-z plane images is very low. On the other hand, it can be seen that the reconstruction through the method of the present invention (TomoGAN) shows a high-resolution view from all angles, and the cut view also shows excellent results in all three planes. That is, it can be seen that the method of the present invention solves the problem of underestimating the RI value.

In conclusion, the method of the present invention can completely solve the problem of missing cones in ODT, and this method of the present invention uses the initial GP reconstruction to generate a projection view, which improves the projection distribution, and provides a simple filter correction after projection. Through projection, the 3D RI distribution can be reconstructed.

As such, the method according to an embodiment of the present invention can generate a high-quality tomography image by removing missing cone artifacts of the tomography image using an unsupervised learning-based neural network.

In addition, since the method according to an embodiment of the present invention is a general algorithm that can be used in tomography images in all fields, it can be applied to all fields using tomography imaging technology, and in particular, can be used even when the correct answer image is unknown. Because it is based on unsupervised learning, it can be applied to various fields. That is, the method of the present invention is used for optical diffraction tomography and optical coherence tomography used to reveal the physicochemical properties of nanoscale particles or cells, and electron microscopic images and medical applications. It can be applied to all fields of tomography imaging, up to the improvement of X-ray tomography images. For example, in an optical diffraction tomography image, a portion that was not visible even with a small number of hologram images can be restored with high resolution through the present invention. It is also possible to reconstruct a 3D image of In this way, it is possible to restore a part that was not seen due to a physical limitation with high quality, and it is also possible to accelerate an image obtained at a low speed.

8 shows the configuration of a tomography image processing apparatus according to an embodiment of the present invention, and shows a conceptual configuration of an apparatus for performing the method of FIGS. 1 to 7 .

Referring to FIG. 8 , an apparatus 800 according to an embodiment of the present invention includes a receiving unit 810 , a generating unit 820 , and a reconfiguration unit 830 .

The receiver 810 receives a tomography image.

Here, the receiver 810 may be projection images of the optical diffraction tomography image, and these projection images may include not only high-quality projection images but also projection images including missing cone artifacts.

The generator 820 removes the missing cone artifact from the received projection images using an unsupervised learning-based neural network generated based on the optimal transport theory, thereby generating high-quality projection images from which the missing cone artifact has been removed. create

Here, the neural network may be trained using a training data set including non-matching data, and the first projection image is input to convert the first projection image into a second projection image from which the missing cone artifact is removed. a neural network, a second neural network for converting a second projection image to a third projection image including a missing cone artifact by inputting the second projection image, a third neural network for discriminating the first projection image from the third projection image The neural network of the present invention may be learned by using a fourth neural network for discriminating the second projection image and the real image corresponding to the second projection image. The neural network of the present invention may be trained based on predefined cyclic loss and adversarial loss with respect to the projection image and the projection image from which the missing cone artifact is removed. For example, the neural network may be trained unsupervised based on a cyclic loss between the first projection image and the third projection image and an adversarial loss between the second projection image and the real image.

In this case, the neural network may include any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.

The reconstruction unit 830 reconstructs a high-quality 3D tomography image through filter-corrected inverse projection (FBP) using the projection images from which the missing cone artifact is removed.

Although the description of the device of FIG. 8 is omitted, each component constituting FIG. 8 may include all the contents described in FIGS. 1 to 7 , which is apparent to those skilled in the art.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

receiving a tomography image; and
By removing the missing cone artifact from the tomography image using an unsupervised learning-based neural network generated based on the optimal transport theory, the projected images from which the missing cone artifact has been removed steps to create
A tomography image processing method comprising a.
According to claim 1,
The receiving step
Receive projection images for the optical diffraction tomography image,
The generating step is
and generating high-resolution projection images by removing the missing cone artifact included in the projection images using the neural network.
According to claim 1,
Reconstructing a 3D tomography image through filtered backprojection (FBP) using the generated projection images
Tomography image processing method, characterized in that it further comprises.
According to claim 1,
The neural network is
A tomography image processing method, characterized in that it is learned using a training data set including non-matching data.
According to claim 1,
The neural network is
a first neural network for converting the first projection image into a second projection image from which the missing cone artifact is removed by receiving a first projection image;
a second neural network that receives the second projection image as an input and converts the second projection image into a third projection image including the missing cone artifact;
a third neural network for discriminating the first projection image and the third projection image; and
a fourth neural network for discriminating the second projection image and a real image corresponding to the second projection image
A tomography image processing method comprising a.
6. The method of claim 5,
The neural network is
tomography image processing, characterized in that unsupervised learning is performed based on a cyclic loss between the first projection image and the third projection image and an adversarial loss between the second projection image and the real image method.
According to claim 1,
The neural network is
A tomographic image processing method, characterized in that the projection image and the projection image from which the missing cone artifact is removed are learned based on predefined cyclic loss and adversarial loss.
According to claim 1,
The neural network is
A tomographic image processing method, comprising: any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.
receiving a tomography image;
By removing the missing cone artifact from the tomography image using an unsupervised learning-based neural network generated based on the optimal transport theory, the projected images from which the missing cone artifact has been removed generating; and
Reconstructing a 3D tomography image through filtered backprojection (FBP) using the generated projection images
A tomography image processing method comprising a.
a receiver for receiving a tomography image; and
By removing the missing cone artifact from the tomography image using an unsupervised learning-based neural network generated based on the optimal transport theory, the projected images from which the missing cone artifact has been removed generating unit
A tomography image processing method comprising a.
11. The method of claim 10,
the receiving unit
Receive projection images for the optical diffraction tomography image,
the generating unit
The tomography image processing apparatus of claim 1, wherein the high-resolution projection images are generated by removing the missing cone artifact included in the projection images using the neural network.
11. The method of claim 10,
A reconstruction unit that reconstructs a 3D tomography image through filtered backprojection (FBP) using the generated projection images
A tomography image processing apparatus further comprising a.
11. The method of claim 10,
The neural network is
A tomography image processing apparatus, characterized in that it is learned using a training data set including non-matching data.
11. The method of claim 10,
The neural network is
a first neural network for converting the first projection image into a second projection image from which the missing cone artifact is removed by receiving a first projection image;
a second neural network that receives the second projection image as an input and converts the second projection image into a third projection image including the missing cone artifact;
a third neural network for discriminating the first projection image and the third projection image; and
a fourth neural network for discriminating the second projection image and a real image corresponding to the second projection image
A tomography image processing apparatus comprising a.
15. The method of claim 14,
The neural network is
tomography image processing, characterized in that unsupervised learning is performed based on a cyclic loss between the first projection image and the third projection image and an adversarial loss between the second projection image and the real image Device.
11. The method of claim 10,
The neural network is
A tomography image processing apparatus, characterized in that the projection image and the projection image from which the missing cone artifact is removed are learned based on predefined cyclic loss and adversarial loss.
11. The method of claim 10,
The neural network is
A tomography image processing apparatus, comprising: any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.

Images