KR102514709B1 - 3d time-of-flight magnetic resonance angiography image processing method using neural network based on two-stage unsupervised learning and apparatus therefor - Google Patents

Abstract

A method and apparatus for processing a 3D time-of-flight magnetic resonance vascular image using a two-step unsupervised learning-based neural network are disclosed. An MRI processing method according to an embodiment of the present invention includes the steps of receiving coronal images of an MRI vessel including artifacts; Artifacts included in the received coronal images are removed using an unsupervised learning-based first neural network generated based on optimal transport (OT) theory, thereby generating coronal images from which artifacts are removed. Reconstructing a 3D volume by doing; generating axial images using the reconstructed 3D volume; and reconstructing a final 3D volume by removing artifacts remaining in the axial images using a second neural network based on unsupervised learning generated based on the optimal transport (OT) theory.

Description

3D time-of-flight magnetic resonance vascular image processing method and apparatus using two-step unsupervised learning-based neural network

The present invention relates to a 3D time-of-flight magnetic resonance imaging (TOF-MRA) processing technology, and more specifically, to improve the quality of a 3D time-of-flight magnetic resonance vascular image using a two-step unsupervised learning-based neural network. It relates to a 3D time-of-flight magnetic resonance angiogram processing method and apparatus capable of processing the same.

Time-of-flight magnetic resonance angiography (TOF-MRA) is widely used in clinical settings to visualize blood flow without contrast injection. Here, the contrast between the blood vessel and the surrounding tissue is amplified by using a flow-related enhancement phenomenon in which the spin enters the image slice.

In 2D TOF, several thin image slices are acquired using flow-corrected gradient-echo sequences, whereas in 3D TOF, image volumes are acquired simultaneously by phase coding in a slice selection direction. These images can be combined using maximum intensity projection (MIP) to obtain a 3D image of blood vessels similar to conventional angiography. Thus, TOF-MRA provides tremendously useful physiological information for the detection of stenosis or occlusion of intracranial arteries.

Fully capturing the k-space when doing TOF-MRA scans is painfully time consuming, especially for 3D scans that have to cover large volumes. In addition, if the patient moves during the extended scan time, artifacts occur in the image. Therefore, accelerating MR scans increases patient throughput and solves the problem of motion artifacts.

To reduce long scan times, k-space can be sub-sampled, but k-space under-sampling introduces aliasing artifacts later. To solve this problem, it is possible to compensate for missing k-space data by merging information from different receiver coils using multiple receiver coils. These parallel MRI (pMRI) techniques are routinely used in clinical practice.

In the case of TOF-MRA, the compressed sensing (CS) algorithm has also been extensively studied using the sparsity of the original image region, which is an inherent characteristic of angiograms. Moreover, the application of CS in conjunction with pMRI has been extensively investigated. CS-MRI has shown its effectiveness in MRI reconstruction, but the iterative nature of this method is slow and computationally expensive. In addition, the inability to learn from a given data distribution is a drawback.

Recently, numerous deep learning algorithms have been proposed for MR reconstruction that show superior performance compared to CS-MRI while significantly reducing computational time. Adversarial generative networks (GANs) have also been mainly investigated to further enhance the reconstruction quality in the MR reconstruction context.

Nonetheless, most deep learning approaches are supervised learning frameworks that require a large number of matched fully-sampled scans to properly train a neural network. This poses a fundamental challenge in neural network training, as matched complete sampling reference data must be acquired under identical conditions, which is not always possible in a clinical setting.

Embodiments of the present invention provide a 3D time-of-flight magnetic resonance vascular image processing method and apparatus capable of improving the quality of a 3D time-of-flight magnetic resonance vascular image using a two-step unsupervised learning-based neural network. .

An MRI processing method according to an embodiment of the present invention includes the steps of receiving coronal images of an MRI vessel including artifacts; Artifacts included in the received coronal images are removed using an unsupervised learning-based first neural network generated based on optimal transport (OT) theory, thereby generating coronal images from which artifacts are removed. Reconstructing a 3D volume by doing; generating axial images using the reconstructed 3D volume; and reconstructing a final 3D volume by removing artifacts remaining in the axial images using a second neural network based on unsupervised learning generated based on the optimal transport (OT) theory.

Each of the first neural network and the second neural network may be learned using a training dataset including mismatched data.

Each of the first neural network and the second neural network may include one of a neural network based on a convolution framelet and a neural network including a pulling layer and an unpooling layer.

In the step of reconstructing the final 3D volume, artifacts remaining in the axial images are removed by inputting a preset number of axial images according to a depth direction from among the generated axial images to the second neural network. Thus, the final 3D volume can be reconstructed.

Furthermore, the magnetic resonance angiogram processing method according to an embodiment of the present invention may further include generating a maximum intensity projection image using the reconstructed final 3D volume.

A method for processing an MRI angiogram according to another embodiment of the present invention includes receiving first plane images of an MRI vessel including artifacts; A first plane image from which artifacts are removed by removing artifacts included in the received first plane images using an unsupervised learning-based first neural network generated based on optimal transport (OT) theory Reconstructing a three-dimensional volume by generating a; generating second plane images using the reconstructed 3D volume; and reconstructing a final 3D volume by removing artifacts remaining in the second plane images using a second neural network based on unsupervised learning generated based on the optimal transport (OT) theory.

The reconstructing of the final 3D volume may include inputting a predetermined number of second plane images according to a depth direction among the generated second plane images to the second neural network, so that the remaining second plane images The final 3D volume can be reconstructed by removing artifacts.

Furthermore, the magnetic resonance angiogram processing method according to another embodiment of the present invention may further include generating a maximum intensity projection image using the reconstructed final 3D volume.

An MRI apparatus for processing an MRI vessel image according to an embodiment of the present invention includes a receiver configured to receive coronal images of an MRI vessel including artifacts; Artifacts included in the received coronal images are removed using an unsupervised learning-based first neural network generated based on optimal transport (OT) theory, thereby generating coronal images from which artifacts are removed. a first restoration unit for reconstructing a 3-dimensional volume; a generation unit generating axial images using the reconstructed 3D volume; and a second reconstruction unit that reconstructs a final 3D volume by removing artifacts remaining in the axial images using a second neural network based on unsupervised learning generated based on the optimal transport (OT) theory. .

Each of the first neural network and the second neural network may be learned using a training dataset including mismatched data.

Each of the first neural network and the second neural network may include one of a neural network based on a convolution framelet and a neural network including a pulling layer and an unpooling layer.

The second reconstruction unit removes artifacts remaining in the axial images by inputting a preset number of axial images according to a depth direction from among the generated axial images to the second neural network, thereby generating the final 3D image. The volume can be reconstructed.

The second restoration unit may generate a maximum intensity projection image using the reconstructed final 3D volume.

According to embodiments of the present invention, the quality of a 3D time-of-flight magnetic resonance vascular image can be improved by using a two-step unsupervised learning-based neural network. That is, the present invention can apply Optimal Transport cycleGAN (OT-cycleGAN) in two stages to accelerate the image of 3D magnetic resonance angiography images, and since unsupervised learning is possible, the unaccelerated answer It has the advantage that it does not require a lot of video.

According to embodiments of the present invention, a projection discriminator is added to improve the quality of not only the MRA source image but also the maximum intensity projection image (MIP image). By providing it, the quality of the image can be dramatically improved.

According to the embodiments of the present invention, it is possible to obtain good image quality without greatly impairing the image quality while dramatically increasing the image acquisition speed by directly mounting it on equipment for taking magnetic resonance angiography, The present invention is a general technique applicable to 3D image reconstruction such as MRI and CT, and can be applied in various medical imaging fields.

According to the embodiments of the present invention, since it is an image reconstruction technique using a neural network, the time required for image reconstruction can be significantly reduced compared to the compression sensing technique used in various medical devices. The present invention can solve the problem in which acceleration is restricted due to the problem of the quality of image being impaired, and high-quality images can be obtained even under higher acceleration conditions.

1 is a flowchart illustrating an operation of a magnetic resonance angiogram processing method according to an embodiment of the present invention.
2 shows an exemplary diagram for explaining a geometric view for unsupervised learning.
3 illustrates an example of a sampling mask used in under-sampling according to acceleration.
Figure 4 shows an exemplary diagram for explaining the method of the present invention.
Figure 5 shows an exemplary diagram for explaining a two-step training process used in the method of the present invention.
Figure 6 shows an exemplary diagram of a neural network architecture for generators used in two stages.
7 shows an exemplary diagram of a neural network architecture for a discriminator used in two steps.
8 shows an example of a maximum intensity projection image according to the method of the present invention.
9 shows an exemplary diagram comparing reconstruction results by the conventional method and the method of the present invention.
10 illustrates the configuration of a magnetic resonance angiogram processing apparatus according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various forms different from each other, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention pertains. It is provided to completely inform the person who has the scope of the invention, and the present invention is only defined by the scope of the claims.

Terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" means that a stated component, step, operation, and/or element is present in the presence of one or more other components, steps, operations, and/or elements. or do not rule out additions.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. The same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

The gist of the embodiments of the present invention is to improve the quality of a 3D time-of-flight magnetic resonance vascular image using a two-step unsupervised learning-based neural network.

Here, the present invention provides an unsupervised learning-based neural network (hereinafter, hereinafter, After removing artifacts included in the received coronal images, that is, coronal slice images, using a "first neural network"), a 3D volume is reconstructed, and an axial image is obtained using the reconstructed 3D volume. That is, an unsupervised learning-based neural network (hereinafter referred to as “second neural network”) generated based on the optimal transport (OT) theory of artifacts remaining in the axial slice images generated in this way after generating the axial slice images. ), it is possible to reconstruct the final 3D volume for the MRI vascular image.

The neural network used in the present invention may include a convolution framelet-based neural network, a neural network including a pooling layer and an unpooling layer, for example, a U-Net, , as well as various types of neural networks applicable to the present invention.

A convolutional framelet refers to a method of expressing an input signal through a local basis and a nonlocal basis, and a study on a new mathematical theory of deep convolutional framelets to reveal the black box characteristics of deep convolutional neural networks (Ye , JC., Han, Y., Cha, E.: Deep convolutional framelets: a general deep learning framework for inverse problems. SIAM Journal on Imaging Sciences 11(2), 991-1048(2018). .

A recent study proposed a systematic framework for designing various types of unsupervised learning architectures for general inverse problems using optimal transport theory, and also provided preliminary results for single-coil 2D MR reconstructions in sparse Fourier samples. The resulting network architecture is similar to cycleGAN, but knowledge of the image physics can greatly simplify the network architecture and training plan.

By extending this idea, the present invention provides a novel multi-planar deep learning technique aimed at reconstruction of unsampled 3D TOF-MRA scans. To overcome the large GPU memory and training data requirements for 3D learning, the present invention can provide a new architecture consisting of two successive unsupervised training steps in 2D space. The first step is to reconstruct the MRA scan in the coronal plane by integrating the complex multi-coil data into the training plan, performing slice-by-slice. In the second stage of reconstruction, the quality of the reconstruction is further increased, especially in terms of maximum intensity projection (MIP) images, through stacked 3D reconstruction with the newly introduced projection discriminator. One of the important advantages of our two-stage unsupervised learning scheme is that each neural network can be trained with a different set of unpaired training data, which maximizes the utility of the data available for training purposes.

Accordingly, the contribution of the present invention can be summarized as follows.

(1) A two-step unsupervised learning process is provided for 3D reconstruction in the coronal plane and the axial plane, respectively, and this sequential learning process is particularly useful for accelerating 3D MR with 4 dimensions (3 spaces, 1 coil). do.

(2) A projection discriminator is provided that learns both volume and maximum full image distributions. This discriminator is used in the second stage of reconstruction and proves to greatly improve the quality of images, especially in terms of MIP images.

(3) By deriving the network architecture using the optimal transport theory, unwanted artifact shapes commonly observed in GAN-type algorithms can be prevented in a top-down manner.

1 is a flowchart illustrating an operation of a magnetic resonance angiogram processing method according to an embodiment of the present invention.

Referring to FIG. 1 , a method for processing an MRI angiogram according to an embodiment of the present invention includes first plane images of an MRI vessel including artifacts, e.g., aliasing artifacts, e.g., a coronal image. Coronal with artifacts removed by receiving and removing artifacts included in the received coronal images using a first neural network based on unsupervised learning generated based on optimal transport (OT) theory A 3D volume is reconstructed by generating images (S110, S120).

When the 3D volume is reconstructed in step S120, second plane images, for example, axial images are generated using the reconstructed 3D volume, and the bus generated based on the optimal transport (OT) theory The final 3D volume is reconstructed by removing artifacts remaining in the axial images using the second neural network based on degree learning (S130 and S140).

Here, in the method of the present invention, each of the first neural network and the second neural network can be learned using a training dataset including mismatched data, and a convolution framelet-based neural network and a pooling layer. and a neural network including an unpooling layer.

At this time, step S140 inputs a preset number of second plane images, for example, 7 second plane images in the depth direction among the generated second plane images to the second neural network, thereby removing artifacts remaining in the second plane images. can be removed to reconstruct the final three-dimensional volume.

Furthermore, the method of the present invention may generate a maximum intensity projection image using the final 3D volume reconstructed in step S140.

The method of the present invention will be described in detail with reference to FIGS. 2 to 9.

Geometry of CycleGAN

Consider the measurement model of Equation 1 below.

[Equation 1]

Y = Fx

Here, y ∈ Y and x ∈ X denote measured values and unknown images, respectively, and F: X→Y may denote known, partially known, or completely unknown image operators.

In contrast to supervised learning, where the goal is to learn the relationship between pairs of images x and measurements y, in unsupervised learning frameworks there are no matching pairs of image measurements. However, since the present invention can have a set of images and undamaged measurements, unsupervised learning is to match a probability distribution rather than each individual sample as shown in FIG. 2 . This is done by finding a transport map that transmits a probability measure between the two spaces.

Unlike supervised learning, which can be applied only when there is matching data, cycleGAN based on optimal transportation has the advantage that it can be generally applied even when there is no matching data. In particular, in the case of magnetic resonance image reconstruction, , has the advantage of being able to replace one generator used in cycleGAN with one operation determined using the physical characteristics of magnetic resonance imaging, reducing the difficulty required for training and obtaining good performance while reducing the dependence on the data required for training. There is an advantage to being

In that respect, optimal transport (OT) provides a rigorous mathematical tool for understanding the geometry of unsupervised learning by CycleGAN. The geometrical perspective for unsupervised learning in the optimal transport theory is shown in FIG. Here, the target image space X has a probability measure μ, while the original image space Y has a probability measure ν. Since there is no paired data, the goal of unsupervised learning is to match probability distributions rather than each individual sample. We just need to find the transport map that transfers the measurement μ to ν and vice versa.

생성기에 의해 수행된다. 그런 다음, 생성기 G_θ는 Y에서 측정 ν를 대상 공간 X의 측정 μ_θ로 밀고 나간다. 마찬가지로, (X, μ)에서 (Y, ν)로의 운송은 다른 뉴럴 네트워크 생성기 F_φ에 의해 수행되므로, 생성기 F_φ가 X에서 측정 μ를 원래 공간 Y의 측정 ν_φ로 밀고 나간다. 그런 다음, 비지도 학습을 위한 최적 운송 맵은 μ와 μ_G 사이의 통계적 거리 dist(μ, μ_G) 및 ν와 ν_F 사이의 통계 거리 dist(ν, ν_F)를 최소화함으로써 달성될 수 있으며, 본 발명은 Wasserstein-1 메트릭을 통계적 거리를 측정하기 위한 수단으로 사용할 수 있다.More specifically, transport from one measurement space (Y, ν) to another measurement space (X, μ) is implemented by a deep neural network parameterized in θ.

performed by generators. The generator G _θ then pushes the measurement ν in Y into the measurement μ _θ in the target space X. Similarly, the transport from (X, μ) to (Y, ν) is performed by another neural network generator F _φ , so that the generator F _φ pushes the measurement μ in X into the measurement ν _φ in the original space Y. Then, the optimal transport map for unsupervised learning can be achieved by minimizing the statistical distance dist(μ, μ _G ) between μ and μ _G and the statistical distance dist(ν, ν _F ) between ν and ν _F and , the present invention can use the Wasserstein-1 metric as a means for measuring statistical distance.

More specifically, if we choose the metric d(x, x') = ||x-x'|| in X, the Wasserstein-1 metric between μ and μ _G can be calculated as shown in Equation 2 below: there is.

[Equation 2]

는 각각 X와 Y의 한계 분포가 μ 및 ν인 공동 측정값의 집합을 의미할 수 있다.here

may mean a set of joint measurement values of which the marginal distributions of X and Y are μ and ν, respectively.

Similarly, the metric between ν and ν _F can be calculated as shown in Equation 3 below.

[Equation 3]

Since the goal is to find a transport map represented by the common distribution π, separate minimization of Equations 2 and 3 is not desirable. Instead, it should be minimized with the same joint distribution π, and can be expressed as in Equation 4 below.

[Equation 4]

Here, the transportation cost can be expressed as in Equation 5 below.

[Equation 5]

At this time, the primitive formulation of unsupervised learning in Equation 4 can be expressed as a double formula as shown in <Equation 6> and <Equation 7> below together with transportation cost.

[Equation 6]

[Equation 7]

Here, λ > 0 means a hyperparameter, and the loss of the cycle coherence term can be expressed as in Equation 8 below, and the loss of the discriminator term can be expressed as in Equation 9 below.

[Equation 8]

[Equation 9]

Here, φ and ψ are often called Kantorovich potentials and can satisfy the 1-Lipschitz condition, the following equation.

In addition, when the forward operator F is known, optimization of F in Equation 6 is no longer necessary, which can be expressed as a simplified discriminator loss as shown in Equation 10 below.

[Equation 10]

The method of the present invention shows that these two types of optimal transport driving cycle GAN (OT-CycleGAN) are useful for the two-step reconstruction method.

forward model

One of the most widely used 3D TOF techniques is MOTSA, which stands for Multiple Overlapping Thin Slab Acquisition. MOTSA involves the sequential capture of several overlapping 3D volumes (or slabs). Each slab contains a relatively small number of slices, so signal loss due to saturation effects is relatively limited. However, some fluctuations in the signal still occur in the end slice due to saturation effects, so MOTSA only extracts the central part for each of the duplicate acquisitions to construct the final dataset to be treated as an MRA projection. End slices are usually discarded or averaged with slices from adjacent MOTSA sections.

In the accelerated MOTSA acquisition, the 3D scan has the same sampling mask as shown in Fig. 3 when viewed in the coronal plane. When Fourier transform is performed along the reading direction, a forward problem as shown in Equation 11 below occurs.

[Equation 11]

Here, x can be defined as in Equation 12 below.

[Equation 12]

는 2D 공간 푸리에 변환을 의미하며,

는 도 3과 같은 샘플링 마스크

의 투사 연산자를 의미할 수 있다.Here, C means the number of coils,

denotes a 2D spatial Fourier transform,

is the sampling mask as shown in FIG.

may mean the projection operator of

Two-step unsupervised 3D TOF reconstruction

As shown in FIG. 3 , in the case of the given forward model of Equation 11 obtained through the sampling plan along the coronal plane, reconstruction should also be performed in the coronal plane direction. However, this causes problems because radiologists usually review images in the axial plane and the reconstruction plane is not aligned with the radiologist's viewing plane, so remaining reconstruction artifacts in the coronal plane direction can degrade diagnostic performance. You can solve this with 3D learning, but the memory requirements for training 3D neural networks are much larger than standard GPU memory, which prohibits their use.

Therefore, the method of the present invention is a two-step approach, in which first-step reconstruction is performed along the coronal direction as shown in Fig. 4a, and axial-direction refinement is performed in the second step. At this time, the size of the image performed in the coronal direction in the first step can be adjusted so that the second step can be performed. In particular, it is worth emphasizing that performing this two-step reconstruction without matched reference data is useful for network design.

Proposition 1. It is assumed that the transport cost for the original OT problem of Equation 4 above is calculated as shown in Equation 13 below.

[Equation 13]

는 결정론적 선형(또는 비선형) 연산자를 의미하고, b와 c는 결정론적 선형(또는 비선형) 함수를 의미한다. 즉 b_x: X→R과 b_y: Y→R이다. 그 후, 해당 이중 OT 문제는 아래 <수학식 14> 및 <수학식 15>와 같이 나타낼 수 있다.here,

denotes a deterministic linear (or non-linear) operator, and b and c denote deterministic linear (or non-linear) functions. That is, b _x : X→R and b _y : Y→R. Then, the double OT problem can be expressed as <Equation 14> and <Equation 15> below.

[Equation 14]

[Equation 15]

In this case, the loss of the cycle coherence term can be expressed as in <Equation 1> below, and the loss of the discriminator term can be expressed as in <Equation 16> below.

[Equation 16]

The last two terms together with the 1-Lipschitz function φ, ψ can be given as <Equation> below.

Step 1: Reconstruct the coronal plane

Proposition 1 can be used to derive the method of the present invention. First, in order to make the dimensions of X and Y the same, the forward model of Equation 11 can be converted into an image domain forward formula by first taking an inverse Fourier transform, and can be expressed as in Equation 17 below.

[Equation 17]

을 의미하고,

은 역 푸리에 변환을 의미할 수 있다. 그런 다음 운송 비용은 아래 <수학식 18>과 같이 정의될 수 있다.here,

means,

may mean inverse Fourier transform. Then, the transport cost can be defined as in Equation 18 below.

[Equation 18]

는 다중 코일 데이터에 대해 제곱합 제곱근(SSOS; square-root of sum of squares) 연산 z =

(X)로 정의될 수 있다. 이 때, 벡터 z의 n번째 성분이 공식적으로 아래 <수학식 19>와 같이 정의될 수 있다.where α and β denote suitable hyperparameters,

is the square-root of sum of squares (SSOS) operation z =

(X) can be defined. At this time, the nth component of the vector z may be formally defined as in Equation 19 below.

[Equation 19]

The transport cost c(x, y; G, F) deserves further discussion. Specifically, the first and second terms of Equation 18 are directly related to the terms of OT cycleGAN, but the loss is calculated after taking the SSOS to make the image comparison less dependent on the coil sensitivity map. On the other hand, the identity (ID) loss, the third term of Equation 18, forces the neural network to normalize so as not to change the image already in the Y region, and the last term of Equation 18 is the data fidelity term in the k-space region. says For each coil, the k-space loss is calculated using the Frobenius norm to apply data consistency to the k-space data acquired in the inherently complex domain.

In addition, if the transportation cost of the present invention is set as shown in <Equation 20> and <Equation 21> below in order to use the dual formula of Proposition 1, it can be seen that it is the same as Equation 13 above.

[Equation 20]

[Equation 21]

는 이미 알려져 있기 때문에 F와 ψ의 경쟁은 필요없고 G와 해당 판별기 φ만 추정하면 된다. 각각 매개변수

와

를 가진 뉴럴 네트워크로 모델링함으로써, 아래 <수학식 23> 및 <수학식 24>와 같은 손실 함수를 얻을 수 있다.Moreover, the k-space sampling mask

Since is already known, there is no need to compete with F and ψ, and only G and the corresponding discriminator ϕ need to be estimated. each parameter

and

By modeling with a neural network having , loss functions such as <Equation 23> and <Equation 24> below can be obtained.

[Equation 22]

[Equation 23]

Here, γ, α, and β denote hyperparameters, and the loss of the cycle coherence term, the loss of the discriminator term, the loss of the identity term, and the loss of the frequency term are expressed in Equation 24, Equation 25, and Equation 26 below. And it can be expressed as in Equation 27.

[Equation 24]

[Equation 25]

[Equation 26]

[Equation 27]

Step 2: Axial Reconstruction

After reconstruction through the first step, the outputs are stacked together to form a single slab. However, when viewing images in the axial plane, the images tend to be blurry and lack proper texture. Accordingly, when performing MIP, thin vascular structures may be omitted or separated, leading to misdiagnosis such as vascular stenosis.

의 닫힌 형태 매핑이 존재하지 않는다. 이 상황은 정방향 및 역방향 연산자를 모두 알 수 없는 OT-cycleGAN 공식에 해당된다. 구체적으로는

에 의해 매개변수화된 뉴럴 네트워크 관점에서 포워드 연산자 F를 정의함으로써, 아래 <수학식 28>과 같은 운송 비용을 정의할 수 있다.Accordingly, the present invention provides an axial image augmentation method using another unsupervised neural network to improve the quality of MIP images in particular. More specifically, as shown in FIG. 4B, after reconstruction in the coronal plane, a 3D volume is constructed for each slab, which is used as an input to the axial image enhancement network. The rationale for using the accumulated volume as input is as follows. First, by performing MIP on the volume using the volume data, the network can learn the distribution of the partially projected image. Second, the network can use the information in the boundary slice because it can be inferred from the adjacent slice. One thing to note here is that the relationship between the input and output areas in the second stage is not well defined. Specifically, using some notation, the distribution of the 3D volume of the SSOS image reconstructed through step I is Y, and the 3D volume of the desired SSOS image distribution is X. Unlike step I where one of the generators can be replaced with a known forward operator, in this case due to SSOS operation and volume stacking

There is no closed form mapping of This situation corresponds to the OT-cycleGAN formulation where both the forward and reverse operators are unknown. Specifically

By defining the forward operator F in terms of the neural network parameterized by

[Equation 28]

Then, the corresponding OT-cycleGAN formula is given by the Kantorovich double formula in Equation 6, where l _cycle and l _Disc are cycle coherence loss and Wasserstein GAN loss represented by Equation 8 and Equation 9, respectively. The resulting network architecture may be as shown in FIG. 5B.

One of the key features of the second step comes from the design of the discriminator φ in Equation 9 above. More specifically, in order to become an OT-cycleGAN, the discriminator φ must satisfy the 1-Lipschitz condition of Equation 29 below.

[Equation 29]

The discriminator architecture in the present invention can be obtained from PatchGAN as shown in FIG. 7b. However, care must be taken as X is composed of 3D slabs. Thus, the slice directions are stacked in the channel dimension, allowing direct exploitation of PatchGAN's 2D convolution. In the first path, as shown in Fig. 7a, the volume data is directly used as input to PatchGAN. In the second path, maximum pooling is applied along the slice direction to generate a 2D image, which is used as an input to PatchGAN as shown in Fig. 7a. This is virtually equivalent to applying PatchGAN to the MIP images of each slab, and is necessary to learn the distribution of MIPs. The quality of MIP images is important in that MIP images are primarily used for vascular pathologists. And, the source video usually serves as an auxiliary tool.

Mathematically, the result discriminator φ can be expressed as in Equation 30 below.

[Equation 30]

Here, φ ₁ and φ ₂ ^max mean a discriminator between the original volume and the maximum parameter image, respectively, and λ ₁ and λ ₂ may mean appropriate hyperparameters. After that, the resulting discriminator loss function of Equation 9 can be decomposed as shown in Equation 31 below.

[Equation 31]

및

에 의해 매개변수화된 뉴럴 네트워크를 사용하여 구현될 수 있고, 반면 판별기는

이며 φ는 각각 가중치

와

를 갖는 뉴럴 네트워크를 사용하여 실현될 수 있다. Here, generators G and F are respectively

and

can be implemented using a neural network parameterized by , while the discriminator is

and φ is each weight

and

It can be realized using a neural network with

By jointly optimizing a set of discriminators that learn the distribution of stack volumes and a MIP discriminator that learns the distribution of MIP images, the method of the present invention can significantly improve the quality of MIP images while maintaining the integrity of the source images. The overall architecture of Phase II reconstruction is as shown in Figure 5b.

training dataset

19 sets of in vivo data can be acquired with a 3T Philips Ingenia scanner from, for example, 10 patients who volunteered for the scan.

In terms of the number of slices used to train the neural network, a total of 18343 full acquisition slices and 18356 under sample slices can be used to train the Phase I neural network. In phase II training, we can train a neural network using full capture 540 slices and 540 under sample slices.

All scans can be directed to an area covering the entire brain with a field of view (FOV) of 180 × 180 mm, and specific parameters for the scan can be defined as follows. Repetition time (TR) = 23.00 ms, echo time (TE) = 3.45 ms, FA = 18.00°. Furthermore, partial Fourier factors can be applied in the frequency encoding direction. Each set was obtained through MOTSA consisting of 6 slabs, and the k-space matrix size can be 774 × 359 × 21 and 30 coils. When the k-space data is filled, the final reconstruction is obtained with a matrix size of 512 × 512 × 45 with zero padding and center cropping. 12 sets of patient data can be used for training, and 7 sets of patient data can be used for simulation studies and in vivo studies.

의 경우, Philips Ingenia 스캐너에서 MR 스캔을 가속화하기 위해 사용되는 것과 동일한 마스크가 수정 없이 사용될 수 있다. 따라서 결정된 두 개의 마스크는 각각 ×4 가속도와 ×8 가속도에 사용될 수 있다.undersampling mask

For , the same mask used to accelerate MR scans in Philips Ingenia scanners can be used without modification. Thus, the determined two masks can be used for ×4 acceleration and ×8 acceleration, respectively.

generator architecture

For a single generator used for Phase I training, the present invention can use a modified U-Net architecture. As shown in FIG. 6, the modified U-Net architecture consists of four convolutional layers, ReLU activation, and group normalization. The pooling and unpooling operations can consist of 3 × 3 convolution with a stride of 2, and bilinear interpolation upscaling. The number of convolution filter channels can be set to 64 in step I, and can be increased by a factor of 2 in each step to reach 1024 in the last step. To cope with the nature of complex MR data, the present invention adheres to the traditional concept by stacking real and virtual components in the channel dimension. Therefore, the dimension of the input channel can be set to 60 (30 coils × 2 = 60).

에서는 하나의 11개의 컨볼루션 계층을 주의 모듈로 활용할 수 있다.Furthermore, we can utilize non-linear attention modules known to improve the expressive power of networks. For G in stage I, we can use the same network architecture of Fig. 6a as for the attention module, because of the large discrepancy between the input and the desired distribution. Furthermore

In , one 11 convolution layers can be used as attention modules.

와

를 위해 두 개의 별도 아키텍처를 사용할 수 있다. 중요한 생성기

에 대해서는 도 6a에 도시된 바와 같이 U-Net 아키텍처를 채택하고 초기 필터 길이를 3단계로 32로 설정할 수 있다. 네트워크 입력은 복수의 슬라이스 영상으로 구성된 3D 볼륨으로, 채널 방향을 따라 쌓는다. 네트워크 출력은 동일한 수의 슬라이스로 3D 볼륨을 향상시킨다. 7의 슬라이스 깊이를 사용할 수 있다. 생성기

의 경우 초기 필터 길이를 2단계만으로 8로 설정하여 네트워크의 표현력을 제한할 수 있다. 규모에 따라 두 네트워크를 차별화하면 동일한 두 네트워크를 사용했을 때보다 효율적이고 안정적인 트레이닝이 이루어질 수 있다. 다시 F의 입출력 역시 3차원 볼륨으로, 각 슬라이스가 채널 방향을 따라 쌓인다.In phase II training, mapping

and

Two separate architectures are available for important generator

As shown in FIG. 6A, the U-Net architecture can be adopted and the initial filter length can be set to 32 in three steps. The network input is a 3D volume composed of multiple slice images, stacked along the channel direction. The network output enhances the 3D volume with the same number of slices. A slice depth of 7 can be used. generator

In the case of , the expressive power of the network can be limited by setting the initial filter length to 8 with only two steps. Differentiating the two networks according to scale can result in more efficient and stable training than using the same two networks. Again, the input and output of F is also a three-dimensional volume, with each slice stacked along the channel direction.

discriminator architecture

The discriminator used in step 2 can use discriminators used in existing studies and can be modified to stabilize the training process. Specifically, patchGAN with a 3-step 4 × 4 convolutional kernel can be used. Each step may consist of a convolutional layer, instance normalization, and a leaky ReLU activation function, as shown in FIG. 7B. Spectral normalization can also be applied to each layer for stability.

는 입력으로 직접 볼륨을 수신하는

과 최대풀링 연산에서 획득한 단일 슬라이스 영상을 수집하는

의 두 경로를 가진다.

과

는 도 7b에 도시된 바와 같이 이중 헤드 판별기

로 볼 수 있다.The discriminator architecture in phase II training is as shown in Fig. 7b. For a given volume data

to receive the volume directly as input

and to collect single slice images obtained in the maximum pooling operation

has two paths.

class

is a double head discriminator as shown in FIG.

Can be seen as.

network training

= 100, α = 0.5, β= 1로 설정될 수 있고, 최적화를 위해 RAdam 옵티마이저를 lookahead 옵티마이저와 함께 사용할 수 있다. RAdam의 매개변수는 β1 = 0.5, β2 = 0.999로 설정될 수 있고, lookahead에 대한 파라미터는 k = 5, α = 0.5로 설정될 수 있다. 초기의 학습율은 0.0001로 설정될 수 있으며, 100epoch을 목표로 트레이닝될 수 있다. 60epoch의 트레이닝에서 학습률은 0.1의 정도로 감쇄할 수 있다.In the first training step, the hyperparameters of the first and second terms of Equation 23 are

= 100, α = 0.5, β = 1, and the RAdam optimizer can be used together with the lookahead optimizer for optimization. The parameters of RAdam can be set to β1 = 0.5, β2 = 0.999, and the parameters for lookahead can be set to k = 5 and α = 0.5. The initial learning rate may be set to 0.0001, and training may be performed with a goal of 100 epochs. In 60 epochs of training, the learning rate can decay by an order of magnitude of 0.1.

In the case of two-step training, the hyperparameters of Equation 31 above can be set to λ1 = 5 and λ2 = 3, and in the second step, the Adam optimizer can be used with parameters β1 = 0.5 and β2 = 0.999, and a consistent result of 0.0001 You can train for 100 epochs with the learning rate.

Figure 8 shows an exemplary view of a maximum intensity projection image according to the method of the present invention, Figure 8a shows the results of two-step learning using supervised learning, and Figure 8b shows learning using only the first step of unsupervised learning. Figure 8c shows the learning result using both stages of unsupervised learning, and Figure 8d shows the actual image that is not accelerated.

As can be seen from FIG. 8, when both the first step and the second step of the method of the present invention are used, the method using both steps makes blood vessels more visible and improves the image quality in the maximum intensity projection image. You can see what helps. In addition, even when compared with the results learned through supervised learning, it can be confirmed that the method of the present invention, learned without matching data, shows better results. In addition, the two numbers marked on each image in FIG. 8 are peak signal to noise ratio (PSNR) and structural similarity (SSIM). If the number is higher, it means that it is similar to the original image, It can be confirmed that these absolute indicators are also the highest in the method of the present invention.

9 shows an exemplary diagram comparing reconstruction results by the existing method and the method of the present invention, and when reconstructing an image accelerated by 8 times, an image obtained through a compression sensing algorithm built into an imaging device of an existing vendor. It shows the result of comparing (a) with the image (b) reconstructed by the method of the present invention.

As can be seen from FIG. 9 , it can be seen that the signal-to-noise ratio is noticeably improved by the method of the present invention compared to the existing built-in compression sensing algorithm, and the image quality is greatly improved, such as invisible blood vessels.

As described above, the method of the present invention uses the optimal transport-based cycleGAN in two steps and uses a projection discriminator to dramatically accelerate the 3D time-of-flight magnetic resonance angiography image and It is possible to provide a deep learning technique based on unsupervised learning that significantly increases image resolution compared to the algorithm.

The method of the present invention divides image reconstruction into two steps as shown in FIG. 5, and in the coronal plane, image reconstruction is performed through optimal transport-based cycleGAN using a measurement operator of magnetic resonance imaging, and then , a three-dimensional volume image is created by combining the data of all coronal planes, and at this time, information obtained from multiple coils is combined into one through root sum square (SSOS) calculation. Image reconstruction in the next step is performed in the axial plane, which is a clinically important region. When constructing an optimal transport-based cycleGAN in the axial plane, it is possible to learn the distribution of projection images, not just the distribution of 3D images, by using thin 3D data rather than 2D data do. At this time, a new projection discriminator is used instead of a discriminator of a general adversarial neural network. The projection discriminator determines whether a given image is realistic in two ways, one is how to determine how realistic a given 3D image is, and the other is how to determine how realistic a maxpooled image is. . The reason for using this discriminator is as follows. When making judgments using magnetic resonance angiography images in clinical practice, the first thing to check before viewing the MRA source image is the Maximum Intensity Projection (MIP) image. The maximum intensity projection image is a method in which magnetic resonance vascular source images are stacked in three dimensions and only the largest value is projected in a predetermined direction. Through this, only blood vessels with strong signal strength can be visualized, and therefore, it is the first method to be used when trying to determine what kind of problem is in the blood vessels. In this way, focusing on the fact that maximum intensity projection images are preferentially used in clinical practice, great improvement can be obtained in both magnetic resonance vascular source images and maximum intensity projection images by using the upper projection image discriminator.

As such, the method according to an embodiment of the present invention can improve the quality of a 3D time-of-flight magnetic resonance vascular image by using a two-step unsupervised learning-based neural network.

In addition, the method according to the embodiment of the present invention uses a projection discriminator to improve the quality of not only the MRA source image but also the maximum intensity projection image (MIP image). By providing additional data, the quality of images can be dramatically improved.

In addition, the method according to the embodiment of the present invention can acquire good image quality without greatly impairing the image quality while dramatically increasing the image acquisition speed by directly mounting it on equipment for taking magnetic resonance angiography. In addition, the present invention is a general technique applicable to 3D image reconstruction such as MRI and CT, and can be applied in various medical imaging fields.

FIG. 10 shows a configuration of a magnetic resonance angiography image processing apparatus according to an embodiment of the present invention, and shows a conceptual configuration of a device performing the methods of FIGS. 1 to 9 .

Referring to FIG. 10 , a magnetic resonance angiogram processing apparatus 1000 according to an embodiment of the present invention includes a receiving unit 1010, a first restoring unit 1020, a generating unit 1030, and a second restoring unit 1040. includes

The receiving unit 1010 receives first plane images, eg, coronal images, of magnetic resonance vessels including artifacts, eg, aliasing artifacts.

The first restoration unit 1020 removes artifacts included in the received coronal images using an unsupervised learning-based first neural network generated based on the optimal transport (OT) theory, so that the artifacts are A three-dimensional volume is reconstructed by generating the removed coronal images.

In this case, the first neural network may be learned using a training dataset including mismatched data, and among neural networks including a convolution framelet-based neural network and a pooling layer and an unpooling layer. Any one neural network may be included.

The generation unit 1030 generates second plane images, for example, axial images, using the reconstructed 3D volume.

The second reconstruction unit 1040 reconstructs the final 3D volume by removing artifacts remaining in the axial images using the second neural network based on unsupervised learning generated based on the optimal transport (OT) theory.

At this time, the second restoration unit 1040 inputs a preset number of second plane images, for example, 7 second plane images in the depth direction among the generated second plane images to the second neural network, thereby generating second plane images. The final three-dimensional volume can be reconstructed by removing the artifacts remaining in the field.

In this case, the second neural network may be learned using a training dataset including mismatched data, and among a neural network based on a convolution framelet and a neural network including a pooling layer and an unpooling layer. Any one neural network may be included.

Furthermore, the second reconstruction unit 1040 may generate a maximum intensity projection image using the reconstructed final 3D volume.

Although the description of the device of FIG. 10 is omitted, each component constituting FIG. 10 may include all of the contents described in FIGS. 1 to 9, which is obvious to those skilled in the art.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (13)

Receiving coronal images of magnetic resonance vessels including artifacts;
Artifacts included in the received coronal images are removed using an unsupervised learning-based first neural network generated based on optimal transport (OT) theory, thereby generating coronal images from which artifacts are removed. Reconstructing a 3D volume by doing;
generating axial images using the reconstructed 3D volume; and
Reconstructing a final 3D volume by removing artifacts remaining in the axial images using a second neural network based on unsupervised learning generated based on the optimal transport (OT) theory
including,
Reconstructing the final three-dimensional volume
Reconstructing the final 3D volume by removing artifacts remaining in the axial images by inputting a predetermined number of axial images according to a depth direction from among the generated axial images to the second neural network; Improving the quality of reconstruction in terms of Maximum Intensity Projection (MIP) images by reconstructing the final 3D volume using a projection discriminator that learns both volume and maximum full image distributions
Magnetic resonance angiography image processing method characterized in that.
According to claim 1,
Each of the first neural network and the second neural network
A method for processing magnetic resonance vascular images, characterized in that learning is performed using a training dataset including mismatched data.
According to claim 1,
Each of the first neural network and the second neural network
An MRI processing method comprising a neural network of any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.
delete According to claim 1,
Generating a maximum intensity projection image using the reconstructed final 3D volume
Magnetic resonance angiography image processing method further comprising a.
Receiving first plane images of magnetic resonance blood vessels including artifacts;
A first plane image from which artifacts are removed by removing artifacts included in the received first plane images using an unsupervised learning-based first neural network generated based on optimal transport (OT) theory Reconstructing a three-dimensional volume by generating a;
generating second plane images using the reconstructed 3D volume; and
Reconstructing a final 3D volume by removing artifacts remaining in the second plane images using a second neural network based on unsupervised learning generated based on the optimal transport (OT) theory
including,
Reconstructing the final three-dimensional volume
By inputting a predetermined number of second plane images according to a depth direction among the generated second plane images to the second neural network, artifacts remaining in the second plane images are removed to obtain the final 3D volume reconstruction, and by using a projection discriminator that learns both the volume and maximum full image distribution, the final 3D volume is reconstructed to improve the quality of reconstruction in terms of Maximum Intensity Projection (MIP) images. thing
Magnetic resonance angiography image processing method characterized in that.
According to claim 6,
Reconstructing the final three-dimensional volume
By inputting a predetermined number of second plane images according to a depth direction among the generated second plane images to the second neural network, artifacts remaining in the second plane images are removed to obtain the final 3D volume A magnetic resonance angiography image processing method characterized by reconstruction.
According to claim 6,
Generating a maximum intensity projection image using the reconstructed final 3D volume
Magnetic resonance angiography image processing method further comprising a.
a receiving unit for receiving coronal images of magnetic resonance vessels including artifacts;
Artifacts included in the received coronal images are removed using an unsupervised learning-based first neural network generated based on optimal transport (OT) theory, thereby generating coronal images from which artifacts are removed. a first restoration unit for reconstructing a 3-dimensional volume;
a generation unit generating axial images using the reconstructed 3D volume; and
A second restoration unit that reconstructs a final 3D volume by removing artifacts remaining in the axial images using a second neural network based on unsupervised learning generated based on the optimal transport (OT) theory.
including,
The second restoring unit
Reconstructing the final 3D volume by removing artifacts remaining in the axial images by inputting a predetermined number of axial images according to a depth direction from among the generated axial images to the second neural network; Improving the quality of reconstruction in terms of Maximum Intensity Projection (MIP) images by reconstructing the final 3D volume using a projection discriminator that learns both volume and maximum full image distributions
Magnetic resonance angiography image processing apparatus, characterized in that.
According to claim 9,
Each of the first neural network and the second neural network
A magnetic resonance vascular image processing apparatus characterized in that learning is performed using a training dataset including mismatched data.
According to claim 9,
Each of the first neural network and the second neural network
A magnetic resonance vascular image processing apparatus comprising a neural network of any one of a convolution framelet-based neural network and a neural network including a pooling layer and an unpooling layer.
delete According to claim 9,
The second restoring unit
Magnetic resonance vascular image processing apparatus, characterized in that for generating a maximum intensity projection image using the reconstructed final three-dimensional volume.

Images