WO2019168310A1

WO2019168310A1 - Device for spatial normalization of medical image using deep learning and method therefor

Info

Publication number: WO2019168310A1
Application number: PCT/KR2019/002264
Authority: WO
Inventors: 이동영; 김유경; 이재성; 변민수; 신성아; 강승관
Original assignee: 서울대학교산학협력단
Priority date: 2018-02-28
Filing date: 2019-02-25
Publication date: 2019-09-06

Abstract

A device for spatial normalization of a medical image comprises: an adaptive template generation unit for generating an adaptive template for spatially normalizing a functional medical image on the basis of pre-stored learning data, if a plurality of functional medical images are inputted to a deep learning architecture; a learning unit which generates, for an inputted functional medical image of a user, an image on the basis of the adaptive template by means of a generative adversarial network (GAN), and which learns by repeating the process of determining whether the generated image is genuine; and a spatial normalization unit for providing a spatially normalized functional medical image of the user on the basis of a learning result.

Description

Spatial Normalization Device and Method for Medical Image Using Deep Learning

Provided are a spatial normalization apparatus and method for medical images using deep learning.

For statistical analysis of medical images, it is reasonable to normalize images from individual subjects in one space and compare them with the values of three-dimensional pixels.

In particular, spatial or quantitative standardization (SN) is an essential procedure for statistical comparison or objective evaluation of brain positron emission tomography (PET) and single photon emission computed tomography (SPECT) images.

Many errors occur when spatial quantitative normalization (SN) is performed using only brain positron emission tomography (PET) images. Specifically, positron emission tomography (PET), single photon emission computed tomography (SPECT), functional magnetic resonance imaging (fMRI), electroencephalogram (EEG), magnetic brain waves Magnetoencephalography (MEG) is difficult to normalize spatially using single information due to the limitations of functional image and the limitation of spatial resolution.

Therefore, in general, magnetic resonance imaging (MRI) or computed tomography (CT) is photographed together and spatially normalized first, and then the transformed vector field is applied to a functional image to perform spatial normalization.

This method has the advantage of obtaining accurate data, but it is time and cost constrained because of the high cost of using expensive equipment.

In addition, spatial quantitative normalization (SN) of brain positron emission tomography (PET) images is performed by using average templates obtained from various samples, but because they do not accurately reflect various characteristics between images such as patients and normal people. Accurate analysis is difficult.

Meanwhile, in recent years, researches using machine learning for deep learning based on big data have been conducted to successfully solve complex and high-level problems.

Thus, there is a need for a technique for performing accurate spatial normalization at low cost without using separate magnetic resonance imaging (MRI) or computed tomography (CT) imaging.

One embodiment of the present invention is to create an individually adaptable template using deep learning, and to spatially normalize individual medical functional images based on the generated template.

In addition to the above objects, it can be used to achieve other objects not specifically mentioned.

The apparatus for spatial normalization of medical images according to an embodiment of the present invention is adapted to generate an adaptive template for spatially normalizing functional medical images based on prestored learning data when a plurality of functional medical images are input into an in-depth learning architecture. Type template generation unit, learns by repeating the process of generating the image based on the adaptive template through the generational adversarial network (GAN) for the functional medical image of the user input and determine the authenticity of the generated image And a spatial normalization unit for providing a functional medical image of the user who is spatially normalized based on the learning result.

The adaptive template generator measures the difference between the adaptive template result derived from the deep learning architecture and the MRI-based spatial normalized image, which is pre-stored learning data, and can be individually adapted through deep learning to minimize the measured difference. You can create an adaptive template.

The learning unit generates a spatial normalized image by spatial normalization of the functional medical image of the user based on the adaptive template, and compares the spatial normalized image with the MRI-based spatial normalized image, which is pre-stored training data, to determine authenticity. When the determination result value is generated, the process of generating and determining a spatial normalized image is repeated in turn. If the determination result does not distinguish between the spatial normalized image and the MRI-based spatial normalized image which is previously stored training data, the corresponding learning is performed. You can finish it.

The learner may repeatedly learn to minimize the Jensen-Shannon Divergence between the data probability distribution of the spatial normalized image and the normalized MRI data probability distribution.

The learning unit may repeatedly learn through a process of solving the min-max problem by using the following equation.

where

Here, D () is a generator for generating an image, G () is a discriminator for determining an image, z is a functional medical image in a native space, x is an MRI-based SN result,

Are the parameters of the constructor and the discriminator, respectively, and E is the expected value for a given probability distribution,

Is the fidelity loss value between MRI-based SN results, m is the batch size, Ii MNI is the image representing the MRI-based SN results in the MNI space, and Ii Native is the functional medical image of the basic space.

Generating an adaptive template by applying a plurality of functional medical images corresponding to various environments to an in-depth learning architecture according to an embodiment of the present invention, Genetic adversarial for a functional medical image of an input user network (GAN) to generate an image based on the adaptive template and repeat the process of determining the authenticity of the generated image, and when the learning is completed, functional medical image of the spatial normalized user based on the learning result Providing a step.

According to one embodiment of the present invention, by generating an optimized template through an image obtained by using deep learning and performing spatial normalization, an error generated can be minimized and a high accuracy result can be obtained.

In addition, by normalizing the image using only positron emission tomography images, quantitative and statistical analysis is possible, so that many experiments can be carried out at low cost for diagnosis and research of patients.

In addition, by performing spatial normalization using an optimally individually adaptable template rather than a template using existing average data, spatial normalization may be accurately reflected to accurately reflect various characteristics of each individual PET image.

1 is a flowchart illustrating a process of spatial normalization of a medical image according to an exemplary embodiment of the present invention.

2 is a block diagram illustrating a spatial normalization apparatus for medical images using deep learning according to an exemplary embodiment of the present invention.

3 is an exemplary view for explaining an adaptive template unit and a learning unit according to an embodiment of the present invention.

4 is a flowchart illustrating a process of spatial normalization of a medical image according to an exemplary embodiment of the present invention.

5 is an exemplary diagram for describing an in-depth learning architecture and a generative host neural network according to an embodiment of the present invention.

FIG. 6 is a diagram for comparing spatial normalized images through a method, an average template method, and a method of utilizing magnetic resonance image information according to an embodiment of the present invention.

7 is a graph showing the ratio of the standardized understanding value of the method and the magnetic resonance image information utilization method and the ratio of the standardized understanding value of the average template method and the magnetic resonance image information utilization method according to an embodiment of the present invention. .

DETAILED DESCRIPTION Embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily practice the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. The drawings and description are to be regarded as illustrative in nature and not restrictive. Like reference numerals designate like elements throughout the specification. In addition, in the case of well-known technology, a detailed description thereof will be omitted.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

Hereinafter, the overall flow of the spatial normalization configuration of a medical image using deep learning according to an embodiment of the present invention will be described in detail with reference to FIG. 1.

As shown in FIG. 1, in the process of generating spatially normalized MRI by spatial normalization (SyN), a deformation field is generally used. Is extracted.

According to an embodiment of the present invention, the spatial normalization apparatus 100 trains a result of generating a spatially normalized PET from a functional medical image by using an extracted deformation field. Deep learning is performed using as a training target.

Then, the spatial normalization apparatus 100 may perform spatial normalization of a functional medical image (PET) without MRI (3D magnetic resonance imaging) or CT through deep learning.

Functional medical images used hereinafter show positron emission tomography (PET), which can represent physiological, chemical, and functional images of the human body in three dimensions using radiopharmaceuticals that emit positrons. It includes all functional medical images that are difficult to normalize spatially using independent information such as fMRI, EEG, and MEG.

Hereinafter, the spatial normalization apparatus 100 that performs spatial normalization on the functional medical image PET through the learned artificial neural network will be described in detail with reference to FIG. 2.

2 is a block diagram illustrating a spatial normalization apparatus for medical images using deep learning according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating an adaptive template unit and a learning unit according to an embodiment of the present invention. It is an illustration.

As shown in FIG. 2, the apparatus 100 for normalizing a medical image includes an adaptive template generator 110, a learner 120, and a spatial normalizer 130.

First, the adaptive template generator 110 generates an adaptive template 200 for spatial normalization of a functional medical image.

When the adaptive template generator 110 receives a plurality of functional medical images, the adaptive template generator 110 inputs a convolutional auto-encoder (CAE) to generate an adaptive template 200 that can be individually adapted through in-depth learning.

In this case, as shown in FIG. 3A, the adaptive template generator 110 may perform in-depth learning using the previously stored learning data 300.

Here, the prestored learning data 300 includes an image obtained by performing spatial normalization based on magnetic resonance imaging (MRI).

In addition, the deep learning architecture includes a convolutional neural network, which represents an artificial neural network that understands an input image through calculation, extracts features to obtain information, or generates a new image.

Next, as shown in (b) of FIG. 3, when the learning unit 120 receives a functional medical image of the user, the learner 120 generates a spatial normalized image using the adaptive template 200. The learner 120 determines the authenticity based on the generated spatial normalized image and the previously stored learning data.

In other words, when the learning unit 120 receives the functional medical image of the user, the learning unit 120 performs repetitive learning on the received functional medical image through a generative adversarial network (GAN).

Here, the Genetic Adversarial Network (GAN) is a machine learning method for generating an image similar to the original data distribution, and is used as a technology for easily and quickly making a real fake. Genetic antagonists (GANs) learn and derive results through competition between two neural network models: generators (G) and discriminators (D). The generator (G) learns the actual data and generates the data based on the purpose of generating data close to the actual, and the discriminator (D) determines whether the data generated by the generator (G) is real or false. Learn.

The learner 120 generates a spatial normalized image based on the functional medical image of the user input using the generated antagonistic neural network, and repeats the process of determining the authenticity of the generated image. Through such iterative learning, the learning unit 120 may derive a result very close to the result of spatial normalization of the functional medical image using the image of performing spatial normalization based on magnetic resonance imaging (MRI).

The learner 120 generates an image by spatially normalizing the functional medical image of the user until the generated spatial normalized image and the previously stored learning data cannot be determined, and determines whether the generated image is authentic. Repeat in turn.

Next, the spatial normalization unit 130 performs spatial normalization of the user functional medical image through the trained completed generative antagonist network. The spatial normalization unit 130 provides a functional medical image of the spatial normalized user.

Meanwhile, the spatial normalization apparatus 100 of the medical image may be a server, a terminal, or a combination thereof.

Terminals are collectively referred to as devices having arithmetic processing capability by providing a memory and a processor. For example, there are a personal computer, a handheld computer, a personal digital assistant, a mobile phone, a smart device, a tablet, and the like.

The server is a memory that stores a plurality of modules (module), the processor is connected to the memory and reacts to the plurality of modules, the processor for processing the service information or action information for controlling the service information provided to the terminal, communication Means, and a user interface (UI) display means.

Memory is a device for storing information. Non-volatile memory, such as highspeed random access memory, magnetic disk storage, flash memory devices, and other non-volatile solid-state memory devices. And various kinds of memories.

The communication means transmits and receives service information or action information with the terminal in real time.

The UI display means outputs service information or action information of the device in real time. The UI display means may be a separate device that directly or indirectly outputs or displays the UI, or may be part of the device.

Hereinafter, a process of generating an adaptive template by the spatial normalization apparatus 100 and generating a spatial normalized functional medical image using the generative antagonist network will be described in detail with reference to FIGS. 4 to 5.

4 is a flowchart illustrating a process of spatial normalization of a medical image according to an exemplary embodiment of the present invention, and FIG. 5 is an exemplary diagram for describing a deep learning architecture and a generative antagonistic neural network according to an exemplary embodiment of the present invention. to be.

As shown in FIG. 4, the spatial normalization apparatus 100 generates an adaptive template by inputting a plurality of functional medical images corresponding to various environments into an in-depth learning architecture (S410).

Here, the deep learning architecture (CAE) is formed of a plurality of layers, and all the convolutional layers extract features in a 3D manner. In addition, deep learning architecture (CAE) performs operations using stride convolution, and applies an exponential linear unit (ELU) activation function after the convolution.

In addition, deep learning architecture (CAE) applies batch normalization except for the final output layer.

The spatial normalization apparatus 100 may measure a difference between the adaptive template result derived through the deep learning architecture and the spatial normalization result based on the pre-stored MRI.

In other words, the spatial normalization apparatus 100 performs deep learning so that the loss function (LCAE) of Equation 1 is minimized in the process of measuring the difference between the output value of the deep learning architecture (CAE) and the pre-stored MRI based spatial normalization result. Can be done.

[Equation 1]

Where m is the batch size, IMNI is the image of MRI based spatial normalization (label) in the MNI space, INative is the functional medical image of the input user in the base space, and N is the number of voxels in the MNI space. Indicates.

As such, the spatial normalization apparatus 100 may generate an adaptive template capable of performing spatial normalization individually through in-depth learning of a deep learning architecture (CAE).

Next, the spatial normalization apparatus 100 receives a functional medical image of the user (S420).

The spatial normalization apparatus 100 may receive a functional medical image of a user from an associated user terminal (not shown) or a server (not shown), wherein the communication network is a wired communication network, a short-range or long-range wireless communication network, They can include any type of communication network that carries data, such as a mixed network.

In operation S430, the spatial normalization apparatus 100 generates an image by applying a functional medical image of the user to an adaptive template, and determines whether the generated image is true or not (S430).

The spatial normalization apparatus 100 may perform training while sequentially updating the generation model and the distinction model through the generative host neural network (GAN).

First, the spatial normalization apparatus 100 spatially normalizes a functional medical image of a user based on an adaptive template through a generator G of a generative host neural network GAN, and generates a spatial normalized image.

In addition, the spatial normalization apparatus 100 may determine the authenticity by comparing the spatial normalized image with the MRI-based spatial normalized image, which is pre-stored training data, through a discriminant (D) of the GN. have.

FIG. 5A is an exemplary diagram illustrating a network structure of a generator G of a generating antagonist neural network, and FIG. 4B is a discriminator of a generating antagonist neural network GAN. It is an exemplary figure which shows the network structure of D).

5, each red orange box represents a 3D strided convolutional kernel, where s is the size of the stride and k is the size of the kernel. Each blue box represents a batch normalization combined with an activation function such as Leak-ReLU or ELU. The green box shows the layer completely connected to the number of devices, and the purple box shows the 3D transposed convolutional layer (deconvolution) with two strides and kernel size 3.

On the other hand, as shown in (a) of FIG. 5, the generator (G) of the GN may be used as the same neural network as the deep learning architecture (CAE), but is not limited thereto.

The spatial normalization apparatus 100 generates the spatial normalized image when the determination result of the discriminator D is generated by using the generator G and the discriminator D of the generative host neural network GAN. Repeat the process in order.

The spatial normalization apparatus 100 is learned by generating a generative antagonist network (GAN) solving a min-max problem as shown in Equation 2 below.

[Equation 2]

Where z is a PET image of native space (input), x is the MRI-based spatial normalization result (label),

Denotes the parameters of the constructor and the discriminator, respectively, and E denotes the expectations for a given probability distribution.

First, max D represents finding the discriminator D that maximizes the objective function.

In Equation 2, the first term E [Log D (x)] is an MRI based spatial normalization result (x), which is a value of an objective function. And the second term E [log (1-D (G (z)))] contains the image G (z) created by the constructor, and for the arg max D, the term 1-D (G (z)) ), Maximizing the second term represents minimizing D (G (z)).

As a result, the spatial normalization apparatus 100 trains the discriminator D to output a large value by inserting a real picture and a small value by inserting a fake picture by using the objective function of the two terms.

Min G then finds the generation network G that minimizes the objective function.

Referring to Equation 2, G is included only in the second term, and G, which minimizes the entire function, is G, which minimizes the second term, and ultimately represents G that maximizes D (G (z)).

As a result, assuming the optimal discriminator D, the objective function for generator G is equal to minimizing Jensen-Shannon Divergence between D (x) and D (G (z).

Therefore, the spatial normalization apparatus 100 iteratively learns and updates the constructor (G) and the discriminator (D) so that the Janson-Shannon divergence between the data probability distribution of the spatial normalized image and the normalized MRI data probability distribution is minimized. Can be.

In addition, as shown in Equation 3, the spatial normalizer 100 may add a loss of fidelity between the image generated by the generator (G) and the MRI-based spatial normalization result (label) to the min-max problem.

[Equation 3]

where

Here, D () is a constructor for generating an image, G () is a discriminator for determining an image, z is a functional medical image in a native space, x is an MRI-based SN result,

Is the fidelity loss value between the MRI-based SN results, m is the batch size, IiMNI is an image representing the MRI-based SN results in the MNI space, and IiNative is a functional medical image of the base space.

As described above, the repeated learner G may generate an image obtained by spatially normalizing a functional medical image of a user having a high similarity with the MRI-based spatial normalization result.

In this process, if the discriminator (D) of the generative host neural network (GAN) cannot distinguish the MRI-based spatial normalization result from the spatial normalized image generated by the generator (G), the spatial normalization apparatus 100 performs the corresponding learning. You can finish it.

Next, the spatial normalization apparatus 100 provides a spatial normalized medical image by spatial normalizing a functional medical image of the user through the learned algorithm (S440).

The spatial normalization apparatus 100 may spatially normalize and provide a functional medical image of a user through the generator G in which the learning is completed in the generative host neural network GAN.

Meanwhile, the spatial normalization apparatus 100 may provide a spatial normalized functional medical image to which the corresponding adaptive template is applied using the adaptive template generated in step S410. In other words, the spatial normalization apparatus 100 may apply only to the deep learning architecture (CAE), generate only the adaptive template 200, and provide the functional medical image by spatial normalization using only the adaptive template 200. .

Hereinafter, comparing the spatial normalization method of the medical image and the spatial normalization method using the average template according to an embodiment of the present invention with reference to FIGS. 6 and 7 will be described in detail with respect to the magnetic resonance image information .

FIG. 6 is a diagram for comparing spatial normalized images through a method, an average template method, and magnetic resonance image information using method according to an embodiment of the present invention, and FIG. 7 is a method according to an embodiment of the present invention. And the ratio of the standardized understanding value of the method of utilizing MRI information and the standardized understanding value of the average template method and the method of using MRI information.

6A is a spatial normalized image of a PET image using an average template, and (b) is a spatial normalized image of a PET image using the method proposed by the present invention, and (c) represents magnetic resonance image information. Spatial normalized images of PET images.

Referring to FIG. 6, while the images of (b) and (c) are almost identical, the image of (a) shows a large difference in the direction of the arrow and the circled area. In other words, when the PET image is spatially normalized using the average template, the accuracy is very low, while the spatial normalization of the PET image through the deep learning proposed in the present invention, the accuracy is very high.

7 illustrates a quantitative analysis result of the spatial normalization method and the spatial normalization method using an average template according to an embodiment of the present invention.

7 is a graph showing a ratio of standardized uptake value (SUVr), and a result of comparing an error with magnetic resonance image information by selecting a certain region of the brain.

Referring to FIG. 7, while the error value appears to be close to 20% of the error value of the spatial normalized result of the average template (white block), the deep learning architecture (CAE: Convolutional auto-encoder) used in the present invention is distinguished. The error of the model (hatched block) is less than 5%. In particular, the error of the GN proposed in the present invention shows the minimum error.

Therefore, the method of generating an adaptive template through the deep learning architecture proposed in the present invention and spatially normalizing the functional medical image through the generating hostile neural network using the generated adaptive template utilizes the actual magnetic resonance image information. You can see that this is most similar to the normalization method.

The program for executing the method according to one embodiment of the present invention can be recorded on a computer readable recording medium.

Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like. The media may be those specially designed and constructed or those known and available to those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, flash memory, and the like. Hardware devices specifically configured to store and execute the same program instructions are included. In this case, the medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although one preferred embodiment of the present invention has been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of the present invention.

100: spatial normalization apparatus of the medical image 110: adaptive template generator

120: learning unit 130: spatial normalization unit

200: adaptive template 300: training data

Claims

An adaptive template generator for generating an adaptive template for spatially normalizing the functional medical image based on pre-stored learning data when a plurality of functional medical images are input into the deep learning architecture;

Learning unit for repeatedly learning the generated image based on the adaptive template for the functional medical image of the user received through a generative adversarial network (GAN) and determining the authenticity of the generated image , And

And a spatial normalization unit for providing a functional medical image of a user who is spatially normalized based on the learning result.
In claim 1

The adaptive template generator,

Measure the difference between the adaptive template result derived from the deep learning architecture and the MRI-based spatial normalized image, which is pre-stored learning data, and create an individually adaptable adaptive template through deep learning to minimize the measured difference. Spatial normalization device for medical imaging.
In claim 1,

The learning unit,

Spatially normalize the functional medical image of the user based on the adaptive template to generate a spatially normalized image,

The authenticity is determined by comparing the spatial normalized image with the MRI-based spatial normalized image, which is pre-stored learning data. When the determination result is generated, the process of generating and determining the spatial normalized image is sequentially performed.

And if the spatial normalized image and the MRI-based spatial normalized image, which are previously stored learning data, are not distinguished as a result of the determination, completing the corresponding learning.
In claim 3,

The learning unit,

And a spatial normalization apparatus of the medical image for repeating learning to minimize the Jensen-Shannon Divergence between the data probability distribution of the spatial normalized image and the normalized MRI data probability distribution.
In claim 1,

The learning unit,

Spatial normalization device for medical images repeatedly learned through the process of solving the min-max problem using the following equation:

where

Here, D () is a generator for generating an image, G () is a discriminator for determining an image, z is a functional medical image in a native space, x is an MRI-based SN result,
Are the parameters of the constructor and the discriminator, respectively, and E is the expected value for a given probability distribution,
Is the fidelity loss value between MRI-based SN results, m is the batch size, Ii MNI is the image representing the MRI-based SN results in the MNI space, and Ii Native is the functional medical image of the basic space.
Generating an adaptive template by applying a plurality of functional medical images corresponding to various environments to an in-depth learning architecture,

Repeating the process of generating an image based on the adaptive template and determining the authenticity of the generated image through a generative adversarial network (GAN) for the received functional medical image of the user; And

And providing a functional medical image of a user who is spatially normalized based on the learning result when the learning is completed.
In claim 6,

Generating the adaptive template,

The space of the medical image which measures the difference between the adaptive template result derived from the deep learning architecture and the pre-stored MRI-based spatial normalization result, and generates the adaptive template which can be individually adapted through in-depth learning to minimize the measured difference. Method of normalization device.
In claim 6,

The learning step,

Spatial normalizing the functional medical image of the user based on the adaptive template, and generating a spatial normalized image;

Comparing the spatial normalized image with the MRI-based spatial normalized image, which is previously stored learning data, to determine authenticity;

Repeating the steps of generating and determining a spatial normalized image when the determination result value is generated, and

If it is not possible to distinguish between the spatial normalized image and the MRI-based spatial normalized image which is pre-stored learning data as a result of the determination, completing the corresponding learning.
In claim 8,

Repeating the step,

And a method of repetitive learning of medical images to minimize the Jensen-Shannon Divergence between the data probability distribution of the spatial normalized image and the normalized MRI data probability distribution.
In claim 6,

The learning step,

Spatial normalization method of medical images repeatedly learned through the process of solving the min-max problem using the following equation:

where

Here, D () is a generator for generating an image, G () is a discriminator for determining an image, z is a functional medical image in a native space, x is an MRI-based SN result,
Are the parameters of the constructor and the discriminator, respectively, and E is the expected value for a given probability distribution,
Is the fidelity loss value between MRI-based SN results, m is the batch size, Ii MNI is the image representing the MRI-based SN results in the MNI space, and Ii Native is the functional medical image of the basic space.