KR102548203B1 - Apparatus and method for analyzing medical image based on artificial neural network and learning method thereof - Google Patents

Abstract

According to the present invention, a first medical image and a second medical image obtained by capturing images of the same subject at different times are received as inputs, and the first medical image and the second medical image are obtained by using a pre-learned medical image analysis model. Disclosed are an artificial neural network-based medical image analysis device and method for discriminating changes between images, and a learning method thereof.

Description

Artificial neural network-based medical image analysis device, method, and learning method thereof

The present invention relates to a medical image analysis apparatus, method, and learning method thereof.

X-ray imaging is a technique that can obtain images of the inside of the human body in the field of radiology and has been used in the field of diagnosis and treatment of various diseases. Depending on the purpose of diagnosis, such as a neck soft tissue scan or mammogram, it can be obtained with anteroposterior, lateral, or diagonal scans.

X-ray imaging is very inexpensive compared to tests such as computed tomography (CT) and magnetic resonance imaging (MRI), and the number of diseases diagnosed through it is very simple and simple.

However, in distinguishing changes through X-ray images, it is difficult to solve using existing CNN networks because it is necessary to distinguish only changes according to diseases from the criteria determined by radiologists while ignoring changes in the patient's breathing, posture, age, etc. There are difficulties.

An object to be solved according to an embodiment of the present invention includes discriminating a change between medical images obtained for the same diagnosis subject using a model trained using read images of a subject.

In addition, it includes making the feature maps of each read image similar to each other.

Other non-specified objects of the present invention may be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

In order to solve the above problems, an artificial neural network-based medical image analysis apparatus according to an embodiment of the present invention includes a memory in which a pre-learned medical image analysis model is stored and a first image obtained by capturing images of the same diagnosis subject at different times. and a processor that receives a medical image and a second medical image and determines a change between the first medical image and the second medical image using the medical image analysis model, wherein the medical image analysis model is pre-read. The first readout image and the second readout image of an arbitrary subject are used as learning input images, and information obtained by reading the change between the first readout image and the second readout image is used as a readout label for learning.

Here, the processor may determine a change region according to a disease among changes between the first medical image and the second medical image.

Here, the determined change area may have a minimized effect due to changes in respiration, posture, and age of the subject to be diagnosed.

A method for learning a medical image analysis model based on an artificial neural network according to another embodiment of the present invention, wherein a processor sets a pre-read first read image and a second read image of a subject as learning input images, and the first read image Based on the step of inputting the information obtained by reading the change between the read image and the second read image as a read label to the first network and the output result of the similarity value between the first read image and the second read image, the first read image 1 adjusting a first parameter of the network.

A method for learning a medical image analysis model based on an artificial neural network according to another embodiment of the present invention includes the steps of receiving a first medical image and a second medical image obtained by imaging the same diagnosis subject at different times and learning in advance and determining a change between the first medical image and the second medical image by using a medical image analysis model, wherein the medical image analysis model includes a first read image of a subject that has been previously read and a second read image of a subject. The readout image is used as a learning input image, and information obtained by reading the change between the first readout image and the second readout image is used as a readout label for learning.

In addition, a first read image and a second read image of an arbitrary subject that have been previously read are used as learning input images, and information obtained by reading a change between the first read image and the second read image is used as a read label, and the first network To perform a medical image analysis model learning method comprising the step of inputting input to and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. It is possible to provide a recording medium in which a computer program including a command to do is stored.

In addition, a first read image and a second read image of an arbitrary subject that have been previously read are used as learning input images, and information obtained by reading a change between the first read image and the second read image is used as a read label, and the first network To perform a medical image analysis model learning method comprising the step of inputting input to and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. It is possible to provide a computer program stored in a computer readable storage medium containing instructions to do.

As described above, according to the embodiments of the present invention, it is possible to determine a change according to a disease in a medical image obtained for the same diagnosis subject using a pre-learned model.

In addition, since the feature maps of each read image are similar to each other, a read error can be reduced.

Even if the effects are not explicitly mentioned here, the effects described in the following specification expected by the technical features of the present invention and their provisional effects are treated as described in the specification of the present invention.

1 is a block diagram illustrating a medical image analysis apparatus based on an artificial neural network according to an embodiment of the present invention.
2 is a flowchart illustrating a method for analyzing a medical image based on an artificial neural network according to an embodiment of the present invention.
3 is a flowchart illustrating a method for learning a medical image analysis model according to an embodiment of the present invention.
4 illustrates an example of a learning input image used for learning of a medical image analysis model according to an embodiment of the present invention.
5 is a diagram for explaining an exemplary structure of a medical image analysis model according to an embodiment of the present invention.
6 is a diagram for explaining a first network and a second network of a medical image analysis model according to an embodiment of the present invention.
7 is a diagram for explaining an effect according to a second network of a medical image analysis model according to an embodiment of the present invention.

Hereinafter, an artificial neural network-based medical image analysis apparatus and method, and a learning method thereof according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the described embodiments. And, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves.

The present invention relates to an artificial neural network-based medical image analysis apparatus, method, and learning method thereof.

1 is a block diagram illustrating a medical image analysis apparatus based on an artificial neural network according to an embodiment of the present invention.

Referring to FIG. 1 , an artificial neural network-based medical image analysis apparatus 10 according to an embodiment of the present invention includes a processor 11 and a memory 12 .

An artificial neural network-based medical image analysis apparatus 10 according to an embodiment of the present invention is a device for reading changes due to a disease from an X-ray image of a subject.

Specifically, it operates like a procedure read by actual radiologists by using a module that allows robust learning to changes in subjects' natural variation (breathing, age, shooting posture), etc.

To this end, it is designed as a CNN structure for distinguishing changes and utilizes multiple labels. The medical image analysis model of the artificial neural network-based medical image analysis apparatus 10 includes a first network, a second network, and a disease information providing neural network. It may include an adjustment label, which is a target pixel value of the line image, and a disease label, which is information obtained by reading a change according to a disease.

The memory 12 may store programs (one or more instructions) for processing and control of the processor 11, store a pre-learned medical image analysis model, flash memory type, hard disk Hard disk type, multimedia card micro type, card type memory (eg SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory) , ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, at least one type of computer-readable storage medium can include

Programs stored in the memory 12 may be divided into a plurality of modules according to functions, and a pre-trained medical image analysis model to be described later may be composed of software modules.

The processor 11 executes one or more instructions stored in the memory 12. Specifically, the processor 11 receives a first medical image and a second medical image obtained by imaging the same diagnosis subject at different times as input, and generates a medical image analysis model. A change between the first medical image and the second medical image is determined using .

Specifically, among the changes between the first medical image and the second medical image, a change region according to a disease is discriminated, and the determined change region has a minimized effect due to changes in breathing, posture, and age of the subject to be diagnosed.

Here, the processor 11 may be divided into a plurality of modules according to functions, or functions may be performed by one processor. A processor is one or more of a central processing unit (CPU), an application processor (AP), a micro controller unit (MCU), or a communication processor (CP). can include

In the medical image analysis model of the artificial neural network-based medical image analysis apparatus 10 according to an embodiment of the present invention, a first read image and a second read image of an arbitrary subject are read as learning input images, and Information obtained by reading the change between the first read image and the second read image is learned as a read label.

The medical image analysis model is described in detail in FIG. 5 below.

2 is a flowchart illustrating a method for analyzing a medical image based on an artificial neural network according to an embodiment of the present invention.

Referring to FIG. 2 , the method of analyzing a medical image based on an artificial neural network according to an embodiment of the present invention is performed by a medical image analysis device, and in step S10, a first medical image obtained by imaging the same diagnosis subject at different time points An image and a second medical image are input.

In step S20, a change between the first medical image and the second medical image is determined using the pre-learned medical image analysis model.

Specifically, among the changes between the first medical image and the second medical image, a change region according to a disease is discriminated, and the determined change region has a minimized effect due to changes in breathing, posture, and age of the subject to be diagnosed.

Here, the medical image analysis model sets a first read image and a second read image of an arbitrary subject as learning input images, and reads information about changes between the first read image and the second read image as a read label. It is learned by

3 is a flowchart illustrating a method for learning a medical image analysis model according to an embodiment of the present invention.

Referring to FIG. 3 , the method for learning a medical image analysis model according to an embodiment of the present invention is performed by a processor, and a first read image and a second read image of an arbitrary subject pre-read in step S110 are used as learning input images. , and information obtained by reading the change between the first read image and the second read image is input to the first network as a read label.

In step S120, a first parameter of the first network is adjusted based on an output result of similarity values between the first readout image and the second readout image.

Here, in the step of adjusting the first parameter of the first network (S120), a first feature vector is extracted from the first read image, and a second feature vector is extracted from the second read image, and the first feature vector and the second feature vector are extracted. 2 After integrating the first feature map and the second feature map generated as the feature vector passes through at least one layer using an integrated layer, the first feature map is generated based on the output result of the similarity value passed through a plurality of layer structures. Adjust the parameters.

In step S130, the first read image and the second read image passing through at least one layer of the first network and the target pixel value are input to the second network as an adjustment label.

In step S140, a second parameter of the second network is adjusted so that a feature map similarity value of each of the first readout image and the second readout image passing through at least one layer of the first network becomes a maximum value.

In step S150, the first read image and the second read image passing through at least one layer of the first network and the second network and information obtained by reading the change according to the disease are set as disease labels and input to the disease information providing neural network.

In step S160, the third parameter of the disease information providing neural network is adjusted based on the output result according to the presence or absence of the disease.

Here, the first parameter, the second parameter, and the third parameter are updated based on output finally returned through the first network, the second network, and the disease information providing neural network.

4 illustrates an example of a learning input image used for learning of a medical image analysis model according to an embodiment of the present invention.

As the learning input image used for learning of the medical image analysis model according to an embodiment of the present invention, a first read image and a second read image of an arbitrary subject that have been previously read are used.

As shown in FIG. 4 , the learning of the medical image analysis model according to an embodiment of the present invention proceeds with overall learning by utilizing chest X-ray follow-up data of about 400,000 pairs.

The learning input image consists of images taken by the patient at different times. refers to

The training input image, which is x-ray data, is transformed into an 8-bit png format and used for network input, and is learned using data augmentation techniques of random cropping, random brightness conversion, and random rotation in the learning process. Input data is normalized to a value between 0 and 1, and is input to train the medical image analysis model.

5 is a diagram for explaining an exemplary structure of a medical image analysis model according to an embodiment of the present invention.

Referring to FIG. 5 , the medical image analysis model according to an embodiment of the present invention includes a first network 110, 120, 130, 140, a second network 200, and a disease information providing neural network 310, 320. do.

Specifically, the first network (110, 120, 130, 140) is at least one layer that returns a similarity value between the first read image (Image A) and the second read image (Image B) based on the first parameter. includes

Specifically, the first network extracts a first feature vector from the first readout image using the first module 110 and extracts the second feature vector from the second readout image using the second module 120. Thus, after integrating the first feature map and the second feature map generated by passing the first feature vector and the second feature vector through at least one layer using the integration layer 130, the multiple layer structure 140 ) to return the feature map similarity value.

Here, the first read image (image A) is the baseline (image A), the second read image (image B) is the follow-up (image B), and the first module 110 and the second module 120 are It is implemented with ResNet.

For example, the first module 110 and the second module 120 of the first network constitute a Siamese network. A Siamese network is a network configuration that runs two convolutional neural networks independently on two inputs and then compares them.

The learning of the Siamese network puts two pictures into the two networks as input and encodes them with a convolutional neural network. It is learned to satisfy the condition.

Through this, it is possible to determine whether there is a change using the similarity value between the first read image and the second read image.

Specifically, baseline (image A) and follow-up (image B) are input and passed through ResNet (110, 120), respectively, and the latent vector generated here is used for normal/abnormal and change/no-change classification.

In addition, the integration layer 130 is implemented as a concat layer, and the multiple layer structure 140 uses a structure passing through the MLP.

That is, in order to distinguish change/no-change, the latent vector from the baseline and the latent vector from the follow-up are concated and a structure passing through the MLP is used.

However, since the configuration of the first network alone cannot exclude changes according to the patient's respiration, age, and photographing posture, the second network and the disease information providing neural network are added to complement each other.

The second network 200 uses the target pixel value as an adjustment label to set the second parameter so that the feature map similarity value of each of the first readout image and the second readout image that has passed through at least one layer of the first network becomes a maximum value. is learned to adjust.

Here, the target pixel value means a pixel value based on the first read image to compensate for a difference that occurs when the first read image and the second read image are partially displaced from the direction of the X-ray according to the subject's posture.

The second network 200 is implemented as an attention module, and has an effect of making each feature map from the baseline and follow-up photos have the same shape as each other.

Here, the reason why the first read image and the second read image are made to have the same shape is to correct the difference that occurs when the first read image and the second read image are partially displaced from the direction of the X-ray according to the subject's posture. The first read image is used as a reference, The shape of the second read image is corrected as if the subject was photographed in a state in which the direction of the X-rays is the same.

To this end, correction is performed so that the pixel values of the two images are similar based on the pixel values of the two images.

The disease information providing neural networks 310 and 320 learn to adjust the third parameter by using information obtained by reading the change according to the disease in the first read image and the second read image as a disease label so as to output the presence or absence of the disease.

Here, the disease information providing neural networks 310 and 320 use the Cross-Entropy Softmax algorithm.

The information obtained by reading the change according to the disease is information on which the type of disease is recorded using the result of analyzing the reading statement on the actual recording sheet.

For example, the type of disease may include lung cancer, pulmonary tuberculosis, pneumonia, and pneumothorax that can be read through a chest X-ray image.

Even if changes in the image are discriminated using this, it is possible to discriminate only changes due to disease, excluding changes due to respiration.

Here, the first parameter, the second parameter, and the third parameter are updated based on output finally returned through the first network, the second network, and the disease information providing neural network.

Loss of total learning can be designed as in Equation 1.

Here, for L _disease and L _change , Cross-Entropy or Pairwise Loss (Triplet, Contrastive, etc.) can be used. λ is used as an element representing the weight of each loss.

In one embodiment of the present invention, L _disease serves to prepare information for distinguishing between normal and abnormal by determining whether each X-ray has a disease, and is used in a disease information providing neural network.

In one embodiment of the present invention, L _change receives two follow-up x-rays as input, enables discrimination between change, change, and no-change, and is used in the first network.

L _matching makes it possible to look at the same place in the baseline and follow-up pictures, and is used in the second network.

By using L _disease and L _matching , after concentrating on the lesion area in one x-ray picture, it is possible to confirm whether the disease is present in the same area in another x-ray picture.

6 is a diagram for explaining a first network and a second network of a medical image analysis model according to an embodiment of the present invention.

5 shows that one ResNet of the first network and the Attention Module of the second network are connected, but according to various embodiments of the present invention, a plurality of modules (111, 112, 113, 114, 115) of the first network ) are connected to the Attention Modules 210, 220, 230, and 240 of the second network, respectively, so that the feature map similarity value of each of the first readout image and the second readout image that has passed through each layer of the first network is the maximum You can sequentially adjust the parameters of the Attention Module so that they become values.

Accordingly, the different pixel values of the first and second read images passing through each layer having a different dimension in the first network may become similar while passing through the attention module of the second network. .

Specifically, we design a Region Matching Loss that makes the two feature maps that passed through the Attention Module look at the same place.

Here, the reason why the first read image and the second read image are made to look at the same place is to correct the difference that occurs when the first read image and the second read image are partially displaced from the direction of the X-ray according to the subject's posture. By correcting the shape of the second read-out image as if the subject was taken in the same direction, design Region Matching Loss to match the attention of the two images so that the cosine similarity is maximized.

Region Matching Loss is defined by Equation 2 below.

Here, F _b and F _f denote feature maps from baseline and follow-up, respectively.

Region Matching Loss has the effect of making the feature maps similar to each other by calculating the similarity values of two different feature maps.

It has a structure in which loss is minimized when the feature maps of the two sheets are similar to each other, so adding this loss function and learning can have the effect of making the feature maps of baseline and follow-up similar to each other.

7 is a diagram for explaining an effect according to a second network of a medical image analysis model according to an embodiment of the present invention.

Figure 7(a) shows the feature maps of baseline and baseline, and Figure 7(b) shows the feature maps of follow-up and follow-up.

In the feature maps obtained using the Region matching module of the second network according to an embodiment of the present invention, it can be confirmed that the same regions are activated in the feature maps, despite the different shooting postures.

In addition, a first read image and a second read image of an arbitrary subject that have been previously read are used as learning input images, and information obtained by reading a change between the first read image and the second read image is used as a read label, and the first network and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image; performing a medical image analysis model learning method comprising: It is possible to provide a recording medium in which a computer program including instructions to do so is stored.

In addition, a first read image and a second read image of an arbitrary subject that have been previously read are used as learning input images, and information obtained by reading a change between the first read image and the second read image is used as a read label, and the first network To perform a medical image analysis model learning method comprising the step of inputting input to and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. It is possible to provide a computer program stored in a computer readable storage medium containing instructions to do.

Such computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be specially designed and configured for the present invention or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, etc. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to act as one or more software modules to perform the operations of the present invention, and vice versa.

The above description is just one embodiment of the present invention, and those skilled in the art can implement it in a modified form without departing from the essential characteristics of the present invention. Therefore, the scope of the present invention should be construed to include various embodiments within the scope equivalent to those described in the claims without being limited to the above-described embodiments.

Claims (18)

In the medical image analysis device,
a memory in which a pre-learned medical image analysis model is stored; and
Receiving a first medical image and a second medical image obtained by capturing images of the same patient at different times, and discriminating a change between the first medical image and the second medical image using the medical image analysis model Including a processor;
The medical image analysis model,
A first read image and a second read image of a pre-read arbitrary subject are used as learning input images, and information obtained by reading a change between the first read image and the second read image is learned as a read label;
and a first network including at least one layer that returns a similarity value between the first read image and the second read image based on a first parameter. According to claim 1,
The processor determines a region of change according to a disease among changes between the first medical image and the second medical image. According to claim 2,
The medical image analysis apparatus of claim 1 , wherein the determined change region has minimal effects due to changes in respiration, posture, and age of the subject to be diagnosed. delete According to claim 1,
The first network extracts a first feature vector from the first read image, and extracts a second feature vector from the second read image, so that the first feature vector and the second feature vector are at least one layer. After integrating the first feature map and the second feature map generated by passing through the integrated layer, the feature map similarity value of the first feature map and the second feature map is based on passing through a plurality of layer structures. A medical image analysis device that is trained to return According to claim 5,
The medical image analysis model includes a second network that is learned to adjust a second parameter using a target pixel value as an adjustment label so that the feature map similarity value has a maximum value. According to claim 6,
The medical image analysis model is trained to adjust a third parameter by using information obtained by reading changes according to diseases in the first read image and the second read image as a disease label so as to output the presence or absence of a disease. Disease information A medical image analysis device including a providing neural network. According to claim 7,
The first parameter, the second parameter, and the third parameter are updated based on an output finally returned through the first network, the second network, and the disease information providing neural network. In the medical image analysis model learning method,
A processor takes a first read image and a second read image of an arbitrary subject as training input images, and reads information obtained by reading a change region between the first read image and a second read image as a read label. Step 1 input to the network; and
and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. According to claim 9,
Adjusting the first parameter of the first network,
A first feature vector is extracted from the first read image, a second feature vector is extracted from the second read image, and the first feature vector and the second feature vector are generated by passing at least one layer. After integrating the first feature map and the second feature map using the integration layer, based on the output result of the feature map similarity value of the first feature map and the second feature map based on passing through a plurality of layer structures A method for learning a medical image analysis model for adjusting the first parameter. According to claim 10,
inputting the first and second read images and target pixel values that have passed through at least one layer of the first network into a second network as adjustment labels; and
and adjusting a second parameter of the second network so that the feature map similarity value has a maximum value. According to claim 11,
inputting the first read image and the second read image passing through at least one layer of the first network and the second network and information obtained by reading the change according to the disease as a disease label into a disease information providing neural network; and
and adjusting a third parameter of the disease information providing neural network based on an output result according to the presence or absence of the disease. According to claim 12,
The first parameter, the second parameter, and the third parameter are updated based on an output finally returned through the first network, the second network, and the disease information providing neural network. In the medical image analysis method performed by the medical image analysis device,
receiving a first medical image and a second medical image obtained by capturing images of the same diagnosis subject at different times; and
Determining a change region between the first medical image and the second medical image by using a pre-learned medical image analysis model;
The medical image analysis model,
A first read image and a second read image of a pre-read arbitrary subject are used as learning input images, and information obtained by reading a change between the first read image and a second read image is learned as a read label;
and a first network including at least one layer that returns a similarity value between the first read image and the second read image based on a first parameter. According to claim 14,
In the step of determining a change region between the first medical image and the second medical image, among the changes between the first medical image and the second medical image, a change region according to a disease is discriminated. According to claim 15,
The medical image analysis method of claim 1 , wherein the determined change region has a minimized effect of at least one of changes in respiration, posture, and age of the subject to be diagnosed. A first read image and a second read image of a subject that have been previously read are set as learning input images, and information obtained by reading a change between the first read image and the second read image is input to the first network as a read label. step of doing; and
Adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image; Includes instructions for performing a medical image analysis model learning method including; A recording medium on which a computer program is stored. A first read image and a second read image of a subject that have been previously read are set as learning input images, and information obtained by reading a change between the first read image and the second read image is input to the first network as a read label. step of doing; and
Adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image; Includes instructions for performing a medical image analysis model learning method including; A computer program stored on a computer-readable storage medium that

Images