KR20220105723A - Apparatus and method for analyzing medical image based on artificial neural network and learning method thereof - Google Patents

Abstract

According to the present invention, disclosed are an apparatus and method for analyzing a medical image based on an artificial neural network and a learning method thereof. The apparatus receives a first medical image and a second medical image imaged at different viewpoints for the same subject to be diagnosed and determines a change between the first medical image and the second medical image by using a pre-learned medical image analysis model.

Description

Artificial neural network-based medical image analysis device, method, and learning method {APPARATUS AND METHOD FOR ANALYZING MEDICAL IMAGE BASED ON ARTIFICIAL NEURAL NETWORK AND LEARNING METHOD THEREOF}

The present invention relates to a medical image analysis apparatus, a method, and a learning method thereof.

X-ray imaging is a technique that can obtain images of the inside of the human body in the field of radiology and has been used in the diagnosis and treatment of various diseases. Depending on the purpose of diagnosis, such as neck soft tissue), mammography, etc., it can be obtained by taking before-and-after photos, lateral or diagonal photos.

Such X-ray imaging is very inexpensive compared to examinations such as computed tomography (CT) and magnetic resonance imaging (MRI), and although the examination method is simple, the number of diseases diagnosed through it is large.

However, in classifying changes through X-ray images, it is difficult to solve using the existing CNN network because it is necessary to distinguish only the changes according to the disease from the criteria judged by the radiologist while ignoring the changes in the patient's breathing, posture, and age. There are difficulties.

An object to be solved according to an embodiment of the present invention includes determining a change between medical images acquired for the same diagnosis subject by using a model learned using read images of an arbitrary subject.

It also includes making the feature maps of each read image similar to each other.

Other objects not specified in the present invention may be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

In order to solve the above problems, an artificial neural network-based medical image analysis apparatus according to an embodiment of the present invention includes a memory in which a pre-learned medical image analysis model is stored and a first image obtained by capturing the same diagnosis subject at different points in time. and a processor that receives a medical image and a second medical image, and determines a change between the first medical image and the second medical image by using the medical image analysis model, wherein the medical image analysis model includes: It is learned by using the first reading image and the second reading image of an arbitrary subject as a learning input image, and using information obtained by reading a change between the first reading image and the second reading image as a reading label.

Here, the processor may determine a change region according to a disease among changes between the first medical image and the second medical image.

Here, in the determined change region, the influence of the change in breathing, posture, and age of the subject to be diagnosed may be minimized.

In a method for training a medical image analysis model based on an artificial neural network according to another embodiment of the present invention, the processor uses a pre-read first read image and a second read image of an arbitrary subject as a learning input image, and inputting information for reading a change between the readout image and the second readout image as a readout label to the first network, and based on an output result of a similarity value between the first readout image and the second readout image, the second readout image 1 adjusting a first parameter of the network.

A method for learning a medical image analysis model based on an artificial neural network according to another embodiment of the present invention includes the steps of: receiving a first medical image and a second medical image obtained by capturing the same diagnosis subject at different points in time; and determining a change between the first medical image and the second medical image by using a medical image analysis model, wherein the medical image analysis model includes a first read image and a second read image of an arbitrary subject. It is learned by using a read image as a learning input image, and information obtained by reading a change between the first read image and the second read image as a read label.

In addition, the first network using the first read image and the second read image of a pre-read arbitrary subject as the learning input image, and the information obtained by reading the change between the first read image and the second read image as the read label and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. It is possible to provide a recording medium in which a computer program including instructions to be stored is stored.

In addition, the first network using the first read image and the second read image of a pre-read arbitrary subject as the learning input image, and the information obtained by reading the change between the first read image and the second read image as the read label and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. It is possible to provide a computer program stored in a computer-readable storage medium containing instructions.

As described above, according to embodiments of the present invention, it is possible to determine a change according to a disease in a medical image acquired for the same diagnosis subject using a pre-trained model.

In addition, since the feature maps of each read image are made to be similar to each other, it is possible to reduce a read error.

Even if effects not explicitly mentioned herein, the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as described in the specification of the present invention.

1 is a block diagram illustrating an artificial neural network-based medical image analysis apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a method for analyzing a medical image based on an artificial neural network according to an embodiment of the present invention.
3 is a flowchart illustrating a method for learning a medical image analysis model according to an embodiment of the present invention.
4 illustrates an example of a learning input image used for learning a medical image analysis model according to an embodiment of the present invention.
5 is a diagram for explaining an exemplary structure of a medical image analysis model according to an embodiment of the present invention.
6 is a diagram illustrating a first network and a second network of a medical image analysis model according to an embodiment of the present invention.
7 is a diagram for explaining an effect of a second network of a medical image analysis model according to an embodiment of the present invention.

Hereinafter, an artificial neural network-based medical image analysis apparatus and method and a learning method thereof according to the present invention will be described in more detail with reference to the drawings. However, the present invention may be embodied in several different forms, and is not limited to the described embodiments. In addition, in order to clearly explain the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves.

The present invention relates to an artificial neural network-based medical image analysis apparatus, method, and learning method thereof.

1 is a block diagram illustrating an artificial neural network-based medical image analysis apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , an artificial neural network-based medical image analysis apparatus 10 according to an embodiment of the present invention includes a processor 11 and a memory 12 .

The artificial neural network-based medical image analysis apparatus 10 according to an embodiment of the present invention is an apparatus for reading a change due to a disease from an X-ray image of a subject.

Specifically, it operates like a procedure read by actual radiologists by using a module that enables strong learning from changes in the subject's natural variation (respiration, age, photographing posture), etc.

To this end, it is designed as a CNN structure to distinguish changes and utilizes multiple labels. The medical image analysis model of the artificial neural network-based medical image analysis apparatus 10 includes a first network, a second network, and a neural network for providing disease information, and the multi-label is information obtained by reading changes in an X-ray image, a read label, X It may include an adjustment label, which is a target pixel value of the line image, and a disease label, which is information obtained by reading a change according to a disease.

The memory 12 may store programs (one or more instructions) for processing and controlling the processor 11 , a pre-trained medical image analysis model is stored, and a flash memory type, hard disk hard disk type, multimedia card micro type, card type memory (eg, SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory) , ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and at least one type of computer-readable storage medium may include

Programs stored in the memory 12 may be divided into a plurality of modules according to functions, and a pre-trained medical image analysis model to be described later may be configured as a software module.

The processor 11 executes one or more instructions stored in the memory 12 , and specifically, receives a first medical image and a second medical image obtained by imaging the same diagnosis subject at different times, and a medical image analysis model is used to determine a change between the first medical image and the second medical image.

Specifically, a change region according to a disease is determined among changes between the first medical image and the second medical image, and in the determined change region, the influence of changes in respiration, posture, and age of a subject to be diagnosed is minimized.

Here, the processor 11 may be divided into a plurality of modules according to functions, and functions may be performed by one processor. The processor is one or more of a central processing unit (CPU), an application processor (AP), a micro controller unit (MCU), or a communication processor (CP). may include

The medical image analysis model of the artificial neural network-based medical image analysis apparatus 10 according to an embodiment of the present invention uses a pre-read first read image and a second read image of an arbitrary subject as learning input images, The information obtained by reading the change between the first read image and the second read image was learned as a read label.

The medical image analysis model will be described in detail with reference to FIG. 5 below.

2 is a flowchart illustrating a method for analyzing a medical image based on an artificial neural network according to an embodiment of the present invention.

Referring to FIG. 2 , the method for analyzing a medical image based on an artificial neural network according to an embodiment of the present invention is performed by a medical image analyzing apparatus, and the first medical image obtained by capturing the same diagnosis subject at a different time in step S10 An image and a second medical image are input.

In step S20, a change between the first medical image and the second medical image is determined using the pre-trained medical image analysis model.

Specifically, a change region according to a disease is determined among changes between the first medical image and the second medical image, and in the determined change region, the influence of changes in respiration, posture, and age of a subject to be diagnosed is minimized.

Here, the medical image analysis model uses the first read image and the second read image of a pre-read arbitrary subject as a learning input image, and reads information obtained by reading a change between the first read image and the second read image as a read label has been learned by

3 is a flowchart illustrating a method for learning a medical image analysis model according to an embodiment of the present invention.

Referring to FIG. 3 , the method for training a medical image analysis model according to an embodiment of the present invention is performed by a processor, and a first read image and a second read image of an arbitrary subject read in step S110 are used as a learning input image. , and information that reads a change between the first read image and the second read image is input to the first network as a read label.

In step S120 , the first parameter of the first network is adjusted based on the output result of the similarity value between the first read image and the second read image.

Here, in the step of adjusting the first parameter of the first network ( S120 ), the first feature vector is extracted from the first readout image and the second feature vector is extracted from the second readout image, so that the first feature vector and the second feature vector are extracted. After the first feature map and the second feature map, which are generated as the two feature vectors pass through at least one layer, are integrated using the integrated layer, the first feature map is first based on the output result of the similarity values that have passed through the plurality of layer structures. Adjust the parameters.

In step S130, the first read image and the second read image that have passed through at least one layer of the first network and the target pixel value are input to the second network as adjustment labels.

In step S140 , the second parameter of the second network is adjusted such that a feature map similarity value of each of the first read image and the second read image that has passed through at least one layer of the first network becomes a maximum value.

In step S150, the first read image and the second read image that have passed through at least one layer of the first network and the second network, and information obtained by reading the change according to the disease are input to the disease information providing neural network as a disease label.

In step S160, based on the output result according to the presence or absence of the disease, the third parameter of the disease information providing neural network is adjusted.

Here, the first parameter, the second parameter, and the third parameter are updated based on the output finally returned through the first network, the second network, and the disease information providing neural network.

4 illustrates an example of a learning input image used for learning a medical image analysis model according to an embodiment of the present invention.

A training input image used for learning a medical image analysis model according to an embodiment of the present invention uses a pre-read first readout image and a second readout image of an arbitrary subject.

As shown in FIG. 4 , in the training of the medical image analysis model according to an embodiment of the present invention, the entire learning is performed using about 400,000 pairs of chest X-ray follow-up data.

The learning input image consists of images taken at different times by the patient, and the first readout image, which is the image taken before (a), is called the baseline, and the second readout image, which is the image taken after (b), is called follow-up. refers to

The learning input image, which is x-ray data, is transformed into an 8-bit png format and used for network input. The input data is normalized to a value between 0 and 1 and is input to train a medical image analysis model.

5 is a diagram for explaining an exemplary structure of a medical image analysis model according to an embodiment of the present invention.

Referring to FIG. 5 , a medical image analysis model according to an embodiment of the present invention includes first networks 110 , 120 , 130 , 140 , a second network 200 , and disease information providing neural networks 310 and 320 . do.

Specifically, the first networks 110 , 120 , 130 , and 140 at least one layer that returns a similarity value between the first readout image (image A) and the second readout image (image B) based on the first parameter includes

Specifically, the first network extracts a first feature vector from the first read image by using the first module 110 , and extracts a second feature vector from the second read image by using the second module 120 . Thus, after integrating the first feature map and the second feature map generated as the first feature vector and the second feature vector pass through at least one layer using the integration layer 130 , the plurality of layer structures 140 ) to pass and learn to return the feature map similarity value.

Here, the first reading image (image A) is the baseline (image A), the second reading image (image B) is the follow-up (image B), and the first module 110 and the second module 120 are It is implemented with ResNet.

For example, the first module 110 and the second module 120 of the first network constitute a Siamese network. A Siamese network is a network configuration that runs two convolutional neural networks independently for two inputs and then compares them.

The learning of the Siamese network is to put two pictures into two networks as input and encode them with a convolutional neural network. If the two pictures are similar, the distance between encodings should be small. learned to satisfy the condition.

Through this, it is possible to determine whether there is a change by using the similarity value between the first read image and the second read image.

Specifically, the baseline (image A) and follow-up (image B) are received and passed through ResNet (110, 120), respectively, and the latent vector obtained therefrom is used to classify normal/abnormal and change/no-change.

In addition, the integration layer 130 is implemented as a concat layer, and the multiple layer structure 140 uses a structure passing through the MLP.

That is, in order to distinguish change/no-change, the latent vector from the baseline and the latent vector from the follow-up are concatenated, and a structure that passes through the MLP is used.

However, since changes according to the patient's respiration, age, and photographing posture cannot be excluded only by the configuration of the first network, the second network and the disease information providing neural network are added to complement each other.

The second network 200 sets the second parameter using the target pixel value as the adjustment label so that the feature map similarity value of each of the first read image and the second read image that has passed through at least one layer of the first network becomes the maximum value. is learned to adjust.

Here, the target pixel value refers to a pixel value based on the first read image in order to correct a difference that occurs when the first read image and the second read image partially deviate from the X-ray direction according to the subject's posture.

The second network 200 is implemented as an attention module, and has the effect of making each feature map from the baseline and the follow-up photo have the same shape.

Here, to have the same shape as each other, in order to correct a difference that occurs when the first read image and the second read image partially deviate from the direction of the X-ray according to the posture of the subject, the first read image is the reference, The shape of the second read image is corrected as if the subject was photographed in a state in which the X-ray directions are the same.

To this end, correction is performed so that the pixel values of the two images are similar to each other based on the pixel values.

The disease information providing neural networks 310 and 320 are trained to output the presence or absence of a disease, and to adjust the third parameter using information obtained by reading changes according to diseases of the first and second reading images as a disease label.

Here, the disease information providing neural networks 310 and 320 use the Cross-Entropy Softmax algorithm.

The information that reads the change according to the disease is information that records the type of disease using the result of analyzing the reading on the actual recording sheet.

For example, the type of disease may include lung cancer, pulmonary tuberculosis, pneumonia, and pneumothorax that can be read through a chest X-ray image.

Using this, even if the change in the image is determined, it is possible to determine only the change according to the disease, excluding the change due to respiration.

Here, the first parameter, the second parameter, and the third parameter are updated based on the output finally returned through the first network, the second network, and the disease information providing neural network.

Loss of overall learning may be designed as in Equation (1).

Here, L _disease and L _change may be Cross-Entropy or Pairwise Loss (Triplet, Contrastive, etc.). λ is used as an element representing the weight of each loss.

L _disease serves to prepare information for classifying normal and abnormal by determining whether each X-ray has a disease in an embodiment of the present invention, and is used in a disease information providing neural network.

L _change receives two follow-up x-rays as an input in an embodiment of the present invention to distinguish change, change, and no-change, and is used in the first network.

L _matching allows to look at the same place in the baseline and follow-up photos, and is used in the second network.

By using L _disease and L _matching , after focusing on the lesion in one x-ray picture, it is possible to check whether there is a disease in the same x-ray picture in another x-ray picture.

6 is a diagram illustrating a first network and a second network of a medical image analysis model according to an embodiment of the present invention.

Although one ResNet of the first network and the Attention Module of the second network are illustrated in FIG. 5 as being connected, according to various embodiments of the present invention, a plurality of modules 111, 112, 113, 114, 115 of the first network ) each of the attention modules 210 , 220 , 230 , 240 of the second network are connected to each other, so that the feature map similarity value of each of the first read image and the second read image that has passed through each layer of the first network is maximized. You can adjust the parameters of the Attention Module sequentially so that it becomes a value.

Accordingly, different pixel values of the first read image and the second read image that have passed through respective layers having different dimensions in the first network pass through the attention module of the second network, and the respective pixel values may be similar. .

Specifically, we design Region Matching Loss to make two feature maps that have passed the Attention Module look at the same place.

Here, to make them look at the same place, the first read image and the second read image are based on the first read image to correct a difference that occurs when the direction of the X-ray is partially shifted depending on the posture of the subject, and the X-ray By correcting the shape of the second read image as if the subject was photographed in the same direction, the Region Matching Loss is designed to match the attention of the two images so that the cosine similarity is maximized.

Region Matching Loss is defined by Equation 2 below.

Here, F _b and F _f denote feature maps from baseline and follow-up, respectively.

Region Matching Loss has the effect of making the feature maps similar to each other by calculating the similarity values of two different feature maps.

Since the two pieces of feature maps have a structure that minimizes loss when they are similar to each other, learning by adding this loss function can have the effect that the baseline and follow-up feature maps are similar to each other.

7 is a diagram for explaining an effect of a second network of a medical image analysis model according to an embodiment of the present invention.

Fig. 7 (a) shows the baseline and basline feature maps, and Fig. 7 (b) shows the follow-up and follow-up feature maps.

In the feature map generated using the region matching module of the second network according to an embodiment of the present invention, it can be confirmed that the same places are activated in the feature map despite the different photographing postures.

In addition, the first network using the first read image and the second read image of a pre-read arbitrary subject as the learning input image, and the information obtained by reading the change between the first read image and the second read image as the read label and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. It is possible to provide a recording medium in which a computer program including instructions to do so is stored.

In addition, the first network using the first read image and the second read image of a pre-read arbitrary subject as the learning input image, and the information obtained by reading the change between the first read image and the second read image as the read label and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. It is possible to provide a computer program stored in a computer-readable storage medium containing instructions.

Such computer-readable media may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above description is only one embodiment of the present invention, those skilled in the art to which the present invention pertains will be able to implement in a modified form within a range that does not depart from the essential characteristics of the present invention. Therefore, the scope of the present invention is not limited to the above-described embodiments, but should be construed to include various embodiments within the scope equivalent to the content described in the claims.

Claims (18)

A medical image analysis apparatus comprising:
a memory storing a pre-trained medical image analysis model; and
receiving a first medical image and a second medical image captured at different points in time for the same diagnosis subject, and determining a change between the first medical image and the second medical image by using the medical image analysis model processor; including;
The medical image analysis model uses a pre-read first read image and a second read image of an arbitrary subject as a learning input image, and reads information obtained by reading a change between the first read image and the second read image as a read label A medical image analysis device that is learned by According to claim 1,
The processor is configured to determine a change region according to a disease among changes between the first medical image and the second medical image. 3. The method of claim 2,
The determined change region is a medical image analysis apparatus in which the influence of the change in breathing, posture, and age of the subject to be diagnosed is minimized. According to claim 1,
The medical image analysis model may include a first network including at least one layer that returns a similarity value between the first read image and the second read image based on a first parameter. 5. The method of claim 4,
The first network extracts a first feature vector from the first readout image and extracts a second feature vector from the second readout image, so that the first feature vector and the second feature vector are at least one layer After integrating the first feature map and the second feature map generated by passing through , the feature map similarity value between the first feature map and the second feature map based on passing through a plurality of layer structures A medical image analysis device that is trained to return . 6. The method of claim 5,
and the medical image analysis model includes a second network trained to adjust a second parameter using a target pixel value as an adjustment label so that the feature map similarity value becomes a maximum value. 7. The method of claim 6,
The medical image analysis model may include information on diseases that have been learned to adjust a third parameter by using, as a disease label, information obtained by reading changes according to diseases of the first read image and the second read image so as to output the presence or absence of a disease. A medical image analysis device comprising a providing neural network. 8. The method of claim 7,
The first parameter, the second parameter, and the third parameter are updated based on outputs finally returned through the first network, the second network, and the disease information providing neural network. In the medical image analysis model training method,
The processor uses the pre-read first reading image and the second reading image of an arbitrary subject as a learning input image, and using information obtained by reading a change region between the first reading image and the second reading image as a reading label, 1 entering the network; and
and adjusting a first parameter of the first network based on an output result of a similarity value between the first read image and the second read image. 10. The method of claim 9,
Adjusting the first parameter of the first network comprises:
A first feature vector is extracted from the first readout image and a second feature vector is extracted from the second readout image, and the first feature vector and the second feature vector are generated as the first and second feature vectors pass through at least one layer. After integrating the first feature map and the second feature map using an integrated layer, based on the output result of the feature map similarity value of the first feature map and the second feature map based on passing through a plurality of layer structures A method of training a medical image analysis model for adjusting the first parameter. 11. The method of claim 10,
inputting the first readout image and the second readout image passing through at least one layer of the first network and target pixel values as adjustment labels into a second network; and
and adjusting a second parameter of the second network so that the feature map similarity value becomes a maximum value. 12. The method of claim 11,
inputting the first read image and the second read image that have passed through at least one layer of the first network and the second network and information obtained by reading a change according to a disease as a disease label to a disease information providing neural network; and
and adjusting a third parameter of the disease information providing neural network based on an output result according to the presence or absence of the disease. 13. The method of claim 12,
The first parameter, the second parameter, and the third parameter are updated based on outputs finally returned through the first network, the second network, and the disease information providing neural network. A medical image analysis method performed by a medical image analysis apparatus, comprising:
receiving a first medical image and a second medical image obtained by capturing the same diagnosis subject at different times; and
determining a change region between the first medical image and the second medical image by using a pre-learned medical image analysis model;
The medical image analysis model uses a pre-read first read image and a second read image of an arbitrary subject as a learning input image, and reads information obtained by reading a change between the first read image and the second read image as a read label A medical image analysis method that was learned by 15. The method of claim 14,
The determining of the change region between the first medical image and the second medical image may include determining a change region according to a disease among the changes between the first medical image and the second medical image. 16. The method of claim 15,
The determined change region is a medical image analysis method in which an effect of at least one of a change in breathing, posture, and age of the subject to be diagnosed is minimized. The first read image and the second read image of a pre-read arbitrary subject are used as a learning input image, and information obtained by reading a change between the first read image and the second read image is input to the first network as a read label making; and
Based on an output result of a similarity value between the first read image and the second read image, adjusting a first parameter of the first network includes instructions to perform a medical image analysis model learning method comprising: A recording medium on which a computer program is stored. The first read image and the second read image of a pre-read arbitrary subject are used as a learning input image, and information obtained by reading a change between the first read image and the second read image is input to the first network as a read label making; and
Based on an output result of a similarity value between the first read image and the second read image, adjusting a first parameter of the first network includes instructions to perform a medical image analysis model learning method comprising: A computer program stored in a computer readable storage medium.

Images