KR20230139257A - Device and method for classifying and segmenting ct image based on machine learning model - Google Patents

Abstract

An electronic device according to an embodiment extracts a feature map by applying an encoder of a machine learning model to a training input image, and trains the feature map by applying a general classification module of the machine learning model to the feature map. Obtain a first likelihood score that a lesion will appear in the input image, extract a lesion area from the training input image by applying a general-purpose segmentation module of the machine learning model to the feature map, and extract the lesion area from the training input image. Based on the first likelihood score and the extracted lesion area, calculate an objective function value with a consistency loss that represents consistency between outputs of the universal classification module and the universal segmentation module, It may include a processor that updates parameters of the machine learning model using the calculated objective function value.

Description

Method and apparatus for classifying and segmenting CT images based on machine learning models {DEVICE AND METHOD FOR CLASSIFYING AND SEGMENTING CT IMAGE BASED ON MACHINE LEARNING MODEL}

Hereinafter, a technology for classifying medical images based on a machine learning model is disclosed.

There may be studies on machine learning models for the single task of classification (CLS) and segmentation (SEG) of intracranial hemorrhage (ICH).

There may be two types of classification and segmentation tasks of ICH at the patient level.

First, a 3D-based model for classification and segmentation of ICH may exist to solve the volumetric problem. A variable 3D U-Net structure can be used for segmentation of ICH. For example, the model's architecture and preprocessing can be selected based on statistical analysis of training CT scans. As another example, a 3D version of the 18 layers ResNet can be used for the classification task of ICH. In 3D-based methods, typically due to limitations in GPU resources, patch-based approaches and small-depth 3D convolutional layers can be used. However, patch-based approaches can often lead to false positives and over-sensitivity.

Second, a model may exist at the 2D slice level for the classification and segmentation tasks of ICH. It could be argued that 2D-based approaches are vulnerable to 3D volumetric information but advantageous for anisotropic CT data with thick slice thickness. A 2D slice-level U-net network can be used, and slice segmentation results can be accumulated for a patient-level ICH segmentation task. Additionally, after the results of the 2D-based CNN network are accumulated, the accumulated features can be transferred to a bidirectional LSTM to expand to the patient-level ICH classification task. It can be argued that when transfer learning for scaling is performed, a three-stage end-to-end learning process should be performed.

Multi-task learning can be used to solve the ICH classification task. For example, classification results can be converted into information about the amount of bleeding, and the performance of the classification task can be improved by connecting this to a Multi-Linear Perceptron (MLP) layer. As another example, a Dilated Residual Network can be employed to maintain resolution as a backbone, and can also be branched at the last layer to perform multiple tasks. As another example, five consecutive CT slices may be used as input. To transfer to Mask R-CNN, 3D features obtained by the 3D convolution operator can be projected into 2D features. Since it must be repeated for all slices to obtain the final 3D result, excessive and redundant 3D convolution operations may occur. As another example, multi-task learning may be performed to increase classification performance. Auxiliary segmentation operations can be added after each DenseNet block, and a bidirectional LSTM layer can be used to combine spatial dependencies between slices. As another example, in the bottleneck of the 3D U-Net [24] architecture, patient-level multi-tasks can be performed using ConvLSTM, and three types of Hounsfield Unit (HU) ranges can be used with three channels. . However, the demand for GPU resources can increase exponentially as the number of CT slices increases. For another example, the decoder for the segmentation task is supervised and the decoder for the reconstruction task is learned unsupervised, so that semi-supervised multi-task learning can be performed to improve the performance of the segmentation task. Multi-task learning may not be applicable to ICH tasks, but it can be used to effectively solve medical problems such as COVID-19.

Finally, there may be models that are closely related to transfer learning approaches.

Transfer learning can often be used for ICH tasks. ImageNet pre-trained models can be popular and easy-to-apply transfer learning models, and can be useful for capturing low-level features of objects (e.g. edges, textures). . However, because the ImageNet pre-trained model deals with 3 RGB-based channels, it may be inappropriate in the medical field where 1 channel is mainly used. Additionally, there may be an inconvenience in that volumetric CT data must be applied only in a slice manner. For example, a framework can be introduced that effectively generates appropriate pre-learning weights for 3D medical domains and 2D medical domains using representation learning through transfer learning. The above-described framework may include two steps in the field of medical imaging: restoring a distorted image to its original state, learning the representation of the CT itself, and then improving the segmentation task through transfer learning. It could be argued that this approach is helpful when solving medical problems with relatively small data sets with one channel.

Intracranial hemorrhage (ICH), even if it occurs mildly, can be a fatal disease that requires urgent treatment. The triage system, when automated by using computer-aided diagnosis (CAD) on non-contrast head CT (NCCT), can reduce the workload of busy emergency rooms, thereby reducing the burden of traumatic injury. ) The survival rate of ICH patients may be increased. Therefore, a high sensitivity model to accurately detect ICH patients as well as a high specificity model to reduce unnecessary workload in emergency situations may be required. Although several studies with good performance have been performed, higher performance may still be required and the unstable performance on external data sets and real incidence data sets may require improvement.

Intracranial hemorrhage (ICH) can be classified according to anatomy into cerebral parenchymal hemorrhage (CPH), intraventricular hemorrhage (IVH), epidural hemorrhage (EDH), subdural hemorrhage (SDH), Subarachnoid hemorrhage (SAH) can be classified into one of five subtypes. All types of ICH, even mild ones, can suddenly become catastrophic and require urgent treatment. A classification system using deep learning-based computer-aided diagnosis (CAD) can be proposed to address preventable deaths due to treatment delays and assessment errors in emergency situations. Radiologists can easily identify patients with bleeding in a classification (CLS) task and accurately recognize the location and amount of bleeding in a segmentation (SEG) task. Through a deep learning model that performs two tasks, a classification task and a segmentation task, in emergency situations, patients with cerebral hemorrhage can be quickly prioritized. Therefore, the model may be required to have both high sensitivity for detecting ICH patients and high specificity for normal patients in order to allocate appropriate treatment to more severely ill patients. Models based on recent and previous ICH research based on deep learning can enable automatic classification and segmentation tasks.

However, problems may still exist when applied to real emergency situations.

Heterogeneity: The texture, intensity, size, and position of hemorrhages in non-contrast head CT (NCCT) vary widely, making detection difficult. can be difficult even for expert radiologists. In particular, the CT intensity of cerebral hemorrhage may decrease over time.

Class-imbalance: The five types of ICH can occur in combination in multiple intracranial locations and can have varying degrees of severity. Because ICH is uneven and may occur rarely in practice, collecting a balanced data set can be difficult.

Robustness to external data: Because robustness is not considered in the models in most existing ICH studies, robustness to external data can be difficult to measure. Existing models are verified by several external data sets, so they may lead to unreliable results when applied to external data or in real clinical environments with a high proportion of normal patients. For example, it may be disclosed that classification performance may be unexpectedly lower in an external validation dataset than in previous reports due to the type and amount of ICH across the dataset.

Patient-level tasks: ICH can primarily be diagnosed using CT. A significant amount of GPU resources may be required to process 3D volumetric CT scans. In existing research, slice-based and patch-based methods can mainly be used to circumvent this problem. Patch-based methods can show excellent sensitivity. However, many false positives can occur. On the other hand, slice-based methods can save GPU resources by stacking features. However, slice-based methods eventually need to be scaled up to the patient level, and learning of slice-based methods can be unstable and complicated while scaling up to 3D CT scans.

Multi-task learning (MTL) can improve the generalization performance of all tasks by utilizing semantic information contained in multiple related tasks. Since a model trained by multi-task learning must capture features that satisfy all tasks, the risk of overfitting can be greatly reduced and synergy effects for other tasks can be obtained. However, although the overall average performance may be improved in a multi-task environment, the performance of some specific tasks in the multi-task model may be worse than that of the single-task model. For example, fine tuning, model capacity, because one task can dominate the entire multi-task learning, resulting in poor performance on other tasks and negative transfer learning effects. and covariance between tasks.

Transfer learning may refer to reusing a model pre-trained by a predefined task to apply it to the target task. Transfer learning can be frequently used in computer vision tasks. In particular, the ImageNet pre-trained approach on a large natural image dataset of 1000 classes can be applied to a variety of tasks. Additionally, training that performs transfer learning can have faster convergence and improved performance than training that does not perform transfer learning. Therefore, pre-trained models can often be used as effective feature extractors in relatively small data sets (e.g., data sets in the medical domain). Transfer learning can vary depending on predefined tasks. In general, like Autoencoder and Model Genesis, a representation is initially learned in a method called representation learning by extracting features from input data or restoring the original image from a distorted or compressed image, and then learning the target. Performing tasks can be effective.

To solve the problem of ICH detection at the patient level, powerful enhancement methods such as changing image scale, permutation intensity, and adding noise can be used to address the heterogeneity of ICH.

To address the class imbalance of ICH, all ICHs can be binarized and classification of normal vs. ICH patients can be performed. However, it may be suitable for the purpose of a triage system.

For robustness to external data, a machine learning model according to one embodiment using multi-task transfer learning may be proposed for classification and segmentation of ICH at the patient level. Machine learning models according to one embodiment may share a 2D-based encoder and perform three multiple tasks: classification (CLS), segmentation (SEG), and reconstruction (REC). Additionally, in learning a machine learning model according to one embodiment, consistency (CON) loss, which indicates consistency between outputs of classification and segmentation tasks, may be used to update parameters of the machine learning model. Multi-task learning allows the encoder to be trained to capture globalized semantic features encompassing all three tasks, increasing balanced accuracy and robustness to external data sets. Finally, for patient-level tasks, the features accumulated by the encoder can be sent to an LSTM-based network, a 3D classifier, or to a Conv3D-based network, a 3D segmenter for patient-level ICH tasks.

However, the technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

A method performed by a processor according to an embodiment includes extracting a feature map by applying an encoder of a machine learning model to a training input image, and general-purpose classification of the machine learning model to the feature map. Obtaining a first likelihood score that a lesion will appear in the training input image by applying a module; obtaining a lesion region from the training input image by applying a general-purpose segmentation module of the machine learning model to the feature map; extracting, based on the obtained first likelihood score and the extracted lesion area, having a consistency loss, indicating consistency between outputs of the universal classification module and the universal segmentation module. It may include calculating an objective function value, and updating parameters of the machine learning model using the calculated objective function value.

Calculating the objective function value may include calculating the objective function value further having a classification loss for the general-purpose classification module and a segmentation loss for the general-purpose segmentation module. there is.

The step of calculating the objective function value includes obtaining a second likelihood score that a lesion will appear in the training input image using the extracted lesion area, and calculating the difference between the first likelihood score and the second likelihood score. It may include calculating the consistency loss based on the consistency loss.

The step of obtaining the second likelihood score includes obtaining the second likelihood score by applying at least one of an average pooling layer and a maximum pooling layer to the extracted lesion area. It can be included.

Calculating the consistency loss includes calculating a square of the difference between the first likelihood score and the second likelihood score, and the first likelihood score calculated for each of the images included in the batch. and calculating the consistency loss by averaging the square of the difference between the likelihood score and the second likelihood score.

The method according to one embodiment may further include obtaining a reconstruction image from the feature map by applying a reconstruction module of the machine learning model to the feature map, and calculating the objective function value. may include calculating a reconstruction loss for the reconstruction module based on the training input image and the reconstruction image.

The step of calculating the restoration loss includes calculating a mean absolute error value of the intensity difference between the pixel of the training input image and the pixel of the restored image corresponding to the pixel of the training input image. May include steps.

The method according to one embodiment includes generating a classification model including the encoder of the machine learning model and a classification module at a patient level connected to the encoder, the target patient of the training data set Obtaining a plurality of feature maps by applying the encoder of the classification model to a plurality of images acquired based on CT imaging for, applying the patient-level classification module to the obtained plurality of feature maps Obtaining a probability score that a lesion appears in at least one image among a plurality of images, and calculating it using the probability score for the plurality of images and a true value class mapped to the plurality of images in the training data set. It may include updating parameters of the classification model based on the objective function value.

Obtaining the plurality of feature maps may include repeating obtaining the feature map by applying the encoder of the classification model to each of the plurality of images.

The step of updating the parameters of the classification model includes updating the parameters of the patient-level classification module while keeping the parameters of the encoder of the classification model fixed in at least some of the training iterations, and the training iterations In another iteration, the step may include unfixing the parameters of the encoder of the classification model and updating the parameters of the encoder of the classification model and the patient-level classification module.

Generating a segmentation model including the encoder of the machine learning model, a decoder connected to the encoder, and a patient level segmentation module according to an embodiment, of the training data set Obtaining a plurality of feature maps by applying the encoder and the decoder to a plurality of images obtained based on CT imaging of a target patient, by applying the patient-level segmentation module to the obtained plurality of feature maps extracting a lesion area from the plurality of images, and the segmentation model based on an objective function value calculated using the extracted lesion area and a true lesion area mapped to the plurality of images in the training data set. The step of updating the parameters may be further included.

Obtaining the plurality of feature maps may include repeating obtaining the feature map by applying the encoder and the decoder of the segmentation model to each of the plurality of images.

The step of updating the parameters of the segmentation model includes updating the parameters of the patient-level segmentation module while fixing the parameters of the encoder of the segmentation model in at least some of the training iterations. In another iteration, it may include unfixing the parameters of the encoder of the segmentation model and updating the parameters of the encoder of the segmentation model and the segmentation module at the patient level.

An electronic device according to an embodiment extracts a feature map by applying an encoder of a machine learning model to a training input image, and trains the feature map by applying a general classification module of the machine learning model to the feature map. Obtain a first likelihood score that a lesion will appear in the input image, extract a lesion area from the training input image by applying a general-purpose segmentation module of the machine learning model to the feature map, and extract the lesion area from the training input image. Based on the first likelihood score and the extracted lesion area, calculate an objective function value with a consistency loss that represents consistency between outputs of the universal classification module and the universal segmentation module, It may include a processor that updates parameters of the machine learning model using the calculated objective function value.

The processor generates a classification model including the encoder of the machine learning model and a classification module at a patient level connected to the encoder, and performs a CT scan on the target patient of the training data set. A plurality of feature maps are obtained by applying the encoder of the classification model to a plurality of images obtained based on the plurality of images, and at least one of the plurality of images is obtained by applying the patient-level classification module to the obtained plurality of feature maps. Obtain a probability score that a lesion appears in one image, and obtain the probability score for the plurality of images and the objective function value calculated using the truth value class mapped to the plurality of images in the training data set. Parameters of the classification model can be updated.

The processor generates a segmentation model including the encoder of the machine learning model, a decoder connected to the encoder, and a segmentation module at a patient level, and the target patient of the training data set. A plurality of feature maps are obtained by applying the encoder and the decoder to a plurality of images acquired based on CT imaging, and the plurality of feature maps are obtained by applying the patient level segmentation module to the obtained plurality of feature maps. Extract a lesion area from images and update parameters of the segmentation model based on an objective function value calculated using the extracted lesion area and a true lesion area mapped to the plurality of images in the training data set. You can.

The encoder of the machine learning model according to one embodiment may be trained to capture globalized features that satisfy all three multiple tasks. Encoders can be improved in terms of balanced accuracy and can be robust on external data sets.

The encoder of the machine learning model according to one embodiment is learned through multiple tasks and can be extended to patient-level tasks (e.g., classification, segmentation) through transfer learning for a 3D operator that uses continuous or volumetric features. there is.

The machine learning model according to one embodiment may have better performance than the machine learning model according to the comparative example through two external data sets reflecting actual clinical environments and a public data set with difficult-to-detect cases of ICH.

Figure 1 shows a patient-level classification model and a patient-level segmentation model according to an embodiment.
Figure 2 shows dictionary learning of an encoder through multi-task learning according to an embodiment.
Figure 3 shows training of a patient-level classification model according to one embodiment.
Figure 4 shows training of a patient-level segmentation model according to one embodiment.
Figure 5 may show the frequency according to the amount of ICH of internal data sets and external data sets according to an embodiment.
Figure 6 may show the volume and intensity of ICH of internal data sets and external data sets according to an embodiment.
Figure 7 may show activation maps of encoders learned using a multi-task model according to one embodiment and a comparative example.
Figure 8 shows a performance comparison of a patient-level classification model according to one embodiment and a classification model according to a comparative embodiment.
Figure 9 may show the difference and robustness between a patient-level classification model according to one embodiment and classification models according to comparative embodiments.
Figure 10 shows the results of a segmentation model according to one embodiment and comparative examples.
Figure 11 shows ICH volumes of a segmented model according to one embodiment and comparative examples.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

For training of machine learning models, transfer learning can be used. Transfer learning may refer to a technique of training a second machine learning model by using at least a portion of a trained first machine learning model as at least a portion of the second machine learning model. The first machine learning model may represent a model for performing a pretext task, and the second machine learning model may represent a model for performing a target task. A freetext task (also referred to as an upstream task) may represent a task performed through a first machine learning model used for training a second machine learning model. The target task (also referred to as a downstream task) is a task that is a goal to be performed and may represent a task performed through a second machine learning model.

The second machine learning model may be difficult to train from the initial state due to various problems such as limitations in training data and/or limitations in training speed.

Instead of training the second machine learning model from an initial state, train the first machine learning model and then use at least a portion of the trained first machine learning model to form a second machine learning model. You can train machine learning models.

A first machine learning model for a freetext task can be trained. Training of the first machine learning model may be performed as pre-training of the second machine learning model.

The initial model for training the second machine learning model may include at least a portion of the trained first machine learning model. For example, a first machine learning model and a second machine learning model may share the structure of an encoder. The encoder of the first trained machine learning model can be used as the encoder of the initial model of the second machine learning model.

An initial model of the second machine learning model may be trained. Training of a second machine learning model using at least a part of the trained first machine learning model as an initial model involves fine-tuning of at least a part of the second machine learning model obtained based on the first machine learning model. It can be included.

Hereinafter, the encoder of the multi-task model is pre-trained using slice-level multi-tasks (e.g., classification task, segmentation task, and reconstruction task) as freetext tasks, and patient-level tasks (e.g., : Apparatus and method for training a machine learning model (e.g., a patient-level classification model and a patient-level segmentation model) as a target task (one of the classification task of the presence or absence of a lesion at the patient level and the segmentation task of the lesion area at the patient level) This will be described later with reference to FIGS. 1 to 11.

Figure 1 shows a patient-level classification model and a patient-level segmentation model according to an embodiment.

The patient level classification device 110 provides a probability score (possibility score) that a lesion is included in a target body part of a target patient based on a plurality of images 112 (e.g., CT slices) in order to classify the presence or absence of a lesion at the patient level. score) (114). The patient level segmentation device 120 divides the lesion area 124 among the target body parts of the target patient based on the plurality of images 122 (e.g., CT slices) in order to segment the lesion area at the patient level. It can represent a device that does this.

The target patient may represent a patient for whom a lesion area is to be divided by classifying whether a lesion exists in the target body part of the target patient or extracting the lesion area from the target body part of the target patient. The target body part may represent a CT scanned body part and, for example, may include the brain. The lesion may represent a physical or physiological change due to intracranial hemorrhage (ICH), and illustratively, the lesion area may represent the location in the brain where the intracranial hemorrhage occurs.

A plurality of images may be acquired based on CT scan of the target patient. A plurality of images may represent CT slices acquired based on one CT scan. A CT slice represents a cross-sectional image obtained by rotating an X-ray generator and an X-ray detector in pairs around the target patient and restoring the You can. Each pixel of a CT slice may represent the attenuation rate of the portion corresponding to the pixel.

Hereinafter, the patient level classification device 110 and the patient level division device 120 will be described, respectively.

The patient level classification device 110 according to one embodiment may include a processor 111. The patient level classification device 110 applies a classification model 113 (also referred to as a classification model) at the patient level to a plurality of images 112 (e.g., CT slices) to determine the possibility of A possibility score (114) can be obtained.

The classification model 113 may represent a machine learning model for classifying the presence or absence of a lesion in a target patient. The classification model 113 may include an encoder 113a and a patient-level classification module 113b connected to the encoder 113a. As will be described later, the encoder 113a may be pre-trained through a multi-task model.

The processor 111 may obtain a plurality of feature maps by applying the encoder 113a to the plurality of images 112. The processor 111 can obtain one feature map by applying the encoder 113a to one image. The processor 111 may obtain a plurality of feature maps by repeating obtaining the feature map by applying the encoder 113a to each of the plurality of images 112.

The processor 111 may obtain the likelihood score 114 by applying the patient-level classification module 113b to the plurality of feature maps. The likelihood score 114 may indicate the possibility that a lesion exists in at least a portion of a target body part of a target patient shown in a plurality of images. For example, the likelihood score 114 may indicate the possibility that a lesion appears in at least one image among a plurality of images.

The processor 111 may classify the presence or absence of a lesion based on the likelihood score 114. For example, if the probability score 114 exceeds a threshold score, the processor 111 may classify the target patient as containing a lesion in the target body part (eg, ICH class). If the probability score 114 is less than the threshold score, the processor 111 may classify the target patient as not containing a lesion in the target body part (eg, normal class). The critical score may represent a score that serves as a standard for classifying the presence or absence of a lesion.

The patient level segmentation device 120 according to one embodiment may include a processor 121. The patient-level segmentation device 120 may obtain a lesion area by applying a patient-level segmentation model 123 (also referred to as a segmentation model) to a plurality of images 122 (e.g., CT slices). You can. Patient-level segmentation tasks can be performed to localize lesions (eg, ICH) and measure cerebral hemorrhage at the patient level.

The patient-level segmentation model 123 may represent a machine learning model for segmenting a lesion area in the target patient's body. The patient-level segmentation model 123 may include an encoder 123a, a decoder 123b connected to the encoder 123a, and a patient-level segmentation module 123c. As will be described later, the encoder 123a may be pre-trained through a multi-task model.

The processor 121 may obtain a plurality of feature maps by applying the encoder 123a and the decoder 123b to the plurality of images 122. The processor 121 can obtain one feature map by applying the encoder 123a and the decoder 123b to one image. The processor 121 may obtain a plurality of feature maps by repeating obtaining the feature map by applying the encoder 123a and the decoder 123b to each of the plurality of images 122.

The processor 121 may obtain the lesion area 124 by applying the patient-level segmentation module 123b to the plurality of feature maps. The lesion area 124 may include one or more lesion areas shown in a plurality of images. The lesion area 124 may represent an area where the lesion areas shown in each of the plurality of images 122 are accumulated.

Figure 2 shows dictionary learning of an encoder through multi-task learning according to an embodiment.

An electronic device according to one embodiment may include a processor. The electronic device may pre-train the encoder of the machine learning model for the target task using the machine learning model for the freetext task. The multi-task model may represent a machine learning model for performing the pretext task when the pretext task is multi-task.

The multi-task model 200 pre-trains the encoder of a classification model (e.g., the patient-level classification model 113 in FIG. 1) and/or a segmentation model (e.g., the patient-level segmentation model 123 in FIG. 1). It can be used to

The multi-task model 200 according to one embodiment includes an encoder 210, a universal classification module 220 connected to the encoder 210, a universal segmentation module 230 connected to the encoder 210, and a restoration connected to the encoder. It may include a module 240 (reconstruction module).

The multi-task model 200 may have a hard parameter sharing architecture including one shared encoder and a plurality of task-specific modules. A hard parameter sharing architecture may represent an architecture that shares a root model (e.g., encoder) and then learns each representation. Encoder 210 can be trained to capture various ICH features. Illustratively, the encoder may be implemented as ResNet-50. The structure of ResNet-50 is relatively lightweight, and ResNet-50 can have four stages that are easy to control with ResBlock to perform operations.

The multi-task model 200 may output a first likelihood score and a segmented region by being applied to the input image. The multi-task model 200 may further output a restored image along with the first likelihood score and the segmentation area.

The training data set of the multi-task model 200 may include a training input image 211, a true value class, and a true lesion area. The true value class and the true lesion area may be mapped to the training input image 211. The true value class may represent a class that indicates whether a lesion appears in the training input image 211. For example, the true value class may be mapped as a lesion class (e.g., ICH class) to the training input image 211 in which the lesion appears, and as a normal class to the training input image 211 in which the lesion does not appear. The true lesion area may represent an area corresponding to an actual lesion in the training input image 211.

The processor of the electronic device may output at least one of the first likelihood score 221, the segmented area 231, and the restored image 241 by applying the training input image 211 to the multi-task model 200. As will be described later, the true value class and the true lesion area may be used to calculate the objective function value of the multi-task model 200.

The processor of the electronic device may extract the feature map 212 by applying the encoder 210 to the training input image 211. The feature map 212 extracts the features of the training input image 211 and may represent a result obtained by performing convolution on the training input image 211.

The processor of the electronic device may obtain the first likelihood score 221 by applying the general-purpose classification module 220 to the feature map 212. The general classification module 220 is a module for a slice-level classification task and may represent a module that outputs the first likelihood score 221 from the feature map 212. The first likelihood score 221 may indicate the possibility that a lesion appears in the training input image 211. Exemplarily, the first likelihood score 221 is a real number greater than or equal to 0 and less than or equal to 1. A value closer to 0 indicates a lower probability that a lesion will appear in the training input image 211, and 1 The closer the value to , the higher the probability that a lesion will appear in the training input image 211.

A slice-level classification task can be performed to determine whether an ICH exists in each CT slice. The encoder can be trained to find key features through a slice-level classification task. Illustratively, the general-purpose classification module 220 may include a linear classifier (eg, linear block).

The processor of the electronic device may extract the lesion area 231 by applying the general-purpose segmentation module 230 to the feature map 212. The general-purpose segmentation module 230 is a module for a slice-level segmentation task and may represent a module that outputs a lesion area 231 extracted from the training input image 211 by applying it to the feature map 212. The lesion area 231 may represent an area corresponding to a lesion in the training input image 211.

Slice-level segmentation tasks can be performed to localize ICH and measure cerebral hemorrhage in each CT slice. The encoder can be trained to find local features through a slice-level segmentation task. Illustratively, the general-purpose segmentation module 230 may include a U-Net decoder including skip connections. The general-purpose segmentation module 230 has 5 depths and can deliver the last feature of each stage of ResNet-50 to the decoder using skip connection. Additionally, spatial and channel squeeze and excitation (SCSE) blocks can be attached to the first and last of each decoder block to more effectively focus attention on the feature.

The processor of the electronic device may obtain the restored image 241 by applying the restoration module 240 to the feature map 212. The restoration module 240 is a module for a slice-level restoration task and can output the restored image 241 from the feature map 212. The restored image 241 may represent an image generated to restore the training input image 211 based on the feature map 212.

A slice-level restoration task can be performed to restore the original CT slide from the output compressed by the encoder for each CT slice. The encoder can learn the representation of the ICH through a slice-level restoration task. By way of example, the restoration module 240 may include a PixelShuffle decoder. The PixelShuffle decoder may be able to super-resolve low-resolution data into high-resolution data at a lower computational cost than a deconvolution layer. The restoration module 240 may have an Autoencoder-like structure without skip connections to prevent trivial solutions. The encoder can extract more meaningful features.

The processor of the electronic device may calculate the objective function of the multi-task model 200.

The objective function of the multi-task model 200 may include classification loss, segmentation loss, reconstruction loss, and consistency loss. For example, the objective function value of the multi-task model 200 can be calculated with equal weights for all four losses as follows:

Here, Total Loss represents the objective function value of the multi-task model 200, SEG Loss represents the segmentation loss, CLS Loss represents the classification loss, CON Loss represents the consistency loss, and REC Loss may represent the restoration loss. there is.

The processor of the electronic device may calculate the classification loss. Classification loss may be related to the performance of the encoder 210 and the general-purpose classification module 220. The processor of the electronic device may calculate a classification loss based on the first likelihood score 221 and the truth class mapped to the training input image 211. Illustratively, the classification loss may be calculated as binary cross-entropy loss (BCE loss).

The electronic device's processor may calculate the segmentation loss. Segmentation loss may be related to the performance of the encoder 210 and the general-purpose segmentation module 230. The processor of the electronic device may calculate the segmentation loss based on the lesion area 231 and the true lesion area. Illustratively, the segmentation loss is a one-to-one combination of binary cross entropy loss (BCE loss) and overlap-based Dice-coefficient loss (DICE loss) (e.g., binary cross entropy loss and can be calculated as the sum of overlap-based Dice coefficient losses).

The processor of the electronic device may calculate the restoration loss. The restoration loss may be related to the performance of the encoder 210 and the restoration module 240. The processor of the electronic device may calculate the segmentation loss based on the training input image 211 and the restored image 241. For example, the processor of the electronic device may perform an average absolute error (mean) of the intensity difference between the pixels of the training input image 211 and the pixels of the reconstructed image 241 corresponding to the pixels of the training input image 211. absolute error) can be calculated. Illustratively, the segmentation loss may be calculated as the average absolute error loss (e.g., L1 loss).

The processor of the electronic device may calculate the coherence loss 250. Consistency loss may indicate consistency between the outputs of the universal classification module 220 and the universal segmentation module 230 (e.g., first likelihood score 221 and lesion area 231).

If the output of the general classification module 220 and the output of the general segmentation module 230 indicate the presence or absence of a lesion differently, the consistency loss 250 may have a high value. For example, if the first likelihood score 221 has a value close to 1 and the lesion area 231 has not been extracted, the consistency loss 250 may have a high value. For another example, when the first likelihood score 221 is extracted with a value close to 0 and the lesion area 231 is extracted, the consistency loss 250 may have a high value.

When the output of the general classification module 220 and the output of the general segmentation module 230 indicate the presence or absence of a lesion equally, the consistency loss 250 may have a low value. For example, when the first likelihood score 221 has a value close to 1 and the lesion area 231 is extracted, the consistency loss 250 may have a low value. For another example, when the first likelihood score 221 has a value close to 0 and the lesion area 231 has not been extracted, the consistency loss 250 may have a low value.

Calculation of consistency loss 250 may be performed for consistency between the universal classification module 220 and the universal segmentation module 230 in each CT slice. The encoder 210 can learn more common features of the slice-level classification task and the slice-level segmentation task through calculation of the consistency loss 250.

The processor of the electronic device may calculate the consistency loss 250 based on the first likelihood score 221 and the second likelihood score. The second likelihood score may indicate the likelihood that a lesion will appear in the training input image 211.

The processor of the electronic device may obtain a second probability score using the extracted lesion area 231. The processor of the electronic device may obtain a second likelihood score by applying at least one of an average pooling layer and a max pooling layer to the extracted lesion area 231. As an example, the processor of the electronic device may apply four average pooling layers and one maximum pooling layer to the extracted lesion area 231 due to the 16x16 size of the last feature of ResNet-50. The processor of the electronic device may match the resolution of the output of the general-purpose segmentation module 230 with the resolution of the output module of the general-purpose classification module 220 by obtaining the second likelihood score.

For reference, both the first likelihood score and the second likelihood score may indicate the possibility that a lesion appears in the training input image 211. However, the first probability score 221 may represent a score obtained based on the general-purpose classification module 220, and the second probability score may represent a score obtained based on the general-purpose segmentation module 230. Accordingly, when the difference between the first likelihood score 221 and the second likelihood score is large, consistency between the outputs of the universal classification module 220 and the universal segmentation module 230 may be low. Conversely, when the difference between the first likelihood score 221 and the second likelihood score is small, consistency between outputs of the general-purpose classification module 220 and the general-purpose segmentation module 230 may be high.

The processor of the electronic device may calculate the consistency loss 250 based on the difference between the first likelihood score 221 and the second likelihood score. For example, the processor of the electronic device may calculate the square of the difference between the first likelihood score 221 and the second likelihood score for each image. The processor of the electronic device may average the square of the difference between the first probability score 221 and the second probability score. A batch may contain one or more images. The processor of the electronic device may calculate the consistency loss by averaging the square of the difference between the first likelihood score 221 and the second likelihood score for the batch.

By way of example, consistency loss ( CON Loss ) can be calculated by the following equation.

here, and represents the output of the universal classification module 220 and the universal segmentation module 230, respectively, N represents the number of CT slices, may represent a combination of four average pooling layers and one maximum pooling layer.

Figure 3 shows training of a patient-level classification model according to one embodiment.

As described above in FIG. 1 , the classification model 300 may include an encoder 310 and a patient-level classification module 320 connected to the encoder. The classification model 300 may output a likelihood score 322 by being applied to a plurality of images 311 .

The training data set of the classification model 300 may include a plurality of images 311 and a true value class mapped to the plurality of images 311 . As will be described later, the true value class can be used to calculate the objective function value of the classification model 300.

The processor of the electronic device may generate a classification model 300. The classification model 300 may include an encoder 310 and a patient-level classification module 320 (eg, 3D Classifier in FIG. 3) connected to the encoder 310. Encoder 310 may be obtained based on the encoder of a trained multi-task model (e.g., multi-task model 200 in FIG. 2). For example, the processor of the electronic device may pre-train the encoder 310 using a multi-task model. The processor of the electronic device may generate a classification model 300 including a pre-trained encoder of a multi-task model.

The processor of the electronic device may obtain a plurality of feature maps by applying the encoder 310 to the plurality of images 311. The processor of the electronic device can obtain one feature map by applying the encoder 310 to one image. The processor of the electronic device may obtain a plurality of feature maps by repeating obtaining the feature map by applying the encoder 310 to each of the plurality of images 311.

One feature map may represent a 2D feature for a two-dimensional medical image (e.g., one CT slice), and multiple stacked feature maps may represent a 2D feature for a three-dimensional medical image (e.g., multiple CT slices). 3D features can be displayed. For example, a processor of an electronic device may extract 3D features by repeatedly stacking 2D features as many as the number of CT slices. The extracted 3D features can be input to the patient-level classification module 320 to supplement volumetric information.

The processor of the electronic device may obtain the likelihood score 322 by applying the patient-level classification module 320 to the plurality of feature maps. The likelihood score 322 may indicate the possibility that a lesion appears in at least one image among a plurality of images.

In a patient-level classification task, a lesion may appear in at least some consecutive images of a plurality of images. The patient-level classification module 320 may be implemented with Long Short Term Memory (LSTM) to capture sequential information of lesions appearing in consecutive images and control the variable length of CT slices. For example, the patient level classification module 320 may have two linear layers and a bi-directional LSTM. The patient-level classification module 320 can improve ICH detection performance through bidirectional LSTM.

The processor of the electronic device may calculate the objective function value of the classification model 300. The processor of the electronic device may calculate the objective function value based on the likelihood score 322 and the truth value class mapped to the plurality of images 311. For example, the processor of the electronic device may calculate the objective function value of the classification model 300 as binary cross-entropy loss (BCE loss).

The processor of the electronic device may update the parameters of the classification model 300 based on the objective function value.

The processor of the electronic device may update the parameters of the remaining non-pre-trained partial models while keeping the parameters of the pre-trained partial model fixed (e.g., frozen) among the classification model 300. For example, the processor of the electronic device may update the parameters of the patient-level classification module 320 while keeping the parameters of the encoder 310 of the classification model 300 fixed in at least some of the training iterations. You can.

The processor of the electronic device may unfreeze (e.g., unfreeze) the parameters of the pre-trained partial model among the classification model 300 and update at least some parameters of the classification model 300. there is. For example, the processor of the electronic device may unfreeze the parameters of the encoder 310 of the classification model 300 in another one of the training iterations, and Parameters of the module 320 can be updated.

The processor of the electronic device according to one embodiment may not immediately perform fine tuning of the encoder 310 in the expansion process. Instead, the processor of the electronic device may perform training of the classification model 300 and fine tuning of the encoder 310 step by step. For example, a processor in an electronic device may use a gradual unfreeze approach. The processor of the electronic device may perform training of the patient-level classification model for 100 epochs while fixing the parameters of the encoder 310. Afterwards, the processor of the electronic device may gradually release the fixation of the parameters of the encoder 310. The processor of the electronic device may release the fixation of the parameters of the encoder 310 every 10 epochs.

Figure 4 shows training of a patient-level segmentation model according to one embodiment.

As described above in FIG. 1, the segmentation model 400 includes an encoder 410, a decoder 420 connected to the encoder 410, and a patient-level segmentation module 430 (e.g., 3D Segmentor in FIG. 4). can do. The segmentation model 400 may output a lesion area 432 by being applied to a plurality of images 411 .

The training data set of the segmentation model 400 may include a plurality of images 411 and a true lesion area mapped to the plurality of images 411 . As will be described later, the true lesion area can be used to calculate the objective function value of the segmentation model 400.

The processor of the electronic device may generate the segmentation model 400. The segmentation model 400 may include an encoder 410, a decoder 420 connected to the encoder 410, and a patient-level segmentation module 430. Encoder 410 may be obtained based on the encoder of a trained multi-task model (e.g., multi-task model 200 in FIG. 2). For example, the processor of the electronic device may pre-train the encoder 410 using a multi-task model. The processor of the electronic device may generate a segmentation model 400 including a pre-trained encoder of a multi-task model.

The processor of the electronic device may obtain a plurality of feature maps by applying the encoder 410 and the decoder 420 to the plurality of images 411. The processor of the electronic device can obtain one feature map by applying the encoder 410 and decoder 420 to one image. The processor of the electronic device may obtain a plurality of feature maps by repeating obtaining the feature map by applying the encoder 410 and the decoder 420 to each of the plurality of images 411.

One feature map may represent a 2D feature for a two-dimensional medical image (e.g., one CT slice), and multiple stacked feature maps may represent a 2D feature for a three-dimensional medical image (e.g., multiple CT slices). 3D features can be displayed. For example, a processor of an electronic device may extract 3D features by repeatedly stacking 2D features as many as the number of CT slices. The extracted 3D features can be input to the patient-level segmentation module 430 to supplement volumetric information.

The processor of the electronic device may extract the lesion area 432 by applying the patient-level segmentation module 430 to the plurality of feature maps. The lesion area 432 may represent an area corresponding to the lesion shown in the plurality of images 411 . The segmentation module 430 may have a 3D convolution layer (Conv3D).

The processor of the electronic device may calculate the objective function value of the segmentation model 400. The processor of the electronic device may calculate an objective function value based on the lesion area 432 and the true lesion area mapped to the plurality of images 411 . For example, the processor of the electronic device may calculate the objective function value of the segmentation model 400 as a combination of binary cross entropy loss (BCE loss) and DICE coefficient loss (DICE loss).

The processor of the electronic device may update the parameters of the segmentation model 400 based on the objective function value.

The processor of the electronic device may update the parameters of the remaining non-pre-trained partial models while keeping the parameters of the pre-trained partial model fixed (e.g., frozen) among the segmentation models 400. For example, the processor of the electronic device may, in at least some of the training iterations, operate the decoder 420 and the patient-level segmentation module 430 while fixing the parameters of the encoder 410 of the segmentation model 400. Parameters can be updated.

The processor of the electronic device may unfix the parameters of the pre-learned partial model of the segmentation model 400 (e.g., unfreeze it) and update at least some parameters of the segmentation model 400. there is. For example, the processor of the electronic device may unfreeze the parameters of the encoder 410 of the segmentation model 400 in another one of the training iterations and , and the parameters of the segmentation module 430 at the patient level can be updated.

The processor of the electronic device according to one embodiment may not immediately perform fine tuning of the encoder 410 in the expansion process. Instead, the processor of the electronic device may perform learning of the segmentation model 400 and fine tuning of the encoder 410 step by step. For example, a processor in an electronic device may use a gradual unfreeze approach. The processor of the electronic device may perform learning of the patient-level segmentation model 400 for 100 epochs while fixing the parameters of the encoder 410. Afterwards, the processor of the electronic device may gradually release the fixation of the parameters of the encoder 410. The processor of the electronic device may release the fixation of the parameters of the encoder 410 every 10 epochs.

Figure 5 may show the frequency according to the amount of ICH of internal data sets and external data sets according to an embodiment. Figure 6 may show the volume and intensity of ICH of internal data sets and external data sets according to an embodiment.

The machine learning model according to one embodiment uses an internal test set (AMC in FIGS. 5 and 6), as well as two external test sets with a high proportion of normal patients (FIG. 5) to demonstrate performance in real emergency situations. and in Figure 6, External1, External2), and another external test set (in Figures 5 and 6, External3) to verify robustness against cases that are difficult to detect due to small, faint hemorrhages (e.g., one open source ICH data It can be evaluated using the PhysioNet data set).

An internal data set may represent a data set used for training, validation, and testing of a machine learning model. Illustratively, the internal data set may have a training data set, a validation data set, and a test data set.

Figure 7 may show activation maps of encoders learned using a multi-task model according to one embodiment and a comparative example.

Activation maps can be obtained by using the feature map output from the last convolutional layer of the encoder, averaged for each channel, normalized with sigmoid activation, and then interpolated to match the input resolution.

Encoders of multi-task models according to one embodiment and comparative embodiments are trained with an encoder of the same structure (e.g., ResNet-50 encoder) and the same training settings (e.g., input size, learning rate, optimization, augmentation, etc.) for fair comparison. and can be evaluated. The only difference between the encoders of the multi-task models according to one embodiment and the comparative embodiment may be a module (eg, a general-purpose classification module, a general-purpose segmentation module, and a restoration module) according to a combination of slice-level tasks.

To perform a comparison according to the severity of ICH cases, 2 severe cases, 2 moderate cases, and 4 mild cases can be compared. .

The encoder can be trained to intensively extract global information through a slice-level classification task, while the encoder can be trained to densely capture local features through a slice-level segmentation task. You can. If the encoder is trained through a multi-task model that performs more diverse tasks, more semantic features can be extracted.

As shown in FIG. 7, the activation map representing the area on which the encoder focuses may vary depending on various combinations of slice-level tasks.

The individuality of each slice-level classification task, segmentation task, restoration task, and consistency loss can be clearly revealed through the activation map. For example, the encoder of a model for a slice-level classification task (in Figure 7, denoted CLS ) may have an activation map that focuses on the most salient points of the ICH. The encoder of the model for the slice-level restoration task (indicated by REC in Figure 7) and the encoder of the model for the slice-level segmentation task (indicated by SEG in Figure 7) can perform pixel-wise dense prediction, , we can have an activation map that captures features for the entire image. The encoder ( REC ) of the model for the slice-level restoration task may have an activation map that is relatively evenly activated compared to the encoder ( SEG ) of the model for the slice-level segmentation task within the brain region. The encoder of the model for the slice-level segmentation task ( SEG ) may have an activation map that is rather sporadically focused among the entire image. The encoder of the model that uses consistency loss with the slice-level classification task and the segmentation task (in Figure 7, denoted as CLS+SEG+CON ) is a prominent part of the encoder of the model for the slice-level classification task ( CLS ). It is possible to have an activation map that has a balance between the tendency to focus on and the tendency for sporadic focusing seen in the model's encoder ( SEG ) for the slice-level segmentation task.

With respect to the encoder of the model for multiple tasks at the slice level, the diversity of the tasks at the slice level may affect the activation map. If the encoder is trained using models for many tasks, noticeable and clear activation regions may emerge.

The encoder of the multi-task model according to one embodiment (represented as MRI-Net in FIG. 7) may have the most ideal activation map compared to the encoders according to comparative embodiments, and the ideal activation map is a balance between the contributions of the tasks. can be matched. The encoder of the multi-task model according to one embodiment may be significantly activated in the edge and border regions of the brain compared to the encoders according to the comparative embodiments (represented as ImageNet and Model Genesis in FIG. 7). Additionally, the encoder (e.g., ResNet-50 encoder) according to the comparative example (denoted as Model Genesis in Figure 7) is an encoder of the model ( REC ) for the slice-level restoration task that is evenly activated within brain regions ( Figure 7 It may have an activation map with a similar pattern to the activation map (indicated by REC in ).

Table 1 can show details of internal data sets and external data sets.

[Table 1]

The robustness of the patient-level classification model and the patient-level segmentation model can be verified through external data sets with a high proportion of normal patients. The robustness of the patient-level classification model and the patient-level segmentation model can be verified for cases that are difficult to detect due to small and faint bleeding through other external data sets described above.

Exemplarily, a data set may include data for patients with five types of ICH (e.g., CPH, IVH, EDH, SDH, and SAH) with high class imbalance. All types of ICH classes can be integrated into one class (e.g., ICH class) for patient-level classification tasks (e.g., binary classification tasks). Integration of ICH classes can resolve class imbalance for ICH, and machine learning models can focus on whether there is cerebral hemorrhage in emergency situations.

Labeling of segmentation masks for ICH patients can be performed by three senior radiologists with more than 14 years of experience. Every CT scan can have a size of 512×512 pixels and can have various depths (e.g., number of CT slices).

CT scans of the four ICH data sets (i.e., internal data set and three external data sets) can be acquired at approximately 5.0 mm thickness according to the emergency room unenhanced NCCT protocol as shown in Table 1. The depth of the internal data set may be distributed in a range greater than or equal to 28 and less than or equal to 50, and the average of the number of slices of the internal data set may be 38.89.

Internal data sets may refer to data sets approved by the Institutional Ethics Committee of Asan Medical Center (AMC). CT scans in the internal data set may be obtained by retrospectively searching the database for patients who underwent consecutive NCCT examinations over a period of time (e.g., September 2009 to June 2017). Based on the clinical radiology reports of the acquired CT scans, the internal data set may contain data corresponding to a total of 811 ICH patients and 521 normal patients.

Two external data sets with high normality rates can be used to validate the model in a real clinical environment. The first external data set (External1) may represent data continuously collected from Nowon Eulji Hospital in Korea from July 2018 to October 2018. The second external data set (External2) may represent data continuously collected at Pohang Stroke and Spine Hospital from March 2019 to June 2019. The first external data set and the second external data set may not be manually selected, and the proportion of ICH patients in the first external data set and the second external data set may represent the incidence in actual clinical practice.

A third external data set (External3) can be used to check the model's performance on a data set that shows a small amount of faint bleeding. For example, the public dataset of PhysioNet version 1.3.1 with 75 cases (e.g. 75 CT scans) could be used as a third external dataset. Among the third external dataset, two CT scans of ICH patients were judged to be of poor quality by the radiologist and could be excluded. The average ICH volume in a third external data set (e.g., PhysioNet data set) may be 15.10 mL. In external data sets, mainly small amounts of cerebral hemorrhage can be observed.

The performance of the machine learning model according to one embodiment and the machine learning model according to the comparative embodiment may be compared.

A machine learning model (e.g., a patient-level classification model and a patient-level segmentation model) according to one embodiment may be implemented in Pytorch accelerated with an NVIDIA TITAN RTX 24GB GPU.

Preprocessing may include applying a brain window 40 HU wide and at the 140 HU level to the image (e.g., CT slice). Additionally, CLAHE (contrast limited adaptive histogram equalization) of the Albummentation library can be applied to increase the contrast of the image. Due to limitations in GPU resources, the image can be reduced to a size of 256×256 through linear interpolation.

Data augmentation may include use of the Albumentation library and the MONAI framework. For example, to address the heterogeneity of ICH, augmentation techniques such as ShiftScaleRotate, RandShiftIntensity, HorizontalFlip, Brightness-Contrast, Gauss Noise, and Blur can be used.

Settings may include setting the batch size. In all experiments, the batch size can be set to the maximum for single GPU memory. Therefore, batch size may vary depending on the model. The machine learning model according to one embodiment and the machine learning model according to the comparative embodiment may be initialized by uniform Xavier, with a warm-up of 5 epochs, a weight reduction of 5e-4, and an interval (0.9, 0.999). It can be trained using the Adam optimizer with a learning rate of 1e-4 using beta. The learning rate can be reduced during training according to the following poly learning rate schedule: . The number of epochs for pre-training can be up to 1000, and the number of epochs for training and fine tuning (e.g. learning for downstream tasks) can be up to 500. However, each model used in the experiment may be the convergence model that recorded the highest verification score.

The model with the highest performance among the machine learning model according to one embodiment and the machine learning models according to comparative embodiments may be determined through evaluation on a verification data set. The validation data set may represent a data set that is separate from the training data set. Experiments on the machine learning model according to one embodiment and the machine learning model according to the comparative embodiment are performed in the same training settings (e.g., input size, learning rate, optimizer, augmentation combinations, etc.). You can.

Table 2 may show a comparison of performance between a patient-level classification model according to one embodiment and a patient-level classification model according to a comparative example.

[Table 2]

The classification models according to Comparative Examples 1 to 6 may represent classification models according to the prior art. The classification models according to Comparative Examples 7 to 14 may represent a model that adjusts (e.g., ablates) the pretext task of the multi-task model for pre-learning of the encoder based on the classification model according to one embodiment. . For each evaluation metric of the data set in Table 2, among the classification models according to one embodiment and comparative examples, underlined indicates the evaluation metric of the model with the largest value, and bold indicates the evaluation of the model with the second largest value. Metrics can be displayed.

Specifically, the classification model according to Comparative Example 1 is described in paper 'A. Patel, S. C. Van De Leemput, M. Prokop, B. Van Ginneken, and R. Manniesing, "Image Level Training and Prediction: Intracranial Hemorrhage Identification in 3D Non-Contrast CT," (in English), Ieee Access, vol. 7, pp. 92355-92364, 2019, doi: 10.1109/Access.2019.2927792.', and the classification model according to Comparative Example 2 is the full version of the paper 'A. Patel, S. C. Van De Leemput, M. Prokop, B. Van Ginneken, and R. Manniesing, "Image Level Training and Prediction: Intracranial Hemorrhage Identification in 3D Non-Contrast CT," (in English), Ieee Access, vol. 7, pp. 92355-92364, 2019, doi: 10.1109/Access.2019.2927792.', and the classification model according to Comparative Example 3 is the paper 'S. P. Singh, L. Wang, S. Gupta, B. Gulyαs, and P. Padmanabhan, "Shallow 3D CNN for detecting acute brain hemorrhage from medical imaging sensors," IEEE Sensors Journal, 2020', Comparative Example 4 The classification model according to represents the Scratch model, and the classification model according to Comparative Example 5 is described in the paper 'K. He, R. Girshick, and P. Dollαr, “Rethinking imagenet pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4918-4927.', and the classification model according to Comparative Example 6 is described in the paper 'Z. Zhou, V. Sodha, J. Pang, M. B. Gotway, and J. Liang, “Models genesis,” Medical image analysis, vol. 67, p. It may be disclosed in 101840, 2021.

To find the best multi-task model for pre-training the encoder, multi-task models for all combinations of slice-level classification task, segmentation task, restoration task, and consistency loss can be tested.

The classification model according to Comparative Example 7 (may be represented by CLS ) represents a model that uses a model that performs a slice-level classification task for pre-training of the encoder, and Comparative Example 8 (may be represented by SEG ) The classification model according to represents a model that uses a model that performs a slice-level segmentation task for pre-learning of the encoder, and the classification model according to Comparative Example 9 (which may be denoted as REC ) uses a slice-level model for pre-learning of the encoder. It can represent a model that uses a model that performs a level restoration task.

The classification model according to Comparative Example 10 (which can be expressed as CLS+SEG ) represents a model that uses a model that performs a slice-level classification task and a segmentation task for pre-learning of the encoder, and the classification model according to Comparative Example 11 ( CLS+ The classification model according to (can be expressed as REC ) represents a model that uses a model that performs a slice-level classification task and restoration task for dictionary learning of the encoder, and Comparative Example 12 (can be expressed as SEG+REC ) The classification model according to may represent a model that uses a model that performs slice-level segmentation tasks and restoration tasks for dictionary learning of the encoder.

The classification model according to Comparative Example 13 (which can be expressed as CLS+SEG+CON ) uses a model that performs a slice-level classification task and a segmentation task for pre-training of the encoder and sets an objective function value with consistency loss. represents a model including a pre-trained encoder, and the classification model according to Comparative Example 14 (can be expressed as CLS+SEG+REC ) includes a slice-level classification task, a segmentation task, and It can represent a model that uses a model that performs a restoration task.

A patient-level classification model according to one embodiment and a classification model according to a comparative example may be quantitatively evaluated using an evaluation metric. Among the evaluation metrics, the evaluation metrics related to the classification task are area under the ROC curve (AUC), sensitivity (SEN), specificity (SPE), balanced accuracy (B-ACC), and It may include an F1 score.

Several comparative experiments can be performed to verify the scalability of transfer learning to patient-level classification tasks and the efficiency of multi-task learning. Comparative Examples 1 to 3 may be classification models according to the prior art, and Comparative Examples 5 and 6 may be classification models using conventional transfer learning.

Table 2 can show the performance of one embodiment and comparative examples for each data set. The top of Table 2 (in Table 2, previous works) corresponds to comparative examples of conventional models and transfer learning approaches, and the bottom of Table 2 (in Table 2, ablation studies) corresponds to machine learning according to one embodiment. It may correspond to comparative examples regarding ablation studies of the model.

The patient-level classification model according to one embodiment is well balanced in terms of AUROC scores and external test sets, and may have much better performance than most of the classification models according to comparative embodiments. Additionally, in terms of F1, B-ACC, SEN, and SPE, the classification model according to one embodiment may have better performance than the classification model according to comparative embodiments. In particular, classification models according to Comparative Examples 1 to 3 may have unstable results in external data sets. The classification model according to one embodiment may be significantly better than the classification model according to Comparative Examples 5 and 6 for most data sets compared to the classification model according to Comparative Examples using conventional transfer learning. The classification models according to Comparative Examples 5 and 6 may have unstable results even in external data sets.

Figure 8 shows a performance comparison of a patient-level classification model according to one embodiment and a classification model according to a comparative embodiment.

In FIG. 8, the ROC of the patient-level classification model (indicated as MRI-Net in FIG. 8) according to one embodiment may be compared with the ROC of the patient-level classification model according to the comparative embodiment. Statistical significance can be determined by the method of DeLong et al. Comparative examples with a p value of less than 0.05 are indicated by *, comparative examples with a p value of less than 0.01 are indicated by **, and comparative examples with a p value of less than 0.001 are indicated by ** It may be displayed as *.

The AUC of the classification model according to one embodiment may be larger than the AUC of the classification models according to Comparative Examples 1 to 6. Differences between the classification model according to one embodiment and the classification models of comparative embodiments may appear more prominently in the ROC graphs 820, 830, and 840 for the external data set than in the ROC graph 810 for the internal data set. there is.

Figure 9 may show the difference and robustness between a patient-level classification model according to one embodiment and classification models according to comparative embodiments.

Regarding pre-training of the encoder of the classification model according to one embodiment, in order to obtain a pretext task of the multi-task model suitable for extending the encoder of the slice-level multi-task model to the patient-level classification task, Ablation studies for all combinations can be performed. The performance of a classification model may depend on the combination of pretext tasks in a multi-task model.

In Table 2, among the comparative examples including a pre-trained encoder using a single task model, the classification model according to Comparative Example 7 ( CLS ) has a value of 0.663 as the average F1 of the four data sets, so the patient This may be the most appropriate task to expand to a level classification task. Among the comparative examples including a pre-trained encoder using a dual task model, the classification model according to Comparative Example 11 ( CSL+REC ) achieved the best performance of 0.766 with an average F1 of the four data sets. You can have it. Among Comparative Examples 13 and 14, the classification model according to Comparative Example 14 ( CLS+SEG+REC ) may have an average F1 of 0.799 for the four data sets.

The classification model according to Comparative Example 10 ( CLS+SEG ) and the classification model according to Comparative Example 13 ( CLS+SEG+CON ) can be compared. The classification model according to Comparative Example 13 ( CLS+SEG+CON ) can have an F1 score increased by 2.8% over the classification model according to Comparative Example 10 ( CLS+SEG ) by using more consistency loss. One embodiment compares the directly trained scratch model to an F1 score increased by 21.1%, balanced accuracy (B-ACC) increased by 13.1%, and sensitivity (SEN) increased by 18.8% compared to the classification model according to Example 4. ), and specificity (SPE) increased by 12.8%. A classification model according to one embodiment may have the best gain through three tasks and consistency loss.

Table 3 shows the performance of the patient-level classification model according to ICH volume.

[Table 3]

In order to determine whether the classification model according to one embodiment can be applied clinically to a classification system, the classification model's ICH presence/absence classification performance (e.g., ICH detection performance) may be analyzed. Among the ICH presence/absence classification performance of the classification model, the performance of classifying ICH over 30mL may be important in terms of mortality. As shown in Table 3, the error rate of ICH patient classification according to ICH volume can be analyzed. The classification model according to one embodiment is capable of classifying all 30mL or greater ICH classes in the internal test set of the internal data set and the three external data sets. The classification model may have false positive rates of 3.0%, 12.3%, 5.4%, and 25.6% on the internal test set of the internal data set and the three external data sets, respectively. The classification model is also able to perfectly classify all patients with ICH bleeding between 15 mL and 30 mL in the internal test set and external data sets. However, the classification model may have false negative cases in the case of ICH patients with a volume of less than 15 mL. The error rate for multiple images of the ICH class with a volume less than 15 mL can have values of 7.1%, 16.6%, 44.4%, and 6.9% in the internal test set and three external data sets, respectively.

Table 4 may show a comparison of performance between a patient-level segmentation model according to one embodiment and a patient-level segmentation model according to a comparative embodiment.

[Table 4]

The split models according to Comparative Examples 4 to 6 and 15 to 17 may represent the split models according to the prior art. The models according to Comparative Examples 7 to 14 may represent a model in which the pretext task of the multi-task model for pre-learning of the encoder is adjusted (e.g., ablation) based on the segmentation model according to an embodiment. For each evaluation metric of the data set in Table 4, among the segmentation models according to one embodiment and comparative examples, underlined indicates the evaluation metric of the model with the largest value, and bold indicates the evaluation of the model with the second largest value. Metrics can be displayed. A paired t-test may be performed on each of the patient-level segmentation models according to one embodiment and the segmentation models according to comparative embodiments. Comparative examples with a p value of less than 0.05 are indicated by *, comparative examples with a p value of less than 0.01 are indicated by **, and comparative examples with a p value of less than 0.001 are indicated by ** It may be displayed as *.

Specifically, the split model according to Comparative Example 4 represents the Scratch model, and the split model according to Comparative Example 5 is described in the paper 'K. He, R. Girshick, and P. Dollαr, “Rethinking imagenet pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4918-4927.', and the splitting model according to Comparative Example 6 is described in the paper 'Z. Zhou, V. Sodha, J. Pang, M. B. Gotway, and J. Liang, “Models genesis,” Medical image analysis, vol. 67, p. It may be disclosed in 101840, 2021. The splitting model according to Comparative Example 15 is described in paper 'A. Patel et al., “Intracerebral Haemorrhage Segmentation in Non-Contrast CT,” Sci Rep, vol. 9, no. 1, p. 17858, Nov 28 2019, doi: 10.1038/s41598-019-54491-6.', and the partitioning model according to Comparative Examples 16 and 17 is described in the paper 'F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation," Nature methods, vol. 18, no. 2, pp. 203-211, 2021. It can represent a 2D version and a 3D version of the model disclosed.

To find the best multi-task model for pre-training the encoder, multi-task models for all combinations of slice-level classification task, segmentation task, restoration task, and consistency loss can be tested.

The segmentation model according to Comparative Example 7 (may be denoted as CLS ) represents a model that uses a model that performs a slice-level classification task for pre-training of the encoder, and Comparative Example 8 (may be denoted as SEG ) The segmentation model according to represents a model that uses a model that performs a slice-level segmentation task for pre-training of the encoder, and the segmentation model according to Comparative Example 9 (which may be denoted as REC ) uses a slice-level segmentation task for pre-learning of the encoder. It can represent a model that uses a model that performs a level restoration task.

The segmentation model according to Comparative Example 10 (which can be expressed as CLS+SEG ) represents a model that uses a model that performs a slice-level classification task and a segmentation task for pre-learning of the encoder, and the segmentation model according to Comparative Example 11 ( CLS+ The segmentation model according to (can be expressed as REC ) represents a model that uses a model that performs a slice-level classification task and restoration task for dictionary learning of the encoder, and Comparative Example 12 (can be expressed as SEG+REC ) The segmentation model according to may represent a model that uses a model that performs slice-level segmentation tasks and restoration tasks for dictionary learning of the encoder.

The segmentation model according to Comparative Example 13 (which can be expressed as CLS+SEG+CON ) uses a model that performs a slice-level classification task and a segmentation task for pre-training of the encoder and sets an objective function value with consistency loss. represents a model including a pre-trained encoder, and the segmentation model according to Comparative Example 14 (can be expressed as CLS+SEG+REC ) includes a slice-level classification task, a segmentation task, and It can represent a model that uses a model that performs a restoration task.

The patient-level segmentation model according to one embodiment and the segmentation model according to the comparative example may be quantitatively evaluated using an evaluation metric.

Among the evaluation metrics, an evaluation metric related to the segmentation task may include a dice similarity coefficient (DSC). Dice Similarity Coefficient (DSC) may represent an overlap metric that is measured only for positive cases. Among the evaluation metrics, an evaluation metric related to the segmentation task may include false positive volumes (FPV). False positive volume (FPV) may represent the volume of an area incorrectly extracted as a lesion area by the model from the normal area of a normal patient. The normal area of a normal patient may represent an area corresponding to the entirety of a plurality of images related to the patient labeled as normal. False positive volume (FPV) can be obtained by converting the pixels of the image into a volume (e.g., a value in units of μL) and can reflect the patient's spacing information in clinical interpretation. For example, false positive volume (FPV) can be calculated according to the formula:

here, represents the output of the patient-level segmentation model, and spacing _xyz may represent meta information of the CT slice for conversion to volume units.

The segmentation model can be evaluated by Dice similarity coefficient (DSC) for ICH patients and false positive volume (FPV) for normal patients. A segmentation model can be evaluated as to whether it is a balanced model by evaluating both the Dice Similarity Coefficient (DSC) and the False Positive Volume (FPV).

Several comparative experiments can be performed to verify the scalability of transfer learning to patient-level segmentation tasks and the efficiency of multi-task learning. Comparative Examples 15 to 17 may be a segmentation model according to the prior art, and Comparative Examples 5 and 6 may be a segmentation model using conventional transfer learning.

Table 4 can show the performance of one embodiment and comparative examples for each data set. The top of Table 4 (in Table 4, previous works) corresponds to comparative examples of conventional models and transfer learning approaches, and the bottom of Table 4 (in Table 4, ablation studies) corresponds to machine learning according to one embodiment. It may correspond to comparative examples regarding ablation studies of the model.

The segmentation model according to one embodiment performs much better than most of the segmentation models according to comparative embodiments in terms of Dice similarity coefficient (DSC) and false positive volume (FPV), not only on the internal test set but also on external data sets. You can have it.

The segmentation model according to one embodiment may have an increase in Dice Similarity Coefficient (DSC) of 11.5% and a reduction in false positive volume (FPV) of 75.8 μL of the four data sets compared to the segmentation model according to Comparative Example 15, For most data sets, it can have much better performance than the segmentation model according to Comparative Example 15.

The segmentation model according to Comparative Example 17 may have unstable results depending on the data set. In particular, the segmentation model according to Comparative Example 17 may have very high false positives despite high sensitivity due to overfitting of the patch-based method with a 3D convolution layer operator (Conv3D Operator). The segmentation model according to Comparative Example 16 has high sensitivity but may have a mean false positive volume (FPV) of 660 μL.

The split model according to Comparative Example 16 may have more balanced sensitivity and specificity than the split model according to Comparative Example 17. The segmentation model according to one embodiment may have an increase in Dice Similarity Coefficient (DSC) of 1.1% and a reduction in false positive volume (FPV) of 75.8 μL for the four data sets over the segmentation model according to Comparative Example 16.

From the perspective of transfer learning, the segmentation model according to Comparative Example 5 is more sensitive than the segmentation model according to Comparative Example 6, but the segmentation model according to Comparative Example 6 has a lower false positive volume (FPV) than the segmentation model according to Comparative Example 5. ) can have better performance when reduced. The patient-level segmentation model according to one embodiment has a Dice Similarity Coefficient (DSC) increase of 2.1% over the segmentation model according to Comparative Example 5 and a false positive volume (FPV) of 50.4 μL compared to the segmentation model according to Comparative Example 6. ) can have a decrease. The p value in the internal data set between the performance of the partition model according to one embodiment and the partition model according to Comparative Example 5 may have a value of less than 0.05 and may have statistical significance.

Regarding the pre-training of the encoder of the segmentation model according to one embodiment, in order to evaluate the pretext task suitable for extending the slice-level multi-task model to the patient-level segmentation task, excision of all combinations of the pretext task Research can be conducted. The performance of the patient-level segmentation model may depend on the combination of pretext tasks.

In Table 4, among the comparative examples including an encoder pre-trained using a single task model, the segmentation model according to comparative example 8 ( SEG ) had an average Dice similarity coefficient (DSC) of 0.554 for the four data sets. value and false positive volume (FPV) of 22.1 μL, the slice-level segmentation task may be the most suitable task to scale to the patient-level segmentation task. Among the comparative examples including a pre-trained encoder using a dual task model, the segmentation model according to Comparative Example 10 ( CLS+SEG ) has the average Dice Similarity Coefficient (DSC) of the four data sets. It may have a maximum value of 0.553, and the segmentation model according to Comparative Example 11 ( CLS+REC ) may have a minimum value of 15.6 μL of the average false positive volume (FPV) of the four data sets.

Among Comparative Examples 13 and 14, the segmentation model according to Comparative Example 14 ( CLS+SEG+REC ) had the maximum value of the average Dice similarity coefficient (DSC) of 0.557 and the minimum average false positive volume (FPV) of the four data sets. It can have a value of 6.8μL.

The split model according to Comparative Example 10 ( CLS+SEG ) and the split model according to Comparative Example 13 ( CLS+SEG+CON ) can be compared. The segmentation model according to Comparative Example 13 ( CLS+SEG+CON ) further exploits the consistency loss, thereby increasing the Dice Similarity Coefficient (DSC) by 0.4% and 52.5 μL over the segmentation model according to Comparative Example 10 ( CLS+SEG ). may have a reduced false positive volume (FPV).

The segmentation model according to one embodiment has a Dice similarity coefficient (DSC) increased by 13.5% and a false positive volume (FPV) reduced by 21.8 μL compared to the segmentation model according to Comparative Example 4 of a directly trained scratch model. You can. In one embodiment the split model may have the best gain with 3 tasks and consistency loss.

Figure 10 shows the results of a segmentation model according to one embodiment and comparative examples. Figure 11 shows ICH volumes of a segmented model according to one embodiment and comparative examples.

For visual inspection, a total of 4 cases were included: severe brain hemorrhage, mild brain hemorrhage, normal case with beam hardening artifact, and normal case. The performance of segmentation models for the cases can be compared.

The segmentation model according to one embodiment is not only numerically superior to the segmentation model according to comparative examples, but may also be visually superior in both severe and mild cerebral hemorrhage. In particular, for normal patients (e.g., a normal case with a beam hardening artifact and one of the normal cases), the segmentation models according to comparative embodiments are sensitive to beam hardening artifacts, but the segmentation model according to one embodiment is sensitive to beam hardening artifacts. Can be robust against positive volume (FPV).

As shown in FIG. 11, the segmentation model according to one embodiment may have significantly greater false positive volume (FPV) reduction performance in normal patients compared to the segmentation model according to the comparative example. The segmentation model according to one embodiment may have better performance than the segmentation model according to the comparative embodiment even in an external data set.

For 3D non-contrast head CT (NCCT), a transfer learning method using a multi-task model to classify the presence or absence of ICH at the patient level and segment the lesion area (e.g., ICH area) can be proposed. Transfer learning methods using multi-task models have high balanced accuracy and robustness to external data sets, and may have the potential to be applied for patient-level ICH tasks (e.g. classification and segmentation) in real clinical environments. The encoder of the patient-level segmentation model and/or classification model according to one embodiment may be trained to capture features from various perspectives through various tasks than a directly trained scratch model. The encoder can be pre-trained using a multi-task model for the freetext task of 4 tasks.

Through ablation studies, it has been shown that the transferability and performance of at least some of the machine learning models can vary depending on the task combination and target task of the multi-task model (e.g., patient-level classification task, patient-level segmentation task). can be found When the pretext task is a single task, an encoder pre-trained with the same type of pretext task as the target task may be most effective in terms of the performance of the machine learning model for the target task. However, when the pretext task is multiple tasks, the more the encoder is trained together by many tasks, the transferability of the encoder to the patient-level task and the performance on the target task can be improved, and the model for the target task can be used in actual clinical trials. Can be more robust to environmental and external data sets.

For example, the slice-level classification task and restoration task (also denoted as CLS+REC ) have the highest performance among encoders pre-trained using dual tasks for the patient-level classification model, while performing well for the patient-level segmentation model. It may have the lowest performance. Consistency loss (also denoted CON ) can balance the sparse predictions of the slice-level classification task and the dense predictions of the slice-level segmentation task, ensuring that the patient-level classification model and segmentation model are balanced and accurate for the target task. You can let it go. In the Model Genesis approach, encoders pre-trained with models for pixel-level dense prediction tasks (e.g., slice-level segmentation task, slice-level restoration task) may exhibit low transferability to patient-level classification models. In particular, the Model Genesis approach may have lower performance in patient-level classification models than the scratch model due to negative spillover effects. Performing representation learning through a dense prediction task may not be suitable for prior learning of a patient-level classification task. For the easily applicable ImageNet approach, extracting low-level features such as edges and textures of objects for ICH detection, both patient-level classification and segmentation models can outperform scratch models.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. It may be possible. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (17)

In a method performed by a processor,
extracting a feature map by applying an encoder of a machine learning model to the training input image;
Obtaining a first likelihood score that a lesion will appear in the training input image by applying a general-purpose classification module of the machine learning model to the feature map;
extracting a lesion area from the training input image by applying a general-purpose segmentation module of the machine learning model to the feature map;
Based on the obtained first likelihood score and the extracted lesion area, calculate an objective function value with a consistency loss that represents the consistency between the outputs of the universal classification module and the universal segmentation module. steps; and
Updating parameters of the machine learning model using the calculated objective function value.
How to include .

According to paragraph 1,
The step of calculating the objective function value is,
Comprising the step of calculating the objective function value further having a classification loss for the general-purpose classification module and a segmentation loss for the general-purpose segmentation module.
method.
According to paragraph 1,
The step of calculating the objective function value is,
obtaining a second probability score that a lesion will appear in the training input image using the extracted lesion area; and
Computing the consistency loss based on the difference between the first likelihood score and the second likelihood score,
method.
According to paragraph 3,
The step of obtaining the second probability score is,
Obtaining the second likelihood score by applying at least one of an average pooling layer and a max pooling layer to the extracted lesion area,
method.
According to paragraph 3,
The step of calculating the consistency loss is,
calculating a square of the difference between the first likelihood score and the second likelihood score; and
Comprising the step of calculating the consistency loss by averaging the square of the difference between the first likelihood score and the second likelihood score calculated for each of the images included in the batch,
method.
According to paragraph 1,
Obtaining a reconstruction image from the feature map by applying a reconstruction module of the machine learning model to the feature map.
It further includes,
The step of calculating the objective function value is,
Comprising a step of calculating a reconstruction loss for the reconstruction module based on the training input image and the reconstruction image,
method.
According to clause 6,
The step of calculating the restoration loss is,
Comprising the step of calculating a mean absolute error value of the intensity difference between the pixel of the training input image and the pixel of the reconstructed image corresponding to the pixel of the training input image,
method.
According to paragraph 1,
generating a classification model including the encoder of the machine learning model and a classification module at a patient level connected to the encoder;
Obtaining a plurality of feature maps by applying the encoder of the classification model to a plurality of images obtained based on CT scan of a target patient in a training data set;
obtaining a probability score of a lesion appearing in at least one image among the plurality of images by applying the patient-level classification module to the obtained plurality of feature maps; and
Updating parameters of the classification model based on a likelihood score for the plurality of images and an objective function value calculated using a truth class mapped to the plurality of images in the training data set.
How to include .
According to clause 8,
The step of acquiring the plurality of feature maps includes:
comprising repeating obtaining a feature map by applying the encoder of the classification model to each of the plurality of images,
method.
According to clause 8,
The step of updating the parameters of the classification model is,
updating parameters of the patient-level classification module while keeping parameters of the encoder of the classification model fixed in at least some of the training iterations; and
Unfreezing the parameters of the encoder of the classification model and updating the parameters of the encoder of the classification model and the patient-level classification module in another of the training iterations.
method.
According to paragraph 1,
Generating a segmentation model including the encoder of the machine learning model, a decoder connected to the encoder, and a segmentation module at a patient level;
Obtaining a plurality of feature maps by applying the encoder and the decoder to a plurality of images obtained based on CT scan of a target patient of a training data set;
extracting a lesion area from the plurality of images by applying the patient-level segmentation module to the acquired plurality of feature maps; and
Updating parameters of the segmentation model based on an objective function value calculated using the extracted lesion area and the true lesion area mapped to the plurality of images in the training data set.
How to include more.
According to clause 11,
The step of acquiring the plurality of feature maps includes:
Comprising repeating obtaining a feature map by applying the encoder and the decoder of the segmentation model to each of the plurality of images,
method.
According to clause 11,
The step of updating the parameters of the segmentation model is,
updating parameters of the patient-level segmentation module while keeping parameters of the encoder of the segmentation model fixed in at least some of the training iterations; and
Unfreezing the parameters of the encoder of the segmentation model and updating the parameters of the encoder of the segmentation model and the segmentation module at the patient level in another of the training iterations.
method.
A computer program combined with hardware and stored in a computer-readable recording medium to execute the method of any one of claims 1 to 13.
In electronic devices,
A feature map is extracted by applying the encoder of the machine learning model to the training input image, and a lesion is detected in the training input image by applying the general classification module of the machine learning model to the feature map. Obtain a first likelihood score to appear, extract a lesion area from the training input image by applying a universal segmentation module of the machine learning model to the feature map, and obtain the first likelihood score and the extracted Based on the lesion area, calculate an objective function value with a consistency loss that represents the consistency between the outputs of the general classification module and the general segmentation module, and use the calculated objective function value. Processor for updating parameters of the machine learning model
Electronic devices containing.
According to clause 15,
The processor,
Generating a classification model including the encoder of the machine learning model and a classification module at a patient level connected to the encoder,
Obtaining a plurality of feature maps by applying the encoder of the classification model to a plurality of images obtained based on CT scan of the target patient of the training data set,
Obtaining a probability score that a lesion appears in at least one of the plurality of images by applying the patient-level classification module to the obtained plurality of feature maps,
updating parameters of the classification model based on a likelihood score for the plurality of images and an objective function value calculated using a truth class mapped to the plurality of images in the training data set,
Electronic devices.
According to clause 15,
The processor,
Generate a segmentation model including the encoder of the machine learning model, a decoder connected to the encoder, and a segmentation module at the patient level,
Obtaining a plurality of feature maps by applying the encoder and the decoder to a plurality of images obtained based on CT scan of the target patient of the training data set,
Extracting a lesion area from the plurality of images by applying the patient-level segmentation module to the acquired plurality of feature maps,
Updating parameters of the segmentation model based on an objective function value calculated using the extracted lesion area and a true lesion area mapped to the plurality of images in the training data set,
Electronic devices.

Images