KR102597647B1 - Method for outputting classification information based on artificial neural network and apparatus therefor - Google Patents

Abstract

A method for a computing device to output classification information about image data according to various embodiments and a device therefor are disclosed. A step of receiving the image data, and inputting the image data into a Fazekas classification model to output the classification information, wherein the Fazekas classification model includes a first feature vector extracted from the image data. A method and apparatus for outputting the classification information based on a second feature vector concatenated with white matter hyperintensity (WMH) segmentation information are disclosed.

Description

Method for outputting classification information based on artificial neural network and apparatus therefor}

A method of outputting classification information based on an artificial neural network, specifically, a method of simultaneously learning a Fazecas classification model and a segmentation model to output the classification information, and a device therefor.

Vascular dementia can be classified as dementia that occurs due to damage to brain tissue due to cerebrovascular disease. Depending on the mechanism of occurrence, cerebrovascular disease can be divided into ischemic cerebrovascular disease (cerebral infarction or cerebral ischemia), which is caused by blockage of cerebral blood vessels, and hemorrhagic cerebrovascular disease (cerebral hemorrhage), which is caused by rupture of cerebral blood vessels. In general, vascular dementia is often caused by repeated occurrences of cerebrovascular disease, but in exceptional cases, dementia symptoms can occur when cerebrovascular disease occurs only once in a major brain region.

The important thing about vascular dementia is that it is more preventable than other causes of dementia, such as Alzheimer's disease. In other words, the risk factors for cerebrovascular disease are relatively well known, and by correcting or controlling these risk factors, cerebrovascular disease can be primarily reduced and, as a result, the occurrence of vascular dementia can be prevented. Representative risk factors for cerebrovascular disease and vascular dementia include high blood pressure, smoking, myocardial infarction, atrial fibrillation, diabetes, and hypercholesterolemia. In addition, associations with increased red blood cell volume (hematocrit), abnormal hemostasis, peripheral vascular disease, and excessive alcohol consumption have also been suggested.

Such early diagnosis of vascular dementia can be performed through classification of medical images of the brain based on a neural network model, object detection, extraction of object boundaries, and registration of different images. Here, the neural network model is a supervised learning algorithm inspired from the structure of neurons in biology. The basic operating principle of a neural network model is to predict the optimal output value for input values by interconnecting multiple neurons. From a statistical perspective, a neural network model can be viewed as a projective trace regression that takes a non-linear function to a linear combination of input variables. In particular, convolution neural networks (CNN) are most widely used to extract features such as the above-mentioned objects from medical images.
- Prior art literature: Domestic Publication No. 10-2017-0144393

The task to be solved is to secure the classification performance and robustness of the two models by simultaneously learning the segmentation model and the Fazecas classification model based on the classification information of the Fazecas classification model that additionally considers the quantitative characteristics of the image data output from the segmentation model. To provide a method and device that can

The technical problems are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

The characteristic configuration of the present invention for achieving the purpose of the present invention as described above and realizing the characteristic effects of the present invention described later is as follows.

According to one aspect, a method for a computing device to output classification information about image data includes receiving the image data, and inputting the image data into a Fazekas classification model to output the classification information. Including, the Fazecas classification model may output the classification information based on a second feature vector obtained by concatenating white matter hyperintensity (WMH) segmentation information with the first feature vector extracted from the image data.

Alternatively, the WMH segmentation information may be calculated by inputting the image data into a segmentation model.

Alternatively, the Fazecas classification model and the segmentation model may be simultaneously learned based on classification loss and segmentation loss calculated using training data.

Alternatively, the learning data may include learning image data, first label information related to the Fazekas scale, and second label information related to the WMH.

Alternatively, the second feature vector may be characterized in that metadata for the image data is further concatenated.

Alternatively, the segmentation information may include first volume information for Deep WMH (DWMH) and second volume information for Periventricular WMH (PVWMH).

Alternatively, the Fazekas classification model is characterized in that it outputs the classification information based on a plurality of variable sets formed by bootstrap sampling of variables included in the second feature vectors.

Alternatively, the Fazecas classification model is characterized in that it further outputs importance information about the degree to which each of the variables influenced the classification information.

Alternatively, the variables include elements included in the first feature vectors, variables related to the first volume for DWMH (Deep WMH) and the second volume for PVWMH (Periventricular WMH) obtained from the segmentation information. It is characterized by

Alternatively, the classification information may include probability values or prediction values for each of the first indicator, second indicator, and third indicator.

Alternatively, the Fazekas classification model includes a sub-model based on a convolutional neural network (CNN), and the sub-model extracts the first feature vector from the image data.

According to another aspect, a computing device that outputs classification information about image data includes a communication unit and a processor connected to the communication unit, wherein the processor receives the image data and applies the image data to a Fazekas classification model. The classification information is output by inputting, and the Fazecas classification model classifies the classification based on a second feature vector that concatenates WMH (white matter hyperintensity) segmentation information with the first feature vector extracted from the image data. Information can be printed.

Various embodiments secure the classification performance and robustness of the two models by simultaneously learning the segmentation model and the Fazecas classification model based on the classification information of the Fazecas classification model that additionally considers the quantitative characteristics of the image data output from the segmentation model. You can.

The effects that can be obtained in various embodiments are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

The drawings attached to this specification are intended to provide an understanding of the present invention, show various embodiments of the present invention, and together with the description of the specification, explain the principles of the present invention.
Figure 1 is a diagram for explaining the structure of CNN.
Figure 2 is a diagram for explaining MRIs, which are image data acquired using an MRI device.
Figure 3 illustrates a computing device that trains a Fazecas classification model based on an artificial neural network.
Figure 4 is a diagram illustrating a method by which a computing device trains a Fazecas classification model.
FIG. 5 is a diagram illustrating a method by which a computing device trains a segmentation model and a Fazecas classification model.
Figures 6 and 7 are diagrams to explain how a computing device trains a segmentation model and a Fazecas classification model using training data.
Figure 8 is a diagram for explaining a method by which a computing device outputs classification information for input image data.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced to make clear the objectives, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

As used throughout the description and claims of this specification, the term “image” or “image data” refers to multidimensional data consisting of discrete image elements (e.g., pixels in two-dimensional images and voxels in three-dimensional images). refers to

For example, “image” may refer to a two-dimensional image corresponding to a slide of a certain tissue observed using a microscope, but “image” is not limited to this and is a (cone-beam) computerized image. It may be a medical image of a subject collected by computed tomography, magnetic resonance imaging (MRI), ultrasound, or any other medical imaging system known in the art. Images may also be provided in non-medical contexts, such as remote sensing systems, electron microscopy, etc.

Throughout the description and claims herein, 'image' is a term that refers to a visible image (e.g., displayed on a screen) or a digital representation of an image.

For convenience of explanation, in the drawings presented, slide image data is shown as an exemplary image modality. However, those skilled in the art will know that the image formats used in various embodiments of the present invention include X-ray images, MRI, CT, PET (positron emission tomography), PET-CT, SPECT, SPECT-CT, MR-PET, 3D ultrasound images, etc. It will be understood that the format includes but is not limited to the format listed as an example.

Medical images described throughout the detailed description and claims of this specification may comply with the 'DICOM (Digital Imaging and Communications in Medicine)' standard. The DICOM standard is a general term for several standards used for digital image expression and communication in medical devices. The DICOM standard is announced by a joint committee formed by the American College of Radiology (ACR) and the National Electrical Manufacturers Association (NEMA).

In addition, the medical images described throughout the detailed description and claims of this specification may be stored or transmitted through the 'Picture Archiving and Communication System (PACS)', and the medical image storage and transmission system complies with the DICOM standard. It may be a system that stores, processes, and transmits medical images appropriately. Medical images acquired using digital medical imaging equipment such as .

In addition, throughout the detailed description and claims of this specification, 'learning' or 'learning' is a term referring to performing machine learning through procedural computing, and is intended to refer to mental operations such as human educational activities. This is not true, and training is used in the generally accepted sense of machine learning. For example, 'deep learning' refers to machine learning using deep artificial neural networks. A deep neural network is a machine that automatically learns the characteristics of each data by learning a large amount of data in a structure consisting of a multi-layer artificial neural network, and through this, learns in a way that minimizes the error in the objective/loss function, that is, classification accuracy. It is a learning model and can extract and classify various levels of features, from low-level features such as points, lines, and surfaces to complex and meaningful high-level features.

And throughout the description and claims herein, the word 'comprise' and variations thereof are not intended to exclude other technical features, attachments, components or steps. Additionally, 'one' or 'one' is used to mean more than one, and 'another' is limited to at least the second or more.

Other objects, advantages and features of the invention will appear to those skilled in the art, partly from the specification and partly from practice of the invention. The examples and drawings below are provided by way of example and are not intended to limit the invention. Accordingly, the details disclosed in this specification regarding specific structures or functions should not be construed in a limiting sense, but are merely representative examples that provide guidance for those skilled in the art to variously practice the present invention with any detailed structures that are practically suitable. It should be interpreted as basic data.

Moreover, the present invention encompasses all possible combinations of the embodiments presented herein. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

In this specification, unless otherwise indicated or clearly contradictory to the context, items referred to in the singular include plural unless the context otherwise requires. Additionally, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a diagram to explain the structure of CNN, an artificial neural network.

CNN (Convolutional Neural Network) is a type of ANN (Artificial Neural Network) that uses convolution operations. To extract image features, CNN calculates a convolution by traversing the input image (or input data) through a filter, and uses the calculation result of the convolution to generate a feature map or activation map. You can.

The CNN maintains the shape of the input and output data of each layer, effectively recognizes features of adjacent images while maintaining spatial information of the image, extracts and learns image features using multiple filters, and collects and collects features of the extracted images. It has the advantage of being able to optionally include a strengthening pooling layer and using filters as shared parameters, requiring very few learning parameters compared to general artificial neural networks.

Specifically, referring to FIG. 1, the CNN may include a feature extraction area for extracting features from an input image (or input data) and an image classification area for classifying the extracted features. The feature extraction area may include at least one convolution layer that finds features of the image while minimizing the number of shared parameters using a filter. Alternatively, the feature extraction area may further include at least one pooling layer that enhances and collects the features. That is, the pooling layer may be omitted.

The convolution layer is a layer that reflects the activation function after applying a filter to the input image (or input data). One or more filters may be applied to the input image flowing into the convolution layer. The one filter can configure a channel of the feature map. For example, when n filters are applied to the convolutional layer, the feature map or the activation map (or output data) has n channels.

The pooling layer is an optional layer located after the convolution layer. The pooling layer receives the output data of the convolution layer as input and can be used as a layer to reduce the size of the output data or emphasize specific data. Methods for processing the pling layer include max pooling, average pooling, and min pooling. The pooling layer has no parameters to learn, can reduce the size of the matrix, and does not change the number of channels.

The CNN can adjust the shape or size of output data depending on the filter size, stride, whether padding is applied, and the maximum pooling size, and determines the channel through the number of filters. You can.

The FC layer is a layer for recognition and classification operations and is a pre-combined layer used to connect each layer in an existing neural network. The FC layer may perform a one-dimensional flattening operation on the two-dimensional array form of the output data in the feature extraction area. The one-dimensional, flattened output data can be classified through the SoftMAx function.

Figure 2 is a diagram for explaining MRIs, which are image data acquired using an MRI device.

A magnetic resonance imaging (MRI) device expresses the strength of the MR (Magnetic Resonance) signal in contrast to the RF (Radio Frequency) signal generated in a magnetic field of a specific strength, and identifies the tomographic area of the object (or patient). It is a device that acquires images of. For example, if an object is placed in a strong magnetic field and an RF signal that resonates only with specific atomic nuclei (e.g., hydrogen nuclei, etc.) is momentarily applied to the object and then stopped, an MR signal is emitted from the specific atomic nucleus, and the MRI device emits this MR signal. MR images can be acquired by receiving . MR signal refers to an RF signal emitted from an object. Here, the size of the MR signal may be determined by the concentration of a certain atom (eg, hydrogen, etc.) included in the object, relaxation time T1, relaxation time T2, and blood flow.

MRI devices include features that are different from other imaging devices. Unlike imaging devices such as CT, where image acquisition depends on the direction of detecting hardware, MRI devices can acquire 2D images or 3D volume images oriented to an arbitrary point. In addition, unlike CT, Neurological images, intravascular images, musculoskeletal images, and oncologic images that are important to describe can be acquired.

Specifically, referring to FIG. 2, the MRI device can acquire T1-w (T1-weighted) MRI, T2-w (T2-weighted) MRI, FLAIR (Fluid attenuated inversion recovery) MRI, etc.

The T1-w and T2-w MRI are images in which the T1 or T2 effect of a specific tissue is emphasized by adjusting TR (repetition time) and TE (time to echo). Here, TR is the repetition time, which is the time interval for applying RF pulses of the same size. The TE is the delay time or echo time. In the process of MRI, when the applied RF (radio frequency) pulse is blocked, the protons in the tissue release the absorbed energy to the surrounding tissue and move in the direction of the external magnetic field (B0) (Z-axis direction). are rearranged.

Here, T1 is the time for the proton spindles to realign along the longitudinal Z-axis, that is, the time constant of the curve for recovery of magnetization in the Z-axis direction. T1 is the time constant for magnetization recovery and is called the longitudinal relaxation time or spin-lattice relaxation time. Meanwhile, when the RF pulse is blocked, the XY component of magnetization collapses.

Additionally, T2 is the time constant of the XY component decay curve of magnetization and is called transverse relaxation time or spin-spin relaxation time. T1 and T2 are tissue-specific values and have different values for water, solids, fat, and protein. Here, lengthening TR reduces the T1 effect. Conversely, if TR is shortened, the T1 effect (contrast) is increased, that is, T1-w MRI is obtained. Shortening TE reduces the T2 effect, and lengthening TE increases the T2 effect, that is, T2-w MRI is obtained.

FLAIR MRI refers to a signal acquisition technique using a magnetic imaging device or images acquired with this technique that weakens the signal of cerebrospinal fluid with a long inversion time and echo time, making it easier to detect lesions that are easy to miss in T2-w MRI. FLAIR MRI may also be referred to as fluid attenuated inversion recovery MRI.

Images obtained through the MRI can be used to diagnose brain diseases and detect or identify brain lesions. In particular, information about vascular dementia can be obtained through the MRIs.

Figure 3 illustrates a computing device that trains a Fazecas classification model based on an artificial neural network.

Referring to FIG. 3, the computing device 100 includes a communication unit 110 and a processor 120, and can communicate directly or indirectly with an external computing device (not shown) through the communication unit 110. Here, the communication unit 110 may correspond to or include a transceiver capable of transmitting and receiving requests and responses to and from other computing devices.

Specifically, computing device 100 is a device that may include typical computer hardware (e.g., a computer processor, memory, storage, input and output devices, and other components of a conventional image processing device; such as a router, switch, etc. Electronic communications devices (electronic information storage systems, such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., software that enables a computing device to function in a particular way) The desired system performance may be achieved by using a combination of instructions.

Such a communication unit 110 can transmit and receive requests and responses with other image processing devices that are linked to it. As an example, such requests and responses may be made through the same TCP (transmission control protocol) session. It is not limited, and may be transmitted and received as, for example, a UDP (user datagram protocol) datagram. Additionally, in a broad sense, the communication unit 110 may include a pointing device such as a keyboard or mouse, other external input devices, printers, displays, and other external output devices for receiving commands or instructions.

Additionally, the processor 120 of the computing device 100 may include a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a cache memory, and a data bus. It may include hardware configuration such as (data bus). In addition, it may further include an operating system and software configuration of an application that performs a specific purpose. The processor 120 may execute instructions to perform the functions of the neural network described below.

The processor 120 can learn models related to the Fazekas scale based on the artificial neural network. Here, the Fazekas scale is one of the indicators that classifies the grade of brain white matter hyperintensity (WMH) and can be used for prediction and analysis related to the diagnosis of VAD (Vascular Dementia), a vascular dementia. The Fazekas scale may be an indicator that is classified into degrees of 0, 1, 2, and 3 (hereinafter, classification information). The Fazekas scale may mean that the symptoms related to VAD become more severe as the value of the indicator increases.

Specifically, the processor 120 may input learning data to the models and extract features of image data such as MRI included in the learning data. Here, the processor 120 may obtain classification information about the Fazecas scale output by the model based on the extracted features. The processor 120 learns the models by adjusting parameters (or weights) for the models to minimize the loss function associated with the models using the obtained classification information and label information included in the learning data. You can do it. The label information may include classification information related to the Fazekas scale evaluated corresponding to the MRI included in the learning data. The processor 120 may perform prediction and analysis on diseases such as VAD based on the classification information.

Figure 4 is a diagram illustrating a method by which a computing device trains a Fazecas classification model.

Referring to FIG. 4, the Fazecas classification model 200 extracts a first feature vector for image features related to Fazecas from image data, and outputs classification information for the image data using the first feature vector. can do.

Specifically, the computing device may receive training data including image data 10 and label information. The computing device may input image data 10 into a Fazecas classification model 200.

The Fazekas classification model 200 may extract the first feature vector related to the Fazekas scale based on the image data 10. For example, the Fazekas classification model 200 may extract the first feature vector from the image data 10 based on the CNN described with reference to FIG. 1. Meanwhile, the Fazecas classification model presents CNN as an artificial neural network that extracts the first feature vector, but is not limited to this and Deep Neural Network (DNN), Recurrent, can extract the first feature vector from the image data. It may be based on any of various types of neural networks, such as Neural Network (RNN), Long Short-Term Memory models (LSTM), BRDNN (Bidirectional Recurrent Deep Neural Network), and Convolutional Neural Networks (CNN).

Next, the Fazecas classification model 200 may output classification information corresponding to the image data 10 based on the extracted first feature vector. For example, the classification information may include probability values or prediction values for indicator values related to the Fazekas scale. The computing device may train the Fazecas classification model 200 so that the loss function for the Fazecas classification model 200 is minimized based on the classification information and label information.

Meanwhile, the learning data may include label information including the value of the Fazekas scale diagnosed or evaluated by an evaluator (such as a doctor) for the image data 10. The label information may include a value of the Fazekas scale that is pre-evaluated by an evaluator (such as a doctor) by considering Deep WMH (DWMH) and/or Periventricular WMH (PVWMH) for the image data.

Here, the label information for the image data 10 may vary depending on the subjectivity of the evaluator. Specifically, the criteria for dividing the values of the Fazekas scale are not clear, and it is difficult for the evaluator to interpret or predict the exact value of the Fazekas scale due to the correlation (or correlation) between the values of the Fazekas scale. In this respect, the value of the Fazekas scale may vary depending on the evaluator's subjective interpretation. In this case, when a computing device trains the Fazekas classification model using the label information, the accuracy and classification ability of the Fazekas classification model may be reduced due to the subjectivity and unclear evaluation criteria of the label value.

Therefore, a learning method for the Fazekas classification model that minimizes concerns about the influence of the evaluator's subjective interpretation is needed so that it can be meaningfully used in a clinical environment. For example, there is a need to consider a method in which the computing device learns the Fazekas classification model by additionally reflecting separate quantitative features that ensure objectivity in order to resolve subjectivity and uncertainty caused by the label information, etc.

In the following, in order to improve the accuracy of prediction and interpretation of the Fazecas scale during the learning process, a segmentation model that outputs quantitative features, which are features with secured objectivity, is additionally defined, and a multi-segmentation model consisting of the segmentation model and the Fazecas classification model is defined. A method of learning the segmentation model and the Fazecas classification model based on the task will be described in detail.

FIG. 5 is a diagram illustrating a method by which a computing device trains a segmentation model and a Fazecas classification model.

Referring to FIG. 5, a segmentation model 300 that outputs segmentation information related to brain white matter hyperintensity (WMH) can be additionally used to reflect quantitative features additionally in the Fazekas classification model 201. In other words, the Fazecas classification model (201) can be trained to increase classification performance and accuracy by configuring multi-task learning including the Fazecas classification model (201) and the segmentation model (300).

The computing device may receive learning data related to multi-task learning. The learning data includes image data 10, first label information for the Fazekas classification model 201 (e.g., an indicator value of the Fazekas scale predetermined for the image data), and second label information for the segmentation model 300. Label information (e.g., may include a segmented mask (or volume value) for WMH (DWMH, PVWMH) in the image data. For example, the second label information is pre-calculated for the image data 10. It may include a first mask segmented for the DWMH for the DWMH and a second mask segmented for the PVWMH for the PVWMH. For example, the second label information may include a masked area associated with the DWMH in the image data 10. The first mask and image data 10 may include information about a second mask in which the area related to the PVWMH is masked. The image data 10 may include a 3D image (or a FLAIR full image image). You can.

Meanwhile, the indicator value for the Fazekas scale included in the first label information may correspond to a Fazekas scale value evaluated based on the first mask and the second mask included in the second label information. For example, a first Fazekas scale value may be pre-evaluated based on the first mask for the image data 10, and a second Fazekas scale value may be pre-evaluated based on the second mask for the image data 10. , the first label information may include a larger value among the first Fazecas scale value and the second Fazecas scale value. Through this, the subjectivity of the first label information can be alleviated. In addition, the Fazekas scale value can generally be determined as any of 0, 1, 2, and 3, but the distinction between (0, 1) does not have significant clinical significance, so in the case of (0,1), it is 1 without distinction. It can be composed of:

The Fazekas classification model 201 may be inputted with some 2D images (eg, 2D-FLAIR in the upper and lower axial direction) obtained from the image data 10 with respect to a specific axial direction. For example, the Fazekas classification model 201 uses a preset range (e.g., Some 2D images in 7) can be input. The Fazecas classification model 201 can extract a first feature vector related to image features (shape, location, distribution, etc.) from the input 2D images. For example, the first feature vector may include a first logit (logit1), a second logit (logit2), etc. related to image features for 2D images.

The segmentation model 300 may input all 2D images obtained from the image data 10 in a specific axis direction. Alternatively, the segmentation model 300 inputs a plurality of 2D images (2D MRI slices) acquired for each of the three axis directions ((Sagittal axis, Axial axis, and Coronal axis) from the image data 10, or the 3D image. The entirety of can be entered.

The segmentation model 300 may extract a feature (W) for outputting segmentation information related to WMH from the image data 10 and output segmentation information (Z) based on the feature (W). Segmentation information (Z) is first volume information (or, a first volume value, a first probability value associated with the first volume value) for the DWMH (Deep WMH) and/or second volume information for the PVWMH (or , a second volume value, and a second probability value associated with the second volume value). Additionally, the segmentation model 300 may output segmentation loss calculated based on segmentation information (Z) and the second label information. Alternatively, the computing device may calculate segmentation loss based on segmentation information (Z) and the second label information.

The Fazecas classification model 201 may output classification information by additionally considering the segmentation information output by the segmentation model 300. For example, the first feature vector may be concatenated with the segmentation information (or variables of volume values included in the segmentation information). The Fazecas classification model 201 may output classification information for the image data 10 using the second feature vector, which is the first feature vector connected to the segmentation information. As described above, the classification information may include probability values or prediction values for each of 1, 2, and 3, which are values related to the Fazekas scale or Fazekas score.

Alternatively, the Fazecas classification model 201 may output the classification information by additionally considering metadata input in relation to the image data 10. Here, the metadata includes information on age, gender, etc., which are characteristics of the subject (patient, etc.) of the image data 10, and/or characteristics of the device (e.g., MRI device) that acquired the image data (MRI vendor, magnetic field strength) may be included. In this case, the Fazecas classification model 201 may output the classification information using a second feature vector, which is the first feature vector in which variables related to the segmentation information and the metadata are concatenated.

In other words, the Fazekas classification model 201 additionally considers segmentation information and/or metadata, which are quantitative features that ensure objectivity, as well as the first feature vector containing image features (shape, location, distribution) for 2D images. You can output classification information.

The computing device may calculate a classification loss related to the Fazecas classification model 201 using the classification information and the first label information. For example, the computing device may calculate the classification loss by reflecting the probability value included in the classification information and the label value included in the first label information in the loss function of the Fazekas classification model 201.

The computing device may learn the Fazecas classification model 201 and the segmentation model 300 based on the segmentation loss output by the segmentation model 300 and the classification loss. For example, the computing device adjusts the parameters or weights of the Fazecas classification model 201 and the segmentation model 300 so that the loss value of the overall loss function including each of the classification loss and the segmentation loss as terms is minimized. The Fazecas classification model (201) and the segmentation model (300) can be trained.

Figures 6 and 7 are diagrams to explain how a computing device trains a Fazecas classification model using a segmentation model and training data.

Referring to FIG. 6, the computing device can train the segmentation model 300 and the Fazecas classification model 201 using the input training data.

Here, the Fazecas classification model 201 is a feature extraction unit 210 that extracts the first feature vector (X) based on the image data 10 included in the learning data, and performs quantitative analysis on the first feature vector (X). It may include a feature connection unit 220 that connects features to generate a second feature vector, and a classification unit 230 that outputs classification information based on the second feature vector. Meanwhile, the feature extraction unit 210 and the classification unit 230 may extract the first feature vector (X) based on an artificial neural network and output the classification information.

As described with reference to FIG. 5, the Fazekas classification model 201 may input some 2D images (e.g., 2D-FLAIR in the upper and lower axial direction) obtained for a specific axial direction from the image data 10. You can. In this case, the feature extractor 200 may extract a first feature vector (X) related to image features (shape, location, distribution, etc.) from the input 2D images. For example, the first feature vector (X) may include a first logit (logit1), a second logit (logit2), etc. related to image features for 2D images. Alternatively, the Fazecas classification model 201 may additionally receive metadata (M) related to the image data (10).

As described with reference to FIG. 5 , the segmentation model 300 may input all 2D images or the entire image data 10 obtained from the image data 10 in a specific axis direction. The segmentation model 300 may extract features for outputting segmentation information related to WMH from the image data 10 and output segmentation information (Z) based on the features. Segmentation information (Z) is first volume information (or, a first volume value, a first probability value associated with the first volume value) for the DWMH (Deep WMH) and/or second volume information for the PVWMH (or , a second volume value, and a second probability value associated with the second volume value).

Additionally, the segmentation model 300 may calculate segmentation loss based on segmentation information (Z) and the second label information. Here, segmentation loss can be calculated based on focal loss, which is a method of adding weight to the loss function according to classification difficulty (or classification error).

For example, the segmentation model 300 assigns a large weight to the loss function when the classification error or classification difficulty is high, and when the classification error or classification difficulty is low (i.e., the degree to which the segmentation information is close to the second label information) For this reason, a small weight may be given to the loss function. For example, the weight may be determined based on the second label information and the segmentation information. The segmentation model 300 can output the segmentation loss for segmentation information (Z) using the loss function defined in Equation 1 below.

Here, α is the weight described above, p is the probability value predicted by the segmentation model (or the probability value included in the segmentation information) and has a value between 0 and 1, and r is a focusing parameter and a value between 0 and 5. You can have

The feature connection unit 220 may receive the first feature vector (X) and segmentation information (Z). The feature connection unit 220 may concatenate variables related to segmentation information (Z) to the first feature vector (X). Here, the variable related to segmentation information (Z) may be a first volume value for the DWMH and/or a second volume value for the PVWMH calculated based on the segmentation information. In this case, the feature connection unit 220 may generate a second feature vector by connecting the first volume value and/or the second volume value to the first feature vector (X). For example, the first feature vector may include variables for a plurality of logits, such as a first logit (logit 1) and a second logit (logit 2), and the feature connection unit 220 may include the first volume value and the The variable for the second volume value may be included in the first feature vector to generate the second feature vector including a plurality of logits, the first volume value, and the variable for the second volume value. Alternatively, when there are a plurality of first feature vectors (X), the feature connection unit 220 may concatenate variables related to segmentation information (Z) to each of the first feature vectors (X).

Alternatively, the feature connection unit 220 may additionally receive metadata (M) related to the image data. Metadata (M) includes information on age, gender, etc., which are characteristics of the subject (patient, etc.) of the image data, and/or characteristics of the device (e.g., MRI device) that acquired the image data (MRI vendor, magnetic field strength) may include. In this case, the feature connection unit 220 may generate the second feature vector by connecting the first volume value, the second volume value, age, gender, etc. to the first feature vector (X). For example, the feature connector 220 may generate the second feature vector using variables such as a plurality of logits, a first volume value, a second volume value, age, and gender. Alternatively, when there are a plurality of first feature vectors ( It can be concatenated.

Meanwhile, the feature connection part 220 may not be included in the Fazecas classification model 201. In this case, the Fazecas classification model 201 outputs a first feature vector (X) and receives a second feature vector, which is the first feature vector ( Classification information can be output. That is, the feature connection unit 220 may be included in the Fazecas classification model 201 or may be a separate unit related to the operation of the computing device.

The classification unit 230 may receive the second feature vectors. The classification unit 230 may output classification information corresponding to the image data based on the input second feature vector. Here, the classification information may include a probability value or prediction value related to the Fazekas scale. For example, the classification information may include probability values or prediction values for each of 1, 2, and 3, which are values related to the Fazekas scale or Fazekas score.

The classification unit 230 may output classification information based on a random forest model. For example, the classification unit 230 generates a plurality of variable sets by bootstrap sampling the elements (or a plurality of variables) of the input second feature vector and based on the plurality of variable sets. This allows decision trees to be formed. The classification unit 230 may output classification information related to the input second feature vector based on voting results of the formed decision trees. In addition, the classification unit 230 may also output importance information about the degree to which each variable included in the second feature vector influenced the classification information. Meanwhile, the operation of the random forest model described above is only an example, and the classification unit 230 may output classification information based on various types or forms of random forest models.

Specifically, referring to FIG. 7, the classification unit 230 determines the degree to which each variable included in the second feature vectors affects the classification information (importance) based on the results of voting of the decision trees, etc. ) can be output. For example, the second feature vector includes variables included in the first feature vector (logit1, logit2), a first volume value (i.e., DWMH), a second volume value (i.e., PVWMH), age, gender, etc. In this case, the classification unit 230 outputs importance information expressing importance in the order of second volume value (PVWMH), age, first volume value (DWMH), first logit (logit1), and second logit (logit2). can do.

Additionally, the classification unit 230 or the computing device may calculate the classification loss for the classification information based on Equation 2 below.

Where K is the number of data sets, y is the ground truth value (or label value), may be a probability value included in the classification information.

The computing device may learn the Pazekas classification model 201 and the segmentation model 300 based on a total loss function that includes each of the classification loss and the segmentation loss as terms. Alternatively, the computing device may learn the feature extraction unit 210 and the segmentation model 300 in the Fazecas classification model 201. Here, the overall loss function can be defined as Equation 3 below.

L _seg is the segmentation loss described above, L _class is the classification loss described above, and λ may be a weight related to the two losses.

The computing device adjusts the parameters (or weights) for each of the Fazecas classification model 201 and the segmentation model 300 so that the loss value due to the overall loss function can be minimized to create the Fazecas classification model 201. and segmentation model 300 can be trained.

In this way, the computing device may provide a classification loss and The Fazekas classification model (201) and the segmentation model (300) can be trained simultaneously by simultaneously considering the segmentation loss. In this case, the Fazekas classification model 201 can be learned to output solid classification information by additionally considering quantitative features and greatly reducing the subjectivity caused by existing label information during the learning process. Additionally, with the mitigation of subjectivity and additional consideration of quantitative features that are objective criteria, the Fazekas classification model (201) and segmentation model (300) can be learned to significantly improve decomposition performance and accuracy. Furthermore, the computing device can simultaneously learn the Fazecas classification model 201 and the segmentation model 300 to output more robust and accurate classification information by additionally including characteristics related to metadata (M) in the second feature vector. You can.

Figure 8 is a diagram for explaining a method by which a computing device outputs classification information for input image data.

Referring to FIG. 8, the computing device can receive image data 11 for classifying indicator values related to Fazecas. As described above, the image data 11 may include a 3D image (or a FLAIR full image image). In this case, the Fazekas classification model 202 uses 2D images acquired for a specific axis direction as described with reference to FIG. 5 (e.g., the middle 2D image among a plurality of 2D images that can be acquired for a specific axis direction). Based on , some 2D images) can be input, and the entire 3D image can be input to the segmentation model 302.

The Fazekas classification model 202 can extract a first feature vector (X) from 2D images. For example, the Fazekas classification model 202 can extract the first feature vector (X), which is an image feature, from the 2D images based on CNN. The first feature vector (X) may include (or include as an element of the vector) information such as a first logit (logit 1) and a second logit (logit 2) related to the image feature.

The segmentation model 302 may generate segmentation information (Z) for calculating PWMH and/or DWMH based on the input 3D image. The segmentation information (Z) may include a probability value (or prediction value) related to calculation of PWMH and/or DWMH values.

The Fazecas classification model 202 is a second feature vector, which is the first feature vector (X), to which the volume values (Z) calculated based on the segmentation information and the input metadata (M) (age, gender, etc.) are connected. Based on this, classification information for the image data can be output. For example, the second feature vector is the variables included in the first feature vector (X) (first logit, second logit, etc.), the first volume value (DWMH value), and the second volume value (PWMH value) Variables for age and gender may be included.

Specifically, the Fazekas classification model 202 may perform bootstrap sampling of variables included in the second feature vectors based on the random forest model, as described with reference to FIG. 6. The Fazekas classification model 202 can output classification information about the Fazekas score or Fazekas scale using a plurality of variable sets generated by bootstrap sampling. Alternatively, the Fazekas classification model 202 may additionally output importance information for each of the variables related to the second feature vectors, as described with reference to FIGS. 6 and 7 .

In this way, the Fazekas classification model (202) considers both image features (shape, location, distribution) for image data and segmentation information (PVWMH volume value, DWMH volume value), which are quantitative features, using the segmentation model (302). You can output classification information. In this case, classification performance and accuracy can be significantly improved compared to the case where the Fazecas classification model 202 outputs classification information related to the Fazecas scale by relying on existing image features (shape, location, distribution). Furthermore, the Fazekas classification model 202 can greatly increase the interpretability of the image data 11 by additionally outputting importance information for each variable included in the second feature vector based on the random forest model.

Based on the description of the above embodiments, those skilled in the art will understand that the method and/or processes of the present invention, and the steps thereof, can be realized with hardware, software, or any combination of hardware and software suitable for a specific application. The point can be clearly understood. The hardware may include general-purpose computers and/or dedicated computing devices or specific computing devices or special features or components of specific computing devices. The processes may be realized by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, or other programmable devices, with internal and/or external memory. Additionally, or alternatively, the processes may be configured as an application specific integrated circuit (ASIC), programmable gate array, programmable array logic (PAL), or to process electronic signals. It can be implemented with any other device or combination of devices. Furthermore, the subject matter of the technical solution of the present invention or the parts contributing to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-viewable recording medium. The machine-viewable recording medium may include program instructions, data files, data structures, etc., singly or in combination. The program instructions recorded on the machine-viewable recording medium may be specially designed and configured for the present invention or may be known and usable by those skilled in the art of computer software. Examples of machine-viewable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and Blu-rays, and magneto-optical media such as floptical disks. (magneto-optical media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include storing and compiling or interpreting them for execution on any one of the foregoing devices, as well as a heterogeneous combination of processors, processor architectures, or combinations of different hardware and software, or any other machine capable of executing the program instructions. machine code, which may be created using a structured programming language such as C, an object-oriented programming language such as C++, or a high- or low-level programming language (assembly language, hardware description languages, and database programming languages and techniques); This includes not only bytecode but also high-level language code that can be executed by a computer using an interpreter.

Accordingly, in one aspect according to the present specification, when the above-described methods and combinations thereof are performed by one or more computing devices, the methods and combinations of methods may be implemented as executable code that performs the respective steps. In another aspect, the method may be practiced as systems that perform the steps, the methods may be distributed in several ways across devices, or all functions may be integrated into one dedicated, standalone device or other hardware. In another aspect, the means for performing steps associated with the processes described above may include any of the hardware and/or software described above. All such sequential combinations and combinations are intended to be within the scope of this specification.

For example, the hardware device may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa. The hardware device may include a processor such as an MPU, CPU, GPU, or TPU coupled with a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, and external devices and signals. It may include a communication unit capable of sending and receiving. In addition, the hardware device may include a keyboard, mouse, and other external input devices to receive commands written by developers.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person skilled in the art to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

Such equivalent or equivalent modifications will include, for example, logically equivalent methods that can produce the same results as performing the method according to the present specification, and the spirit and scope of the present invention should not be limited by the foregoing examples, but should be understood in the broadest sense permissible by law.

Claims (12)

In a method for a computing device to output classification information for image data,
receiving the image data; and
Including the step of inputting the image data into a Fazekas classification model and outputting the classification information,
The Fazecas classification model outputs the classification information based on a second feature vector that concatenates the PVWMH (Periventricular white matter hyperintensity) volume and the DWMH (Deep WMH) volume with the first feature vector extracted from the image data. do,
The Fazecas classification model further outputs importance information about the degree to which each of the first feature vector, the PVWMH volume, and the DWMH volume contributed to the classification information. According to paragraph 1,
The method, characterized in that the PVWMH volume and the DWMH volume are calculated by inputting the image data into a segmentation model. According to paragraph 2,
The method is characterized in that the Fazecas classification model and the segmentation model are simultaneously learned based on classification loss and segmentation loss calculated using training data. According to paragraph 3,
The method, wherein the training data includes training image data, first label information related to the Fazekas scale, and second label information related to the WMH. According to paragraph 1,
The method is characterized in that the second feature vector is further concatenated with metadata for the image data. delete According to paragraph 1,
The Fazecas classification model is characterized in that it outputs the classification information based on a plurality of variable sets formed by bootstrap sampling of variables included in the second feature vectors. delete In clause 7,
The method is characterized in that the variables include elements included in the first feature vectors, variables related to the DWMH volume and the PVWMH volume. According to paragraph 1,
The method is characterized in that the classification information includes probability values or prediction values for each of the first indicator, second indicator and third indicator. According to paragraph 1,
The Fazekas classification model includes a sub-model based on a convolutional neural network (CNN), and the sub-model extracts the first feature vector from the image data. In a computing device that outputs classification information about image data,
Ministry of Communications; and
Includes a processor connected to the communication unit,
The processor receives the image data, inputs the image data into a Fazekas classification model, and outputs the classification information,
The Fazecas classification model outputs the classification information based on a second feature vector that concatenates the PVWMH (Periventricular white matter hyperintensity) volume and the DWMH (Deep WMH) volume with the first feature vector extracted from the image data. do,
The Fazecas classification model further outputs importance information about the degree to which each of the first feature vector, the PVWMH volume, and the DWMH volume contributed to the classification information.

Images