WO2020138932A1 - Machine learning-based method and system for classifying thrombi using gre image - Google Patents

Abstract

The present invention relates to a machine learning-based method and system for classifying thrombi using a gradient echo (GRE) image, the method comprising the steps in which: an image acquisition unit acquires a GRE image; a lesion detection unit detects a lesion region from the acquired GRE image by using an artificial neural network model; a patch region configuration unit configures the detected lesion region into a patch region having a predetermined size, and reconfigures the patch region through a three-dimensional directional projection; and a thrombi classification unit classifies thrombi in the patch region by using the artificial neural network model.

Description

Thrombus classification method and system using machine learning based GRE image

The present invention relates to a thrombus classification method and system using machine learning based GRE (Gradient echo) images, in particular, detecting a thrombus region from a GRE image through an artificial neural network model, and automatically classifying and providing the type of thrombus It relates to a method and a system.

Research into analyzing and diagnosing medical images using a computer has been actively conducted. In particular, the technology for diagnosing through medical images has been developed due to the innovative development of artificial intelligence technology based on deep learning.

Deep learning-based medical image analysis starts with classifying images, and detection of objects, segmentation of objects, and registration of different images are important issues in medical image analysis. Convolutional neural networks (CNN), which are specialized in extracting features from images, are used most often because images are used as input.

On the other hand, GRE (Gradient Echo) images are widely used as MRI sequences that can sensitively view blood clots by signaling and measuring the magnetic components of magnetic resonance imaging (MRI). There is a discomfort that the doctor must judge the type of thrombus by directly viewing the image.

Prior art has published Patent No. 10-2018-0021635 (a method and system for analyzing and expressing lesion features using depth direction recursive learning in 3D medical images), but lesions using convolutional and recursive neural networks in 3D medical images It only discloses a method for extracting feature expressions.

The present invention was devised to solve the above-described problems, and a machine learning based GRE image that detects a thrombus region from a GRE (Gradient echo) image through an artificial neural network model and automatically classifies the thrombus type. To provide a thrombus classification method and system utilizing.

The method according to an aspect of the present invention for solving the above technical problem is a method of classifying blood clots using a machine learning-based gradient echo (GRE) image, wherein the image acquisition unit acquires a GRE image, and a lesion detection unit A step of detecting a lesion area in a GRE image obtained using an artificial neural network model, and setting the detected patch area to the patch area of a constant size, and resetting the patch area through 3D projection And a step of classifying the thrombus in the patch region using the artificial neural network model.

A method according to another aspect of the present invention for solving the above technical problem is a method of classifying blood clots using a machine learning-based gradient echo (GRE) image, comprising: (a) an image acquisition unit obtaining a GRE image; (b) detecting a lesion region in the GRE image acquired by the lesion detection unit using an artificial neural network model; (c) setting the lesion area in which the patch area setting unit is detected as a patch area of a predetermined size, and resetting the patch area through projection in a 3D direction; (d) classifying the thrombus in the lesion region including the patch region using the artificial neural network model; And (e) generating an image including projection information of any one of RED-CLOT and WHITE-CLOT based on the result of the classification by the image generation unit, wherein in the step (c), the patch area setting unit comprises Comparing the shape of the lesion feature expression that appears in the patch region of the predetermined size reset through dimensional projection, and in step (d), the thrombus classification unit is RED in the lesion region according to the comparison result of the patch region setting unit. -CLOT and WHITE-CLOT.

In one embodiment, the thrombus classification unit may classify RED-CLOT and WHITE-CLOT according to an artificial neural network model previously learned through cognition using a YOLO neural network in the patch region. The YOLO neural network is a kind of object detection algorithm, and after training algorithms that detect each of RED-CLOT and WHITE-CLOT, it is possible to generate a target vector of the training set according to the final output grid cell.

A system according to another aspect of the present invention for solving the above technical problem is a system for classifying blood clots using a machine learning based gradient echo (GRE) image, an image acquisition unit for acquiring a GRE image; A lesion detection unit for detecting a lesion region in a GRE image obtained using an artificial neural network model; A patch area setting unit that sets the detected lesion area as a patch area of a constant size and resets the patch area through projection in 3D direction; And a thrombus classification unit for classifying thrombus in a lesion region including a patch region using an artificial neural network model.

In one embodiment, the patch region setting unit may compare the shape of the lesion feature expression appearing in the patch region of the predetermined size reset through projection in 3D direction. In addition, the thrombus classification unit may classify any one of RED-CLOT and WHITE-CLOT in the lesion region according to the comparison result of the patch region setting unit.

In one embodiment, the thrombus classification unit may classify RED-CLOT and WHITE-CLOT according to an artificial neural network model previously learned through cognition using a YOLO neural network in the patch region. Here, the YOLO neural network is a kind of an object detection algorithm, and after training an algorithm for detecting each of the RED-CLOT and WHITE-CLOT, a target vector of the training set is generated according to the final output grid cell.

In one embodiment, based on the classification result of the thrombus classification unit may further include an image generation unit for generating an image including the projection information of either RED-CLOT or WHITE-CLOT.

In the case of using the method and system for classifying thrombi by utilizing the machine learning-based gradient echo (GRE) image of the present invention described above, lesion area detection and thrombus type are automatically classified and provided to provide convenience to the user. It is possible to increase the accuracy of diagnosis by projecting and analyzing a lesion region reconstructed in three dimensions in various directions.

1 is a block diagram of a thrombus classification system using a machine learning-based GRE image according to an embodiment of the present invention.

2 is a block diagram of a data processing unit of a thrombus classification system according to an embodiment of the present invention.

3 is a flowchart illustrating a thrombus classification method using a machine learning-based GRE image according to an embodiment of the present invention.

4 is an exemplary view of a GRE image showing RED-CLOT and WHITE-CLOT according to an embodiment of the present invention.

5 is an exemplary diagram of a structure for a convolutional neural network employable in the system and method of the present embodiment.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are exemplified only for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention It can be implemented in various forms and is not limited to the embodiments described herein.

Embodiments according to the concept of the present invention can be applied to various changes and can have various forms, so the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosure forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

The terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “include” or “have” are intended to indicate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described herein, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, actions, components, parts or combinations thereof are not excluded in advance.

Terms used in the present specification are as follows.

T2-weighted imaging (T2-weighted imaging): refers to a technique obtained from a specific pulse sequence (magnetic pulse imaging) from magnetic resonance imaging (MRI) or an image obtained by this technique. Provide structural information.

FLAIR (Fluid attenuated inversion recovery): A signal acquisition technique using magnetic imaging devices or images obtained by this technique that makes it easier to detect lesions that are easily missed in T2-enhanced images by weakening the signals of the cerebrospinal fluid with long reversal and echo times Refers to. FLAIR can be referred to as fluid attenuation inversion recovery.

DWI (Diffusion weighted imaging): Diffusion-weighted imaging mainly refers to diffusion-weighted images obtained from magnetic resonance imaging, and provides information on the degree and extent of diffusion of water molecules in cell tissues in a specific direction.

Perfusion weighted imaging (PWI): refers to perfusion-weighted images (simply, perfusion images) obtained from magnetic resonance imaging, and informs the change in concentration over time of the injected contrast agent.

Penumbra: A semi-shaded region in an image caused by an ischemic event or embolism, which indicates that the oxygen transport function is locally reduced to cause hypoxic cell death or to be viable upon proper treatment within a few hours.

ADC (Apparent diffusion coefficient): This is an apparent diffusion coefficient obtained from a magnetic resonance image, and provides information on a diffusion impeding factor in internal tissues of the human body.

Arterial phase (AP): A period of specific perfusion obtained from magnetic resonance imaging, indicating the time when contrast medium injected over time passes through the artery.

Capillary phase (CP): A period of specific perfusion obtained from magnetic resonance imaging, which indicates the time when the contrast medium injected over time passes through the capillary portion.

Venous phase (VP): A period of specific perfusion obtained from magnetic resonance imaging, which indicates when the contrast medium injected over time passes through the vein.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a thrombus classification system using a machine learning-based GRE image according to an embodiment of the present invention.

Referring to FIG. 1, the thrombus classification system using a machine learning-based GRE image according to this embodiment includes a control unit 2, a storage unit 4, an image acquisition unit 6, a display unit 8, and a data processing unit ( 10). A thrombus classification system using a GRE image can automatically detect a lesion area in a GRE (Gradient Echo) image, automatically classify thrombus, and provide an image including projection information according to the classification result.

The control unit 2 implements a method of detecting a lesion area and automatically classifying blood clots by executing a program or a software module stored in the storage unit 4, and can control each component of the system.

The storage unit 4 may store a program or a software module for implementing a method of detecting a lesion area and automatically classifying thrombi. The storage unit 4 may store the GRE image transmitted from an external device.

In addition, the storage unit 4 may store programs or software modules for machine learning, deep learning, or artificial intelligence. Deep learning or artificial intelligence may have an architecture to increase accuracy. For example, deep learning or artificial intelligence architectures include a convolutional neural network (CNN) and a pooling structure, a deconvolution structure for upsampling, and a skip connection structure to improve learning efficiency. And the like.

The image acquisition unit 6 may acquire a GRE image from an external device. The image acquisition unit 6 may be connected to a magnetic resonance images (MRI) device, an MRA device, or a CT device to obtain a 3D image of a patient.

The display unit 8 generates data information stored in the storage unit 4, image information acquired by the image acquisition unit 6, lesion area detection results processed by the data processing unit 10, patch area setting results, thrombus classification results, and generation. It can be made to output the image in a visual, audible or a mixture of them. The display unit 8 may include a display device.

The data processing unit 10 may detect a lesion area in a GRE image using machine learning, classify a thrombus within the patch region by setting a patch region, and generate an image including projection information based on the classification result. have.

2 is a block diagram of a data processing unit of a thrombus classification system according to an embodiment of the present invention. Referring to FIG. 2, the data processing unit 10 includes a lesion area extraction unit 100, a patch area setting unit 200, a blood clot classification unit 300, and an image generation unit 400.

The lesion region extracting unit 100 may detect a lesion region in a GRE image using an artificial neural network model. The lesion region extracting unit 100 may extract the lesion from any one of a 2D convolutional neural network (CNN), a 3D convolutional neural network, and a virtual 3D convolutional neural network from the GRE image. Specifically, the lesion region extracting unit 100 may extract the lesion region through a deep learning structure composed of CNN, pooling, deconvolution, and skip connection. That is, the lesion region is detected through an artificial neural network model in which an artificial neural network is trained through an annotation and CAM method in a GRE image signal.

The patch area setting unit 200 may set the detected lesion area as a patch area of a predetermined size. In addition, the patch area setting unit 200 may reset the patch area through projection in a 3D direction.

The thrombus classification unit 300 may classify a thrombus in a patch region using an artificial neural network model. In this case, the thrombus can be classified as either RED-CLOT or WHITE-CLOT. The thrombus classification unit 300 may perform classification through recognition using the YOLO neural network in the patch region. That is, the thrombus classification unit may classify RED-CLOT and WHITE-CLOT according to an artificial neural network model previously learned through recognition using the YOLO neural network in the patch region.

The YOLO neural network is a kind of object detection algorithm, and after training algorithms that detect each of RED-CLOT and WHITE-CLOT, it is possible to generate a target vector of the training set according to the final output grid cell. The size of the target vector can be made of the product of height, width, number of anchor boxes, and vector. And the result vector may include the existence of an object, the coordinates (x, y) of the center value, the height of the bounding box value, and classes. Here, in the case of classification through the YOLO neural network, if training is good, the probability of existence of RED-CLOT or WHITE-CLOT when the classification object exists will be close to 1, and the center value and the bounding box value corresponding to the grid cell, And you can print out the class probability. At this time, non-maximum suppression may be applied to all grid cells.

Thrombus (CLOT) refers to the process of forming a lump or lump in which blood is tangled in a solid state. It can block blood vessel passages and induce a decrease in blood flow, and can be classified into WHITE-CLOT with a predominant platelet component and RED-CLOT with a predominant red blood cell component. In the case of the WHITE-CLOT, it is possible to perform stent treatment, which is a non-surgical treatment, but it is important to distinguish between WHITE-CLOT and RED-CLOT because the RED-CLOT is not capable of non-surgical treatment.

The image generator 400 may generate a 3D image by visualizing projection information of the patch region extracted from the lesion region extraction unit and the thrombus classification unit. According to an embodiment, an image including projection information of RED-CLOT may be generated in a W shape, but is not limited thereto.

3 is a flowchart illustrating a thrombus classification method using a machine learning-based GRE image according to an embodiment of the present invention.

Referring to FIG. 3, the image acquisition unit acquires a gradient echo (GRE) image (S310 ). The GRE image is an image measured by signaling the magnetization component of a 3D magnetic resonance image (MRI). Thereafter, the lesion area detection unit detects the lesion area in the GRE image using the artificial neural network model (S320). At this time, the artificial neural network model may be at least one of a 2D convolutional neural network (CNN), a 3D convolutional neural network, and a virtual 3D convolutional neural network.

The lesion area in which the patch area setting unit is detected is set as a patch area of a predetermined size (S330). At this time, the patch area can be set to a size already specified by the user. Subsequently, the patch region setting unit resets the patch region through projection in various 3D directions again (S340 ).

The thrombus classification unit uses the artificial neural network model to classify the patch region as either RED-CLOT or WHITE-CLOT (S350). RED-CLOT and WHITE-CLOT can be classified according to a pre-trained artificial neural network model. The thrombus classification unit may perform classification through recognition using the YOLO neural network in the patch region.

Based on the classification result, the image generator generates an image including projection information of one of RED-CLOT or WHITE-CLOT (S360).

4 is an exemplary view of a GRE image showing RED-CLOT and WHITE-CLOT according to an embodiment of the present invention. 5 is an exemplary diagram of a structure for a convolutional neural network employable in the system and method of the present embodiment.

4(a) is a GRE image in which WHITE-CLOT is found, and FIG. 4(b) is a GRE image in which RED-CLOT is found, and an artificial neural network model may be trained using the corresponding image as training data. In other words, the learning module consists of a CNN and a pooling structure for summing lesion information, a deconvolution structure for upsampling, and a skip connection structure for smooth learning. I can learn.

As an example, the deep learning architecture may have a form including a convolutional network, a deconvolutional network, and a shortcut. As shown in FIG. 5, the deep learning architecture stacks a 3x3 size color convolution layer and an activation layer (ReLU) and extracts a 2x2 size filter to extract local features of the medical image (X). By applying (stride) 1 to perform the operation of the convolution block that is connected to the next lower depth level 4 times, and then apply a 2x2 size deconvolution layer and activation layer (ReLU) After connecting to the next higher depth level, the operation of the inverse convolution block that stacks the 3x3 size color convolution layer and the activation layer is repeated 4 times, where each of the convolution networks, including the operation of the convolution block of each level, is performed. It may be made to copy and contatenate the convolution result of the corresponding level of the inverse convolution network of the same level to the image of the convolution block of the level and perform convolution operations in each block.

The convolution block in the convolution network and the deconvolution network may be implemented by a combination of conv-ReLU-conv layers. Further, the output of the deep learning architecture may be obtained through a classifier connected to a convolutional network or a deconvolutional network, but is not limited thereto. The classifier may be used to extract local features from an image using a fully connectivity network (FCN) technique.

Further, the deep learning architecture may be implemented to additionally use an insulation module or a multi filter pathway in a convolution block depending on the implementation. Different filters in the inception module or multi-filter path may include 1x1 filters.

For reference, when the input image in the deep learning architecture has 32 horizontal, 32 vertical, and RGB channels, the size of the input image X corresponding to the medical image corresponding to the target vector may be [32x32x3]. . When applied to the YOLO algorithm, these sizes may correspond to the product of height, width, and number of anchor boxes and classes, respectively, in the order described. Here, the last 3 of [32x32x3] may be, for example, a value obtained by adding a class (eg, 3) to a predetermined value (eg, 0) multiplied by the number of actor boxes (eg, 1), that is, (3). .

In the deep learning architecture's convolutional neural network (CNN), the convolutional (CONV) layer is connected to some areas of the input image, and can be designed to calculate the dot product of the connected areas and their weights. .

Here, the ReLU (rectified linear unit) layer is an activation function applied to each element, such as max(0,x). The ReLU layer does not change the size of the volume. The POOLING layer may output a reduced volume by performing downsampling or subsampling on a dimension represented by (horizontal, vertical).

Then, the fully-connected (FC) layer may calculate class scores and output a volume having a size of [1x1x10], for example. In this case, 10 numbers correspond to class scores for 10 categories. The pre-connection layer is connected to all elements of the previous volume. There, some layers may have parameters, while some layers may not. CONV/FC layers may include weight and bias as an activation function, not just input volume. Meanwhile, the ReLU/POOLING layers are fixed functions, and the parameters of the CONV/FC layer can be learned with a gradient descent so that the class score for each image is the same as the label of the corresponding image.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

Claims (5)

1. In the method of classifying blood clots using a machine learning-based GRE image,

   (a) obtaining a gradient echo (GRE) image by an image acquisition unit;

   (b) detecting a lesion region in the GRE image acquired by the lesion detection unit using an artificial neural network model;

   (c) setting the lesion area in which the patch area setting unit is detected as a patch area of a predetermined size, and resetting the patch area through projection in a 3D direction;

   (d) classifying the thrombus in the lesion region including the patch region using the artificial neural network model; And

   (e) the image generating unit includes generating an image including projection information of either RED-CLOT or WHITE-CLOT based on the classification result,

   In the step (c), the patch region setting unit compares the shape of the lesion feature expression appearing in the patch region of the constant size reset through the projection in the 3D direction,

   In the step (d), the thrombus classification unit uses a machine learning based GRE image to classify any one of RED-CLOT and WHITE-CLOT in the lesion area according to the comparison result of the patch area setting unit.
2. The method according to claim 1,

   The classifying step classifies the RED-CLOT and WHITE-CLOT according to an artificial neural network model previously learned through recognition using the YOLO neural network in the patch region, wherein the YOLO neural network is a type of object detection algorithm, and the RED-CLOT and A thrombus classification method using a machine learning-based GRE image that generates a target vector of a training set according to a final output grid cell after training an algorithm that detects each WHITE-CLOT.
3. In a system for classifying blood clots using machine learning-based GRE (Gradient echo) images,

   An image acquisition unit that acquires a GRE image;

   A lesion detection unit for detecting a lesion region in a GRE image obtained using an artificial neural network model;

   A patch area setting unit that sets the detected lesion area as a patch area of a constant size and resets the patch area through projection in 3D direction; And

   Contains a thrombus classification unit for classifying thrombus in a lesion region including a patch region using an artificial neural network model.

   The patch region setting unit compares the shape of the lesion feature expression that appears in the patch region of the constant size reset through projection in 3D direction,

   The thrombus classification system uses a machine learning based GRE image to classify any one of RED-CLOT and WHITE-CLOT in the lesion region according to the comparison result of the patch region setting unit.
4. The method according to claim 3,

   The thrombus classification unit classifies RED-CLOT and WHITE-CLOT according to an artificial neural network model previously learned through recognition using a YOLO neural network in the patch region, wherein the YOLO neural network is a type of object detection algorithm, RED-CLOT and WHITE -A thrombus classification system using machine learning based GRE images that trains an algorithm that detects each CLOT and then generates a target vector of the training set according to the final output grid cell.
5. The method according to claim 3,

   A thrombus classification system using a machine learning-based GRE image, further comprising an image generation unit generating an image including projection information of either RED-CLOT or WHITE-CLOT based on the classification result of the thrombus classification unit.

Images