KR20230127762A - Device and method for detecting lesions of disease related to body component that conveyes fluid from medical image - Google Patents

Abstract

Embodiments extract spatial features and connectivity between segments from a medical image of a detection target using a pre-learned disease detection model, and calculate features for predicting body components based on the connectivity between the spatial features and segments, It relates to an apparatus and method for detecting a lesion of a disease related to a body component that delivers blood from a medical image.

Description

DEVICE AND METHOD FOR DETECTING LESIONS OF DISEASE RELATED TO BODY COMPONENT THAT CONVEYES FLUID FROM MEDICAL IMAGE }

Embodiments of the present application relate to an apparatus and method for detecting disease lesions related to body components that deliver blood, such as blood vessels, from a medical image based on inter-slice connectivity between 2D object-slice images.

Time-of-flight Magnetic resonance angiography (TOF-MRA), a non-invasive diagnostic method that does not use a contrast agent, is widely used to diagnose intracranial artery occlusion and stenosis, which is known to be the biggest cause of ischemic cerebral infarction in Asians. . However, since intracranial arteries are numerous and have complex shapes, it is difficult to accurately detect all of them when there are numerous obstructive or stenotic lesions.

In addition, with the recent expansion of health insurance benefits, the number of magnetic resonance imaging examinations has increased rapidly, and radiologists are suffering from work overload.

Recently, with the development of deep learning algorithms and hardware, the performance of machine learning has rapidly improved, and attempts to apply it to medical images are actively being made. In particular, various algorithms for extracting blood vessel regions from medical images are being developed.

However, most of these algorithms basically have high detection performance for a single image, but low detection performance for a sequence composed of a series of images or a 3D image. Therefore, when applied to a blood vessel having a complex and elongated structure compared to other body-based structures, there is a limitation in that the blood vessel area and the vessel-related lesion area cannot be accurately detected.

In particular, it is difficult to accurately detect signs of diseases (eg, stenotic lesions) associated with blood vessels having complex shapes and small apertures, such as intracranial arteries. Moreover, it is more difficult to accurately locate the symptoms of these diseases.

Patent Registration Publication No. 10-2250688 (2021.05.12.)

Conference paper “U-Net: Convolutional Networks for Biomedical Image Segmentation, Olaf Ronneberger, Philipp Fischer, and Thomas Brox” (2015.05.18.)

According to embodiments of the present application, an apparatus and method for detecting disease lesions related to body components that deliver blood, such as blood vessels, from a medical image based on inter-slice connectivity between 2D object-slice images.

In addition, it is intended to provide a computer readable recording medium recording a program including instructions for performing the method.

A method of detecting a disease lesion related to a body component that delivers blood from a medical image according to an aspect of the present application is performed by a computing device including a processor. The method includes extracting spatial features and connectivity between segments from a medical image of a detection target using a pre-learned disease detection model, and calculating features for predicting body components based on the connectivity between the spatial features and segments. may also include The medical image is a plurality of 2-dimensional slice images constituting the body components of a 3-dimensional volume, and the plurality of 2-dimensional slice images are sequentially input to the disease detection model according to the order of the slice images, and the spatial features and slice images are sequentially input to the disease detection model. Features for predicting connectivity and body components are obtained for each 2D slice image.

In one embodiment, the step of extracting spatial features and connectivity between slices from the medical image of the detection target and calculating features for predicting body components based on the spatial features and connectivity between slices includes a plurality of 2D slices. sequentially extracting spatial features for each image; extracting connectivity between slices of adjacent 2D slice images based on spatial features of the adjacent 2D slice images; and calculating a feature for predicting a body component based on the connectivity between the spatial feature and the slice.

In an embodiment, a correlation between at least one 2D slice image having a stored spatial feature among spatial features of at least some previously extracted 2D slice images and a 2D slice image from which a current spatial feature is extracted is calculated. By doing so, connectivity between the segments may be extracted.

In an embodiment, the spatial features may be extracted by extracting spatial features from the 2D slice image for each hierarchical level and performing a pooling operation on the extracted spatial features. Connectivity between the segments is extracted based on the result of the pooling operation at the deepest hierarchical level.

In one embodiment, the step of calculating a feature for predicting a body component based on the connectivity between the spatial feature and the intercept may include performing deconvolution on an operation result of a previous hierarchical level for each hierarchical level, and performing deconvolution on the result of the deconvolution operation. restoring an input 2D slice image by combining the result and spatial features of the same hierarchical level, and reducing a channel of the combined result; and extracting features for predicting body components from the reconstructed input image.

In one embodiment, the reconstructed input image corresponds to the results of the deconvolution operation, the combination operation, and the convolution operation at the highest hierarchical level, and the features for predicting body components extracted from the reconstructed image are individual segments. It may be an image feature reflected in the spatial feature of the image and the connectivity between the slices.

In one embodiment, the method includes the steps of segmenting a body component region in an input medical image based on the extracted feature for predicting the body component, and predicting the spatial feature, the connectivity between segments, and the body component. One or more of the steps of detecting the lesion area of the disease based on one or more of the characteristics for the disease may be further included.

In an embodiment, the dividing of the body component region may include: calculating a body component probability value in an input image based on a feature for predicting the body component region; generating a segmentation mask based on the calculated body component probability values; and dividing the body component region using the segmentation mask.

In an embodiment, the detecting of the lesion area of the disease may include calculating a global feature through a convolution operation and a global pooling operation based on connectivity between segments; calculating a global score based on the global feature; calculating a regional feature through a convolution operation and a regional pooling operation based on the feature for predicting the body component; calculating a regional score based on the regional characteristics; and determining a lesion area by calculating a lesion score based on the regional score and the global score. Here, the global feature is a feature for detecting a lesion area in the input image in a global aspect, and the regional feature is a feature for detecting a lesion area in the input image in a regional aspect. The global score quantifies the possibility of detecting a lesion area in an input image in a global aspect. The regional score quantifies the possibility of detecting a lesion area in an input image in a regional aspect.

In one embodiment, the step of calculating regional features through a convolution operation and a regional pooling operation based on the features for predicting the body components includes the image in which the features for predicting the body components are calculated is converted into a plurality of regions. partitioning into; and extracting regional features through a convolution operation and a regional pooling operation of features for predicting body components for each region.

In one embodiment, the global score is calculated from global features using a probability function. The regional score is calculated from regional features using a probability function.

In an embodiment, the determining of a lesion area by calculating a lesion score based on the regional score and the global score may include calculating a lesion score by calculating the local score and the global score through an amplification operator; and determining a lesion area based on the lesion score.

In one embodiment, the lesion score may be calculated by multiplying a regional score and a global score through a multiplication operator.

In one embodiment, the method includes: calculating positions of at least one of a position of a segmented body component region and a position of a detected lesion area in a corresponding 2D slice image, and calculating the position of the segmented body component region and the detected lesion area. One or more steps of generating a 3D model of one or more of the regions may be further included.

A computer readable recording medium according to another aspect of the present application may have a program recorded thereon to perform a method of detecting a lesion of a disease related to a body component that delivers blood from a medical image according to the above-described embodiments. .

According to another aspect of the present application, an apparatus for detecting a lesion of a disease related to a body component that delivers blood from a medical image includes a medical image including a plurality of 2-dimensional slice images constituting a body component of a 3-dimensional volume. a receiving image acquisition unit; and an analyzer configured to extract spatial features and connectivity between segments from a medical image of a detection target using a pre-learned disease detection model, and to calculate features for predicting body components based on the spatial features and connectivity between segments. may also include The disease detection model includes a division unit forming a backbone data processing path having a depth structure composed of a plurality of hierarchical levels; a first detector forming a first auxiliary data processing path branched from the subnetwork of the backbone data processing path; and a second detection unit forming a second auxiliary data processing path branched from the division feature block of the backbone data processing path.

In one embodiment, the division unit may include a plurality of encoding blocks and a plurality of decoding blocks located for each hierarchical level; a sub-network for extracting connectivity between segments by connecting encoding blocks and decoding blocks of a previous hierarchical level at the deepest hierarchical level; a division feature block that calculates a feature for predicting the body component from an operation result of a decoding block of the highest hierarchical level; and a segmentation block dividing a body component region in an input image based on a feature for predicting the body component. The encoding block for each hierarchical level propagates an operation result to an encoding block of the next hierarchical level through a first connection path and to a decoding block of the same hierarchical level through a second connection path, and the decoding block for each hierarchical level An input image is gradually restored by receiving an operation result of a decoding block or sub-network of a previous hierarchical level through a connection path and an operation result of an encoding block of the same hierarchical level through the second connection path.

In an embodiment, the first detection unit may include a first extraction block calculating a global feature through a convolution operation and a global pooling operation based on connectivity between segments; and a first score block that calculates a global score based on the global feature.

In one embodiment, the second detection unit may include a second extraction block calculating a regional feature through a convolution operation and a regional pooling operation based on the feature for predicting the body component; and a second extraction block for calculating a regional score based on the regional feature.

In one embodiment, the disease detection model may be configured to calculate a lesion score by calculating the regional score and the global score through a multiplication operator, and to detect a lesion area based on the calculated lesion score.

In one embodiment, the disease detection model is trained to minimize the difference between the predicted value and the actual value of the dividing unit, the difference between the predicted value and the actual value of the first detection unit, and the difference between the predicted value and the actual value of the second detection unit. .

According to the embodiments of the present application, deep learning technology can be used to extract blood vessels and lesions and quantitatively present the degree of stenosis, which is a great help for radiologists to more accurately and efficiently read TOF-MRA examinations. can give

In particular, it is possible to accurately extract blood vessels in which the central line is difficult to see due to occlusion or lesions with a very narrow width can be accurately extracted, which is important in the diagnosis of ischemic stroke patients and is essential for determining the treatment strategy. can be measured

BRIEF DESCRIPTION OF THE DRAWINGS To describe the technical solutions of the embodiments of the present invention or the prior art more clearly, drawings required in the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for the purpose of explaining the embodiments of the present specification and not for the purpose of limitation. In addition, for clarity of explanation, some elements applied with various modifications, such as exaggeration and omission, may be shown in the drawings below.
1 is a schematic block diagram of an apparatus for detecting a lesion of a disease related to a body component that delivers blood, such as a blood vessel, from a medical image according to an aspect of the present application.
2 illustrates a medical image input to a disease detection model according to an embodiment of the present application.
3 shows a schematic network architecture of a vascular-related disease detection model, according to an embodiment of the present application.
4 is a diagram illustrating a learning process of a disease detection model according to an embodiment of the present application.
5A and 5B are diagrams illustrating prediction results of a blood vessel area and a lesion area according to an embodiment of the present application.
6 is a flowchart of a method of detecting a disease lesion related to a body component that delivers blood from a medical image according to another aspect of the present application.
7 illustrates a result of displaying the location of a divided blood vessel region and the location of a detected lesion region on a 2D slice image according to an embodiment of the present application.
8 illustrates a 3D model in which positions of divided blood vessel regions and positions of detected lesion regions are rendered according to an embodiment of the present application.
9A to 9C show model performance according to localization size according to an experimental example of the present application.
FIG. 10 illustrates segmentation performance and detection performance of the model of FIG. 3 and other modified models according to an experimental example of the present application.

Hereinafter, the embodiments of the present application will be described in detail with reference to the drawings.

However, it should be understood that this disclosure is not intended to limit the disclosure to the specific embodiments, and includes various modifications, equivalents, and/or alternatives of the embodiments of the disclosure. . In connection with the description of the drawings, like reference numerals may be used for like elements.

In this specification, expressions such as “has,” “may have,” “includes,” or “may include” refer to corresponding characteristics (eg, numerical values, functions, operations, steps, parts, elements, and/or components). elements), and does not preclude the presence or addition of additional features.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Expressions such as "first", "second", "first" or "second" used in various embodiments may modify various elements regardless of order and/or importance, and define the elements. I never do that. The above expressions may be used to distinguish one component from another. For example, the first element and the second element may represent different elements regardless of order or importance.

As used herein, the expression “configured (or configured) to” means, depending on the circumstances, for example, “suitable for,” “having the capacity to” ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or set) to perform A, B, and C" may include a dedicated processor (eg, an embedded processor) to perform those operations, or one or more software programs stored in a memory device that executes By doing so, it may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

In this specification, a target object that a user wants to segment is a body component that carries blood between axial slices of a three-dimensional volume. In certain embodiments, the object to be segmented includes blood vessels and/or blood supply organs.

A blood vessel is a typical body component that transports blood and has a tubular shape. The blood-supplying organ is an organ that is not a blood vessel but forms a circulation path of blood, and includes, for example, a ventricle and/or an atrium. Hereinafter, for clarity of explanation, the present invention will be described in embodiments in which a division target is a blood vessel, but it will be clear to those skilled in the art that the division target in this specification is not limited to an anatomical blood vessel.

1 is a schematic block diagram of an apparatus for detecting a lesion of a disease related to a body component that delivers blood, such as a blood vessel, from a medical image according to an aspect of the present application.

Referring to FIG. 1 , a device (1, hereinafter referred to as “disease detection device”) for detecting disease lesions related to body components that deliver blood, such as blood vessels, from medical images is used to detect diseases using a disease detection model 20. an analysis unit 10 that detects a corresponding lesion area; It includes an image acquisition unit 30. In addition, the device 1 may further include an output unit 50 .

The disease detection device 1 according to embodiments may be entirely hardware, entirely software, or partially hardware and partially software. For example, the device may collectively refer to hardware equipped with data processing capability and operating software for driving the hardware. In this specification, terms such as “unit”, “module”, “apparatus”, or “system” are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing computing device including a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), or another processor. Also, software may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

The analysis unit 10 includes a processor and memory. The memory of the analyzer 10 executes the previously learned disease detection model 20 and a method of detecting disease lesions related to body components that deliver blood, such as blood vessels, from medical images when executed by the processor. It can also store a program consisting of instructions for

The operation of the analyzer 10 will be described in more detail with reference to FIGS. 3 to 5 below.

2 illustrates a medical image input to a disease detection model according to an embodiment of the present application.

Referring to FIG. 2 , the image acquisition unit 30 directly/indirectly obtains a medical image from a medical device. The medical image is input to the disease detection model 20 . Then, the analysis unit 10 may divide the blood vessel area and/or detect a lesion area of a blood vessel-related disease.

The medical device is a device that generates a medical image including body components by photographing the inside of the body of a target to be detected, such as a blood vessel, such as a magnetic resonance angiography image. For example, the medical device may be a medical device that generates a fluid velocity-weighted magnetic resonance angiography (TOF-MRA) image.

The image acquisition unit 30 may be connected to receive information from the medical device. The image acquisition unit 30 may be implemented as a connection interface. Then, the device 1 may directly acquire the medical image by the image acquiring unit 20 .

Alternatively, the image acquisition unit 30 may be connected to an external device for wired/wireless electrical communication. To this end, the image acquisition unit 30 may be implemented as a communication module.

The external device may be a medical device capable of electrical communication with the communication module itself, a device connected to the medical device, or another device that communicates electrically with a device directly connected to the medical device. Then, when the device 1 acquires a medical image from an external device by the image acquiring unit 30, it may be treated as indirectly acquiring the medical image from the medical device.

The display unit 50 receives the analysis result of the analyzer 10 or stored information of the device 1 and displays the information. The display unit 50 may include, for example, an LCD, OLED, flexible screen, etc., but is not limited thereto.

The apparatus 1 may display a result of dividing a blood vessel area and a result of detecting a lesion area through the display unit 50 .

Network architecture of disease detection model

3 shows a schematic network architecture of a vascular-related disease detection model, according to an embodiment of the present application.

Referring to FIG. 3 , the disease detection model 20 includes a backbone data processing path for extracting a blood vessel region from an input image; It consists of one or more auxiliary data processing paths for detecting lesions based on data (eg, features) calculated in the backbone data processing path.

In certain embodiments, the disease detection model 20 includes the backbone data processing path; a first auxiliary data processing path for detecting a lesion area in a global aspect; and a second auxiliary data processing path for detecting a lesion area in a regional aspect.

The disease detection model 20 includes a division unit 100; a first detection unit 310; and a second detection unit 330.

The division unit 100 is a network constituting the backbone data processing path. The segmentation unit 100 may be configured to extract features (hereinafter referred to as blood vessel features) for predicting a blood vessel area from an input image of the model 20 (eg, a plurality of 2-dimensional slice images constituting a 3-dimensional volume). may be Through this, the analysis unit 10 predicts a region of a body component (ie, a blood vessel), which is an object related to a disease, as a segmentation target in the input image of the model 20 .

The features for predicting blood vessels include spatial features of blood vessels in individual slice images; and a context for blood vessels between consecutive slice images. The segmentation unit 100 may more accurately predict a blood vessel region based on the captured context and extracted features.

Spatial features of blood vessels in the slice image are features related to blood vessels represented in the 2D image. The spatial features are image features for identifying a blood vessel region and a non-vascular region composed of an anatomical structure of a blood vessel expressed in an image. The spatial features are geometric features extractable from a single input image input to the encoding block 110, and include, for example, edges, points, and/or curves. The spatial feature corresponds to a feature extracted from an input image in a general deep learning model such as CNN.

The connectivity between slices is a context of continuous data between an input 2D slice image and at least one slice image among other slice images in an adjacent order. The adjacent sequence may be at least one slice image previously input. The primary anatomical role of blood vessels is to transport blood without leakage. Because of this, blood vessels have strong inter-slice connectivity. An image sequence input to the disease detection model 20 includes continuous information of blood vessels from one body point to be photographed to the next adjacent body point to be photographed.

The division unit 100 includes a plurality of encoding blocks 110; subnetwork 120; a plurality of decoding blocks 130; and a segmentation feature block 140 . In certain embodiments, the division unit 100 may further include a division block 150.

The number of decoding blocks 130 may match the number of encoding blocks 110 . For example, the division unit 100 includes three encoding blocks 110a to 110c; and three decoding blocks 130a to 130c.

The backbone data processing path of the division unit 100 has a depth structure formed of a plurality of hierarchical level structures. A plurality of hierarchical levels are formed by a plurality of encoding blocks 110 and decoding blocks 130, respectively. One encoding block 110 and one decoding block 130 are located at one hierarchical level. The value of the hierarchical level is set as a hyperparameter.

Operation results processed at the hierarchical level are propagated to the next hierarchical level. Output data of the plurality of encoding blocks 110 may be propagated from the highest layer to the lowest layer. Output data of the plurality of decoding blocks 130 may be propagated from the lowest layer to the highest layer.

Encoding block 110 may be configured to perform convolution and pooling operations.

The encoding block 110 extracts features through a convolution operation. The features include spatial features of blood vessels; and to calculate a context for blood vessels between successive slice images.

The encoding block 110 may include a plurality of convolution filters (or referred to as kernel filters) that perform a convolution operation on input data. Then, the encoding block 110 may increase the channel of input data. Data resulting from the convolution operation has the number of channels according to the number of convolution filters.

The plurality of convolution filters may include a 3*3 convolution filter, but is not limited thereto.

In one embodiment, the encoding block 110 may include a plurality of convolutional layers. Then, the encoding block 110 may sequentially perform a convolution operation on the input data multiple times.

In one example, the encoding block 110 may include two convolutional layers. The convolution layer may be configured to process each 2D sequence image with a convolution operation, Rectified Linear Unit (ReLU) activation, and batch normalization.

The encoding block 110 may reduce the size of input data (eg, convolution operation result) through a pooling operation. Then, the encoding block 110 may reduce the resolution of the input image compared to before the pooling process.

In certain embodiments, the encoding block 110 may be configured to perform a max-pooling operation.

The window size of max pooling of the encoding block 110 may be, for example, 2 Х 2, but is not limited thereto.

In the division unit 100, the encoding block 110 for each hierarchical level is configured to generate data having more channels as the hierarchical level becomes deeper.

For example, the encoding block 110 may have 64, 128, or 256 filters for each hierarchical level. However, the number of filters of the encoding block 110 for each hierarchical level is merely illustrative and may be set to various values based on the pixel size of the input image and/or the depth of the hierarchical level.

In the division unit 100 , operation results of the plurality of encoding blocks 110 are propagated to the plurality of decoding blocks 130 through the first connection path.

The first connection path is a data processing path formed by an input terminal of the encoding block 110 of the highest hierarchical level and an output terminal of the decoding block 130 of the highest hierarchical level, and the encoding block 110 of the highest hierarchical level It is sequentially connected to the remaining encoding blocks 110 and a plurality of decoding blocks 130 from the input end of . The first connection path is a data processing path under a single encoder-decoder structure utilized in a general deep learning field.

In the division unit 100, the encoding block 110 performs a pooling operation on the convolution operation result data and supplies it to the encoding block 110 of the next level. Before being supplied to the encoding block 110 of the next level, the convolution operation result data is subjected to a pooling operation, so that the resolution is reduced. As a result, the encoding block 110 reduces the resolution of the image input to the block and outputs it.

Since the decrease in resolution corresponds to the enlargement of the image, the encoding block 110 at a deeper hierarchical level extracts features by performing a convolutional operation on the enlarged image from the input image. As a result, as data is propagated between the plurality of encoding blocks 110 in the divider 100, it is gradually down-sampled.

On the first connection path, at least one encoding block 110 and at least one decoding block 130 are connected through a subnetwork 120 .

The sub-network 120 extracts connectivity between at least some segments of the series of 2D target segment images based on spatial features extracted from the connected encoding block 110 .

In certain embodiments, the subnetwork 120 may connect the encoding block 110 of the previous hierarchical level and the decoding block 130 of the deepest hierarchical level at the deepest hierarchical level. As shown in FIG. 3 , output data of the encoding block 110C of the previous hierarchical level is input to the decoding block 130C of the previous hierarchical level through the deepest sub-network 120 .

When a series of sequence images are input to the segmentation unit 100, the subnetwork 120 calculates connectivity between at least some of the segments based on the result of operation of the connected encoding block 110.

The sub-network 120 is configured to process input in the form of a sequence to model time-varying dynamic characteristics. To this end, the subnetwork 120 may store information for a certain period of time. The disease detection model 20 uses the sub-network 120 to further emphasize connectivity between adjacent slice images to segment blood vessels or detect lesions.

In one embodiment, the subnetwork 120 may be a bi-directional convolutional long short-term memory (LSTM) network.

Alternatively, in other embodiments, the subnetwork 120 may be, for example, a recurrent neural network (RNN), long short-term memory (LSTM), or GRU.

The sub-network 120 stores the feature of at least one sequence image among the features of a plurality of sequence images extracted previously, and connects the at least one sequence image having the stored feature with the sequence image from which the current feature is extracted. By calculating the association of , the connectivity between these segments is extracted.

When a series of sequence images are sequentially input to the segmentation unit 100, the connectivity between the segments is calculated based on an operation result of the plurality of encoding blocks 110 along the first connection path for each of the sequence images.

The subnetwork 120 is a partial flow of information processed by a plurality of encoding blocks 110 among the flow of information calculated by processing each sequence image input to the segmentation unit 100 according to the first connection path. and a data processing path that traverses the partial flow of information processed by decoding block 130 . A partial flow of processing information formed along the plurality of encoding blocks 110 (order of 110A, 110B, 110C, and 120 in FIG. 3) and a partial flow of processing information formed along the plurality of decoding blocks 130 (shown in FIG. 3) 120, 130C, 130B, 130A sequence) may be referred to as a vertical warp path part, and the partial flow of information processed inside the subnetwork 120 (input to output sequence of 120 in FIG. 3 ) may be referred to as a weft path part that transversely connects the warp path part. At this time, since the sub-network 120 is located in a layer deeper than the encoding block 110 and decoding block 130 of the deepest level among the plurality of encoding blocks 110 and decoding blocks 130, the division unit 100 ) may be treated as being located at the deepest level of the hierarchy.

In addition, the divider 100 may be configured to propagate calculation results of the plurality of encoding blocks 110 to the plurality of decoding blocks 130 through the first connection path and the second connection path.

The second connection path is a skip connection formed between the encoding block 110 and the decoding block 130 on the same hierarchical level. The second connection path connects each of the plurality of encoding blocks 110 and the plurality of decoding blocks 130 . The skip connection propagates the operation result of the encoding block 110 at each hierarchical level to the decoding block 130 . In this process, the video features of the input data extracted from each encoding block 110 are passed to the corresponding decoding block 130 .

The division unit 100 may propagate the output data of the encoding block 110 to the corresponding input terminal of the decoding block 130 through the first connection path or the first connection path and the second connection path. . For example, a spatial feature may propagate through the first connection path and the second connection path. Connectivity between segments may be propagated through a first connection path.

The decoding block 130 is configured to perform a deconvolution operation.

The decoding block 130 is configured to up-sample input data through a deconvolution operation. The decoding block 130 may perform a deconvolution operation by performing a convolution operation by adding a padding pixel (eg, zero-padding) around the target pixel.

The decoding block 130 performs deconvolution on data input through the subnetwork 120 along the first data processing path.

In one embodiment, the decoding block 130 may include a deconvolution layer. The deconvolution layer is configured to perform deconvolution processing and activation processing. The deconvolution layer may be configured to perform, for example, deconvolution and ReLU activation operations on each 2D sequence image.

In one embodiment, the decoding block 130 may be further configured to perform a convolution operation and a convolution operation in addition to a deconvolution operation.

The decoding block 130 may combine its own deconvolution operation result and the corresponding encoding block 110's convolution operation result. The result of the convolution operation of the corresponding encoding block 110 is obtained through the second connection path.

The decoding block 130 is configured to reduce a channel of input data by performing a convolution operation. For example, the decoding block 130 may reduce the number of channels of input data from 128 to 64 through a convolution operation.

In some embodiments, the decoding block 130 may have a filter having the same size as that of the convolution layer of the corresponding encoding block 110 . The reduced channel corresponds to the increased number of channels through a pooling operation at the same hierarchical level. In FIG. 3 , the number of channels (eg, 256 -> 128) in the decoding block 130C may decrease by the increased number of channels (eg, 128 -> 256) in the encoding block 110C.

The decoding block 130 may perform a convolution operation on a combination result of its own deconvolution operation result and the corresponding encoding block 110's convolution operation result. Then, the decoding block 130 may reduce the channel of the combined result (ie, combined data of the deconvolution result of the previous hierarchical level and the operation result of the encoding block 11 of the same hierarchical level).

If the deconvolution operation, the combination operation, and the convolution operation are continuously performed in the decoding block 130 for each layer level, the plurality of decoding blocks 130 obtains the spatial features extracted from the plurality of encoding blocks 130 and the subnetwork 120. ), the blood vessel area is gradually restored based on the connectivity between the slices calculated in . Finally, the plurality of decoding blocks 130 restores the resolution of the image down-sampled by the plurality of encoding blocks 110 . The image reconstructed in the decoding block 130 of the highest hierarchical level has the resolution (or size) of the initial input image input to the corresponding encoding block 110.

When a plurality of 2D slice images are sequentially input to the segmentation unit 100, the segmentation feature block 140 extracts a common feature of each 2D slice image and a blood vessel feature based on connectivity between slices.

In one embodiment, the segmentation feature block 140 is a feature for predicting a blood vessel area (ie, a blood vessel feature) from an operation result of the decoding block 130 of the highest layer obtained through all of the plurality of decoding blocks 130. ) is extracted. An operation result of the decoding block 130 of the highest layer corresponds to a reconstructed image. The image reconstructed according to the operation result of the decoding block 130 of the highest layer input to the segmentation feature block 140 is inter-segment captured as a spatial feature of each 2D segment image and a context between the 2D segment image. It is a video that reflects connectivity. For this reason, extraction of blood vessel features from the decoding block 130 of the highest layer by the split feature block 140 may be treated as extraction of blood vessel features from the reconstructed image. When the segmentation feature block 140 extracts image features for predicting blood vessel regions, spatial features of individual slice images and connectivity between slices are extracted from the extracted image features.

In one embodiment, the segmentation feature block 140 may include a convolutional layer. The segmented feature block 140 may extract blood vessel features from an image reconstructed by the convolution layer. Since the principle of the convolution layer of the segmentation feature block 140 is similar to that of the encoding block 110, a detailed description thereof will be omitted.

The division block 150 may divide the blood vessel region by using the blood vessel characteristics calculated in the division feature block 140 as final features.

In one embodiment, the partitioning block 150 may include a stochastic layer. The segmentation block 150 calculates a blood vessel probability value representing a possibility of being a blood vessel region in the input image corresponding to the reconstructed image based on the blood vessel feature by a probability layer, and the restoration is performed based on the calculated blood vessel probability value. A blood vessel region may be detected from an input image corresponding to the processed image.

The probability layer may generate a blood vessel probability map. The blood vessel probability map includes a probability value that each pixel is a blood vessel region based on the blood vessel probability value calculated for each pixel. A blood vessel area may be determined based on the probability value.

The probability layer may use a sigmoier function, but is not limited thereto and may use other probability functions. Based on the blood vessel probability map, the location of the blood vessel region for each input image is determined.

In addition, a segmentation mask may be generated in the blood vessel probability map. The segmentation mask image corresponds to a structure obtained by filtering a blood vessel region in an input image.

A segmentation mask composed of pixels having a probability value of the blood vessel probability map greater than a preset threshold probability may be generated. For example, when the threshold probability is 0.5, a segmentation mask composed of pixels having a probability value greater than 0.5 may be generated from the probability map. The segmentation mask may be associated with the position of the blood vessel region through the blood vessel probability value calculated for each pixel.

In the middle or finally calculated on the backbone data processing path made of the division unit 100

In certain embodiments, an operation result of the subnetwork 120 of the division unit 100 is propagated to the first detection unit 310 . The first auxiliary data processing path branches from the middle of the backbone data processing path, the weft path of the subnetwork 120 .

The calculation result of the division feature block 140 of the division unit 100 is propagated to the second detection unit 330 . For example, the operation result of the convolution layer of the division block 140 is propagated to the second detector 330 . The second auxiliary data processing path branches from the final block of the backbone data processing path.

The disease detection model 20 may detect a lesion area in a global aspect and a lesion area in a regional aspect for the same input image.

The first detection unit 310 detects the lesion area by extracting features for predicting the lesion area (hereinafter referred to as global features) from the global aspect of the input image.

The first detection unit 310 includes a first extraction block 311; A first score block 315 is included.

The first extraction block 311 is configured to calculate global features. The first extraction block 311 may calculate a global feature through a convolution operation from an operation result of the subnetwork 120 . The global feature corresponds to an image feature for predicting a lesion in the input image of the model 20 in a global aspect.

In one embodiment, the first extraction block 311 may include two convolutional layers. The first extraction block 311 may operate input data through the two convolution layers.

Also, the first extraction block 311 may be configured to perform a global pooling operation so that the first detection unit 310 detects a lesion in a global aspect. For example, the first extraction block 311 may be configured to perform a global average pooling operation.

Final global features may be calculated by performing a global average pooling operation on the image features calculated through the convolution operation in the first extraction block 311 .

The first score block 315 receives the operation result of the first extraction block 311 and calculates a global score representing the predicted probability of the lesion area in the global aspect in the input image. The global score quantifies the possibility of detecting a lesion area in an input image in a global aspect.

In one embodiment, the first score block 315 may include a fully connected layer and a probability layer.

In the fully connected layer, a plurality of nodes are completely connected to each other, and the fully connected layer may receive the calculation data of the first extraction block 311 as input data and perform an operation based on a parameter assigned to each node. Said parameters include, for example, weights and/or biases.

In one example, the fully connected layer may perform a matrix multiplication operation on the operation result of the first extraction block 311 in order to predict vascular-related diseases (eg, stenotic and occlusive lesions) learned through samples. .

Then, an analysis value for classifying whether the input image has a lesion area is calculated based on the global feature by the fully connected layer.

The probability layer may calculate a first lesion probability value by which a lesion area is predicted in the input image based on an operation result of the fully connected layer using a probability function. The first lesion probability value is a lesion prediction probability value based on global features. The probability function may be a sigmoir function, but is not limited thereto.

Then, the first score block 315 may calculate the first lesion probability value itself or a value obtained by converting it into a designated scale range as a global score value.

In one example, when the first lesion probability value is calculated as a value of a global score using a sigmoid function, the global score may be expressed by the following equation.

X _t is a 2D slice image, t is the sequence number of sequence images, and T is the total number of sequence images. H, W, and C represent the height, width, and channel of the sequence image, respectively. F is a function that transforms X _t to globally predict the lesion area, and is applied to operations on the weft path of the backbone data processing path and through fully connected layers of the first extraction block 311 and the first score block. It may be the result of The first score block 315 may calculate a global score of 0 to 1 through a sigmoid function σ. The higher the predictability of the lesion, the closer the global score is to 1.

The second detection unit 330 is configured to detect a lesion from an input image in a local aspect. The second detection unit 330 may calculate a localization map and locally detect a lesion area of a vascular-related disease such as a stenotic lesion.

The second detection unit 330 includes a second extraction block 331 and a second score block 335 .

The second extraction block 331 is configured to calculate regional features for predicting regional lesions in the input image.

The second extraction block 331 may perform a convolution operation. For example, the second extraction block 331 may be configured to sequentially perform a convolution operation, batch normalization, ReLU activation, and a convolution operation.

The operation result of the second extraction block 331 is extracted as a regional feature.

A pooling operation is processed in the second auxiliary data processing path. The second detection unit 330 may perform a pooling operation on the operation result of the second extraction block 331 .

The second detector 330 may localize the reconstructed image to a plurality of regions before processing the pooling operation. The area is formed by reducing the size of the input image. A lesion detection operation in the regional aspect is performed on a regional basis.

In one embodiment, the second detection unit 330 may be configured to localize an input image having 256*256 pixels by reducing the size by 16 times. Then, an image reconstructed with 256*256 pixels may be partitioned into 256 regions having a total size of 16*16. This area size will be described in more detail with reference to FIG. 9 below.

The second detection unit 330 is configured to perform a regional pooling operation process on each of the partitioned regions. For example, the second detector 330 may be configured to perform a region average pooling operation on 256 regions. Unlike the global average pooling performed by the first detector 310, the second detector 330 divides the 2D slice image by region and performs a pooling operation.

The image feature calculated through the convolution operation in the second extraction block 331 is subjected to a global average pooling operation, so that a final regional feature may be calculated. The result of this pooling operation (i.e., the final regional feature) is fed into the second score block 335.

The second score block 335 receives the regional features according to the pooling operation and calculates a regional score representing a predicted probability of a lesion region in the input image in terms of the region. The regional score is a score that quantifies the possibility of detecting a lesion area in an input image in a regional aspect.

In one embodiment, the second score block 335 may include a probability layer.

The probability layer may calculate a second lesion probability value by which a lesion area is predicted in the input image based on a regional feature according to the pooling operation using a probability function. The second lesion probability value is a lesion prediction probability value based on regional characteristics. The probability function may be a sigmoir function, but is not limited thereto.

Then, the second score block 335 may calculate the second lesion probability value itself or a value obtained by converting it into a designated scale range as a regional score value.

In one embodiment, the second score block 335 may calculate a localization map that locally predicts a lesion area in an input image of the model 20 using a probability function. The local map is composed of probability values of lesion areas for each pixel of the region.

In one example, the local map Zt may be expressed by the following equation.

Here, H' and W' denote unit height and unit width for partially predicting a lesion area (eg, stenotic lesion) in the input image Xt (eg, 2D slice image). H' and W' are hyperparameters set to values less than or equal to H and W. According to the above embodiment, H' and W' may be set to 16, respectively.

The values of H' and W' are determined according to the size of the region divided by global average pooling. G is a function of converting the sizes of H' and W' to partially predict a lesion in an input image (eg, a 2D slice image), and corresponds to a local pooling operation.

In the above embodiment, Z _t consists of 256 (=16Х16) grid cells and is expressed as a square-shaped localization score.

When an (i, j) th grid cell in Z _t is referred to as Z _tij , Z _tij is a localization score for each grid cell. Z _tij represents the predictability of regional lesions for 256 regions of the input image X _t [iХ16: (i+1)Х16, jХ16: (j+1)Х16].

The disease detection model 20 may detect the lesion area of the detection target in the input image based on the global score calculated by the first detection unit 310 and the regional score calculated by the second detection unit 330.

In one embodiment, the disease detection model 20 may use the amplification operator 350 to calculate a lesion score based on a regional score and a global score.

The amplification operator 350 determines whether the detection result of the first detection unit 310 and the detection result of the second detection unit 330 match (eg, detection of all lesions or non-detection of all lesions). It is an operator that allows a more amplified value to be calculated as a lesion score.

In one embodiment, the amplification operator 350 may be a multiplication operator. Then, the lesion area may be finally determined by calculating the lesion score by multiplying the regional score and the global score of the input image. The determined lesion area is detected as a lesion area of a detection target.

However, the amplification operator is not limited thereto, and may include a weight multiplication operator or other amplification operators.

In one example, using multiplicative operator 350, disease detection model 20 may calculate the following lesion map D _t .

* indicates an operation that multiplies all elements of Z _t by Y _t pixel by pixel. Similar to the localization map, D _t consists of 256 lesion scores. D _tij is expressed by the following equation.

Under D _t above, the area of Z _tij predicted from X _t is a subset of the area predicted from Y _t . The lesion map represents predictive lesions locally while reflecting the global score.

The analysis unit 10 determines a lesion area based on the lesion score. For example, the analysis unit 10 may determine the lesion area by comparing a predetermined threshold with a lesion score. The threshold may be set statistically or empirically.

5A and 5B are diagrams illustrating prediction results of a blood vessel area and a lesion area according to an embodiment of the present application.

Referring to FIGS. 5A and 5B , a blood vessel area and a lesion area for each detection target may be predicted with high accuracy matching actual values.

Learning model parameters

The disease detection model 20 of FIG. 1 extracts spatial features and connectivity between slices from an input image, and calculates blood vessel features based on the spatial features and connectivity between slices; It is learned to detect a lesion area of a vascular-related disease based on at least one of the spatial features, connectivity between slices, and blood vessel features. Vascular-related diseases are diseases that are expressed by having non-normal blood vessel shapes, such as blood vessels being narrowed or blocked compared to normal blood vessels. The disease detection model 20 detects the lesion area of such a vascular-related disease, so that the device 1 generates disease information of a detection target. If the lesion area of the disease is detected, the input image is treated as having the disease.

In certain embodiments, the disease detection model 20 extracts spatial features, connectivity between segments and/or vascular features to segment a specific vascular structure through multi-task learning, while simultaneously extracting the specific vascular structure. It is trained to perform multi-tasking, detecting lesion areas based on features for segmenting vascular structures (spatial features, inter-segment connectivity and/or vascular features). In some embodiments, the particular vascular structure may be an intracranial artery. The lesion associated with the specific vascular structure may be a stenotic or occlusive lesion of the intracranial artery.

4 is a diagram illustrating a learning process of a disease detection model according to an embodiment of the present application.

Referring to FIG. 4 , the disease detection model 20 of FIG. 3 is learned using a training data set composed of a plurality of training samples. The range of divisible vascular structures and the range of detectable disease information of the disease detection model 20 depend on the training data set. Each of the plurality of training samples includes a training image, first label data, and second label data.

The training image is a medical image of a blood vessel, and may be a medical image of a sample target corresponding to a medical image of a detection target. The medical image may be a magnetic resonance angiography image, but is not limited thereto. Various medical images in which blood vessels and non-vascular regions are distinguished may be used as training images and input images.

In certain embodiments, the disease detection model 20 may be trained to receive a plurality of 2D slice images representing a 3D volume as a set of input images. When the 3D volume includes a 3D blood vessel, each 2D slice image may include a plane of the 3D blood vessel.

In this case, the training image of each training sample may be a plurality of 2D segment images. Then, all training images of the training data set may be subsetted into a plurality of 2D segment images for each training sample. In some of the above embodiments, the plurality of 2D slice images may be slice images of a body part obtained by imaging an intracranial artery.

The first label data is data representing a blood vessel region in a training image. As shown in FIG. 4 , the first label data may indicate whether a pixel of a training image is a blood vessel or not. In some of the above embodiments, the first label data may be data representing an intracranial artery in a training image.

The second label data is data representing a lesion area in a training image. As shown in FIG. 4 , the second label data may indicate whether a pixel of a training image is a related lesion or not. In some of the above embodiments, the second label data may indicate a stenotic or occlusive lesion of an intracranial artery as a lesion related to the specific vascular structure.

The disease detection model 20 sequentially receives a plurality of 2D slice images within a sample and proceeds with learning. This learning continues until subsets of a plurality of 2D slice images of all training samples of the training data set are input. In this process, parameters of the dividing unit 100 are learned. In addition, the parameters of the first lesion network 310 and the second lesion network 330 are learned at the same time as the parameters of the dividing unit 100 are learned.

In one embodiment, the disease detection model 20 may include a first error between a blood vessel prediction value and an actual blood vessel value according to calculation by the dividing unit 100; The second error between the predicted value of the lesion and the actual value of the lesion of the disease detection model 20 based on the calculation result by the first detection unit 310 and the calculation result by the second detection unit 330 is learned to be minimized. The blood vessel prediction value may be a blood vessel region extracted by the division unit 100 in a learning process. The blood vessel actual value may be first label data. The lesion prediction value is final based on the calculation result of the first detection unit 310 (ie, the predicted lesion area in the global aspect) and the calculation result of the second detection unit 330 (ie, the predicted lesion area) in the learning process. It may also be a predicted lesion area. The actual value of the lesion may be second label data.

In another embodiment, the disease detection model 20 is a difference between the predicted value and the actual value of the division unit 100, the difference between the predicted value and the actual value of the first detection unit 310, and the second detection unit 330 It may be learned to minimize the difference between the predicted value and the actual value of . Actual values of the first detection unit 310 and the second detection unit 330 may be the same second label data.

In one example, when the length T of a plurality of 2D slice images and the interval S between slice images (T<S) are set, the disease detection model 20 calculates the following loss function It can also be learned to minimize.

Here, L _seg , L _cls , and L _loc are loss functions for the prediction of the divider 100 , the prediction operation of the first detector 310 , and the prediction of the second detector 330 . W _seg , W _cls , and W _loc are weights corresponding to the loss of each operation.

The disease detection model 10 is trained to have parameters that minimize the loss function of Equation 5.

If the disease detection model 20 learned in this way is used, blood vessels can be more accurately segmented. This is because the disease detection model 20 can preserve the comprehensive anatomical structure of blood vessels when blood vessels are divided.

At the same time, the disease detection model 20 can accurately capture the location of a lesion of a vascular-related disease such as a stenotic lesion or an occlusive lesion. This is because the narrowed or completely obscured portion of a blood vessel (i.e., the location of a lesion) represented as an ablated portion in a medical image such as a magnetic resonance angiography image can be accurately inferred based on the divided blood vessel structure.

disease information detection

A method of detecting a lesion of a disease related to a body component that delivers blood from a medical image according to another aspect of the present application may be performed by a computing device including a processor. In certain embodiments, the method may be performed by device 1 of FIG. 1 .

6 is a flowchart of a method of detecting a disease lesion related to a body component that delivers blood from a medical image according to another aspect of the present application.

Referring to FIG. 6 , the method: extracts spatial features and connectivity between segments from a medical image of a target to be detected using a pre-learned disease detection model 20, and obtains blood vessel characteristics based on the connectivity between the spatial features and segments. A step of calculating (S100) is included.

In addition, the method includes: segmenting a blood vessel region in the input medical image based on the blood vessel feature extracted in the step (S100) (S150), and/or at least one of the spatial feature, the connectivity between slices, and the blood vessel feature. It may also include a step of detecting a lesion area of a blood vessel-related disease based on (S300).

As described above, the medical image may be a plurality of 2D slice images forming a 3D volume. The 3-dimensional volume is a 3-dimensional volume in which a body component of a detection target is photographed. In some embodiments, the medical image may be a plurality of 2D slice images of intracranial arteries. The medical image may be an image captured by magnetic resonance angiography.

As described above, the disease detection model 20 uses a training data set including a training image corresponding to a medical image of a detection target to segment a specific vascular structure and to detect disease lesions related to the divided blood vessels in advance. As a learned machine learning model, it is in a state learned to extract spatial features and connectivity between segments from a medical image of a detection target, and to calculate blood vessel characteristics based on the connectivity between the spatial features and segments.

In step S100, the plurality of 2D slice images are sequentially input to the disease detection model 20 according to the order of the slice images. Spatial features and connectivity between slices are extracted by the division unit 100 of the model 20, and blood vessel features are calculated (S100).

In an embodiment, the step of extracting spatial features and connectivity between slices from the medical image of the detection target and calculating blood vessel features based on the spatial features and connectivity between slices (S100) is performed on each of a plurality of 2D slice images. sequentially extracting spatial features (S110); extracting connectivity between slices of the adjacent 2D slice images based on spatial features of the adjacent 2D slice images (S120); and calculating blood vessel features based on the spatial features and connectivity between slices ( S130 ).

The step of sequentially extracting spatial features from each of the plurality of 2D slice images (S110) may include extracting spatial features from the 2D slice images for each hierarchical level, and performing a pooling operation on the extracted spatial features. there is.

The hierarchical level is the hierarchical level of the dividing unit 100 . Spatial feature extraction and pooling operation processing is performed n times. In the case of the division unit 100 of FIG. 3 , n is set to 3.

The spatial features extracted in step S110 include spatial features for each hierarchical level. The spatial feature for each hierarchical level may be a spatial feature extracted from the corresponding hierarchical level and subjected to a pooling operation. For example, the spatial feature extracted in step S110 may be an operation result output from the encoding block 110 for each hierarchical level in the segmentation unit 100 of FIG. 3 .

In an embodiment, the connectivity between the slices may be generated based on spatial features of adjacent 2D slice images extracted n times from the 2D slice images and subjected to a pooling operation (S120). The spatial features of the adjacent 2D segment images extracted n times and subjected to the pooling operation may be an operation result of the encoding block 110 of the deepest hierarchical level of FIG. 3 .

In one embodiment, the step of calculating the blood vessel feature based on the connectivity between the spatial features and the slices (S130), deconvolution is performed on the calculation result of the previous hierarchical level for each hierarchical level, and the result of the deconvolution operation and the same Restoring an input 2D slice image by combining spatial features of a hierarchical level and reducing a channel of the combined result (S133); and extracting blood vessel features from the reconstructed input image (S134).

An operation result of the previous hierarchical level is an operation result along the first connection path. For example, in FIG. 3, the calculation result of the previous hierarchical level of the decoding block 130C is the calculation result of the sub-network 120, and the calculation result of the previous hierarchical level of the decoding block 130B is the result of the decoding block 130C. It may be the result of an operation.

The spatial feature of the same hierarchical level is the result of operation of the corresponding encoding block 110 obtained through the second connection path.

In one embodiment, the reduced number of channels may correspond to the increased number of channels through a pooling operation at the same hierarchical level (S131).

These deconvolution operations, combination operations, and convolution operations are continuously performed for each hierarchical level (S131).

When the deconvolution operation, the combination operation, and the convolution operation are performed, the down-sampled input image is reconstructed while extracting spatial features (S131). The reconstructed image may correspond to results of deconvolution, combination, and convolution operations in the decoding block 130A of the highest hierarchical level in the division unit 100 of FIG. 3 (S133).

The blood vessel features extracted from the reconstructed image are image features that reflect connectivity between slices as well as spatial features of individual slice images (S134).

In an embodiment, the dividing of the blood vessel region (S150) includes: calculating a blood vessel probability value based on the blood vessel characteristics; generating a segmentation mask based on the calculated blood vessel probability value; and dividing the blood vessel region using the segmentation mask.

As described above, the blood vessel probability value may be calculated by generating a blood vessel probability map composed of blood vessel probability values for each pixel. The operation of the steps S130 and S150 has been described above with reference to the decoding block 130, the division feature block 140, and the division block 150, and detailed descriptions thereof are omitted.

In certain embodiments, the detecting of the lesion area of the vascular-related disease (S300) includes: calculating a global feature through a convolution operation and a global pooling operation based on connectivity between segments (S311); Calculating a global score based on the global feature (S315); Calculating a regional feature through a convolution operation and a regional pooling operation based on the blood vessel characteristics of the step S100 (S331); Calculating a regional score based on the regional characteristics (S335); and determining a lesion area by calculating a lesion score based on the regional and global scores (S350).

In step S311 of calculating a global feature through a convolution operation and a global pooling operation based on the connectivity between the segments, the operation result of the subnetwork 120 may be subjected to a convolution operation and a global pooling operation.

In one embodiment, calculating a global score based on the global feature (S315) may calculate a global score from the global feature using a probability function.

Since the operation of the steps S311 and 315 has been described above with reference to the first detection unit 310, a detailed description thereof will be omitted.

In one embodiment, calculating regional features through a convolution operation and regional pooling operation based on the blood vessel characteristics of step S100 (S331) divides the reconstructed image from which the blood vessel features are calculated into a plurality of regions. step; and extracting regional features through a convolution operation and a regional pooling operation of blood vessel characteristics for each region.

In an embodiment, in step S331 of calculating a regional score based on the regional feature, a regional score may be calculated from the regional feature using a probability function. The regional score may be calculated as a localization map.

Since the operation of the steps S331 and 335 has been described above with reference to the second detection unit 330, a detailed description thereof will be omitted.

In an embodiment, determining a lesion area by calculating a lesion score based on the regional score and the global score (S350) includes calculating a lesion score by calculating the local score and the global score through an amplification operator; and determining a lesion area based on the lesion score.

In one embodiment, the lesion score may be calculated by multiplying a regional score and a global score through a multiplication operator.

In addition, the method may further include: calculating the location of the divided blood vessel region and/or the location of the detected lesion region from the corresponding 2D slice image (S510). The position may be calculated as a coordinate value on a plane of the 2D slice image. The calculated position is displayed through the display unit 50 .

7 illustrates a result of displaying the location of a divided blood vessel region and the location of a detected lesion region on a 2D slice image according to an embodiment of the present application.

In FIG. 7 , the middle image and the right image are partial views in which the brightness is edited so that the blood vessel area and the lesion area can be more identified.

Referring to FIG. 7 , the location of the divided blood vessel area and/or the location of the detected lesion area may be displayed on a 2D slice image (S510).

In addition, the method may further include: generating a 3D model of the divided blood vessel region and/or the detected lesion region (S550).

8 illustrates a 3D model in which positions of divided blood vessel regions and positions of detected lesion regions are rendered according to an embodiment of the present application.

Referring to FIG. 8 , the divided blood vessel area and/or lesion area may be created as a 3D rendered model. The generated model may be displayed through the display unit 50 (S550).

In one embodiment, the device 1 calculates the location of the segmented blood vessel region and/or the detected lesion region on the 2D slice image (eg, by the analysis unit 10), and places the location at the calculated location. Based on this, a 3D blood vessel model and a lesion model may be created. In this case, the 3D blood vessel model and the lesion model are related to the position of each region calculated in step S510. For example, as shown in FIG. 5 , the positions of blood vessels and lesions calculated in step S510 may be coordinated on the blood vessel model and the lesion model.

The blood vessel area and lesion area divided in steps S510 and S550 may be marked with a designated marker to distinguish them from other areas in the medical image. The blood vessel area and the lesion area may also be marked with different markers and distinguished from each other.

The marker may be a color, but is not limited thereto.

Experimental Example

9A to 9C show model performance according to localization size according to an experimental example of the present application, and FIG. 10 shows the model of FIG. 3 and other models modified therefrom, respectively, according to an experimental example of the present application. Shows the segmentation performance and detection performance of

In the above experimental example, for 180 patients with intracranial constrictive lesions confirmed by cerebral angiography, 120 were used for training, 30 were tested, and 30 were tested internally at Seoul National University Bundang Hospital, the present applicant. As a result, 30 people were used as external test subjects in other external hospitals.

As shown in FIGS. 9A to 9C , the highest performance is shown when the localization size of a region is 16x16 for a 256*256 slice image.

10 is a prediction result of the disease detection model 20 (full model in FIG. 9) of FIG. 3 using an amplification operator with a backbone data processing path and first and second auxiliary data processing paths, first to third modified models. The prediction result of (without SP, without DP, without MO in FIG. 9) is shown.

here, The first modified model (without SP in FIG. 10 ) is one in which the backbone data processing path is removed from the entire model of FIG. 3 . The second modified model (without DP in FIG. 10) is a model in which the lesion detection path of the global aspect is removed, and instead of multiplicative operation, the stenotic lesion is predicted only locally in the localization map. The third modified model (without MO in FIG. 10) is a model in which the constrictive lesion is predicted only locally in the localization map (similar to the second modified model) by not using the multiplicative operator.

As shown in FIG. 10, it is confirmed that the predictive performance of the disease detection model 20 of the present application is the most accurate compared to other modified models.

It will be clear to those skilled in the art that the device 1 and method for detecting disease lesions related to body components that deliver blood from the medical images may include other components not described herein. For example, it may further include data input devices, output devices such as display and/or printing, networks, network interfaces and protocols, and the like.

The operation by the device 1 and method for detecting disease lesions related to body components that deliver blood from medical images according to the embodiments described above are at least partially implemented as a computer program and can be read by a computer. It can be recorded on a recording medium. For example, implemented together with a program product consisting of a computer-readable medium containing program code, which may be executed by a processor to perform any or all steps, operations, or processes described.

The computer may be any device that may be integrated with or may be a computing device such as a desktop computer, laptop computer, notebook, tablet pc, smart phone, or the like. A computer is a device that has one or more alternative and special purpose processors, memory, storage, and networking components (whether wireless or wired). The computer may run, for example, an operating system compatible with Microsoft's Windows, Apple's OS X or iOS, a Linux distribution, or an operating system such as Google's Android OS.

The computer-readable recording medium includes all types of recording devices in which data readable by a computer is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, computer-readable recording media may be distributed in computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present invention reviewed above has been described with reference to the embodiments shown in the drawings, but this is only exemplary, and those skilled in the art will understand that various modifications and variations of the embodiments are possible therefrom. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

Claims (21)

A method for detecting a disease lesion related to a body component that delivers blood from a medical image performed by a computing device including a processor, the method comprising:
Extracting spatial features and connectivity between segments from a medical image of a detection target using a pre-learned disease detection model, and calculating features for predicting body components based on the connectivity between the spatial features and segments,
The medical image is a plurality of 2-dimensional slice images constituting the body components of a 3-dimensional volume,
The plurality of 2D slice images are sequentially input to the disease detection model according to the order of the slice images, and the spatial features, connectivity between slices, and features for predicting body components are obtained for each 2D slice image. How to.
The method of claim 1, wherein the step of extracting spatial features and connectivity between slices from the medical image of the detection target and calculating features for predicting body components based on the spatial features and connectivity between slices comprises:
sequentially extracting spatial features from each of the plurality of 2D slice images;
extracting connectivity between slices of adjacent 2D slice images based on spatial features of the adjacent 2D slice images; and
and calculating a feature for predicting a body component based on the connectivity between the spatial feature and the slice.
According to claim 2,
Connectivity between the slices by calculating the correlation between at least one 2D slice image having a stored spatial feature among the spatial features of at least some 2D slice images previously extracted and the 2D slice image from which the current spatial feature is extracted. A method characterized in that for extracting.
According to claim 2,
The spatial features are extracted by extracting spatial features from the 2D slice image for each hierarchical level and performing a pooling operation on the extracted spatial features;
Wherein the connectivity between the segments is extracted based on a result of a pooling operation at the deepest hierarchical level.
The method of claim 2, wherein calculating a feature for predicting a body component based on the connectivity between the spatial feature and the slice comprises:
The input 2D slice image is restored by performing deconvolution on the operation result of the previous hierarchical level for each hierarchical level, combining the result of the deconvolution operation with spatial features of the same hierarchical level, and reducing the channel of the combined result. doing; and
A method comprising extracting features for predicting body components from a reconstructed input image.
The method of claim 5,
The reconstructed input image corresponds to the result of the deconvolution operation, combination operation, and convolution operation at the highest hierarchical level,
A method characterized in that the features for predicting body components extracted from the reconstructed image are image features reflected in spatial features of individual slice images and connectivity between slices.
According to claim 1,
Segmenting the body component region in the input medical image based on the extracted feature for predicting the body component, and based on one or more of the spatial features, connectivity between segments, and features for predicting the body component A method further comprising at least one step of detecting a lesion area of the disease.
The method of claim 7, wherein the dividing of the body component region comprises:
calculating a body component probability value in an input image based on a feature for predicting the body component;
generating a segmentation mask based on the calculated body component probability values; and
and segmenting body component regions using the segmentation mask.
The method of claim 7, wherein the step of detecting the lesion area of the disease,
calculating a global feature through a convolution operation and a global pooling operation based on connectivity between segments;
calculating a global score based on the global feature;
calculating a regional feature through a convolution operation and a regional pooling operation based on the feature for predicting the body component;
calculating a regional score based on the regional characteristics; and
determining a lesion area by calculating a lesion score based on the regional score and the global score;
The global feature is a feature for detecting a lesion area in an input image in a global aspect, the regional feature is a feature for detecting a lesion area in an input image in a regional aspect, and the global score is an input in a global aspect. A method characterized in that the probability of detecting a lesion area in an image is quantified, and the regional score digitizes the probability of detecting a lesion area in an input image in a regional aspect.
The method of claim 9, wherein calculating a regional feature through a convolution operation and a regional pooling operation based on the feature for predicting the body component comprises:
dividing an image from which features for predicting body components are calculated into a plurality of regions; and
and extracting regional features through a convolution operation and a regional pooling operation of features for predicting body components for each region.
The method of claim 9,
The global score is calculated from global features using a probability function;
Wherein the regional score is calculated from regional features using a probability function.
The method of claim 9 , wherein determining a lesion area by calculating a lesion score based on the regional score and the global score comprises:
Calculating a regional score and a global score through an amplification operator to calculate a lesion score; and
and determining a lesion area based on the lesion score.
The method of claim 12,
Wherein the lesion score is calculated by multiplying the local score and the global score through a multiplication operator.
The method of claim 1 , wherein the method:
calculating positions of at least one of the positions of the segmented body component regions and the positions of the detected lesion areas from the corresponding 2D slice image, and a 3D model of at least one of the segmented body component regions and the detected lesion regions. A method further comprising one or more of the steps of generating.
A computer readable recording medium having a program recorded thereon for performing the method of detecting a disease lesion on a body component that delivers blood from a medical image according to any one of claims 1 to 14.
An apparatus for detecting a lesion of a disease related to a body component that delivers blood from a medical image,
an image acquisition unit receiving medical images including a plurality of 2-dimensional slice images constituting body components of a 3-dimensional volume; and
An analysis unit that extracts spatial features and connectivity between segments from a medical image of a detection target using a pre-learned disease detection model, and calculates features for predicting body components based on the spatial features and connectivity between segments. do,
The disease detection model,
a division unit forming a backbone data processing path having a depth structure composed of a plurality of hierarchical levels;
a first detector forming a first auxiliary data processing path branched from the subnetwork of the backbone data processing path; and
and a second detection unit forming a second auxiliary data processing path branched from the division feature block of the backbone data processing path.
The method of claim 16, wherein the dividing unit
a plurality of encoding blocks and a plurality of decoding blocks located for each hierarchical level;
a sub-network for extracting connectivity between segments by connecting encoding blocks and decoding blocks of a previous hierarchical level at the deepest hierarchical level;
a division feature block that calculates a feature for predicting the body component from an operation result of a decoding block of the highest hierarchical level; and
A segmentation block for segmenting a body component region in an input image based on a feature for predicting the body component;
The encoding block for each hierarchical level propagates the operation result to the encoding block of the next hierarchical level through a first connection path and to the decoding block of the same hierarchical level through a second connection path;
The decoding block for each hierarchical level gradually converts an input image by receiving an operation result of a decoding block or subnetwork of a previous hierarchical level through the first connection path and an operation result of an encoding block of the same hierarchical level through the second connection path. A device characterized by restoring.
The method of claim 16, wherein the first detection unit,
a first extraction block that calculates a global feature through a convolution operation and a global pooling operation based on connectivity between segments; and
and a first score block for calculating a global score based on the global characteristic.
The method of claim 18, wherein the second detection unit,
a second extraction block calculating a regional feature through a convolution operation and a regional pooling operation based on the feature for predicting the body component; and
and a second extraction block for calculating a regional score based on the regional feature.
The method of claim 19, wherein the disease detection model,
calculating a lesion score by calculating the regional score and the global score through a multiplication operator; and
An apparatus characterized in that configured to detect a lesion area based on the calculated lesion score.
The method of claim 16,
The disease detection model is an apparatus characterized in that the difference between the predicted value and the actual value of the dividing unit, the difference between the predicted value and the actual value of the first detection unit, and the difference between the predicted value and the actual value of the second detection unit are minimized. .

Images