TWI768483B - Method and apparatus for identifying white matter hyperintensities - Google Patents

Abstract

The present disclosure is related to a method and an apparatus for identifying white matter hyperintensities (WMH). Some embodiments of the present disclosure are related to a method for identifying white matter hyperintensities (WMH). The method comprises: receiving an input image; performing a first set of convolutions; and performing an output convolution so as to produce at least two image segmentations, the output convolution including performing a convolution with at least two kernel maps.

Description

Method and device for identifying white matter hyperintensity

The present disclosure generally relates to a method and apparatus for image recognition. More specifically, the present disclosure relates to a method and apparatus for identifying white matter hyperintensities (WMH).

In the field of medical imaging, image interpretation is generally performed by specialist physicians. For example, after obtaining a brain image by Magnetic Resonance Imaging (MRI), a brain surgeon can circle the lesions on the image. By referring to the selected lesions, patient complaints and other medical examination results, specialists can determine the etiology and pathogenesis of the disease, and make a diagnosis and plan treatment accordingly. In the field of brain medicine, white matter hyperintensities (WMH) are often considered by physicians as possible white matter leisons.

Due to the development of image recognition technology, image recognition technology has been used in the preliminary selection of medical images to increase the efficiency and accuracy of medical image interpretation by specialists.

Some embodiments of the present disclosure relate to a method of identifying white matter hyperintensity. The method includes: receiving an input image; performing a first set of convolutions; and performing an output convolution to generate at least two image segmentations, the output convolution comprising performing a convolution with at least two kernel maps.

Some embodiments of the present disclosure relate to a method for quantifying white matter hyperintensity. The method includes: receiving a plurality of input images; performing a method of identifying white matter hyperintensity for each of the plurality of input images to generate a plurality of gray matter image segmentations, a plurality of white matter image segmentations, a plurality of white matter hyperintensity images segmentation, segmentation of a plurality of cerebrospinal fluid images; segmentation based on the plurality of gray matter images to generate a gray matter volume; segmentation based on the plurality of white matter images to generate a white matter volume; segmentation based on the plurality of white matter hyperintensity images to generate a white matter hyperintensity volume; Based on the plurality of cerebrospinal fluid image segmentations, a cerebrospinal fluid volume is generated; and image grading is performed based on the gray matter volume, the white matter volume, the white matter hyperintensity volume, the cerebrospinal fluid volume, and slice thicknesses of the plurality of input images.

Some embodiments of the present disclosure relate to devices for identifying white matter hyperintensities. The apparatus includes: a processor; and a memory coupled to the processor. The processor executes computer-readable instructions stored in the memory to perform the following operations: receive at least one input image. The following operations are performed for each of the at least one input image: performing a first set of convolutions; and performing output convolutions to generate at least two image segmentations, the output convolutions including performing convolutions with at least two core maps.

Methods, systems, and other aspects of the present invention are described. Reference will be made to certain embodiments of the invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with the embodiments, it will be understood that the intention is not to limit the invention to these specific embodiments. On the contrary, the invention is intended to cover alternatives, modifications and equivalents within the spirit and scope of the invention. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense.

Furthermore, in the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, one of ordinary skill in the art will be able to practice the present invention without these specific details. In other instances, methods, procedures, operations, components and networks well known to those of ordinary skill in the art have not been described in detail in order to avoid obscuring aspects of the present invention.

Some embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

FIG. 1 illustrates a flowchart of image recognition according to some embodiments of the present disclosure. FIG. 1 may illustrate input image 101 and output images 201 , 203 , 205 , 207 , 209 , 211 . According to FIG. 1 , the input image 101 may be processed by the image processing program 300 . After the input image 101 is processed by the image processing program 300 , two or more images can be output, such as output images 201 , 203 , 205 , 207 , 209 , and 211 .

The input image 101 can be a flat image or a 2D image. In other embodiments, the input image 101 may be a stereoscopic image or a 3D image. In some embodiments, the input image 101 may be an MRI image. For example, the input image 101 may be a FLAIR (Fluid Attenuated Inversion Recovery) image. The input image 101 can also be a T2-FLAIR image.

The image processing program 300 can divide the input image 101 into multiple images according to the characteristics of different tissues. The output images 201, 203, 205, 207, 209, 211 may indicate different tissues. In some embodiments, output image 201 may indicate gray matter; output image 203 may indicate white matter; output image 205 may indicate white matter hyperintensity; output image 207 may indicate cerebrospinal fluid; output image 209 may indicate scalp and/or Skull; output image 211 may indicate air.

In some embodiments, the input image 101 may output two or more images through the image processing program 300 . For example, the input image 101 may output two images through the image processing program 300 , one output image may indicate white matter, and the other output image may indicate white matter hyperintensity. For example, the input image 101 can output three images through the image processing program 300 , one output image can indicate gray matter, one output image can indicate white matter, and one output image can indicate white matter hyperintensity.

The image processing program 300 may be a 2D image recognition program or a 3D image recognition program for recognizing features on the image. The image processing program 300 may be an image recognition program generated by machine learning. The image processing program 300 may be an image recognition program generated by deep learning. The image processing procedure 300 may be an image recognition procedure generated by backpropagation of a convolutional neural network. The image processing program 300 may be a convolutional neural network. The image processing program 300 may be a convolution neural network for image semantic segmentation. The image processing program 300 may be a convolutional neural network with a U-Net architecture. The image processing program 300 may be a multi-class output convolutional neural network with a U-Net architecture. The image processing program 300 may be a convolutional neural network with a U-SegNet architecture. The image processing program 300 may be a multi-class output convolutional neural network with a U-SegNet architecture.

In some embodiments of the present disclosure, the image processing program 300 may be a convolutional neural network generated by backpropagation of a convolutional neural network. In the embodiment of the present disclosure, the cases for training and testing the convolutional neural network are shown in Table 1. This disclosure does not detail the process of generating a convolutional neural network through backpropagation of the convolutional neural network. Fazekas Grading training dataset test dataset Grade I 834 65 Grade II 339 twenty two Grade III 195 13 total number of cases 1368 100 Table 1

FIG. 2 illustrates a hierarchical flow diagram according to some embodiments of the present disclosure. According to the image recognition flowchart shown in FIG. 1 , when multiple input images 101 are input to the image processing program 300 , the image processing program 300 can correspondingly output multiple output images 201 , multiple output images 203 , and multiple output images 205 . , multiple output images 207 , multiple output images 209 and/or multiple output images 211 . For example, 100 brain images (for example, 100 images of the same patient) can be sequentially input into the image processing program 300, and the image processing program 300 can output 100 output images 201, 100 output images 203, and 100 correspondingly Output images 205 , 100 output images 207 , 100 output images 209 , and/or 100 output images 211 .

Referring to FIG. 2 , a volume 231 can be calculated from multiple output images 201 ; a volume 233 can be calculated from multiple output images 203 ; a volume 235 can be calculated from multiple output images 205 ; and a volume 237 can be calculated from multiple output images 207 . In some embodiments, output image 201 is indicative of gray matter; output image 203 is indicative of white matter; output image 205 is indicative of white matter hyperintensity; and output image 207 is indicative of cerebrospinal fluid. In these embodiments, by calculating the gray matter area of each of the plurality of output images 201 , the gray matter volume 231 can be obtained; by calculating the white matter area of each of the plurality of output images 203 , the obtained white matter volume 233; by calculating the white matter hyperintensity area of each of the multiple output images 205, the white matter hyperintensity volume 235 can be obtained; by calculating the cerebrospinal fluid area of each of the multiple output images 207 , cerebrospinal fluid volume 237 can be obtained.

In some embodiments, the input image 101 may output two or more images through the image processing program 300 . For example, 100 input images 101 can be output by the image processing program 300 to two groups of images, the first group of output images may include 100 images indicating white matter, and the second group of output images may include 100 images indicating white matter hyperintensity. In these embodiments, by calculating the white matter area of each of the first set of output images, the white matter volume can be obtained; by calculating the white matter hyperintensity area of each of the second set of output images, the Obtain white matter hyperintensity volumes.

In some embodiments, the input image 101 can output three images through the image processing program 300 . For example, 100 input images 101 can be outputted by the image processing program 300 to three sets of images, the first set of output images may include 100 images indicating the gray matter of the brain, the second set of output images may include 100 images indicating the white matter of the brain, and the third set of output images may include 100 images indicating the white matter of the brain. A set of images may contain 100 images indicative of white matter hyperintensity. In these embodiments, the gray matter volume can be obtained by calculating the gray matter area of each of the first set of output images; the brain gray matter volume can be obtained by calculating the white matter area of each of the second set of output images. White matter volume; by calculating the area of white matter hyperintensity in each of the third set of output images, the volume of white matter hyperintensity was obtained.

According to some embodiments of FIG. 2 , gray matter volume 231 , white matter volume 233 , white matter hyperintensity volume 235 , cerebrospinal fluid volume 237 may be input to grading program 800 . After being processed by the grading program 800, the grading result 900 can be output. In some embodiments, in addition to gray matter volume 231 , white matter volume 233 , white matter hyperintensity volume 235 , and cerebrospinal fluid volume 237 , imaging parameters 239 may optionally be input into grading program 800 to add grading program 800 efficiency or increase the accuracy of the classification result 900. Image parameters 239 may include image parameters of input image 101 . Image parameters 239 may include slice thickness of input image 101 .

In some embodiments, the staging procedure 800 may be a gradient boosting procedure. Gradient boosting is a machine learning technique used for regression and classification problems. In some embodiments, the grading procedure 800 may be an XGBoost (eXtreme Gradient Boosting) procedure. In some embodiments of the present disclosure, the grading procedure 800 may be a gradient boosting procedure generated through machine training. In the embodiments of the present disclosure, the examples used for training and testing the gradient boosting procedure or the XGBoost procedure are shown in Table 1. This disclosure does not detail the process of generating gradient boosting programs or XGBoost programs by machine learning.

The graded result 900 may be one of the results according to the Fazekaz grades. On the Fezex scale, a grade of 0 may indicate no white matter hyperintensity or no white matter lesion; a grade of 1 may indicate that the area of white matter hyperintensity (or white matter lesion area) is punctate or thin; and a grade of 2 may indicate cerebral White matter hyperintensity areas (or white matter lesion areas) begin to confluence; a grade of 3 may indicate white matter hyperintensity areas (or white matter lesion areas) as large confluent areas.

FIG. 3A illustrates an image processing procedure 300 according to some embodiments of the present disclosure. The input image 101 may be 512×512×1. The input image 101 can be a 512×512 image with one channel, or a 512×512 image with one dimension. The units of 512×512, 256×256, 128×128 or 64×64 described in this disclosure may be pixels or other suitable units.

Operation 301 is performed on the input image 101 to output a feature map set 102 . Operation 301 may be a convolution operation with a plurality of kernal maps. In some embodiments, operation 301 may be a convolution operation with 32 core maps, and the output feature map group 102 may include 32 feature maps. In some embodiments, operation 301 may include zero padding. Each feature map in feature map set 102 may be 512x512. In some embodiments, operation 301 may include batch normalization and Rectified Linear Unit (ReLU) function processing.

Operation 302 is performed on the feature map group 102 to output the feature map group 103 . Operation 302 may be a convolution operation with a plurality of core graphs. In some embodiments, operation 302 may be a convolution operation with 64 core maps, and the output feature map group 103 may include 64 feature maps. In some embodiments, operation 302 may include zero padding. Each feature map in feature map set 103 may be 512x512. In some embodiments, operation 302 may include batch normalization and rectified linear unit function processing. Operation 301 and operation 302 may be viewed as a set of convolution operations.

Operation 307 is performed on the feature map set 103 to output a set of output images. Operation 307 may be to perform output convolution. In some embodiments, operation 307 may include zero padding. Operation 307 may be a convolution operation with the plurality of core graphs. Operation 307 may be a convolution operation with at least two core graphs. In some embodiments, operation 307 may be a convolution operation with the six core maps to generate output images 201 , 203 , 205 , 207 , 209 , 211 . Output image 201 may indicate gray matter; output image 203 may indicate white matter; output image 205 may indicate white matter hyperintensity; output image 207 may indicate cerebrospinal fluid; output image 209 may indicate scalp and/or skull; output image 211 may indicate Indicates air.

In some embodiments, operation 307 may be a convolution operation with 2 core maps to generate two output images, one output image may be indicative of white matter and one output image may be indicative of white matter hyperintensity.

In some embodiments, operation 307 may be a convolution operation with three core maps to generate three output images, one output image may be indicative of gray matter, one output image may be indicative of white matter, and one output image may be indicative of white matter high signal.

After the input image 101 is processed by the image processing program 300 , two or more output images can be output to generate multi-category semantic segmentation for the input image 101 . A single class of semantic segmentation may be processed by an image processing program to output only one output image, and the output image may be indicative of white matter hyperintensity. Compared with single-category semantic segmentation, multi-category semantic segmentation can achieve better tissue morphological discrimination and reduce false positive lesion determination.

FIG. 3B illustrates an image processing procedure 300 according to some embodiments of the present disclosure. The input image 101 may be 512×512×1. The input image 101 can be a 512×512 image with one channel, or a 512×512 image with one dimension.

Operation 301 is performed on the input image 101 to output a feature map set 102 . Operation 301 may be a convolution operation with a plurality of core graphs. In some embodiments, operation 301 may be a convolution operation with 32 core maps, and the output feature map group 102 may include 32 feature maps. In some embodiments, operation 301 may include zero padding. Each feature map in feature map set 102 may be 512x512. In some embodiments, operation 301 may include batch normalization and rectified linear unit function processing.

Operation 302 is performed on the feature map group 102 to output the feature map group 103 . Operation 302 may be a convolution operation with a plurality of core graphs. In some embodiments, operation 302 may be a convolution operation with 64 core maps, and the output feature map group 103 may include 64 feature maps. In some embodiments, operation 302 may include zero padding. Each feature map in feature map set 103 may be 512x512. In some embodiments, operation 302 may include batch normalization and rectified linear unit function processing. Operation 301 and operation 302 may be viewed as a set of convolution operations.

Operation 401 is performed on the feature map group 103 to output the feature map group 104 . Operation 401 may be a down-sampling process. Operation 401 may be a max pooling process. In some embodiments, operation 401 may be 2×2 max pooling, and the output feature map group 104 may include 64 feature maps, wherein each feature map may be 256×256.

Operation 303 is performed on the feature map group 104 to output the feature map group 105 . Operation 303 may be a convolution operation with a plurality of core graphs. In some embodiments, operation 303 may be a convolution operation with 64 core maps, and the output feature map group 105 may include 64 feature maps. In some embodiments, operation 303 may include zero padding. Each feature map in feature map set 105 may be 256x256. In some embodiments, operation 303 may include batch normalization and rectified linear unit function processing.

Operation 304 is performed on the feature map set 105 to output the feature map set 106 . Operation 304 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 304 may be a convolution operation with 128 core maps, and the output feature map set 106 may include 128 feature maps. In some embodiments, operation 304 may include zero padding. Each feature map in feature map set 106 may be 256x256. In some embodiments, operation 304 may include batch normalization and rectified linear unit function processing. Operations 303 and 304 may be viewed as a set of convolution operations.

Operation 601 is performed on the feature map group 106 to output the feature map group 107 . Operation 601 may be an up-sampling process. Operation 601 may be a max unpoolling process. In some embodiments, operation 601 may be 2×2 max-unpooling, and the output feature map group 107 may include 128 feature maps, wherein each feature map may be 512×512. In some embodiments, operation 601 may be performed according to the downsampling index (or pooling index) passed in operation 501 . For example, when performing the (2×2) maximum pooling in operation 401, the data of the maximum pooling index can be stored (for example, operation 501), and when performing operation 601, a (2×2) maximum inversion can be performed according to the maximum pooling index. Pooling (eg filling back the position indicated by the index).

The downsampling index (such as max pooling indices; max pooling indices) is obtained from the downsampling of a resolution, and the value can be filled back to the same value according to the downsampling index when upsampling (such as max unpooling) back to the same resolution Location. This de-pooling can replace deconvolution, which can shorten the required training time, reduce the required training data, and reduce a large number of operations. In addition, performing de-pooling according to the index can reduce the loss of high-frequency data, can provide better segmentation of tissue boundaries, and can increase the location accuracy of small lesions (lesions), and can provide better segmentation of tissue boundaries.

Operation 701 is performed on the feature map set 103 and the feature map set 107 to output the feature map set 108 . Operation 701 may be a skip connection process. For example, the feature map group 103 is stored before the pooling process in operation 401 is performed, and the feature map group 103 is connected with the feature map group 107 generated after performing the de-pooling in operation 601 to generate the feature map group 108 . According to the exemplary embodiment of FIG. 3B , the number of feature maps in the feature map group 108 is 64 plus 128, ie, 192.

Through skip connection processing, feature maps can be extended to combine multi-scale image information. Skip connection processing reduces the loss of high-frequency data and provides better segmentation of tissue boundaries.

Operation 305 is performed on the feature map set 108 to output the feature map set 109 . Operation 305 may be a convolution operation with a plurality of core graphs. In some embodiments, operation 305 may be a convolution operation with 64 core maps, and the output feature map group 109 may include 64 feature maps. In some embodiments, operation 305 may include zero padding. Each feature map in feature map set 109 may be 512x512. In some embodiments, operation 305 may include batch normalization and rectified linear unit function processing.

Operation 306 is performed on the feature map set 109 to output the feature map set 110 . Operation 306 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 306 may be a convolution operation with 64 core maps, and the output feature map set 110 may include 64 feature maps. In some embodiments, operation 306 may include zero padding. Each feature map in feature map set 110 may be 512x512. In some embodiments, operation 306 may include batch normalization and rectified linear unit function processing. Operations 305 and 306 may be viewed as a set of convolution operations.

Operation 307 is performed on the set of feature maps 110 to output a set of output images. Operation 307 may be to perform output convolution. In some embodiments, operation 307 may include zero padding. Operation 307 may be a convolution operation with the plurality of core graphs. Operation 307 may be a convolution operation with at least two core graphs. In some embodiments, operation 307 may be a convolution operation with the six core maps to generate output images 201 , 203 , 205 , 207 , 209 , 211 . Output image 201 may indicate gray matter; output image 203 may indicate white matter; output image 205 may indicate white matter hyperintensity; output image 207 may indicate cerebrospinal fluid; output image 209 may indicate scalp and/or skull; output image 211 may indicate Indicates air.

In some embodiments, operation 307 may be a convolution operation with 2 core maps to generate two output images, one output image may be indicative of white matter and one output image may be indicative of white matter hyperintensity.

In some embodiments, operation 307 may be a convolution operation with three core maps to generate three output images, one output image may be indicative of gray matter, one output image may be indicative of white matter, and one output image may be indicative of white matter high signal.

After the input image 101 is processed by the image processing program 300 , two or more output images can be output to generate multi-category semantic segmentation for the input image 101 . A single class of semantic segmentation may be processed by an image processing program to output only one output image, and the output image may be indicative of white matter hyperintensity. Compared with single-category semantic segmentation, multi-category semantic segmentation can achieve better tissue morphological discrimination and reduce false-positive lesion determination.

FIG. 4 illustrates an image processing procedure 300 according to some embodiments of the present disclosure. In FIG. 4, the technical contents of operation 301, operation 302, operation 401, operation 303, operation 304, input image 101, feature map group 102, feature map group 103, feature map group 104, feature map group 105, feature map group 106 and The operations using the same reference symbols in FIG. 3B, the technical contents of the input images or feature map groups are similar or identical. Operation 301, operation 302, operation 401, operation 303, operation 304, input image 101, feature map group 102, feature map group 103, feature map group 104, feature map group 105, and feature map in FIG. Group 106 repeats the description.

Referring to FIG. 4 , operation 402 is performed on the feature map group 106 including 128 feature maps to output the feature map group 111 . Operation 402 may be a downsampling process. Operation 402 may be a max pooling process. In some embodiments, operation 402 may be 2x2 max pooling, and the output feature map group 111 may include 128 feature maps, wherein each feature map may be 128x128.

Operation 308 is performed on the feature map set 111 to output the feature map set 112 . Operation 308 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 308 may be a convolution operation with 128 core maps, and the output feature map set 112 may include 128 feature maps. In some embodiments, operation 308 may include zero padding. Each feature map in feature map set 112 may be 128x128. In some embodiments, operation 308 may include batch normalization and rectified linear unit function processing.

Operation 309 is performed on the feature map set 112 to output the feature map set 113 . Operation 309 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 309 may be a convolution operation with 256 core maps, and the output feature map group 113 may include 256 feature maps. In some embodiments, operation 309 may include zero padding. Each feature map in feature map set 113 may be 128x128. In some embodiments, operation 309 may include batch normalization and rectified linear unit function processing. Operations 308 and 309 may be viewed as a set of convolution operations.

Operation 403 is performed on the feature map group 113 including 256 feature maps to output the feature map group 114 . Operation 403 may be a downsampling process. Operation 403 may be a max pooling process. In some embodiments, operation 403 may be 2x2 max pooling, and the output feature map group 114 may include 256 feature maps, wherein each feature map may be 64x64.

Operation 310 is performed on the set of feature maps 114 to output the set of feature maps 115 . Operation 310 may be a convolution operation with a plurality of core graphs. In some embodiments, operation 310 may be a convolution operation with 256 core maps, and the output feature map group 115 may include 256 feature maps. In some embodiments, operation 310 may include zero padding. Each feature map in feature map set 115 may be 64x64. In some embodiments, operation 310 may include batch normalization and rectified linear unit function processing.

Operation 311 is performed on the feature map set 115 to output the feature map set 116 . Operation 311 may be a convolution operation with a plurality of core graphs. In some embodiments, operation 311 may be a convolution operation with 512 core maps, and the output feature map group 116 may include 512 feature maps. In some embodiments, operation 311 may include zero padding. Each feature map in feature map set 116 may be 64x64. In some embodiments, operation 311 may include batch normalization and rectified linear unit function processing. Operation 310 and operation 311 may be viewed as a set of convolution operations.

Operation 602 is performed on the feature map set 116 to output the feature map set 117 . Operation 602 may be an upsampling process. Operation 602 may be a max unpooling process. In some embodiments, operation 602 may be 2x2 max unpooling, and the output feature map set 117 may include 512 feature maps, where each feature map may be 128x128. In some embodiments, operation 602 may be performed according to the downsampling index (or pooling index) passed in operation 502 . For example, when performing the (2×2) max pooling in operation 403 , the data of the max pooling index can be stored (for example, in operation 502 ), and when performing operation 602 , the (2×2) max inversion can be performed according to the max pooling index. Pooling (eg filling back the position indicated by the index).

The downsampling index (such as max pooling indices; max pooling indices) is obtained from the downsampling of a resolution, and the value can be filled back to the same value according to the downsampling index when upsampling (such as max unpooling) back to the same resolution Location. This de-pooling can replace deconvolution, which can shorten the required training time, reduce the required training data, and reduce a large number of operations. In addition, performing de-pooling according to the index can reduce the loss of high-frequency data, can provide better segmentation of tissue boundaries, and can increase the location accuracy of small lesions (lesions), and can provide better segmentation of tissue boundaries.

Operation 702 is performed on the feature map set 113 and the feature map set 117 to output the feature map set 118 . Operation 702 may be skip connection processing. For example, the feature map group 113 is stored before the pooling process in operation 403 is performed, and the feature map group 113 is connected with the feature map group 117 generated after performing the de-pooling in operation 602 to generate the feature map 118 . According to the exemplary embodiment of FIG. 4 , the number of feature maps in the feature map group 118 is 256 plus 512, ie, 768.

Through skip connection processing, feature maps can be extended to combine multi-scale image information. Skip connection processing reduces the loss of high-frequency data and provides better segmentation of tissue boundaries.

Operation 312 is performed on feature map set 118 to output feature map set 119 . Operation 312 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 312 may be a convolution operation with 256 core maps, and the output feature map group 119 may include 256 feature maps. In some embodiments, operation 312 may include zero padding. Each feature map in feature map set 119 may be 128x128. In some embodiments, operation 312 may include batch normalization and rectified linear unit function processing.

Operation 313 is performed on the feature map set 119 to output the feature map set 120 . Operation 313 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 313 may be a convolution operation with 256 core maps, and the output feature map group 120 may include 256 feature maps. In some embodiments, operation 313 may include zero padding. Each feature map in feature map set 120 may be 128x128. In some embodiments, operation 313 may include batch normalization and rectified linear unit function processing. Operation 312 and operation 313 may be viewed as a set of convolution operations.

Operation 603 is performed on the feature map group 120 to output the feature map group 121 . Operation 603 may be an upsampling process. Operation 603 may be a max unpooling process. In some embodiments, operation 603 may be 2×2 max-unpooling, and the output feature map group 121 may include 256 feature maps, wherein each feature map may be 256×256. In some embodiments, operation 603 may be performed according to the downsampling index (or pooling index) passed in operation 503 . For example, when performing the (2×2) maximum pooling in operation 402, the data of the maximum pooling index can be stored (for example, in operation 503), and when performing operation 603, a (2×2) maximum inversion can be performed according to the maximum pooling index. Pooling (eg filling back the position indicated by the index).

Operation 703 is performed on the feature map set 106 and the feature map set 121 to output the feature map set 122 . Operation 703 may be skip connection processing. For example, the feature map group 106 is stored before the pooling process in operation 402 is performed, and the feature map group 106 is connected with the feature map group 121 generated after performing the de-pooling in operation 603 to generate the feature map 122 . According to the exemplary embodiment of FIG. 4 , the number of feature maps in the feature map group 122 is 128 plus 256, ie, 384.

Operation 314 is performed on the feature map set 122 to output the feature map set 123 . Operation 314 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 314 may be a convolution operation with 128 core maps, and the output feature map set 123 may include 128 feature maps. In some embodiments, operation 314 may include zero padding. Each feature map in feature map set 123 may be 256x256. In some embodiments, operation 314 may include batch normalization and rectified linear unit function processing.

Operation 315 is performed on the feature map set 123 to output the feature map set 124 . Operation 315 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 315 may be a convolution operation with 128 core maps, and the output feature map group 124 may include 128 feature maps. In some embodiments, operation 315 may include zero padding. Each feature map in feature map set 124 may be 256x256. In some embodiments, operation 315 may include batch normalization and rectified linear unit function processing. Operations 314 and 315 may be viewed as a set of convolution operations.

Operation 601 is performed on the feature map set 124 to output the feature map set 107 . Operation 601 may be an upsampling process. Operation 601 may be a max unpooling process. In some embodiments, operation 601 may be 2×2 max-unpooling, and the output feature map group 107 may include 128 feature maps, wherein each feature map may be 512×512. In some embodiments, operation 601 may be performed according to the downsampling index (or pooling index) passed in operation 501 . For example, when performing the (2×2) maximum pooling in operation 401, the data of the maximum pooling index can be stored (for example, operation 501), and when performing operation 601, a (2×2) maximum inversion can be performed according to the maximum pooling index. Pooling (eg filling back the position indicated by the index).

Operation 701 is performed on the feature map set 103 and the feature map set 107 to output the feature map set 108 . Operation 701 may be skip connection processing. For example, the feature map group 103 is stored before the pooling process in operation 401 is performed, and the feature map group 103 is connected with the feature map group 107 generated after performing the de-pooling in operation 601 to generate the feature map 108 . According to the exemplary embodiment of FIG. 4 , the number of feature maps in the feature map group 108 is 64 plus 128, ie, 192.

Operation 305 is performed on the feature map set 108 to output the feature map set 109 . Operation 305 may be a convolution operation with a plurality of core graphs. In some embodiments, operation 305 may be a convolution operation with 64 core maps, and the output feature map group 109 may include 64 feature maps. In some embodiments, operation 305 may include zero padding. Each feature map in feature map set 109 may be 512x512. In some embodiments, operation 305 may include batch normalization and rectified linear unit function processing.

Operation 306 is performed on the feature map set 109 to output the feature map set 110 . Operation 306 may be a convolution operation with the plurality of core graphs. In some embodiments, operation 306 may be a convolution operation with 64 core maps, and the output feature map set 110 may include 64 feature maps. In some embodiments, operation 306 may include zero padding. Each feature map in feature map set 110 may be 512x512. In some embodiments, operation 306 may include batch normalization and rectified linear unit function processing. Operations 305 and 306 may be viewed as a set of convolution operations.

Operation 307 is performed on the set of feature maps 110 to output a set of output images. Operation 307 may be to perform output convolution. In some embodiments, operation 306 may include zero padding. Operation 307 may be a convolution operation with the plurality of core graphs. Operation 307 may be a convolution operation with at least two core graphs. In some embodiments, operation 307 may be a convolution operation with the six core maps to generate output images 201 , 203 , 205 , 207 , 209 , 211 . Output image 201 may indicate gray matter; output image 203 may indicate white matter; output image 205 may indicate white matter hyperintensity; output image 207 may indicate cerebrospinal fluid; output image 209 may indicate scalp and/or skull; output image 211 may indicate Indicates air.

In some embodiments, operation 307 may be a convolution operation with 2 core maps to generate two output images, one output image may be indicative of white matter and one output image may be indicative of white matter hyperintensity.

In some embodiments, operation 307 may be a convolution operation with three core maps to generate three output images, one output image may be indicative of gray matter, one output image may be indicative of white matter, and one output image may be indicative of white matter high signal.

After the input image 101 is processed by the image processing program 300 , two or more output images can be output to generate multi-category semantic segmentation for the input image 101 . A single class of semantic segmentation may be processed by an image processing program to output only one output image, and the output image may be indicative of white matter hyperintensity. Compared with single-category semantic segmentation, single-category semantic segmentation can achieve better tissue morphological discrimination and reduce false positive lesion determination.

3A, 3B and 4 of the present disclosure are exemplary embodiments of the image processing program 300, but these exemplary implementations are not intended to limit the present invention. Those skilled in the art should understand that the image size, feature map size, number of feature maps, number of core maps, number of convolution operations, number of down-sampling, up-sampling The number of times, the number of skip connections, and the number of pooled index passes can vary.

5A-5E show brain MRI 2D images according to some embodiments of the present disclosure. 5A-5E show brain T2-FLAIR images according to some embodiments of the present disclosure. Each of FIGS. 5A-5E may be the input image 101 shown in FIG. 1 , FIG. 3A , FIG. 3B or FIG. 4 for the present disclosure.

6A-6E show brain MRI 2D images according to some embodiments of the present disclosure. 6A-6E show brain T2-FLAIR images according to some embodiments of the present disclosure. Each of FIGS. 6A-6E includes a region circled as white matter hyperintensity.

The green circled portions in each of FIGS. 6A-6E indicate areas of white matter hyperintensity circled only by the encephalologist. The red circled portions in each of FIGS. 6A-6E indicate white matter hyperintense regions only circled by the image processing program 300 (as shown in FIG. 1 , FIG. 3A , FIG. 3B or FIG. 4 ). The yellow circled portions in each of FIGS. 6A-6E indicate white matter elevations circled by both the neurologist and the image processing program 300 (as shown in FIGS. 1 , 3A, 3B, or 4 ) signal area.

The green circled portion in each of FIGS. 6A-6E may indicate a false negative result of the image processing routine 300 . The red circled portion in each of FIGS. 6A-6E may indicate a false positive result of the image processing procedure 300 . The yellow circled portion in each of FIGS. 6A-6E may indicate a true positive result for the image processing routine 300 .

Referring to FIG. 6A to FIG. 6E , most of the parts circled in yellow, that is, the white matter hyperintensity region circled by the image processing program 300 substantially overlaps the white matter hyperintensity region circled by the neurologist. In other words, the ability of the image processing program 300 to circle the white matter hyperintensity regions is approximately the same as the ability of the neurologist to circle the white matter hyperintensity regions.

Referring to FIG. 6A to FIG. 6E , the green circled parts are rare, which means that false negative results are rare. Referring to FIG. 6A to FIG. 6E , the red circled part is rare, which means that false positive results are rare.

7A and 7B show statistical results of a single class output U-Net according to the present disclosure. Single-class output may refer to a single-class semantic segmentation of an image. The single-class output U-Net can be an image processing procedure that does not perform operation 501 in FIG. 3B and uses only one core map in operation 307 (output convolution), so only one output image is output, and the output image indicates white matter hyperintensity area. A single-class U-Net can be an image processing procedure in FIG. 4 that does not perform operations 501, 502, and 503, and uses only one core map in operation 307 (output convolution), so only one output image is output, and the output image is Indicates areas of white matter hyperintensity. A single-class U-Net can be an image processing procedure that does not perform pooling index transfer as in Figure 3B or Figure 4, and uses only one core image in the output convolution, so only one output image is output, and the output image indicates white matter hyperintensity area.

7A is a regression analysis diagram of the actual white matter lesion area (or actual white matter hyperintensity area) and the predicted white matter lesion area (or predicted white matter hyperintensity area) of the single-class output U-Net. In FIG. 7A , the X-axis is the actual white matter lesion area, and the Y-axis is the predicted white matter lesion area of the single-category output U-Net, and the unit is cm ² .

According to the regression analysis graph shown in Figure 7A, the r value is 0.996, which means that 99.6% of the variation in the Y-axis value (ie, the predicted white matter lesion area) can be explained by the variation in the X-axis value (ie, the actual white matter lesion area). According to the regression analysis graph shown in FIG. 7A , the p value approaches 0, which means that the relationship between the Y-axis value (ie, the predicted white matter lesion area) and the X-axis value (ie, the actual white matter lesion area) is significant. According to the regression analysis diagram shown in FIG. 7A , the regression line equation is y=1.074x+0.036.

Figure 7B is a Bland-Altman plot. In Figure 7B, the X-axis is the average of the actual white matter lesion area and the area of the white matter lesion predicted by the single-category output U-Net, and the Y-axis is the actual white matter lesion area minus the white matter lesion area predicted by the single-category output U-Net. , and its equivalent unit is cm ² .

According to Fig. 7B, the deviation value (Bias) between the actual white matter lesion area and the single-class output U-Net predicted white matter lesion area is -0.206. According to Figure 7B, the positive 1.96 standard deviation (+1.96SD) of the distribution of the actual white matter lesion area minus the single-class output U-Net predicted white matter lesion area is 0.624, and the negative 1.96 standard deviation (-1.96SD) is -1.036. The 95% limits of agreement for the actual white matter lesion area minus the single category output U-Net predicted white matter lesion area were 0.624 and -1.036.

8A and 8B show statistical results of a single class output U-SegNet according to the present disclosure. Single-class output may refer to a single-class semantic segmentation of an image. The single-class output U-SegNet may be the image processing procedure in FIG. 3B that uses only one core map in operation 307 (output convolution), so only one output image is output, and the output image indicates white matter hyperintense regions. A single-class U-SegNet may be the image processing procedure in FIG. 4 that uses only one core map at operation 307 (output convolution), so only one output image is output, and the output image indicates white matter hyperintense regions. A single-class U-SegNet can be the image processing procedure of Figure 3B or Figure 4 that uses only one core map in the output convolution, so only one output image is output, and the output image indicates white matter hyperintense regions.

8A is a regression analysis diagram of the actual white matter lesion area (or actual white matter hyperintensity area) and the predicted white matter lesion area (or predicted white matter hyperintensity area) of the single-category output U-SegNet. In FIG. 8A , the X-axis is the actual white matter lesion area, and the Y-axis is the predicted white matter lesion area by the single-category output U-SegNet, and the unit is cm ² .

According to the regression analysis graph shown in Figure 8A, the r value is 0.995, which means that 99.5% of the variation in the Y-axis value (ie, the predicted white matter lesion area) can be explained by the variation in the X-axis value (ie, the actual white matter lesion area). According to the regression analysis graph shown in FIG. 8A , the p value approaches 0, which means that the relationship between the Y-axis value (ie, the predicted white matter lesion area) and the X-axis value (ie, the actual white matter lesion area) is significant. According to the regression analysis diagram shown in FIG. 8A , the regression line equation is y=1.077x+0.037.

Figure 8B is a Bland-Altman plot. In Figure 7B, the X-axis is the average of the actual white matter lesion area and the area of the white matter lesion predicted by the single-category output U-SegNet, and the Y-axis is the actual white matter lesion area minus the single-category output U-SegNet predicted white matter lesion area , and its equivalent unit is cm ² .

According to Fig. 8B, the deviation value (Bias) between the actual white matter lesion area and the single-class output U-SegNet predicted white matter lesion area is -0.213. According to Figure 8B, the positive 1.96 standard deviation (+1.96SD) of the distribution of the actual white matter lesion area minus the single-class output U-Net predicted white matter lesion area is 0.685, and the negative 1.96 standard deviation (-1.96SD) is -1.111. The 95% limits of agreement for the actual white matter lesion area minus the single-class output U-SegNet predicted white matter lesion area were 0.685 and -1.111.

9A and 9B show statistical results of the multi-class output U-Net according to the present disclosure. Multi-class output may refer to multi-class semantic segmentation of images. The multi-class output U-Net can be an image processing procedure that does not perform operation 501 in FIG. 3B and uses multiple core maps in operation 307 (output convolution), so multiple output images can be output, and the multiple output images are Each may be indicative of gray matter, white matter, white matter hyperintense regions, cerebrospinal fluid, scalp and/or skull, and air (eg, output images 201, 203, 205, 207, 209, and 201), respectively. The multi-class U-Net can be an image processing program that does not perform operations 501, 502, and 503 in FIG. 4, and uses multiple core images in operation 307 (output convolution), so multiple output images can be output, and the Each of the multiple output images may be indicative of gray matter, white matter, white matter hyperintense regions, cerebrospinal fluid, scalp and/or skull, and air, respectively (eg, output images 201, 203, 205, 207, 209, and 201) . The multi-class U-Net can be an image processing program that does not perform pooling index transfer in Figure 3B or Figure 4, and uses multiple core images in the output convolution, so multiple output images can be output, and the multiple output images Each may be indicative of gray matter, white matter, white matter hyperintense regions, cerebrospinal fluid, scalp and/or skull, and air (eg, output images 201, 203, 205, 207, 209, and 201), respectively.

9A is a regression analysis diagram of the actual white matter lesion area (or actual white matter hyperintensity area) and the predicted white matter lesion area (or predicted white matter hyperintensity area) of the multi-class output U-Net. In FIG. 9A , the X-axis is the actual white matter lesion area, and the Y-axis is the predicted white matter lesion area by the multi-class output U-Net, and the unit is cm ² .

According to the regression analysis graph shown in Figure 9A, the r value is 0.995, which means that 99.5% of the variation in the Y-axis value (ie, the predicted white matter lesion area) can be explained by the variation in the X-axis value (ie, the actual white matter lesion area). According to the regression analysis graph shown in FIG. 9A , the p value approaches 0, which means that the relationship between the Y-axis value (ie, the predicted white matter lesion area) and the X-axis value (ie, the actual white matter lesion area) is significant. According to the regression analysis diagram shown in FIG. 9A , the regression line equation is y=1.046x+0.045.

Figure 9B is a Bland-Altman plot. In Figure 9B, the X-axis is the average value of the actual white matter lesion area and the area of the white matter lesion predicted by the multi-category output U-Net, and the Y-axis is the actual white matter lesion area minus the white matter lesion area predicted by the multi-category output U-Net. , and its equivalent unit is cm ² .

According to Fig. 9B, the deviation value (Bias) between the actual white matter lesion area and the predicted white matter lesion area of the single-class output U-Net is -0.150. According to Fig. 9B, the positive 1.96 standard deviation (+1.96SD) of the distribution of the actual white matter lesion area minus the predicted white matter lesion area of the multi-class output U-Net is 0.600, and the negative 1.96 standard deviation (-1.96SD) is -0.901. The 95% limits of agreement for the actual white matter lesion area minus the single category output U-Net predicted white matter lesion area were 0.600 and -0.901.

10A and 10B show statistical results of multi-class output U-SegNet according to the present disclosure. Multi-class output may refer to multi-class semantic segmentation of images. The multi-class output U-SegNet can be the image processing program of FIG. 3B, so it can output multiple output images, and each of the multiple output images can respectively indicate the gray matter, the white matter, the white matter hyperintensity area, and the cerebrospinal cord. Fluid, scalp and/or skull, and air (eg, output images 201, 203, 205, 207, 209, and 201). The multi-class U-SegNet can be the image processing program of FIG. 4, so it can output multiple output images, and each of the multiple output images can respectively indicate gray matter, white matter, white matter hyperintensity area, cerebrospinal fluid , scalp and/or skull, and air (eg, output images 201, 203, 205, 207, 209, and 201).

10A is a regression analysis diagram of the actual white matter lesion area (or actual white matter hyperintensity area) and the predicted white matter lesion area (or predicted white matter hyperintensity area) of the multi-class output U-SegNet. In FIG. 9A , the X-axis is the actual white matter lesion area, and the Y-axis is the predicted white matter lesion area by the multi-class output U-SegNet, and the unit is cm ² .

According to the regression analysis graph shown in Figure 10A, the r value is 0.996, which means that 99.6% of the variation in the Y-axis value (ie, the predicted white matter lesion area) can be explained by the variation in the X-axis value (ie, the actual white matter lesion area). According to the regression analysis graph shown in FIG. 10A , the p value approaches 0, which means that the relationship between the Y-axis value (ie, the predicted white matter lesion area) and the X-axis value (ie, the actual white matter lesion area) is significant. According to the regression analysis diagram shown in FIG. 10A , the regression line equation is y=1.000x+0.063, which is close to y=x.

Figure 10B is a Bland-Altman plot. In Figure 10B, the X-axis is the average of the actual white matter lesion area and the area of the white matter lesion predicted by the multi-category output U-SegNet, and the Y-axis is the actual white matter lesion area minus the area of the white matter lesion predicted by the multi-category output U-SegNet. , and its equivalent unit is cm ² .

According to Fig. 10B, the deviation value (Bias) between the actual white matter lesion area and the predicted white matter lesion area of the single-class output U-SegNet is -0.206. According to Figure 10B, the positive 1.96 standard deviation (+1.96SD) of the distribution of the actual white matter lesion area minus the multi-class output U-SegNet predicted white matter lesion area is 0.624, and the negative 1.96 standard deviation (-1.96SD) is -1.036. The 95% limits of agreement for the actual white matter lesion area minus the single category output U-Net predicted white matter lesion area were 0.624 and -1.036.

11A-11E show brain MRI 2D images according to some embodiments of the present disclosure. 11A-11E show brain T2-FLAIR images according to some embodiments of the present disclosure. FIG. 11A may be an input image for input to an image processing program. FIG. 11A can be the input image 101 in FIG. 1 , FIG. 3A , FIG. 3B or FIG. 4 of the present disclosure. Each of FIGS. 11B-11E includes a region circled as white matter hyperintensity. Each of FIGS. 11B-11E may indicate white matter hyperintensity identification results based on FIG. 11A .

The green circled portions in each of FIGS. 11B-11E indicate white matter hyperintensity regions circled only by the neurologist. The red circled portions in each of FIGS. 11B-11E indicate white matter hyperintense regions only circled by the image processing program. The yellow circled portions in each of FIGS. 11B-11E indicate areas of white matter hyperintensity circled by both the neurologist and the image processing program.

The green circled portion in each of FIGS. 11B-11E may indicate a false negative result of the image processing procedure. The red circled portion in each of FIGS. 11B-11E may indicate a false positive result of the image processing procedure. The yellow circled portion in each of FIGS. 11B-11E may indicate a true positive result of the image processing procedure.

Figure 11B shows the recognition results that may include a single class of U-Net image processing procedures. Arrows in Figure 11B indicate false positive results (ie, the red circled portion). Pseudo-negative results (ie, the green circled portion) are also included in FIG. 11B.

Figure 11C shows the recognition results that may include a single class of U-SegNet image processing procedures. The two arrows in Figure 11C indicate false positive results (ie, the red circled portion).

FIG. 11D shows the recognition results that may include multi-class U-Net image processing procedures. Arrows in Figure 11D indicate false positive results (ie, the red circled portion). Compared to FIG. 11B , FIG. 11D does not include false negative results (ie, the portion circled in green).

FIG. 11E shows the recognition result which may include multiple types of U-SegNet image processing programs (ie, the image processing programs shown in FIG. 3B or FIG. 4 ). Figure 11E does not contain red circled parts or green circled parts. False positive or false negative results are not included in Figure 11E.

Compared with the recognition result of the single-type U-Net image processing program shown in FIG. 11B , the recognition result of the multi-type U-SegNet image processing program shown in FIG. 11E (ie, the image processing program shown in FIG. 3B or FIG. 4 ) is better . Compared with the recognition result of the single-type U-SegNet image processing program shown in FIG. 11C , the recognition result of the multi-type U-SegNet image processing program shown in FIG. 11E (ie, the image processing program shown in FIG. 3B or FIG. 4 ) is better . Compared with the recognition result of the multi-class U-Net image processing program shown in FIG. 11D , the recognition result of the multi-class U-SegNet image processing program shown in FIG. 11E (ie, the image processing program shown in FIG. 3B or FIG. 4 ) is better .

12A-12E show brain MRI 2D images according to some embodiments of the present disclosure. 12A-12E show brain T2-FLAIR images according to some embodiments of the present disclosure. Figure 12A may be an input image for input to an image processing program. FIG. 12A may be the input image 101 shown in FIG. 1 , FIG. 3B or FIG. 4 of the present disclosure. Each of Figures 12B-12E includes a region circled as white matter hyperintensity. Each of FIGS. 12B-12E may indicate white matter hyperintensity identification results based on FIG. 11A .

The green circled portions in each of Figures 12B-12E indicate white matter hyperintensity regions circled only by the neurologist. The red circled portions in each of Figures 12B-12E indicate white matter hyperintense regions only circled by the image processing program. The yellow circled portions in each of Figures 12B-12E indicate white matter hyperintense regions circled by both the neurologist and the image processing program.

The green circled portion in each of Figures 12B-12E may indicate a false negative result of the image processing procedure. The red circled portion in each of Figures 12B-12E may indicate a false positive result of the image processing procedure. The yellow circled portion in each of Figures 12B-12E may indicate a true positive result of the image processing procedure.

Figure 12B shows the recognition results that may include a single class of U-Net image processing procedures. The three arrows in Figure 12B indicate false positive results (ie, the red circled portion). Figure 12B also includes false negative results (ie, the green circled portion).

Figure 12C shows the recognition results that may include a single class of U-SegNet image processing procedures. The three arrows in Figure 12C indicate false positive results (ie, the red circled portion). Pseudo-negative results (ie, the green circled portion) are also included in Figure 12C.

FIG. 12D shows the recognition result of the U-Net image processing program which may include multiple classes. The two arrows in Figure 12D indicate false positive results (ie, the red circled portion). Pseudo-negative results (ie, the green circled portion) are also included in Figure 12D. The red circled portion of Fig. 12D is less than the red circled portion of Fig. 12B. The false positive results of Figure 12D are less than the false positive results of Figure 12B. The green circled portion of Fig. 12D is less than the green circled portion of Fig. 12B. The false negative results of Figure 12D are less than the false negative results of Figure 12B.

FIG. 12E shows the recognition result that may include multiple types of U-SegNet image processing programs (ie, the image processing programs shown in FIG. 3B or FIG. 4 ). The red circled portion is not included in Figure 12E. False positive results are not included in Figure 11E. The green circled portion of Fig. 12E is less than the green circled portion of Fig. 12C. The false negative results of Figure 12E are less than the false negative results of Figure 12C.

Compared with the recognition result of the single-type U-Net image processing program shown in FIG. 12B , the recognition result of the multi-type U-SegNet image processing program shown in FIG. 12E (ie, the image processing program shown in FIG. 3B or FIG. 4 ) is better . Compared with the recognition result of the single-type U-SegNet image processing program shown in FIG. 12C , the recognition result of the multi-type U-SegNet image processing program shown in FIG. 12E (ie, the image processing program shown in FIG. 3B or FIG. 4 ) is better . Compared with the recognition result of the multi-class U-Net image processing program shown in FIG. 12D , the recognition result of the multi-class U-SegNet image processing program shown in FIG. 12E (ie, the image processing program shown in FIG. 3B or FIG. 4 ) is better .

Based on True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN), the machine learning results are evaluated. Quantitative indicators include the following: Accuracy = (tp+tn)/(tp+fp+fn+tn); Precision = tp/(tp+fp) (that is, how many of the results predicted to be positive are correct); Recall rate (Recall) = tp/(tp+fn) (that is, several of the actual positive samples are predicted correctly); F1 score = 2/((1/Precision)+(1/Recall)) (that is, the harmonic mean of precision and recall); Sensitivity = TP/(TP+FN) (same as recall); and Specificity=TN/(FP+TN).

Regarding the single-class output U-Net (the image processing program used in Figure 7A, Figure 7B, Figure 11B, Figure 12B), the F1 score was 88.19%; the sensitivity was 90.61%; the specificity was 99.96%; and the accuracy was 99.93% .

Regarding the single-class output U-SegNet (the image processing program used in Figure 8A, Figure 8B, Figure 11C, Figure 12C), the F1 score was 87.75%; the sensitivity was 90.40%; the specificity was 99.95%; and the accuracy was 99.93% .

Regarding the multi-class output U-Net (image processing program used in Figure 9A, Figure 9B, Figure 11D, Figure 12D), the F1 score was 88.73%; the sensitivity was 92.21%; the specificity was 99.95%; the accuracy was 99.93% .

Regarding the multi-class output U-SegNet (the image processing program used in Figure 10A, Figure 10B, Figure 11E, Figure 12E), the F1 score was 90.01%; the sensitivity was 94.04%; the specificity was 99.95%; the accuracy was 99.94% . The F1 score of the multi-class output U-SegNet is higher than the F1 score of the single-class output U-Net, the F1 score of the single-class output U-SegNet and the F1 score of the multi-class output U-Net.

FIG. 13 shows a confusion matrix of the classification result 900 output from the classification process 800 of FIG. 2 . Figure 13 shows the confusion matrix of the binning result 900 output from XGBoost. The input data of XGBoost includes gray matter volume 231 , white matter volume 233 , white matter hyperintensity volume 235 , cerebrospinal fluid volume 237 , and image parameters 239 (eg, image slice thickness). In some embodiments, gray matter volume 231 , white matter volume 233 , white matter hyperintensity volume 235 , cerebrospinal fluid volume 237 are based on inputting a series of input images 101 to the image processing program 300 of FIG. 3B or FIG. 4 of the present disclosure (such as the multi-class U-SegNet image processing program). In some embodiments, the image parameter 239 is the image slice thickness of the series of input images 101 .

Referring to FIG. 13 , the X-axis of the confusion matrix may be the Fezex level predicted by XGBoost, and the Y-axis of the confusion matrix may be the actual Fazex level. When the actual Fezex level is 1, the predicted probability of a Fezex level of 1 is 0.8294, the predicted probability of a Fezex level of 2 is 0.1706, and the predicted probability of a Fezex level of 3 is 0. When the predicted Fezex level is 1, the probability of the actual Fezex level is 0.8294, the predicted Fezex level is 0.1412, and the predicted Fezex level is 0. The probability is 0.

When the actual Fezex level is 2, the predicted probability of Fezex level 2 is 0.7406; when the actual Fezex level is 3, the predicted probability of Fezex level 3 is 0.8254.

According to the gray matter volume 231 , the white matter volume 233 , the white matter hyperintensity volume 235 , the cerebrospinal fluid volume obtained by the image processing program 300 (eg, the multi-class U-SegNet image processing program) as shown in FIG. 3B or FIG. 4 of the present disclosure 237 for grading, the probability of correct prediction of Fazex grade is high.

14 illustrates a system 1400 according to some embodiments of the present disclosure. The system 1400 includes a client 1410 , a database 1420 and a server 1430 . The client 1410 includes a processor 1411 and a memory 1412, and the memory 1412 can store the programs or operations described in the present disclosure for the processor 1411 to be stored. The database 1420 includes a processor 1421 and a memory 1422. The memory 1422 can store the programs or operations described in the present disclosure for the processor 1421 to be stored. The server side 1430 includes a processor 1431 and a memory 1432. The memory 1432 can store the programs or operations described in the present disclosure for the processor 1431 to be stored.

In some embodiments, the client 1410 can access data from the database 1420 . For example, client 1410 may access images from database 1420 for use by a user (eg, a specialist). The client 1410 can access the brain MRI 2D image or the brain T2-FLAIR image from the database 1420 for the user to use. The client 1410 may send a request to the database 1420 so that the database 1420 transmits one or more images selected by the client 1410 to the server 1430 for image recognition and/or classification.

After the server 1430 receives the image recognition request and the associated one or more images sent from the database 1420, the server 1430 uses each of the one or more images as the input image 101 (as shown in FIG. 1 ). , FIG. 3B or FIG. 4 ) and input to the image processing program 300 (as shown in FIG. 1 , FIG. 3B or FIG. 4 ) for image recognition.

After being processed by the image processing program 300, corresponding to the one or more images received from the database 1420, the server 1430 generates one or more output images 201, one or more output images 203, one or more output images Image 205 , one or more output images 207 , one or more output images 209 , one or more output images 211 .

In some embodiments, the server 1430 may further grade one or more images transmitted by the database 1420 according to the request transmitted by the database 1420 . The server 1430 can calculate the volume of the tissue according to the one or more output images 201 , one or more output images 203 , one or more output images 205 , and one or more output images 207 output by the image processing program 300 to generate Volume 231, Volume 233, Volume 235, and Volume 237. The server side 1430 inputs the volume 231 , the volume 233 , the volume 235 and the volume 237 to the classification process 800 . In some embodiments, in addition to volume 231 , volume 233 , volume 235 , and volume 237 , server 1430 also inputs image parameters 239 to grading process 800 . Image parameter 230 may be an image slice thickness for one or more images received from database 1420 . The grading program 800 outputs a grading result 900 according to the input data.

The server side 1430 can be based on one or more output images 201, one or more output images 203, one or more output images 205, one or more output images 207, one or more output images 209, one or more output images The output image 211 generates lesion edge coordinates and lesion volume. The server 1430 can generate a rating report according to the rating result 900 . The server 1430 can transmit the lesion edge coordinates, the lesion volume and the grading report to the database 1420 or the client 1410 . The server side can transmit the lesion edge coordinates, lesion volume and grading report to the client terminal 1410 through the database 1420 .

The database 1420 can circle the lesions on the one or more corresponding images according to the lesion edge coordinates, and transmit the images including the circled lesions to the user terminal 1410 for the user to use. The user terminal 1410 can circle the lesion on the one or more corresponding images according to the lesion edge coordinates for the user to use. The lesion volume and grading report may be transmitted to the client 1410 for use by the user.

While the invention has been described and illustrated with reference to specific embodiments of the present disclosure, such description and illustration do not limit the invention. It will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the true spirit and scope of the invention as defined by the appended claims. The illustrations may not necessarily be drawn to scale. Due to manufacturing processes and tolerances, differences may exist between the artistic representation in this application and the artistic representation in the actual invention. There may be other embodiments of the invention not specifically described. The specification and drawings are to be regarded in an illustrative rather than a restrictive sense. Modifications may be made to adapt a particular situation, material, composition of matter, method or process to the object, spirit and scope of the invention. All such modifications are intended to be within the scope of the claims appended hereto. Although the methods disclosed herein have been described with reference to specific operations performed in a specific order, it is to be understood that such operations may be combined, subdivided, or reordered to form equivalent methods without departing from the teachings of the present invention . Accordingly, unless specifically indicated otherwise herein, the order and grouping of operations are not limitations of the invention. Furthermore, the effects detailed in the above-described embodiments and the like are merely examples. Therefore, the present application can further have other effects.

Additionally, the logic flows depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Additionally, other steps may be provided, or steps may be eliminated from the illustrated flows, and other components may be added to or removed from the illustrated systems. Accordingly, other embodiments are within the scope of the appended claims.

101: Input image 102: Feature map group 103: Feature Map Group 104: Feature Map Group 105: Feature Map Group 106: Feature Map Group 107: Feature Map Group 108: Feature map group 109: Feature Map Group 110: Feature map group 111: Feature map group 112: Feature map group 113: Feature map group 114: Feature map group 115: Feature map group 116: Feature map group 117: Feature map group 118: Feature map group 119: Feature map group 120: Feature map group 121: Feature map group 122: Feature map group 123: Feature map group 124: Feature map group 201: Output image 203: output image 205: Output image 207: Output image 209: Output image 211: output image 300: Image processing program 231: Volume 233: Volume 235: volume 237: Volume 239: Image parameters 301: Operation 302: Operation 303: Operation 304: Operation 305: Operation 306: Operation 307: Operation 308: Operation 309:Operation 310: Operation 311: Operation 312: Operation 313: Operation 314: Operation 315: Operation 401: Operation 402: Operation 403: Operation 501: Operation 502: Operation 503: Operation 601: Operation 602: Operation 603: Operation 701: Operation 702: Operation 703: Operation 800: Grading Procedures 900: Grading Results

FIG. 1 illustrates a flowchart of image recognition according to some embodiments of the present disclosure.

FIG. 2 illustrates a hierarchical flow diagram according to some embodiments of the present disclosure.

FIG. 3A illustrates an image processing procedure according to some embodiments of the present disclosure.

FIG. 3B illustrates an image processing procedure according to some embodiments of the present disclosure.

FIG. 4 illustrates an image processing procedure according to some embodiments of the present disclosure.

5A-5E show brain images according to some embodiments of the present disclosure.

6A-6E show brain images according to some embodiments of the present disclosure.

7A and 7B show statistical results according to some embodiments of the present disclosure.

8A and 8B show statistical results according to some embodiments of the present disclosure.

9A and 9B show statistical results according to some embodiments of the present disclosure.

10A and 10B show statistical results according to some embodiments of the present disclosure.

11A-11E show brain images according to some embodiments of the present disclosure.

12A-12E show brain images according to some embodiments of the present disclosure.

13 shows statistical results according to some embodiments of the present disclosure.

14 illustrates a system according to some embodiments of the present disclosure.

For a better understanding of the foregoing aspects of the present disclosure, as well as additional aspects and embodiments thereof, reference should be made to the following description in conjunction with the above drawings. In the various figures, like reference characters indicate similar elements.

1410: Client Device

1411: Processor

1412: Memory

1420: Database Device

1421: Processor

1422: Memory

1430: Server side device

1431: Processor

1432: Memory

Claims (14)

A method for identifying white matter hyperintensities, comprising: receiving an input image; performing a first set of convolutions; generating a first set of feature maps after performing the first set of convolutions feature map set); perform a first pooling on the first feature map set to generate a first set of max pooling indices and a second feature map set and performing a second set of convolutions on the second set of feature maps to generate a third set of feature maps; and performing an output convolution to generate at least two image segmentations , the output convolution involves performing a convolution with at least two kernel maps. The method of claim 1, further comprising: performing a first unpooling (a first unpooling) on the third feature map group based on the first set of max pooling indices (a first set of max pooling indices), to generate a fourth feature map set; skip connect the first feature map set to the fourth feature map set to generate a fifth feature map set; and perform a first feature map set on the fifth feature map set Three sets of convolutions and the output convolution produce the at least two image segmentations. The method of claim 1, wherein: the input image comprises a T2-weighted fluid-attenuated inversion recovery (FLAIR; fluid-attenuated inversion recovery) image generated by magnetic resonance imaging (MRI). The method of claim 1, wherein: the at least two image segmentations comprise gray matter image segmentation, white matter image segmentation, white matter hyperintensity image segmentation, cerebrospinal fluid image segmentation, scalp and skull image segmentation, or air image segmentation. The method of claim 1, further comprising: performing a second pooling on the third feature map set to generate a second set of max-pooling indices and a fourth feature map set; and the fourth feature map set performing a third set of convolutions to generate a fifth set of feature maps; performing a third pooling on the fifth set of feature maps to generate a third set of max-pooling indices and a sixth set of feature maps; and A fourth set of convolutions is performed on the sixth set of feature maps to generate a seventh set of feature maps. The method of claim 5, further comprising: performing a first de-pooling on the seventh set of feature maps based on the third set of max-pooling indices to generate an eighth set of feature maps; The graph group is jump-connected to the eighth feature graph group to generate a ninth feature graph feature map group; perform a fifth set of convolutions on the ninth feature map set to generate a tenth feature map set; based on the second set of maximum pooling indices, perform a second inversion on the tenth feature map set pooling to generate an eleventh feature map group; jump-connecting the third feature map group to the eleventh feature map group to generate a twelfth feature map group; executing the twelfth feature map group a sixth set of convolutions to generate a thirteenth feature map set; based on the first set of max-pooling indices, a third de-pooling is performed on the thirteenth feature map set to generate a fourteenth feature map group; skip connecting the first feature map group to the fourteenth feature map group to generate a fifteenth feature map group; and performing a seventh group of convolutions on the fifteenth feature map group and the output convolution to generate the at least two image segmentations. A method of white matter hyperintensity quantification, comprising: receiving a plurality of input images; performing the method of claim 4 for each of the plurality of input images to generate a plurality of gray matter image segmentations, a plurality of white matter image segmentations, Segmenting a plurality of white matter hyperintensity images, segmenting a plurality of cerebrospinal fluid images; segmenting a plurality of gray matter images to generate a gray matter volume; segmenting a plurality of white matter images to generate a white matter volume; based on the plurality of white matter hyperintensity images segmentation, resulting in a white matter hyperintensity volume; generating a cerebrospinal fluid volume based on the plurality of cerebrospinal fluid image segmentation; and performing image grading based on the gray matter volume, the white matter volume, the white matter hyperintensity volume, the cerebrospinal fluid volume, and slice thicknesses of the plurality of input images . A device for identifying white matter hyperintensities, comprising: a processor; and a memory coupled to the processor, wherein the processor executes computer-readable instructions stored in the memory to perform the following operations: receive at least one input image; and performing the following operations for each of the at least one input image: performing a first set of convolutions; generating a first set of feature maps after performing the first set of convolutions; the first feature A first pooling is performed on the set of graphs to generate a first set of max-pooling indices and a second set of feature maps; and a second set of convolutions are performed on the second set of feature maps to generate a third feature map and performing an output convolution to generate at least two image segmentations, the output convolution including performing a convolution with at least two core maps. The apparatus of claim 8, wherein for each of the plurality of input images further comprising the following operations: performing a first inverse pooling on the third set of feature maps based on the first set of max-pooling indices to generate a fourth set of feature maps; jump-connect the first set of feature maps to the fourth set of feature maps to generate a fifth set of feature maps; and perform a third set of the fifth set of feature maps convolution and the output convolution to generate the at least two image segmentations. 8. The apparatus of claim 8, wherein: the input image comprises a T2-weighted-liquid-attenuated inversion recovery image generated via MRI. The apparatus of claim 8, wherein: the at least two image segmentations include gray matter image segmentation, white matter image segmentation, white matter hyperintensity image segmentation, cerebrospinal fluid image segmentation, scalp and skull image segmentation, or air image segmentation. The apparatus of claim 8, wherein for each of the plurality of input images further comprising the following operations: performing a second pooling on the third set of feature maps to generate a second set of max-pooling indices and a first Four feature map sets; perform a third set of convolutions on the fourth feature map set to generate a fifth feature map set; perform a third pooling on the fifth feature map set to generate a third set of maximum pooling the index and a sixth set of feature maps; and performing a fourth set of convolutions on the sixth set of feature maps to generate a seventh set of feature maps. The apparatus of claim 12, wherein for each of the plurality of input images further comprising the following operations: based on the third set of max-pooling indices, performing a first unpooling on the seventh set of feature maps to generate an eighth set of feature maps; jump-connect the fifth set of feature maps to the eighth set of feature maps to generate a ninth set of feature maps; perform a fifth set of convolutions on the ninth set of feature maps to generate a tenth feature map group; based on the second set of maximum pooling indices, perform a second de-pooling on the tenth feature map group to generate an eleventh feature map group; skip the third feature map group connecting to the eleventh feature map set to generate a twelfth feature map set; performing a sixth set of convolutions on the twelfth feature map set to generate a thirteenth feature map set; based on the first group max pooling index, perform a third de-pooling on the thirteenth feature map group to generate a fourteenth feature map group; jump connect the first feature map group to the fourteenth feature map group, to generate a fifteenth feature map set; and perform a seventh set of convolutions and the output convolution on the fifteenth feature map set to generate the at least two image segmentations. The apparatus of claim 11, wherein the processor further performs the following operations: generating a plurality of gray matter image segmentations, a plurality of white matter shadows based on the plurality of input images Image segmentation, multiple white matter hyperintensity image segmentation, multiple cerebrospinal fluid image segmentation; based on the multiple gray matter image segmentation, a gray matter volume is generated; based on the multiple white matter image segmentation, a white matter volume is generated; based on the multiple white matter image segmentation segmenting the hyperintensity image to generate a white matter hyperintensity volume; segmenting the plurality of cerebrospinal fluid images to generate a cerebrospinal fluid volume; based on the gray matter volume, the white matter volume, the white matter hyperintensity volume, the cerebrospinal fluid volume, and Image grading is performed on slice thicknesses of the plurality of input images.

Images