KR20210131551A - Method and apparatus for diagnosing white blood cells based on deep learning - Google Patents

Abstract

The present invention relates to a deep learning-based leukocyte analysis method and device. The method comprises the steps of: segmenting a leukocyte image into sub-images; and classifying the sub-image by a convolutional neural network (CNN)-based classifier to analyze leukocytes. The CNN-based classifier learns the weights of all FC-layers except for an FC-layer following a convolutional layer among a VGG16 network structure and a ResNet50 network structure, wherein the number of data is supplemented by using a five-fold cross-validation technique. Whether the white blood cells are normal can be analyzed.

Description

Deep learning-based leukocyte analysis method and device {METHOD AND APPARATUS FOR DIAGNOSING WHITE BLOOD CELLS BASED ON DEEP LEARNING}

The present invention relates to a leukocyte analysis method and apparatus, and more particularly, to a deep learning-based leukocyte analysis method and apparatus capable of analyzing whether white blood cells are normal by analyzing a leukocyte image based on deep learning.

There are five main types of white blood cells. These are neutrophils, lymphocytes, monocytes, eosinophils, and basophils. Individual leukocytes have very different functions as shown below, and their amounts are basophil (0-1%), eosinophil (1-5%), lymphocyte (20-45%), monocyte (2-10%) and neutrophil (50 -70%).

1 shows the normal values and clinical significance of the leukocyte differential calculation, FIG. 2 shows normal leukocytes, and FIG. 3 shows abnormal leukocytes.

In the case of the recently developed automatic blood cell analyzer, in addition to the differential calculation of the five types of leukocytes normally present in peripheral blood, indicators for detecting the presence of malignant blood cells are being developed. However, the automatic blood cell analyzer's ability to discriminate white blood cells is not perfect, so it is necessary when abnormal results are suspected from the equipment and should be confirmed by a collection method.

In the case of the collection method, the difference in accuracy between doctors with experience and those without experience is quite large. To overcome this, analysis methods using artificial intelligence have been recently introduced. Among them, the most commonly applied AI method is CNN, and the traditional method for measuring white blood cells is shown in FIG. 4 .

An object of the present invention is to provide a method and apparatus for analyzing white blood cells based on deep learning that can analyze whether white blood cells are normal by analyzing a white blood cell image based on deep learning.

In particular, an object of the present invention is to provide a deep learning-based leukocyte analysis method and apparatus capable of more efficiently analyzing whether leukocytes are normal by improving an existing convolutional neural network (CNN)-based classifier.

In order to solve the above problems, the present invention provides a deep learning-based leukocyte analysis method, comprising: segmenting a leukocyte image into sub-images; and analyzing the leukocytes by classifying the sub-images by a Convolutional Neural Network (CNN)-based classifier, wherein the CNN-based classifier is a VGG16 network structure and a ResNet50 network structure following a convolutional layer We provide a deep learning-based leukocyte analysis method that learns the weights of all FC-layers except for the FC-layer of , but supplements the number of data using a five-fold cross-validation technique.

According to another aspect of the present invention, there is provided a deep learning-based leukocyte analysis apparatus, comprising: a segmentation unit for segmenting a leukocyte image into sub-images; and a CNN-based classifier that analyzes leukocytes by classifying the sub-images based on a Convolutional Neural Network (CNN), wherein the CNN-based classifier is a convolutional layer of a VGG16 network structure and a ResNet50 network structure. It provides a deep learning-based leukocyte analysis device that learns the weights of all FC-layers except the -layer, but supplements the number of data using a five-fold cross-validation technique.

According to another aspect of the present invention, there is provided a deep learning-based leukocyte analysis method, comprising the step of inputting a leukocyte image to a CNN (Convolutional Neural Network)-based detector to analyze the leukocyte, wherein the CNN-based detector comprises: R -CNN, Faster R-CNN, Faster R-CNN, provides a deep learning-based leukocyte analysis method characterized in that it operates by any one of YOLO algorithm.

According to another aspect of the present invention, as a deep learning-based leukocyte analysis apparatus, comprising a CNN-based detector that analyzes leukocytes based on a convolutional neural network (CNN) based on a leukocyte image, wherein the CNN-based detector comprises: R-CNN; It provides a deep learning-based leukocyte analysis device, characterized in that it operates by an algorithm by any one of Fast R-CNN, Faster R-CNN, and YOLO.

According to the present invention, it is possible to provide a method and apparatus for analyzing white blood cells based on deep learning that can analyze whether white blood cells are normal by analyzing a white blood cell image based on deep learning.

In addition, the present invention can provide a deep learning-based leukocyte analysis method and apparatus capable of more efficiently analyzing whether leukocytes are normal by improving an existing convolutional neural network (CNN)-based classifier.

1 to 19 are views for explaining the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

First, we explain the basics of deep learning and CNN.

Deep learning is a field of machine learning algorithms that attempts to summarize key contents or functions in large amounts of data or complex data through a combination of several nonlinear transformation techniques. Among them, the deep learning technique used for image classification is a Convolutional Neural Network (CNN) algorithm.

When an image is input to CNN, the CNN algorithm is used for the purpose of classifying the image. In the case of an existing feed-forward neural network, it receives image pixel values as they are as a one-dimensional vector and classifies which type they belong to. However, even if the same cat image is slightly rotated, different in size, or slightly deformed, it is difficult to classify the image. In this case, a lot of training data was required, and there was a disadvantage that the training time was considerably longer.

The CNN algorithm creates a feature map that finds out the unique features of the image from the input image. The value passed through the image to the feature map is input to the neural network to classify which class label the image belongs to. Therefore, the forward neural network used in CNN is also called a classifier. In other words, the CNN algorithm is a structure that connects a feature extraction neural network and a classifier.

5 is a diagram showing what happens in a CNN network, and FIG. 6 shows a basic CNN structure.

As shown in Figures 5 and 6, the feature extraction neural network in the CNN algorithm repeats the convolution, activation, and sub-sampling processes, starting with the low-dimensional feature, deriving the higher-dimensional feature, and then performing the classification task with the final feature. will do

The CNN algorithm derives characteristics that can represent images and puts them into the neural network rather than receiving image pixel values as they are. If the front-feed neural network is used, the input value determined by the user's judgment is used to determine which characteristics should be derived to help classification. For example, if it is a problem of classifying apples and bananas, the user must create an expression that derives the characteristics of color, length, and shape. The CNN algorithm automates this feature derivation process through a training process using data.

Meanwhile, a well-known network structure among CNN algorithms for classifying images is the VGG16 network structure. 7 is a simple schematic diagram of the VGGNET structure.

The VGG algorithm is a model developed by VGG, a research team at Oxford University. This network structure took second place in the 2014 ImageNet image recognition contest, and the name changes depending on the number of layers 16 and 19. Among them, the VGG network structure with 16 layers is called VGG-16. This paper on the VGG algorithm was published in "Very deep convolutional networks for large-scale image recognition" at ICLR 2015 by Karen Simonyan and Andrew Zisserman.

The basic network structure of the VGG-16 is shown in FIG. 8 . In Fig. 8, the table is a table in the paper, and the figure shows the VGG-16 network structure using [224x224x3] image data.

As shown in FIG. 8, VGG-16 has 13 convolution layers and 3 fully connected layers, and consists of a total of 16 layers.

Here, the convolution layer plays a role in extracting features from the input image data. This layer consists of a filter that extracts features and an activation function that converts the value into a nonlinear value. Also, the input image data is usually a three-dimensional tensor. This is because image data has RGB values per pixel. For example, 4x4 image data is a 3D tensor of [4x4x3]. When such an image is input to the convolution layer, it is calculated as shown in FIG. 9 .

VGG-16 has 13 convolution layers, and the size of the input data of this structure is [224x224x3] image.

Meanwhile, ResNet is the algorithm that won the ILSVRC in 2015. ResNet is an algorithm developed by Microsoft, and an algorithm developed by Chinese researchers at the Beijing Research Institute. The ResNet related paper is "Deep Residual Learning for Image Recognition", and the authors are Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jiao Sun. The number of layers deepened rapidly from the ResNet algorithm. GooLgeNet in ILSVRC in 2014 consisted of 22 layers, but ResNet in ILSVRC in 2015 participated with 152 layers.

Referring to FIG. 10 , it can be seen that the top-5 error decreases as the network deepens. In other words, it can be seen that the performance improves as the depth increases.

Also, referring to the graphs of FIG. 11 , it can be seen that the 56-layer network having a deeper structure shows worse performance than the 20-layer network. These graphs confirm that it is not possible to unconditionally deepen the layer in the conventional way. The authors of ResNet introduce the residual block and the ResNet algorithm using this block.

1) Residual block

The residual block is the core of the ResNet algorithm. In Fig. 12, the right side shows the Residual Block, and if there is a difference from the existing layer, only one shortcut is made so that the input value can be added to the output value.

를 타겟 값

로 사상하는 함수

을 얻는 것이 목적이었다. 그러나 ResNet은

를 최소화하는 것을 목적으로 하며, 입력 값

는 변할 수 없는 값이므로

를 0에 가깝게 만드는 것이 목적이 된다.

가 0이 되면 출력과 입력이 모두

로 같아지게 된다. ResNet에서는

이므로

을 최소로 해준다는 것은

를 최소로 해주는 것과 동일한 의미를 지닌다. 여기서

를 residual라고 부른다. 즉, residual을 최소로 만드는 것이므로 ResNet이란 이름이 붙게 되었다.Conventional neural networks are input values

is the target value

function that maps to

The purpose was to obtain However, ResNet

It aims to minimize the input value

is an immutable value, so

The goal is to make it close to 0.

When is 0, both output and input are

becomes equal to In ResNet

Because of

that minimizes

It has the same meaning as minimizing here

is called residual. In other words, since the residual is minimized, the name ResNet was given.

2) ResNet structure

The ResNet algorithm basically has a VGG-19 network structure. ResNet 34, for example, is a structure that adds convolutional layers to the VGG-19 network structure to deepen it, and then adds shortcuts. The structure of a plain network, which is a version excluding shortcuts in ResNet-34, is shown in FIG. 13 .

As can be seen from the figure of FIG. 13, ResNet-34 used a 3X3 convolutional filter uniformly except for the first. And when the size of the feature map is halved, the depth of the feature map is doubled.

ResNet algorithm developers compared the performance of ResNet with plain networks of layers 18 and 34 in ImageNet to see if shortcuts, that is, residual blocks, are effective.

In the graph of FIG. 14 , the plain network increased the error as the layer deepened, but the error decreased as the layer deepened in ResNet. This shows that there is an effect of learning to minimize residuals by connecting shortcuts.

The table of FIG. 15 is a table showing well how the ResNet structure of 18 layers, 34 layers, 50 layers, 101 layers, and 152 layers is configured. The deeper the structure, the better the performance. In other words, ResNet with 152 layers has the best performance.

In order to classify white blood cells (WBC) using this configuration, a WBC-related dataset should be used.

The publicly available WBC dataset constitutes the entire dataset by combining the blood cell images and the LISC dataset on the kaggle site.

First, the blood cell images on the kaggle site have about 100 images for each of the 4 different leukocyte types grouped into 4 different folders according to the leukocyte type. The cell types are Eosinophil, Lymphocyte, Monocyte and Neutrophil. We provide a total of 410 images in this data set.

The LISC dataset was created by collecting 400 samples from 100 microscope slides from the peripheral blood of 8 normal subjects. The microscope slide used Gismo-Right technique, and images were acquired by a light microscope (Microscope-Acioscope 40) from peripheral blood stained using an achromatic lens of magnification 100. These images were then recorded with a digital camera (Sony Model No. SSCDC50AP) and saved in BMP format. The size of the image is 720×576 pixels. All color images were taken at the Imam Khomeini Hospital Hematology Department BMT Lab in Tehran, Iran. And these images were classified into five categories by hematologists: basophil, eosinophil, lymphocyte, monocyte, and neutrophil. In addition, nuclear and cytoplasmic-related regions were manually segmented by experts.

The VGG-16 algorithm and ResNet50 algorithm were trained using both datasets above, and this algorithm classifies basophil, eosinophil, lymphocyte, and monocyte. Applicant, DeepHelix, Inc., has learned algorithms.

3) WBC classification using VGG-16 and ResNet50

Since the VGG-16 algorithm and ResNet50 algorithm trained CIFAR-1000, a dataset with 1000 classes, the last FC-layer has a structure with 1000 outputs. Based on this structure, WBC cannot be classified. The reason is that the WBC to be classified by the present applicant is a total of four classes, including basophil, eosinophil, lymphocyte, and monocyte.

16 and 17, the network structure was changed by adjusting the output of the VGG16 network structure for classifying 1000 and the output of the ResNet50 network structure from 1000 to 4 outputs.

With reference to these points, this invention is demonstrated.

Example 1:

The inventors of the present application classify (classify) using a pipeline in which VGG-16 and ResNet50 are mounted on a white blood cell photograph, as shown in FIG. 18 .

The present inventor has an algorithm that is not disclosed by learning the weights of all FC-layers except for the FC-layer following the convolutional layer among the VGG16 network structure and the ResNet50 network structure. The two WBC datasets (BCCD & LISC dataset) described above were used for learning the two algorithms, and the training was carried out by dividing it into 80% training dataset and 20% test dataset. In addition, the existing images were applied to learning including rotated and inverted images. In the training process, a five-fold cross-validation technique was used to compensate for the insufficient number of data, and an optimized model could be selected.

As a result of learning the two algorithms, the number of epochs of VGG-16 was 44 and the number of epochs of ResNet-50 was 47. An accuracy of 75% was confirmed. As a result, it can be seen that the ResNet-50 algorithm shows better performance than VGG-16. ResNet-50 takes a lot of time to learn compared to its performance. On the other hand, since VGG-16 has a depth of 16, the learning time can be reduced. The present inventor dramatically reduced the fully connected stage corresponding to the last stage of VGG-16, thereby reducing the learning time.

Example 2:

The present inventor intends to utilize the method of the object detector for leukocyte classification as shown in FIG. 19 . The methods of R-CNN [1], Fast R-CNN [2], Faster R-CNN [3], and YOLO [4], which are representative CNN-based detectors, are used to classify leukocytes.

reference book:

[1] Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. Preprint. Available from: arXiv:1311.2524v3 Cited 7 May 2014.

[2] Girshick R. Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision.2015:1440-1448.

[3] Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017; 39(6):1137-1149.

[4] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: Unified, real-time object detection.

Preprint. Available from: arXiv: 1506.02640v5 Cited 9 May 2016

Hereinafter, an embodiment according to the present invention has been described, but the present invention is not limited to the above embodiment, and other various modifications and variations are possible.

Claims (4)

A deep learning-based leukocyte analysis method, comprising:
segmenting the leukocyte image into sub-images; and
Classifying the sub-image by a Convolutional Neural Network (CNN)-based classifier to analyze leukocytes
including,
The CNN-based classifier learns the weights of all FC-layers except for the FC-layer following the convolutional layer among the VGG16 network structure and the ResNet50 network structure, but using a five-fold cross-validation technique. A deep learning-based leukocyte analysis method characterized by supplementing the number of data. As a deep learning-based leukocyte analysis device,
a segmentation unit that segments the white blood cell image into sub-images; and
A CNN-based classifier that classifies the sub-images based on a Convolutional Neural Network (CNN) and analyzes leukocytes
including,
The CNN-based classifier learns the weights of all FC-layers except for the FC-layer following the convolutional layer among the VGG16 network structure and the ResNet50 network structure, but using a five-fold cross-validation technique. Deep learning-based leukocyte analysis device, characterized in that the number of data is supplemented. A deep learning-based leukocyte analysis method, comprising:
Step of analyzing the white blood cells by inputting the white blood cell image into a CNN (Convolutional Neural Network) based detector
including,
The CNN-based detector, R-CNN, Fast R-CNN, Faster R-CNN, Deep learning-based leukocyte analysis method, characterized in that it operates by an algorithm by any one of YOLO. As a deep learning-based leukocyte analysis device,
CNN-based detector that analyzes leukocytes based on convolutional neural network (CNN) images of leukocytes
including,
The CNN-based detector, R-CNN, Fast R-CNN, Faster R-CNN, Deep learning-based leukocyte analysis device, characterized in that it operates by any one algorithm by YOLO.

Images