KR102412337B1 - Dynamic learning apparatus and method for the classification of human epithelial cell image - Google Patents

Abstract

The present invention relates to a dynamic learning apparatus and method for classifying images of human epithelial cells, and more particularly, when an image is input, performing discrete wavelet transformation; normalizing the discrete wavelet transform coefficients to feed the network; 4 different CNNs are used in parallel so that each CNN forms a final residual building block with different normalized wavelet coefficients as input; and passing the global average pooling and classification layer.

Description

DYNAMIC LEARNING APPARATUS AND METHOD FOR THE CLASSIFICATION OF HUMAN EPITHELIAL CELL IMAGE

The present invention relates to a dynamic learning apparatus and method for classifying human epithelial cells images, and more particularly, to human epithelial cell images that minimize intra-class discrepancies by encouraging specific homogenization with respect to the intensity level and shape of human epithelial cells. It relates to a dynamic learning apparatus and method for classification.

Computer-aided diagnostic (CAD) systems refer to any technology for integrating automation of the disease diagnosis process. These systems have attracted great interest since the development of various machine learning methods over the past few decades due to their ability to perform automated tasks in a much more reliable manner. One of the most difficult tasks associated with these CAD systems is to fully analyze and understand images that represent biological organisms. For autoimmune diseases, indirect immunofluorescence (IIF) for human epithelial type 2 (HEp-2) cell pattern is the most recommended diagnostic method. However, manual analysis of IIF images represents a arduous task that can take considerable time. In addition, the complexity of the images introduces an important part of the pathologist's subjectivity, which can lead to some discrepancies in diagnostic results. This is the reason why CAD systems are of great interest to pathologists to aid in diagnosis, mainly for the automatic classification of different types of HEp-2 cells.

Various methods were discussed in the literature and methods presented in other editions of the HEp-2 cell sorting contest held by the International Conference on Pattern Recognition (ICPR). As a classic pattern recognition task, the HEp-2 cell sorting method is a feature extraction or selection process followed by a sorting method. Feature extraction remains the most important part of the procedure as it consists of extracting relevant information that can aid in the accurate identification of different cell types. In general, we can distinguish two method groups in the literature: conventional machine learning-based methods and deep learning-based methods.

It is the person who adopted the codebook generated from the descriptor. Nosaka et al. used the local binary pattern (LBP) as a feature and provided it as an input to a linear support vector machine (SVM) in the classification step. Hwang et al. utilized organizational and statistical features in a hybrid manner and provided their own construct maps for the classification process.

For other statistical features such as gray-level magnitude band matrix, nearest neighbor classifier was adopted for the discrimination part. The same statistical features were fed to the SVM by Wiliem et al. A linear local distance coding method was used to extract the features used as inputs of the linear SVM by Xu et al.

Researchers in this field have utilized a hybrid feature learning method. Indeed, Cataldo et al. proposed the use of combinations of various features such as morphological features, global texture descriptors such as rotational invariant disappearance features, and other kinds of LBP descriptors such as rotational invariant uniform LBPs, simultaneous occurrence of adjacent LBPs, and completed Rotation-invariant co-occurrence of adjacent LBPs also adopted in LBPs. As another hybrid feature extraction method, the authors proposed a combination of LBP and SIFT descriptors by Theodorakopoulos et al. It can be seen that the performance of the method of Foggia et al depends entirely on the identifiability provided by the extracted features, leaving an important part to the subjectivity of the user.

Since deep learning has been deployed, automatic feature learning methods have been widely adopted. They showed excellent results in the object recognition problem, and many researchers have adopted it as a major tool for HEp-2 cell sorting. Unlike conventional methods in which accuracy is determined by subjective selection of features, deep learning methods such as Convolutional Neural Networks (CNNs), which are currently widely used, have the advantage of providing an automatic feature learning process. Indeed, many studies have demonstrated the superiority of deep learning-based functions over hand-made ones for HEp-2 cell sorting tasks.

The first work to apply CNN to the HEp-2 cell sorting problem was presented by Foggia et al. in the 2012 ICPR HEp-2 cell sorting contest. Although the results were excellent, the available data sets at the time were not heterogeneous, and many improvements were needed. Since then, the many data sets available have varied considerably, and other proposed CNN models continue to push their limits in terms of classification accuracy. Gao et al. presented a simple CNN architecture that was tested on different datasets. They were the first to test data augmentation techniques, such as rotation at different angles, on HEp-2 cell images. Li et al. adopted the Deep Residual Inception Model (DRI), which combines ResNet, the most widely used CNN model, and one of GoogleNet's "Inception" modules. Phan et al performed transfer learning, which consisted of using an already trained network on a new dataset using a model trained on the ImageNet dataset.

Lei et al. proposed a complex transfer learning method. We used the different proposed architectures of pre-trained ResNet models and blended them together to create what we termed a cross-modal transfer learning approach. The results obtained through this method represent one of the latest advances in sorting HEp-2 cells today. Another state-of-the-art performance was obtained in the study presented by Shen et al. The authors used the ResNet approach again, but with a deeper residual module called the Deep Cross Residual (DCR) module with large data extension.

HEp-2 cell image data sets generally exhibit significant heterogeneity. This is explained by the variation within the class due to the presence of different intensity levels.

1A to 1D are diagrams illustrating an example of a cellular image of one of a data set.

The two images in FIGS. 1A and 1B show microscopically stained cell types. The first image comes from a positive intensity level and the second image comes from a negative intensity level. The variation within the class is clearly visible in these two images, which are assumed to belong to the same class. We can image the complexity faced by classifiers trying to output these two images from the same class.

The heterogeneity discussed here becomes even clearer when looking at the images shown in Figures 1c and 1d. The previously mentioned types belong to allogeneic cells, which are the same cell type in both images. Again you can notice the variation within the class. Also, the complexity becomes even more noticeable when comparing the two negative intensity images shown in Figs. 1b and 1d. The two images are similar in that the cell shape is poorly perceived and the lighting is poor. The two images are not easy to distinguish, but they must belong to two different classes.

This marked heterogeneity can cause certain problems in terms of differential efficiency. Some of the real state-of-the-art methods still have some problems with solving this problem, especially when the data sets show strong differences. The method proposed in this study specifically attempts to solve this heterogeneity-related problem. Two data sets with strong intra-class variation were selected. Some methods, such as the one proposed by Nigam et al., try to solve this problem by performing preliminary intensity-based separation prior to the cell type classification itself. While this approach can make the end result reasonably good, the fact that the classification part uses two steps after burdening the future extraction process obviously extends the global processing time.

Therefore, there is a need to develop a technology capable of minimizing the heterogeneity in one step.

Therefore, the present invention has been proposed to solve the above problems, and dynamic learning for human epithelial cell image classification to minimize intra-class discrepancy by encouraging specific homogenization with respect to the intensity level and shape of human epithelial cells To provide an apparatus and method.

Objects of the present invention are not limited to those mentioned above, and other objects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description.

A dynamic learning method for human epithelial cell image classification according to the present invention for achieving the above object includes: performing discrete wavelet transformation when an image is input; normalizing the discrete wavelet transform coefficients to feed the network; 4 different CNNs are used in parallel so that each CNN forms a final residual building block with different normalized wavelet coefficients as input; and passing the global average pooling and classification layer.

In addition, a dynamic learning apparatus for human epithelial cell image classification according to the present invention for achieving the above object includes: a transformation unit that performs discrete wavelet transformation when an image is input; a regularizer that normalizes the discrete wavelet transform coefficients to supply the network; a block forming unit in which four different CNNs are used in parallel to form a final residual building block with each CNN receiving different normalized wavelet coefficients as input; and a classifier that passes the global average pooling and classification layer.

In accordance with the present invention, a specific homogenization with respect to the intensity level and shape of human epithelial cells is encouraged to minimize intraclass discrepancies.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description.

1 is a diagram illustrating an example of a cellular image of one of a data set;
2 is a diagram schematically showing a dynamic learning step for human epithelial cell image classification according to an embodiment of the present invention;
3 is a diagram illustrating wavelet coefficients of positive intensity cellular images in a data set according to an embodiment of the present invention;
4 is a diagram illustrating wavelet coefficients of negative intensity cellular images in a data set according to an embodiment of the present invention;
5 is a diagram showing the structure of four different networks according to an embodiment of the present invention;
6 is a view embodying the structure of two remaining building blocks of FIG. 5 according to an embodiment of the present invention;
7 is a diagram illustrating a summary hierarchy structure according to an embodiment of the present invention;
8 is a diagram schematically illustrating a dynamic learning method according to an embodiment of the present invention.
9 is a block diagram illustrating a dynamic learning apparatus according to an embodiment of the present invention.

Objects and effects of the present invention, and technical configurations for achieving them will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings. In the description of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. And the terms described below are terms defined in consideration of donation in the present invention, which may vary according to the intention or custom of the user or operator.

However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. Only the present embodiments are provided so that the disclosure of the present invention is complete, and to completely inform those of ordinary skill in the art to which the present invention belongs, the scope of the invention, the present invention is defined by the scope of the claims will only be Therefore, the definition should be made based on the content throughout this specification.

On the other hand, throughout the specification, when a certain part is said to be "connected" with another part, it is not only "directly connected" but also "indirectly connected" with another member interposed therebetween. include In addition, when a part "includes" a certain component, this means that other components may be further provided without excluding other components unless otherwise stated.

The present invention proposes a dynamic learning process with different networks taking different input image transformations simultaneously. Discrete wavelet transform (DWT) is used to generate different transforms. A two-dimensional DWT is performed on the images. The DWT coefficients are normalized to feed the network and expressed in image form. Four different CNNs are used in parallel, each taking a different normalized wavelet coefficient as input. The propagation of the error is done in parallel, and at a specific point in the network, high-level features from the feature maps of the four CNNs are combined to mix different information learned by the four networks and passed through a non-linear function. do. Then, the final remaining building block is formed using the output of the floor performing the linking process. Finally, this nonlinearity passes through the global average pooling layer and the fully connected layer to perform the final classification.

Different wave coefficients are expected to expose and significantly reinforce changes in local intensity, and especially in the case of cell types showing strong differences in the level of luminous illumination, the intensity as well as the shape are homogenized. After most of the images of a given cell, four different CNNs dynamically learn these published properties. This method was tested on publicly available SNPHEp-2 and ICPR datasets, as a result of which strong disparities in the datasets were actually minimized, increasing the finally obtained discrimination results.

2 is a diagram schematically illustrating a dynamic learning step for classifying images of human epithelial cells according to an embodiment of the present invention.

In FIG. 2, AC, HDC, VDC, and DDC represent an approximate coefficient, a horizontal detail coefficient, a vertical detail coefficient, and a diagonal detail coefficient, respectively.

Hereinafter, the operation of each step shown in FIG. 2 will be described in detail with reference to FIGS. 3 to 7 .

When an image is input, two-dimensional discrete wavelet transform decomposition is performed in step 201 to apply DWT decomposition on the input image. The two-dimensional decomposition of the first level produces four different outputs. After the high and low frequency band decomposition provided by the DWT, the approximation coefficient contains the low frequency band information. High frequency related information can be retrieved mainly by the three remaining coefficients: horizontal, vertical and diagonal detail. These four different coefficient matrices are represented as images and then considered as inputs to four different CNNs to be used in parallel.

3 is a diagram illustrating wavelet coefficients of positive intensity cellular images in a data set according to an embodiment of the present invention.

Referring to Figure 3, it shows the four output results of the DWT digestion with the positive intensity microstained cells. (a) shows the original image with strong illumination, and (b) shows the approximate coefficients. The results of the low-pass filtering process involved in the DWT decomposition are shown. This low-pass filtering has the effect of enhancing areas that do not exhibit significant gray changes. On the other hand, it can be seen that the intensity value of the pixel is slightly enhanced in the result image shown in (b). Comparing the original image shown in (a) with the approximation coefficients shown in (b), a slight enhancement of the overall illumination becomes clear.

Meanwhile, the images shown in FIGS. (c), (d), and (e) show detailed coefficients. This coefficient represents the high-frequency component of the original image, while all three other detailed coefficients actually represent the shape of the cell.

4 is a diagram illustrating wavelet coefficients of a negative intensity cellular image in a data set according to an embodiment of the present invention.

Referring to Figure 4, it shows the four output results of the negative-intensity fine speckled cell and DWT digestion. (a) shows the original image with very poor lighting. It can be seen how the negative intensity image does not provide a significant signal about cell shape. The image in (b) shows the approximation coefficient, in which the intensity level is noticeably enhanced in the image. Comparing the images shown in Fig. 3 (b) and Fig. 4 (b), it is possible to clearly identify the homogenization occurring in terms of intensity. Homogenization of images under two different fluorescent lights. Although the original images shown in FIGS. 3A and 4A show a large difference in terms of global illumination, their approximations shown in FIGS. 3B and 4B, respectively The coefficient represents a specific homogenization in illumination.

The images shown in (c), (d) and (e) of Fig. 4 show three different coefficients of detail. Here, we have horizontal, vertical and diagonal details, respectively. Although the original image in Fig. 4(a) does not provide striking information regarding the cell shape, it can explain how the coefficient of detail exposes the cell's geometric signal. And, by comparing the detailed coefficients of the positive illumination image of FIG. 3 and the negative illumination image of FIG. 4 , (c), (d), (e) and FIGS. 4 (c), (d), (e) ), the homogenization that occurs in terms of perception of geometric signals provides a comprehensive understanding of the shape of cells. Knowing that the geometrical patterns of cells are indeed important for identification, it seems that enhancement is indeed necessary. Also, the three sub-coefficients act like gradients in different directions.

Finally, the approximation and detail coefficients minimize the differences between the two kinds of illumination (intensity levels) and the approximate coefficients are used to promote specific homogenization of intensity and geometric patterns into three details. A count representing the cell morphology. These four different coefficients are used as inputs to four different networks, simultaneously learning how to dynamically encapsulate all the necessary information that will help distinguish different cellular types.

Thereafter, in step 203, four different coefficients are supplied to other networks. CNNs are used for that purpose. Networks will be utilized in a parallel paradigm. As shown in Fig. 2, in the first part of the proposed scheme, there are four networks that extract features from different coefficients. After this process, features learned from the network are linked, blending the various information contained therein and providing the final advanced feature vector. This vector is obtained after the overall average pooling process and passes through the fully connected layer to perform the final classification. The four CNNs have the same structure as shown in FIG. 5 .

5 is a diagram showing the structure of four different networks according to an embodiment of the present invention.

The first level of the DWT decomposition process is reduced in size to half of the original image. Since the size of the original cellular image is 112x112, the network receives a 56x56 input as shown in FIG. We used a 3x3 receive file for the whole of the convolution operation. The convolution operation is performed in such a way that the output volume maintains the spatial extent of the input volume so that the downsampling process is performed only by the pooling layer. This is to avoid loss of spatial information during the feature extraction process performed by the convolution operation. In FIG. 5, "Conv 3x3, m" indicates a convolution operation having a filter size of 3x3 and m different filters. The value of m is set to 32.

In Fig. 5, the input coefficients are first given to the convolutional layer, and the properties of which are described in the previous paragraph are indicated by the next pooling layer. This pooling layer has a stride length and padding of 2 to downsample the input volume by half in the convolutional layer. Thus, the output volume in this part of the network has a spatial extent of 28x28 and contains 32 different maps. Then, the remaining building blocks are constructed using the output of the previous layer. The output of the first residual building block (residual building block 1 in Fig. 5) has the same characteristics as the input volume. This means that it still contains 32 functional maps of size 28x28. The second pooling operation is performed just before the second remaining building block. The output of the residual building block 2 has 64 (2m in Fig. 5) functional maps and a spatial extent of 14x14.

The remaining building blocks consist of two convolutional layers, two Rectified Linear Unit (ReLU) activation layers and two Batch Normalization (BN) layers. For a given variable x, the ReLU activation function is defined as in Equation 1 below.

<Equation 1>

BN is a type of regularization method used between layers of CNNs to solve the internal covariate shift problem. Indeed, the distribution of functional maps in different layers is constantly changing during the training process. The BN layer aims to find variables that minimize distribution changes during the training process.

FIG. 6 is a detailed view of the structures of the two residual building blocks of FIG. 5 according to an embodiment of the present invention, (a) shows the structures of the residual building blocks 1 and 2 in four different networks, ( b) shows the structure of the final residual building block after the summing layer (SumL).

Here, the identity short-cut is defined as in Equation 2 below.

<Equation 2>

)이라는 용어는 4 개의 다른 네트워크에서 4 개의 볼륨을 합한 것을 나타낸다.Here, the term f (x, W) denotes the output of the final convolution operation, x denotes the input volume, W denotes the parameters to be learned in the learning process, and y denotes the final output volume of the residual building block. On the other hand, (b) of FIG. 6 shows an identity short-cut when performing after summarizing the functional maps in four different networks. In this case, it is defined as in the following <Equation 3>, where Sum(

) refers to the sum of 4 volumes from 4 different networks.

<Equation 3>

7 is a diagram illustrating a summary hierarchical structure according to an embodiment of the present invention.

7 shows what is called a summary layer. After the second residual building block of 4 networks, all final volumes from the network have 64 feature maps of size 14x14, as discussed above and shown in FIG. 5 . The first operation of the summary layer is to perform the concatenation of features. Maps from other networks. Useful patterns learned from different networks are organized into a volume. There are 256 different functional maps in this single volume. To encapsulate all information and reduce the number of parameters in the next layer, a convolution operation with filter size 1x1 is performed on the inherited volume in the concatenation process. This task is similar to a linear combination of various features learned from other networks. The 1x1 convolution operation reduces depth by combining features, but preserves the spatial extent of the input volume, so no spatial information is lost. In the final volume after this summing process, there are 64 functional maps of size 14x14 as shown in Fig.

This final volume is used as an input of the final building block shown in Fig. 6 (b) and <Equation 3>. As can be clearly seen in FIG. 6(b), two convolution operations are included in this residual block. Each of these convolution operations uses 128 different filters. Since the convolution operation maintains the spatial extent, the output volume of the residual building block 3 will be in the form of 14x14x128. After the BN and ReLU activation layers, this volume is used for global average pooling. After applying full mean pooling, the final high-level feature vector is of the form 1x1x128, giving a one-dimensional vector containing 128 elements.

Advanced feature vectors are provided to fully connected layers that will perform the classification process. This layer uses the softmax function defined as follows, and is defined as in Equation 4 below.

<Equation 4>

값은 softmax 함수의 출력을 나타내며, 이는 데이터가 주어진 모든 단일 클래스의 정규화 된 확률로 해석될 수 있다. 네트워크는 오류를 역전 파하고 다음에 의해 정의된 교차 엔트로피 오류 함수를 사용하여 학습하며, 하기 <수학식 5>와 같이 정의된다.Here, N is the number of classes, which means the length of the input feature vector of the softmax function. The zj value is the j-th element of the corresponding input vector. That is, it represents the function associated with the j-th class in the input feature vector of the function. where j varies from 1 to the total number of classes N

The value represents the output of the softmax function, which can be interpreted as the normalized probability of any single class given the data. The network backpropagates the error and learns using the cross-entropy error function defined by the following, and is defined as in Equation 5 below.

<Equation 5>

Here, the value yj represents the actual label of class N for the given data. On the other hand, the value zj represents the elements of the output vector z of the softmax function computed using Equation 4.

8 is a diagram schematically illustrating a dynamic learning method according to an embodiment of the present invention. To avoid duplication, the four networks are identical before the summing layer. For only the first network we mentioned different volume sizes (output of the layer), the same size applies to the remaining 3 networks. As mentioned earlier, all inputs are the same size.

We start with the results obtained by using one single network with the original image as input. The structure of this network follows the same structure as the proposed method except for the fact that there is no aggregation layer because there is only a single network.

Data augmentation was applied to the dataset to improve the learning ability of the network. For all split sets of training data, the cells were rotated 360º in 18º steps, similar to the experiment by Bayramoglu et al. This means that the original training set has been expanded by a factor of 20 in the 360º quadrant containing 20 parts of 18º. Reinforcing the training set actually improves the accuracy on the test set.

Therefore, the present invention is fed to four different deep networks in a parallel learning paradigm to efficiently homogenize features extracted from images with approximations and different intensity levels, and the feature maps from those different networks are connected and passed to the classification layer. to produce a final type of cellular image.

By performing a dynamic learning method to minimize intra-class discrepancies by encouraging specific homogenization with respect to the intensity level and shape of cells, this problem is minimized to improve differential outcomes.

In the present specification and drawings, preferred embodiments of the present invention have been disclosed, and although specific terms are used, these are only used in a general sense to easily explain the technical content of the present invention and help the understanding of the present invention. It is not intended to limit the scope. It is apparent to those of ordinary skill in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

1: Dynamic learning device 10: Transformer
30: normalization unit 50: block forming unit
70: classification unit

Claims (2)

In the dynamic learning method for human epithelial cell image classification,
when an image is input, performing discrete wavelet transform to generate four different discrete wavelet transform coefficients;
normalizing the four different discrete wavelet transform coefficients to feed a network;
4 different CNNs are used in parallel so that each CNN forms a final residual building block with different normalized wavelet transform coefficients as input; and
passing the final residual building blocks through a global average pooling and classification layer to produce a final type of cellular image;
The different normalized wavelet coefficients include approximate coefficients and detailed coefficients,
The detail coefficient includes a horizontal detail coefficient, a vertical detail coefficient and a diagonal detail coefficient,
the intensity value of the pixel is enhanced over the original of the image by the approximation coefficient, thereby facilitating a specific homogenization in terms of intensity;
Facilitates a specific homogenization of the geometric pattern of cells into three details, including horizontal, vertical and diagonal, compared to the original of the image by the factor of detail,
In the step of forming the final residual building block, a volume consisting of features extracted from wavelet transform coefficients by each CNN operating in parallel is concatenated, and then a convolution operation is performed, and generated through the convolution operation A dynamic learning method for human epithelial cell image classification, characterized in that the final volume is used as an input of the final building block to form the final residual building block.
A dynamic learning device for human epithelial cell image classification, comprising:
a transform unit that generates four different discrete wavelet transform coefficients by performing discrete wavelet transform when an image is input;
a normalizer for normalizing the four different discrete wavelet transform coefficients to supply the network;
a block forming unit in which four different CNNs are used in parallel to form a final residual building block with each CNN receiving different normalized wavelet transform coefficients as input; and
a classifier for passing the final residual building blocks through a global average pooling and classification layer to generate a final type of cellular image;
The different normalized wavelet coefficients include approximate coefficients and detailed coefficients,
The detail coefficient includes a horizontal detail coefficient, a vertical detail coefficient and a diagonal detail coefficient,
the intensity value of the pixel is enhanced over the original of the image by the approximation coefficient, thereby facilitating a specific homogenization in terms of intensity;
Facilitates a specific homogenization of the geometric pattern of cells into three details, including horizontal, vertical and diagonal, compared to the original of the image by the factor of detail,
The block forming unit performs a convolution operation after concatenating a volume composed of features extracted from wavelet transform coefficients by each CNN, and using the final volume generated through the convolution operation as an input of the final building block. Dynamic learning device for human epithelial cell image classification, characterized in that it forms the final residual building block.

Images