WO2023191202A1

WO2023191202A1 - Tissue slide image analysis system and method

Info

Publication number: WO2023191202A1
Application number: PCT/KR2022/013247
Authority: WO
Inventors: 정요셉
Original assignee: 가톨릭대학교 산학협력단
Priority date: 2022-03-29
Filing date: 2022-09-05
Publication date: 2023-10-05
Also published as: KR20230140160A

Abstract

The present specification provides a tissue slide image analysis system and a method therefor, the tissue slide image analysis system comprising: an extraction unit which extracts a plurality of tile images from a tissue slide image; a probability map generation unit which inputs the plurality of tile images into an artificial intelligence model and generates an expected probability map for the plurality of tile images on the basis of output information of the artificial intelligence model; and a class classification unit which classifies the tissue slide image as one of a plurality of classes on the basis of the expected probability map.

Description

Tissue slide image analysis system and method

The present invention relates to tissue slide image analysis systems and methods.

Many tests for pathologists are subject to the risk of misdiagnosis. This can cause problems such as rapid increases in medical costs, increased misdiagnosis rates, decreased medical productivity, and the risk of cancer diagnosis.

To solve this problem, research is underway to automatically analyze pathological images. Automatic analysis of pathological images has the advantage of reducing human effort, saving time, and providing the basis for surgery and treatment.

However, although the speed and memory capacity of central processing units (CPUs) and graphics processing units (GPUs) continue to increase, the development of pathological image analysis technology is hindered by the large size of pathological images.

Embodiments provide a tissue slide image analysis system and method that can analyze tissue slide images more quickly and accurately.

However, the technical challenge that this embodiment aims to achieve is not limited to the technical challenges described above, and other technical challenges may exist.

One embodiment includes an extraction unit that extracts a plurality of tile images from a tissue slide image, inputs a plurality of tile images into an artificial intelligence model, and creates an expected probability map for the plurality of tile images based on the output information of the artificial intelligence model. It provides a tissue slide image analysis system including a probability map generator that generates a probability map and a class classification unit that classifies the tissue slide image into one of a plurality of classes based on the expected probability map.

Another embodiment includes an extraction step of extracting a plurality of tile images from a tissue slide image, an input step of inputting a plurality of tile images into an artificial intelligence model, and a prediction of the plurality of tile images based on output information of the artificial intelligence model. Provided is a tissue slide image analysis method including a probability map generation step of generating a probability map and a class classification step of classifying the tissue slide image into one of a plurality of classes based on the expected probability map.

According to the tissue slide image analysis system and method according to embodiments, tissue slide images can be analyzed more quickly and accurately.

1 is a diagram showing an example of a tissue slide image analysis method described in embodiments of the present invention.

FIG. 2 is a diagram illustrating an example of an operation of extracting a plurality of tile images from a tissue slide image using a sliding window algorithm.

Figure 3 is a diagram illustrating an example of a compression operation for a plurality of tile images.

Figure 4 is a diagram showing an example of an operation for analyzing a tissue slide image using a wavelet weighted ensemble.

Figure 5 is a diagram showing the Dice score and pixel accuracy for each class according to the model used.

Figure 6 is a diagram showing the change in Dice scores for low-frequency weights in all classes.

Figure 7 is a diagram comparing segmentation prediction for annotations using heatmaps and line profiles for tumor probability, methods without compression, WAE, and WWE.

Figure 8 is a diagram showing expected results for tumors by class.

Figure 9 is a diagram showing the schematic structure of a tissue slide image analysis system.

Figure 10 is a diagram showing a tissue slide image analysis method.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this does not mean excluding other components unless specifically stated to the contrary, but may further include other components, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The terms "about", "substantially", etc. used throughout the specification are used to mean at or close to that value when manufacturing and material tolerances inherent in the stated meaning are presented, and are used to enhance the understanding of the present invention. Precise or absolute figures are used to assist in preventing unscrupulous infringers from taking unfair advantage of stated disclosures. The term “step of” or “step of” as used throughout the specification of the present invention does not mean “step for.”

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware.

In this specification, some of the operations or functions described as being performed by a terminal, apparatus, or device may instead be performed on a server connected to the terminal, apparatus, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is identifying data of the terminal. It can be interpreted as:

Hereinafter, details will be described through examples.

I. Introduction

Recently, deep learning technology has been applied to various image analysis technologies, and computer vision technology is developing rapidly. In particular, convolutional neural networks (CNN) are used to automatically diagnose many diseases. However, as mentioned above, despite the continued increase in the speed and memory capacity of central processing units (CPUs) and graphics processing units (GPUs), advancements in pathological image analysis techniques are hampered by the large size of pathological images. I'm receiving it.

To solve the memory limitation problem caused by the size of pathological images, methods such as decimation, cropping, and compression are used to reduce the size of images during preprocessing of pathological images. can be used

Decimation is a process of downsampling a large image, and can reduce noise power and improve signal to noise ratios (SNR) using an anti-aliasing filter. There is an advantage. However, decimation can cause loss of high-frequency information, resulting in lower resolution due to reduced signal bandwidth.

Cropping is a method of extracting a desired area as a tile from the entire image. When cropping is used, missing information for a single tile does not occur, but information about the spatial relationship between different tiles may be lost. This can be an important issue considering that object judgment in pathological images depends on the relative size and color of each cell.

Compression is widely used to minimize the size of image files and reduce inadequacy and redundancy of image data without degradation of image quality, and is often the preferred method for processing large-sized images.

For example, to compress large-capacity images with high resolution (e.g. satellite images for detecting ships), discrete wavelet transform (DWT, Discrete Wavelet Transform) can be used.

The compression technology called discrete wavelet transform has the advantage of solving difficulties in the process of analyzing large amounts of high-resolution images and providing better performance than existing computer vision algorithms.

Also the discrete wavelet transform. It is also useful for texture classification because it can provide locality for frequency and space due to its limited duration. Therefore, the discrete wavelet transform can be applied to classify tumors by analyzing histopathology images.

Hereinafter, we propose a method for analyzing histopathological images using the above-described compression method. In order to compress large volumes of histopathology images, in addition to the discrete wavelet transform method described above, principal component analysis (PCA) based on hematoxylin and eosin (H&E) staining characteristics was used to convert three-channel RGB images into one channel. A reduction method may be used.

When using the methods disclosed in the embodiments of this specification, histopathological images can be more usefully analyzed as follows. 1) Average dice score and pixel accuracy can be improved. 2) When using discrete wavelet transform, an artificial neural network can be trained not only with spatial information but also with texture information. 3) It can provide more powerful performance because the ROI of training after compression is high. This type of compression method can be useful in applications where large-scale data and texture information are important (e.g. remote sensing, microscopy).

II. METHODS

A.Data preparation

In the examples herein, whole slide images (WSI) of colon biopsy specimens using hematoxylin and eosin staining were used. Annotations include adenocarcinoma (ADENOCA), high grade adenoma with dysplasia (TAH), low grade adenoma with dysplasia (TAL), and carcinoid (CARCINOID). ), hyperplastic polyp (HYPERP) can be classified into five classes.

As an example, the above-mentioned full slide image may be a 20-fold magnified image taken of a specimen of colonic biopsy tissue using a digital full slide camera (Aperio AT2, Leica biosystems, USA).

B. Compressed image analysis

In the examples of this specification, a compression area was applied based on the wavelet transform used in JPEG2000 to analyze tissue slide images.

In Figure 1, (a) is an operation to extract a tile, (b) is an operation to compress the image depth, (c) is a forward conversion to the compression domain, (d) is a learning and prediction operation using a convolutional neural network, (e) is a diagram showing the operation of predicting the entire image from one tile, and (f) is a diagram showing the operation of applying a wavelet-weighted ensemble (WWE).

1) Tile extraction based on sliding window algorithm (Figure 1(a) & Figure 2)

When extracting a plurality of tiles, that is, tile images, from one whole slide image (WSI), information about the location and adjacent tiles may be lost due to the limited field of view. However, morphological information between adjacent regions is a very important factor in diagnostic decisions.

To overcome this problem, multiple Region of Interest (ROI) and sliding window methods can be used.

The multiple ROI method is faster than the sliding window method due to low redundancy, but the sliding window method has the following advantages.

First, the redundancy of the sliding window method can support data augmentation, an essential preprocessing step in deep learning-based approaches.

Second, this method can indirectly overcome the limited field of view problem because overlapping areas depend on adjacent tiles.

Lastly, the overall accuracy can be increased by averaging the probability of overlapping areas while summing the entire image based on the tiles.

In the embodiments of this specification, the sliding window method was used as a tile extraction method. FIG. 2 is a diagram illustrating an example of an operation of extracting a plurality of tile images from a tissue slide image using a sliding window algorithm.

For example, in the embodiments of this specification, the maximum acceptable tile size is 512 × 512 pixels due to limitations in GPU memory size, but tiles with a size of 1,024 × 1,024 pixels were extracted before the compression step.

At this time, the stride of the sliding window was set to 256 pixels horizontally and vertically.

2) z-axis compression based on principal component analysis (PCA) (Figure 1(b) & Figure 3)

Referring to Figure 3, the histopathological image has three red (R), green (G), and blue (B) channels. At this time, the correlation is high for each color.

The color of histopathological images can be changed by hematoxylin and eosin staining, which stains cell nuclei blue and the extracellular matrix and cytoplasm pink.

Therefore, z-axis compression can only be applied to the R and B channels of the tissue region.

In embodiments of the present specification, the Otsu algorithm may first be applied to extract RGB values from the histopathological image, and then the value for the G (Green) channel may be removed.

Principal component analysis (PCA) can be applied to maximize the variation between the values for the R (Red) channel and the values for the B (Blue) channel and minimize the mean squared error. (Figure 3(d)). This process allows us to reduce the dimensionality of the image and perform a normalization process for color, which is widely used in histopathology.

3) Neural network training in compression domain (x-axis and y-axis compression) (Figure 1(c) & Figure 1(d))

After image depth compression (z-axis), a discrete wavelet transform (DWT) can be performed on each tile using the Haar wavelet basis function to compress information along the x- and y-axes. The sub-band of the 2D wavelet transform can be calculated using Equations 1 to 4.

here

is the coordinate of the input tile,

is the coordinate of the output subband,

and

is the 2D wavelet basis function of level j,

is an approximation of the original image, called LL (low-low) subband,

,

, and

are high-frequency components whose directions are horizontal, vertical, and diagonal, respectively.

These components may be referred to as low-high (LH) subband, high-low (HL) subband, and high-high (HH) subband, respectively. This method of using wavelet domain analysis has the following advantages.

First, the image size is reduced (e.g. from 1,024 × 1,024 pixels to 512 × 512 pixels), but all necessary information can be retained to completely reconstruct the original image. After reconstruction, the ROI can be increased without information loss, which is proportional to the generalization performance.

Second, this method can capture texture information in wavelet subbands according to cancer grading, so the result of 2D gray-level co-occurrence matrix (GLCM) in wavelet domain can capture texture. Useful for classification.

In the embodiments herein, four discrete wavelet transform subbands were input in parallel to each individual segmentation model, U-Net++.

In embodiments of the present specification, the DiceCE loss function combining the dice coefficient and cross-entropy may be used. As an example, each subband model uses two NVidia Titan

4) Full image prediction using wavelet weighted ensemble (Figure 1(e) & Figure 1(f) & Figure 4)

The reconstruction process is described below.

In embodiments of the present specification, after generating a whole probability map for each subband, a wavelet weighted ensemble (WWE) is applied to four trained artificial intelligence neural networks for each subband, as shown in FIG. 1(e). )-based ensemble learning was applied.

First, a binary mask image (FIG. 4(b)) can be obtained from the original image using the Otsu algorithm (FIG. 4(a)).

After 2D wavelet transformation based on the Haar wavelet, four wavelet subbands for the binary tissue mask were generated (Figure 4(c)). In the embodiments herein, they are defined as wavelet weights, namely LL weight, LH weight, HL weight and HH weight.

In the embodiments of this specification, a small value ε was added to each wavelet weight and then multiplied by the assigned weight (FIG. 4(d)).

Finally, the weights were multiplied by the corresponding probability map (Figure 4(e)) and then the inverse discrete wavelet transform, which also uses the Haar wavelet, was applied to obtain the final prediction (Figure 4(f)).

Parameters such as W1, W2, W3 and W4 were determined empirically. Ideally, if the same region of each subband has a probability of 1, the reconstruction probability of that region should also be 1 without these parameters. However, because the LL subband contains the basic characteristics of the original image, more weight (i.e., 1.8) was given to the LL subband. Then, ε was added to remove the zero-terms.

The wavelet weighted ensemble method used in the embodiments of the present specification can be expressed as Equation 5 below.

here

and

represents the 2D wavelet basis function of level 1,

represents an approximation of the binary tissue mask (LL subband weights) and

are the high-frequency components (LH, HL, and HH subband weights) for the binary tissue mask with directions horizontal, vertical, and diagonal, respectively.

and

describes the probability map for each subband.

is the final prediction result after wavelet weighted ensemble (WWE).

,

, and

To optimize the weight parameter such as W = (

,

) ego

Optimization that satisfies Equation 6, which is a function that determines the average of the Dice scores of x, was applied.

In the embodiments herein, the range of each parameter is 0.3 to 3.0 and the step size may be 0.3. As an example,

= 1.8;

= 0.9;

= 0.9, and

The parameter value was chosen as = 0.3.

C.Experiment setup

The quality of the prediction can be quantified using dice score (Dice) and pixel accuracy (Acc) as follows:

here

,

and

is the number of pixels for true positives, false positives, true negatives, and false negatives.

In the examples herein, the ensemble method described above (i.e., wavelet weighted ensemble (WWE)) on the WSI test data set was compared to other models in three ways:

(1) Apply sliding window tile extraction instead of using and compressing data after decimation (pixel size: 512 x 512 pixels, stride of sliding window: 128 pixels, x10 scale)

(2) Use compressed data such as LL, LH, HL and HH subbands

(3) using weighted average ensemble (WAE) for each subband result;

WAE can be expressed in Equation 9 as follows.

here

,

and

describes the probability map for each subband,

is the final prediction result after weighted average ensemble.

,

and

describes the probability maps for LL, LH, HL and HH subbands, respectively. Set the same weight value in WAE as in WWE. (

=1.8;

=0.9,

=0.9, and

=0.3).

To verify the superiority of the method proposed in the examples of this specification, an experiment was conducted as follows. (1) Compare the average Dice and Acc for each method, (2) observe the distribution of Dice and Acc for all classes, (3) low-frequency weight (

) and high-frequency weighting (

,

, and

), (4) compared sample images and line profiles according to each method, and (5) compared with the uncompressed method according to WWE's Dice, WAE, and tumor probability thresholds.

III. EXPERIMENTAL RESULTS AND DISCUSSIONS

To verify the method proposed in the embodiments of this specification, as an example, whole slide images (WSI) of 390 colon biopsy specimens were used. At this time, the average size of the entire slide image (WSI) was 43,443 x 28,645 pixels. The data set was divided into three groups: 274 train data, 77 validation data, and 39 test data.

In embodiments herein, a pipeline was implemented to achieve binary segmentation of normal and abnormal regions in colorectal cancer (CRC) tissue images using the dataset described above.

When comparing the average Dice and Acc for each method, the average Dice and Acc results for the LL subband increased by 5.6% and 1.6%, respectively, in the case of the model using compressed data compared to the uncompressed model.

However, the average Dice and Acc result: 1.5% for Acc) The subband carrying high frequency information was reduced compared to before compression.

Figure 5 shows Dice and Acc for each class according to the model used. For all tumor classes, the average results for the LL subband are relatively high. Additionally, the average results of LH, HL, and HH subbands carrying high-frequency components are relatively high in ADENOCA and TAH, and are relatively easy to detect due to progressive disease progression and subsequent pathological transformation. However, results for the LH, HL, and HH subbands are less predictive for TAL, CARCINOID, and HYPERP. TAL (relatively less advanced), CARCINOID (malignant tumor but originating in the submucosa), and HYPERP (benign tumor) are difficult to predict accurately using only high-frequency components.

Based on these results, embodiments of the present invention propose an ensemble method that can improve results by using both low-frequency information and high-frequency information.

For Weighted Average Ensemble (WAE), a widely used ensemble technique, even when weighting the results in the LL subband, the average Dice and Acc are lower than the average Dice and Acc in the LL subband (-5.7% for Dice and Acc -2.3%). Compared to the uncompressed results, ADENOCA and TAH showed good performance after WAE because Dice and Acc in high-frequency subbands such as LH, HL, and HH were higher than those in the uncompressed case. (ADENOCA: +4.9% for Dice, +0.0% for Acc, TAH: +3.7% for Dice, +0.2% for Acc). However, for TAL, CARCINOID, and HYPERP, which show poor performance in high frequency bands, Dice and Acc after weighted average ensemble (WAE) are lower than without compression (TAL: +0.2% for Dice, -0.4% for Acc CARCINOID : +4.3% for Dice, +2.3% for Acc, HYPERP: +11.1% for Dice, +4.1% for Acc;).

On the other hand, for wavelet weighted ensemble (WWE), the average Dice and Acc increase by about 0.6% and 0.2%, respectively, compared to LL. For each class: ADENOCA (+0.4% dice and +0.1% Acc), TAH (+0.7% dice and +0.2% Acc), TAL (+0.5% dice and +0.1% Acc) CARCINOID (+1.5 dice % and +0.1% for Acc) and HYPER (+0.8% for dice and +1.8% for Acc) gradually increase the results. Notably, unlike WAE, WWE has TAL (+9.2% for Dice and +1.6% for Acc), CARCINOID (+8.7% for Dice and +3.7% for Acc), and HYPERP (+14% for Dice; +4.9% for ACC), respectively, showing higher performance than WAE.

Low-frequency weighting (

) and high-frequency weighting (

,

and

), each weight was optimized by empirically verifying the change in Dice scores of all classes. In WWE, the best weight can be determined by average dice score.

Figure 6 shows the low-frequency weights (

) explains the change in dice score.

Referring to Figure 6, the dice scores of all classes increase relatively steeply from 0.6 to 0.9. In particular, the Dice score increase rate of HYPERP and ADENOCA is relatively high.

Above a value of 1.2, dice scores start to become saturated for all classes. Additionally, high-frequency weighting (

,

and

), but the change in dice score can be ignored.

Figure 7 shows the results of comparing segmentation prediction for annotations, a method without compression, WAE, and WWE using a heatmap and line profile for tumor probability (Figures 7a-d). Color bars represent tumor probability for each pixel. The heatmap is overlaid on the original histology image, and an enlarged image of the colored border area is located on top of the main image. A line profile of tumor probability cut along the red dotted line is below the main image.

Figure 7(a) is the ground truth annotated by a pathologist. The pixel value of the annotation is 1 and the value of other areas is 0.

Figure 7(b) shows the segmentation results of the uncompressed method. For efficient training, there is some loss of high-frequency information after decimation, but the ROI used for single training is the same as other methods. The enlarged image in Figure 7(b) predicts a larger area than in the annotation, and the tumor probability of each pixel is relatively low.

As shown in Figure 7(c), the segmentation results for WAE are clearly qualitatively better than those without compression. WWE's final segmentation results have accurate edges and high probability at each pixel compared to other methods. Tumor probability line profiles processed with WWE are most similar to the original annotated profiles.

WWE's Dice score for wavelet subbands, WAE for wavelet subbands and methods without compression can be compared over a range of critical tumor probability values. Between thresholds 0.1 and 0.4, the Dice score of the uncompressed method is slightly higher than WAE. However, beyond the threshold of 0.5, the dice score of this method drops sharply compared to other methods. WAE and WWE continue to perform strongly for all thresholds, and WWE's Dice score is consistently higher than WAE's due to high-frequency information.

Finally, we compared the WWE predicted images with the images annotated by the pathologist.

Figure 8 is a diagram showing expected results for tumors by class.

Figures 8a-e show histological images of five different tumor classes. The pathologist's annotations are shown in Figures 8f-j. The corresponding predicted probability maps using WWE are shown in Figures 8K-O. The proposed WWE method generally segmented the affected area, which matched well with the actual image.

WWE's average Dice Score and Acc are 0.852 ± 0.086 and 0.962 ± 0.027, respectively. The highest Dice score (0.887 ± 0.101) is TAH, where high-frequency information is important. On the other hand, the worst Dice score (0.830 ± 0.057) is TAL, where low-frequency information is important. In the case of HYPERP, it was observed that the normal area where dead nuclei were gathered was abnormally predicted, as can be seen in the yellow dotted box (Figure 8o-t). These abnormal predictions may be caused by artifacts such as tissue folds, ink, dust, and air bubbles and may require additional artifact removal. Despite these anomalies, the overall prediction of colorectal cancer using WWE was not biased toward any one class and performed well for all classes.

Through the method described above, diagnostic accuracy (e.g., Dice, Acc) can be increased by using a compressed region to reduce high-frequency information loss.

In an embodiment of the present specification, a WWE method was proposed that learns and combines low-frequency components and high-frequency components separately in the compression region. Using the NVIDIA TITAN So without compression

To learn the experimental ROI size, the resolution of the original image (x20 magnification) must be lowered (x10 magnification). In this process, loss of high-frequency components cannot be avoided.

On the other hand, the method proposed in the embodiment of this specification can process a tile size of 1024 x 1024 before compression. Therefore, there is no need to lower the resolution to learn the same ROI size, and learning is possible at 20x magnification. Additionally, the method proposed in the embodiment of this specification can learn tiles four times larger than the hardware limit compared to the method without compression.

On the other hand, the method described in the embodiments herein may require four times more GPUs at the same time. From the perspective of time resources, in the case of a general 2D convolution-based CNN, the amount of computation increases exponentially as the input size increases. Therefore, it is faster to learn one image by dividing it into four images than to learn an image four times larger at once. This case is similar to the principle of the Cooley-Tukey FFT algorithm.

IV. CONCLUSION

Due to hardware limitations, research was conducted to prevent loss of high-frequency information that occurs during the process of resizing images and to increase the accuracy of the final result by using protected high-frequency information. Using the wavelet weighted ensemble method, we found that accuracy was improved over uncompressed images. The overall accuracy was determined by the low-frequency component, while the high-frequency component affected the margin.

The processing methods described in the examples herein were applied to colorectal cancer histopathology images. This processing method can also be applied to general histopathology images and shows similar increases in accuracy. The proposed wavelet-weighted ensemble method can also be applied to process large-scale images (e.g., astronomy and satellite images) and to other margin-critical fields (e.g., radiotherapy).

Figure 9 is a diagram showing the schematic structure of the tissue slide image analysis system 100.

Referring to FIG. 9 , the tissue slide image analysis system 100 may include an extraction unit 110, a probability map generation unit 120, and a class classification unit 130.

The extraction unit 110 may extract a plurality of tile images from the input tissue slide image.

At this time, the tissue slide image is, for example, a slide image of a colon (colorectal) biopsy tissue, and may be the above-described whole slide image (WSI).

When the tissue slide image is a slide image of a colon biopsy tissue, the colon biopsy tissue may be a tissue to which the above-described hematoxylin and eosin (H&E) staining has been applied. As described above, when hematoxylin and eosin staining is applied, cell nuclei may be stained blue and the extracellular matrix and cytoplasm may be stained pink in colonic biopsy tissue.

Meanwhile, the plurality of tile images may be images of a smaller size than the tissue slide image. At this time, the width and height of the plurality of tile images may be the first size (e.g. 1024).

The extractor 110 may generate a plurality of tile images based on the sliding window algorithm described above. That is, the extraction unit 110 extracts the portion overlapping the sliding window on the tissue slide image as a tile image, then moves the position of the sliding window, and repeats the process of extracting the tile image again to generate a plurality of tile images. You can.

For example, the plurality of tile images may be RGB images having a red (R) channel, a blue (B) channel, and a green (G) channel.

Meanwhile, the extractor 110 may selectively perform a compression operation on a plurality of tile images.

When the plurality of tile images are RGB images, the G channel component for the plurality of tile images can be removed. In this case, only R-channel and B-channel components remain in the plurality of tile images.

The extractor 110 can compress a plurality of tile images from which the G-channel component has been removed into a 1-channel image. At this time, the channel of the plurality of compressed tile images may be, for example, a monochrome channel or a binary channel.

The probability map generator 120 may input a plurality of tile images into an artificial intelligence model and generate an expected probability map for the plurality of tile images based on output information of the artificial intelligence model. At this time, the expected probability map for the plurality of tile images may be set as a two-dimensional map indicating the probability that a pixel or a specific size area on the plurality of tile images is an area where a disease (e.g. tumor) has occurred. As an example, the probability map may be the same two-dimensional image as the tile image. At this time, the value of pixel (i, j) of the probability map may indicate the probability that pixel (i, j) of the tile image is included in the area where a disease (e.g. tumor) has occurred.

Meanwhile, the probability map generator 120 may additionally perform learning on the artificial intelligence model by comparing the output result of the artificial intelligence model with whether a tumor has occurred in the actual tissue slide image.

At this time, the artificial intelligence model may be a convolutional neural network network, for example. However, the artificial intelligence model may be a deep learning model other than a convolutional neural network.

A deep learning model may be a model in which artificial neural networks are stacked in multiple layers. In other words, the deep learning model automatically learns the characteristics of the input value by learning a large amount of data in a deep neural network made up of multi-layer networks, and through this, the network is trained to minimize the error in the objective function, that is, prediction accuracy. It is a model of

In the present invention, the deep learning model is a convolutional neural network (CNN) as an example, but the present invention is not limited to this and can use various deep learning models that can be used now or in the future.

Deep learning models can be implemented through deep learning frameworks. The deep learning framework provides commonly used functions in the form of a library when developing deep learning models and supports the good use of system software or hardware platforms. In this embodiment, the deep learning model can be implemented using any deep learning framework that is currently published or will be released in the future.

The class classification unit 130 may classify the tissue slide image into one of a plurality of classes based on the expected probability map generated by the probability map generator 120. As an example, the class classification unit 130 may compare the pattern of the expected probability map for the tissue slide image with the reference pattern set for each of the plurality of classes and set the most similar class as the class of the corresponding tissue slide image.

At this time, the plurality of classes include the aforementioned i) adenocarcinoma (ADENOCA), ii) high grade adenoma with dysplasia (TAH), and iii) low grade adenoma with dysplasia (TAL). ), iv) carcinoid (CARCINOID), and v) hyperplastic polyp (HYPERP).

Figure 10 is a diagram showing a tissue slide image analysis method 1000.

Referring to FIG. 10, the tissue slide image analysis method 1000 may first include an extraction step (S1010) of extracting a plurality of tile images from the tissue slide image.

At this time, the tissue slide image may be a slide image of colon biopsy tissue. For example, the colon biopsy tissue may be a tissue to which hematoxylin and eosin staining has been applied.

Meanwhile, the extraction step (S1010) may extract a plurality of tile images based on a sliding window algorithm.

For example, the extracted plurality of tile images may be RGB images having an R (Red) channel, a B (Blue) channel, and a G (Green) channel.

Meanwhile, the extraction step (S1010) may optionally include compressing a plurality of tile images. At this time, compressing the plurality of tile images may include removing the G-channel component of the plurality of tile images and compressing each of the plurality of tile images from which the G-channel component has been removed into a 1-channel image. .

And the tissue slide image analysis method 1000 may include an input step (S1020) of inputting a plurality of tile images into an artificial intelligence model. The artificial intelligence model may be, for example, a convolutional neural network network.

And the tissue slide image analysis method 1000 may include a probability map generation step (S1030) of generating an expected probability map for a plurality of tile images based on output information of the artificial intelligence model.

Additionally, the tissue slide image analysis method 1000 may include a class classification step (S1040) of classifying the tissue slide image into one of a plurality of classes based on the expected probability map.

At this time, the plurality of classes may include adenocarcinoma, high-grade adenoma with dysplasia, low-grade adenoma with dysplasia, carcinoid, and hyperplastic polyp.

The tissue slide image analysis system 100 described above may be implemented by a computing device including at least some of a processor, memory, user input device, and presentation device. Memory is a medium that stores computer-readable software, applications, program modules, routines, instructions, and/or data that are coded to perform specific tasks when executed by a processor. The processor may read and execute computer-readable software, applications, program modules, routines, instructions, and/or data stored in memory. A user input device may be a means for allowing a user to input a command that causes the processor to execute a specific task or to input data required to execute a specific task. User input devices may include a physical or virtual keyboard, keypad, key buttons, mouse, joystick, trackball, touch-sensitive input means, or microphone. Presentation devices may include displays, printers, speakers, or vibrating devices.

Computing devices may include a variety of devices such as smartphones, tablets, laptops, desktops, servers, and clients. A computing device may be a single stand-alone device or may include multiple computing devices operating in a distributed environment comprised of multiple computing devices cooperating with each other through a communication network.

In addition, the tissue slide image analysis method 1000 described above includes a processor, and is coded to perform an image diagnosis method using a deep learning model when executed by the processor, computer readable software, applications, program modules, It can be executed by a computing device having a memory storing routines, instructions, and/or data structures.

The above-described embodiments can be implemented through various means. For example, the present embodiments may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the image diagnosis method using the deep learning model according to the present embodiments includes one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), It can be implemented by Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers, or microprocessors.

For example, the tissue slide image analysis method 1000 according to embodiments may be implemented using an artificial intelligence semiconductor device in which neurons and synapses of a deep neural network are implemented with semiconductor devices. At this time, the semiconductor device may be currently used semiconductor devices, such as SRAM, DRAM, or NAND, or may be next-generation semiconductor devices such as RRAM, STT MRAM, or PRAM, or a combination thereof.

When implementing the tissue slide image analysis method 1000 according to the embodiments using an artificial intelligence semiconductor device, the results (weights) of learning the deep learning model with software are transferred to a synapse mimic device arranged in an array or an artificial intelligence semiconductor device. You can also learn on your device.

In the case of implementation by firmware or software, the tissue slide image analysis method 1000 according to the present embodiments may be implemented in the form of a device, procedure, or function that performs the functions or operations described above. Software code can be stored in a memory unit and run by a processor. The memory unit is located inside or outside the processor and can exchange data with the processor through various known means.

Additionally, terms such as "system", "processor", "controller", "component", "module", "interface", "model", or "unit" described above generally refer to computer-related entities hardware, hardware and software. It may refer to a combination of, software, or running software. By way of example, but not limited to, the foregoing components may be a process, processor, controller, control processor, object, thread of execution, program, and/or computer run by a processor. For example, both an application running on a controller or processor and the controller or processor can be a component. One or more components may reside within a process and/or thread of execution, and the components may be located on a single device (e.g., system, computing device, etc.) or distributed across two or more devices.

Meanwhile, another embodiment provides a computer program stored in a computer recording medium that performs the tissue slide image analysis method 1000 described above. Another embodiment also provides a computer-readable recording medium on which a program for implementing the tissue slide image analysis method described above is recorded.

The program recorded on the recording medium can be read, installed, and executed on the computer to execute the above-described steps.

In this way, in order for the computer to read the program recorded on the recording medium and execute the functions implemented by the program, the above-mentioned program is a C, C++ program that the computer's processor (CPU) can read through the computer's device interface (Interface). , may include code coded in computer languages such as JAVA and machine language.

These codes may include functional codes related to functions defining the above-mentioned functions, etc., and may also include control codes related to execution procedures necessary for the computer processor to execute the above-described functions according to predetermined procedures.

In addition, these codes may further include memory reference-related codes that determine which location (address address) in the computer's internal or external memory the additional information or media required for the computer's processor to execute the above-mentioned functions should be referenced. .

In addition, if the computer's processor needs to communicate with any other remote computer or server in order to execute the above-mentioned functions, the code is It may further include communication-related codes for how to communicate with other computers, servers, etc., and what information or media should be transmitted and received during communication.

Recording media that can be read by a computer recording the above-described program include, for example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical media storage, etc., and also include carrier wave (e.g. , transmission via the Internet) may also be implemented.

Additionally, computer-readable recording media can be distributed across computer systems connected to a network, so that computer-readable code can be stored and executed in a distributed manner.

In addition, the functional program for implementing the present invention and the code and code segments related thereto are designed by programmers in the technical field to which the present invention belongs, taking into account the system environment of the computer that reads the recording medium and executes the program. It can also be easily inferred or changed by .

The tissue slide image analysis method 1000 described with reference to FIG. 12 may also be implemented in the form of a recording medium containing instructions executable by a computer, such as an application or program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include all computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The above-described tissue slide image analysis method can be executed by an application installed by default on the terminal (this may include programs included in the platform or operating system, etc. installed by default on the terminal), and the user can use an application store server, application or It may also be executed by an application (i.e. program) installed directly on the master terminal through an application providing server such as a web server related to the service. In this sense, the above-described tissue slide image analysis method may be implemented as an application (i.e., program) installed by default in the terminal or directly installed by the user and recorded on a computer-readable recording medium, such as in the terminal.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

CROSS-REFERENCE TO RELATED APPLICATIONCROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority under Article 119(a) of the U.S. Patent Act (35 U.S.C § 119(a)) to Patent Application No. 10-2022-0038949 filed in Korea on March 29, 2022. All contents are hereby incorporated by reference into this patent application. In addition, if this patent application claims priority for a country other than the United States for the same reasons as above, the entire contents thereof will be incorporated into this patent application by reference.

Claims

An extraction unit that extracts a plurality of tile images from the tissue slide image;

a probability map generator that inputs the plurality of tile images into an artificial intelligence model and generates an expected probability map for the plurality of tile images based on output information of the artificial intelligence model; and

A tissue slide image analysis system comprising a class classification unit that classifies the tissue slide image into one of a plurality of classes based on the expected probability map.
According to paragraph 1,

The tissue slide image is,

Tissue slide image analysis system, which is a slide image of colonic biopsy tissue.
According to paragraph 2,

A tissue slide image analysis system wherein the colonic biopsy tissue is a tissue to which hematoxylin and eosin staining has been applied.
According to paragraph 1,

The extraction unit,

A tissue slide image analysis system that generates the plurality of tile images based on a sliding window algorithm.
According to paragraph 1,

A tissue slide image analysis system wherein the plurality of tile images are RGB images having an R (Red) channel, a B (Blue) channel, and a G (Green) channel.
According to paragraph 1,

The artificial intelligence model is a tissue slide image analysis system that is a convolutional neural network.
According to paragraph 1,

The plurality of classes are,

Image analysis system for tissue slides including adenocarcinoma, high-grade adenoma with atypia, low-grade adenoma with atypia, carcinoid, and hyperplastic polyp.
An extraction step of extracting a plurality of tile images from the tissue slide image;

An input step of inputting the plurality of tile images into an artificial intelligence model;

A probability map generating step of generating an expected probability map for the plurality of tile images based on output information of the artificial intelligence model; and

A tissue slide image analysis method comprising a class classification step of classifying the tissue slide image into one of a plurality of classes based on the expected probability map.
According to clause 8,

The tissue slide image is,

Method for analyzing tissue slide images, which are slide images of colonic biopsy tissue.
According to clause 9,

A tissue slide image analysis method wherein the colon biopsy tissue is a tissue to which hematoxylin and eosin staining has been applied.
According to clause 8,

The extraction step is,

A tissue slide image analysis method for extracting the plurality of tile images based on a sliding window algorithm.
According to clause 8,

A tissue slide image analysis method wherein the plurality of tile images are RGB images having an R (Red) channel, a B (Blue) channel, and a G (Green) channel.
According to clause 8,

The artificial intelligence model is a tissue slide image analysis system that is a convolutional neural network.
According to clause 8,

The plurality of classes are,

Image analysis system for tissue slides including adenocarcinoma, high-grade adenoma with atypia, low-grade adenoma with atypia, carcinoid, and hyperplastic polyp.