WO2021073279A1

WO2021073279A1 - Staining normalization method and system for digital pathological image, electronic device and storage medium

Info

Publication number: WO2021073279A1
Application number: PCT/CN2020/112366
Authority: WO
Inventors: 南洋; 王佳平; 李风仪; 侯晓帅; 谢春梅
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-10-18
Filing date: 2020-08-31
Publication date: 2021-04-22
Also published as: CN111028923A; CN111028923B

Abstract

A staining normalization method and system for a digital pathological image, an electronic device and a storage medium. The staining normalization method for a digital pathological image comprises: performing data parsing on a pre-stored digital pathological slice image to generate an RGB image I(x,y) (S110); performing HSD transform on the RGB image I(x,y) according to a preset conversion rule, and converting the RGB image into an HSD image (S120); using the HSD image to continuously train a deep convolutional Gaussian mixture model to extract and solve Gaussian mixture models for different styles of images until the optimal deep convolutional Gaussian mixture model is obtained (S130); and using the optimal deep convolutional Gaussian mixture model to perform staining normalization on a histopathology HE stained digital pathological image to be detected (S140). By using the staining normalization method for a digital pathological image, the effectiveness and stability of staining normalization may be effectively improved.

Description

Digital pathological image staining normalization method, system, electronic device and storage medium

This application claims the priority of the Chinese patent application filed with the Chinese Patent Office on October 18, 2019, the application number is 201910993386.3, and the invention title is "Digital Pathology Image Staining Normalization Method, Electronic Device and Storage Medium", and its entire content Incorporated in this application by reference.

Technical field

This application relates to the field of image processing of digital medicine, and in particular to a method, system, electronic device, and storage medium for normalizing digital pathological image staining.

Background technique

Due to the wide variety of kidney diseases, the etiology and pathogenesis are complex, the clinical manifestations of many kidney diseases are not completely consistent with the histological changes of the kidney, and the treatment plans and the development results of the disease are also very different. At present, the results of renal pathological examinations have become a gold indicator for the diagnosis of kidney diseases. When pathologists perform renal biopsy, they need to obtain some important medical indicators based on visual and empirical observations. Now they plan to assist pathologists in reading pictures through artificial intelligence. However, the inventor found that different technicians cannot guarantee that the slides produced in each batch will have the same color distribution during the production and dyeing process. This will cause greater interference to the AI + medical AI model of digital medical care and cause predictions. Fluctuations in results.

At present, after the image is cut by the feature block, the size of the feature block generated by the cutting is different. At this time, it is necessary to adopt a normalization operation to unify the size of the feature block image. The existing normalization of coloring is usually achieved through methods such as confrontation generation networks and variational autoencoders. Although this method can perform style transfer, However, uncontrollable noise points are easily generated, which may change the image structure. Therefore, a more stable and natural dyeing normalization method is urgently needed.

Summary of the invention

This application provides a method, system, electronic device and storage medium for staining and normalization of digital pathological images. Its main purpose is to solve the problem that the current staining normalization can be achieved by confrontation generation networks, variational autoencoders and other methods. Perform style transfer, but it is easy to produce uncontrollable noise points, which may change the problem of image structure, and effectively improve the stability of the stained normalized image.

In order to achieve the above objective, the present application provides a method for normalizing digital pathological image staining, which is applied to digital pathological images, and includes the following steps:

S110: Perform data analysis on the pre-stored digital pathological slice image to generate an RGB image I(x,y),

Among them, I _R is the two-dimensional matrix of the R channel in the _{RGB image, I G} is the two-dimensional matrix of the G channel in the RGB image, and I _B is the two-dimensional matrix of the B channel in the RGB image;

S120: Perform HSD conversion on the RGB image I(x, y) according to a preset conversion rule, and convert the RGB image into an HSD image;

S130: Use the HSD image to continuously train a deep convolutional Gaussian mixture model to extract Gaussian mixture models for solving images of different styles, until an optimal deep convolutional Gaussian mixture model is obtained;

S140: Perform staining normalization on the HE stained digital pathology image of the histopathology to be detected by the optimal deep convolution Gaussian mixture model.

In order to achieve the above objectives, this application provides a digital pathological image staining normalization system, including:

The RGB unit performs data analysis on the pre-stored digital pathological slice image to generate an RGB image I(x,y),

The HSD unit performs HSD conversion on the RGB image I(x, y) according to a preset conversion rule, and converts the RGB image into an HSD image;

The convolutional neural network unit uses the HSD image to continuously train a deep convolutional Gaussian mixture model to extract Gaussian mixture models for solving different styles of images, until an optimal deep convolutional Gaussian mixture model is obtained;

The staining normalization unit performs staining normalization on the HE stained digital pathology image of the histopathology to be detected through the optimal depth convolution Gaussian mixture model.

In order to achieve the above objective, the present application provides an electronic device, which includes a memory, a processor, and a coloring normalization program for a digital pathological image stored in the memory, and the coloring of the digital pathological image is normalized When the program is executed by the processor, the following steps are performed:

In addition, in order to achieve the above object, the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a staining normalization program for a digital pathological image, and the staining of the digital pathological image is unified. When the transformation program is executed by the processor, the following steps are implemented:

The digital pathological image staining normalization method, system, electronic device, and computer-readable storage medium proposed in this application generate RGB image I(x,y) by analyzing the digital pathological slice image, and then calculate the RGB image I(x,y) according to the conversion rules. The RGB image I(x,y) is subjected to HSD transformation, and then the RGB image is converted into an HSD image. The HSD image is used to continuously train the deep convolutional Gaussian mixture model to extract the Gaussian mixture model for solving different styles of images, and then through the depth The convolutional Gaussian mixture model performs staining normalization on HE stained digital pathological images of histopathology to be detected, which effectively improves the effectiveness and stability of staining normalization.

In order to achieve the above and related objects, one or more aspects of the present application include features that will be described in detail later. The following description and drawings illustrate some exemplary aspects of the present application in detail. However, these aspects indicate only some of the various ways in which the principles of the present application can be used. Furthermore, this application is intended to include all these aspects and their equivalents.

Description of the drawings

Fig. 1 is a flowchart of an embodiment of a method for normalizing digital pathological image staining according to the present application;

FIG. 2 is a schematic diagram of RGB image generation according to an embodiment of a method for normalizing staining of digital pathological images according to the present application;

Figure 3 is a system framework diagram of an embodiment of the application;

Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

The realization, functional characteristics, and advantages of the purpose of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the application, and not used to limit the application.

At present, after the feature block is cut in the image, the size of the feature block generated by the cutting is different. At this time, it is necessary to adopt a normalization operation to unify the size of the feature block image. The existing normalization of coloring is usually achieved through methods such as confrontation generation networks and variational autoencoders. Although this method can perform style transfer, However, it is easy to produce uncontrollable noise points, which may change the image structure. In order to avoid the inability to ensure that the glass slides produced in each batch have the same color distribution during the dyeing process, this will cause greater interference to the AI model and cause predictions Due to the fluctuation of the results, this application provides a method for normalizing digital pathology image staining, which combines DCGMM with deep convolutional neural networks to enhance the interpretability of the model and improve the stability of training, so as to perform on the premise of preserving structural information Style transfer provides a guarantee for the accuracy of medical diagnosis.

Fig. 1 is a flowchart of a method for normalizing staining of digital pathological images in this application. In this embodiment, the method for normalizing staining of digital pathological images includes the following steps:

S110: Perform data analysis on the pre-stored digital pathological slice image to generate an RGB image I(x, y),

The pre-stored digital pathological slice image can be a digital pathological slice image that is scanned by a slice scanner to a computer for storage, or it can be a digital pathological slice image extracted from a pathological slice database. Read these digital pathology slice images used as training materials into the opencv image processing library, and generate color images in RGB space in the opencv image processing library, that is, RGB images;

FIG. 2 is a schematic diagram of RGB image generation according to an embodiment of a method for normalizing staining of digital pathological images of the present application. As shown in Figure 2, the pathological slice on the left is finally processed into a color image in the RGB space on the right.

S120: Perform HSD transformation on the RGB image I(x, y) generated in step S110 according to the preset conversion rule, and convert the RGB image into HSD image; among them, the conversion process of HSD into RGB image into HSD image, RGB image , HSD images are all images in a specific format.

In RGB images, the mixed information of color (chromatic) and intensity (intensity) hinders the standardization of color recognition, and HSD images will not hinder the standardization of color recognition. Therefore, before image training, the RGB image needs to be converted to HSD. image;

Among them, the conversion rules are:

Where I _R represents the two-dimensional matrix of the R channel in the RGB image, I _G represents the two-dimensional matrix of the G channel in the RGB image, and I _S represents the two-dimensional matrix of the S channel;

S130: Use the HSD image to continuously train the deep convolutional Gaussian mixture model to extract the Gaussian mixture model for solving different style images, until the optimal solution is obtained, and the optimal deep convolutional Gaussian mixture model is obtained;

GMM (Gaussian Mixture Model, Gaussian Mixture Model),

The essence of the image is a three-dimensional matrix. The characteristics of some objects in the image (such as the color of cells, tissues, base fluid, morphological characteristics, etc.) can be expressed by a single Gaussian model or a mixture of multiple single Gaussian models. GMM is to use Gaussian probability density function to accurately quantify something, and decompose a class of objects into a number of models based on Gaussian probability density function (normal distribution curve). For pathological images, the gray distribution value in each channel is generally multi-peak. By treating the multi-peak characteristics of the histogram as the superposition of multiple Gaussian distributions, image segmentation can be performed to distinguish the category to which each point in the image belongs. In order to solve the GMM (Gaussian mixture model Gaussian mixture model) of different styles of stained images, it is necessary to solve π _k , ∑ _k , μ _k . In the traditional method, the parameters of GMM are processed by the EM (Expectation-Maximization algorithm, EM, maximum expected value) algorithm. To solve the problem, first, perform E-step to extract the posterior probability γ, and then use the extracted γ to perform M-step calculation to obtain π _k , ∑ _k , and μ _k that maximize the log-likelihood function.

In this application, the extraction of γ in the E-step is completed through a deep convolutional neural network, and the solution of the M-step is regarded as an optimization problem, and π _{k that minimizes the negative log-likelihood function is obtained} , ∑ _k , μ _k , the negative log likelihood function is regarded as Loss, and the optimization problem is solved.

Specifically, each pixel in the original image is a certain type of probability value p(x),

Where π _k is the probability of taking the k-th Gaussian distribution, and N represents _{the multivariate normal distribution with the mean μ k} and the covariance matrix Σk. In the GMM solution of digital pathology images for histopathological HE staining (hematoxylin-eosin staining, HE staining for short), p(x) is the category represented by a certain point in the current image. The formula indicates that for each point x in the image, it can be expressed as a superposition of k Gaussian mixture models, so it is necessary to use a neural network to replace the traditional E-step to calculate the parameter γ, and then use the existing γ to calculate μ Evaluate with ∑ to solve the category corresponding to each point in the image.

S131: Extract the posterior probability vector γ through the convolutional neural network, and convert each pixel in the HSD image into a k-dimensional vector γ with a value between (0, 1) through the normalization operation, and this value represents 3 The probability value corresponding to each category, that is, the input image is separated into different channels according to the tissue structure; in this application, k=3 is set to correspond to the three categories in the image, corresponding to the nucleus, tissue, and background respectively; this Planning operations generally include: convolution, pooling, nonlinear activation functions, etc.;

S132: Acquire the pixel point X of the image = {x ₁ ,x ₂ ,...,x _p }, the probability that the pixel point x ₁ , x ₂ ,..., x _{p in} the image is generated by the kth Gaussian distribution is:

Among them, the hidden variable z represents the distribution of colors _{with mean μ=[μ 1} ,.., μ _k ] and covariance ∑=σ ^{2 I;}

_{S133: Solve π k} , ∑ _k , μ _k from the vector γ and the HSD image in combination with the probability, and repeatedly train the deep convolutional Gaussian mixture model.

Specifically, the deep convolutional Gaussian mixture model (DCGMM) obtains gamma through the forward propagation of the network, and then solves π _k , ∑ _k , μ _k through gamma, and then calculates the log-likelihood function, and then minimizes the log-likelihood Function, use gradient descent algorithm to update π _k , ∑ _k , μ _k ; make the negative log likelihood function as small as possible, that is, keep updating until π _k , ∑ _k , μ _k are obtained to maximize the log likelihood function; Among them, gamma is the posterior probability;

_{Calculate and obtain π k} , ∑ _k , μ _k that minimize the log-likelihood function in formula (2). The calculation details are as follows:

for i=1,2,...,k and j=1,2...,p:

μ _i ＝E(x _i |argmax(γ _j )=i)

∑ _i ＝E[(x _j -E[x _j ])(x _j -E[x _j ]) ^T )|argmax(γ _j )=i]

z=N(μ,∈·∑)

Combined with formula (1), its log likelihood function is:

S134: Training the deep convolutional Gaussian mixture model (DCGMM) repeatedly based on the deep learning convolutional neural network until the optimal deep convolutional Gaussian mixture model is obtained, so that DCGMM can predict which GMM each pixel of the current picture belongs to according to the current picture A single Gaussian model, and then transform the HSD of the pixel to be transformed into the HSD value of the template image to complete the style transfer, such as judging which category the current pixel belongs to (background, cell, tissue).

In the process of extracting the posterior probability vector γ through the convolutional neural network, and repeatedly training the deep convolutional Gaussian mixture model based on the deep learning convolutional neural network, the neural network layers used include: convolutional layer, batch processing layer, activation layer , Pooling layer and Upsampling layer;

The principle of the convolutional layer is:

F _con (I)=I·W+b (4)

In formula (4), the size of the image I is (H*W*C), W represents the weight of the convolution kernel and filter in the convolution layer, and b represents the bias term. The convolutional layer is based on the kernel and extracts the key features of the image.

In addition to the normal convolution, in order to increase the Reception Field of the neural network, the Dilated Convolution (Conv _dilated ) is also used in the middle layer of the network. The convolutional layer adds a hyperparameter expansion rate (Dilation Rate), which represents the number of intervals between the convolution kernel filters. Compared with the MaxPoolingLayer, the hole convolution can retain the internal data structure, avoiding the use of the pooling layer to cause information loss, and increasing the receptive field ；

The principle of the batch layer is:

F _BN-recale =γ*F _{BN-normalization} +β (6)

I is the input feature image, formula (5) standardizes I, E(I) and Var(I) are the mean and variance of I respectively; formula (6) performs scaling transformation, γ and β are scaling factors and Offset. Batch processing can prevent gradient explosion and gradient disappearance, and speed up model convergence;

The principle of the activation layer is:

F _ReLU =max(α×I,I) (7)

The formula (7) indicates that when I is a positive number, no transformation is performed, and when I is a negative number, it is output with a certain probability. The ReLU layer makes the convolution operation nonlinear, so that the model can fit complex actual results;

The principle of the pooling layer is:

Maximum pooling down-samples the input tensor, selecting the maximum value of each region, and reducing the tensor dimension after maximum pooling;

The principle of UpSampling layer is:

By adopting the bilinear interpolation method, that is, on the basis of the original image pixels, using a suitable interpolation algorithm to insert new elements between the pixels, to achieve the purpose of enlarging the size of the feature map;

After the network goes through the up-sampling and down-sampling steps, it passes through the last Softmax activation layer. Assuming that there is an array V containing j elements in total, Vi represents the i-th element in V. The calculation method is as follows:

After the activation function is activated, the feature vector γ that is consistent with the length and width of the input image is output.

The digital pathological image staining normalization method in this embodiment generates RGB image I (x, y) by analyzing the digital pathological slice image data, and performs HSD transformation on the RGB image I (x, y) according to the conversion rule, and then Convert RGB images to HSD images, use HSD images to continuously train the deep convolutional Gaussian mixture model to extract Gaussian mixture models for different styles of images, and then use the deep convolutional Gaussian mixture model to stain digital pathology images to be detected with HE staining. Unified, effectively improve the effectiveness and stability of dyeing normalization.

S140: Perform staining normalization on the HE stained digital pathology image of the histopathology to be detected through the optimal deep convolution Gaussian mixture model.

After training the DCGMM, apply DCGMM to the image to be tested. DCGMM automatically calculates the category of each pixel in the image to be detected. According to the category of each pixel, perform H, The conversion of S and D completes the normalization of dyeing.

Among them, the conversion includes changing the average value, whitening and color transformation; the color change transformation refers to scaling the whitening Gaussian distribution through singular value decomposition (SVD) to obtain the same covariance matrix as the template image.

FIG. 3 is a frame diagram of the digital pathological image staining normalization system involved in the digital pathological image staining normalization method according to an embodiment of the application. As shown in FIG. 3, the digital pathological image staining normalization system involved in this embodiment includes an RGB unit, an HSD unit, and a convolutional neural network unit;

The RGB unit is used to scan the pathological slice through the slice scanner into a computer for storage to generate a digital pathological slice image, so that the digital pathological slice image is read into the opencv image processing library, and the RGB space color image is generated in the opencv image processing library. That is, the RGB image I(x,y),

The HSD unit is used to perform HSD conversion on the RGB image I(x,y), and convert the RGB image into an HSD image. The conversion rules are:

Convolutional neural network unit includes E-step module and M-step module, composed of many neural network layers;

The E-step module is used to complete the extraction of γ in E-step through the deep convolutional neural network. Specifically, the vector γ of the posterior probability is extracted through the convolutional neural network, and each pixel in the HSD image is processed through the normalization operation. Converted into a k-dimensional vector γ with a value between (0,1), which represents the probability values corresponding to the three categories, that is, the input image is separated into different channels according to the organizational structure; in this application, k= 3. Corresponding to the three categories in the image, respectively corresponding to the three regions of cell nucleus, tissue, and background; this planning operation generally includes: convolution, pooling, non-linear activation function, etc.;

The M-step module is used to obtain the pixel points X={x ₁ ,x ₂ ,...,x _p } of the image, and the pixels x ₁ ,x ₂ ,...,x _{p in} the image are generated by the kth Gaussian distribution The probability is:

_{Combine probability to solve π k} , ∑ _k , μ _k from the vector γ and HSD image, and train the deep convolutional Gaussian mixture model repeatedly.

for i=1,2,...,k and j=1,2...,p:

μ _i ＝E(x _i |argmax(γ _j )=i)

∑ _i ＝E[(x _j -E[x _j ])(x _j -E[x _j ]) ^T )|argmax(γ _j )=i]

z=N(μ,∈·∑)

Combined with formula (1), its log likelihood function is:

The neural network layer includes a convolutional layer, a batch processing layer, an activation layer, a pooling layer and an Upsampling layer. The principle of the convolutional layer is:

F _con (I)=I·W+b

In this formula, the size of the image I is (H*W*C), W represents the weight of the convolution kernel and filter in the convolutional layer, and b represents the bias term. The convolutional layer is based on the kernel and extracts the key features of the image.

The principle of the batch layer is:

F _BN-recale =γ*F _{BN-normalization} +β

I is the input feature image, the formula

Standardize I, E(I) and Var(I) are the mean and variance of I, respectively; the formula F _BN-recale = γ*F _{BN-normalization} + β for scaling transformation, γ and β are the scaling factor and bias, respectively Shift. Batch processing can prevent gradient explosion and gradient disappearance, and speed up model convergence;

The principle of the activation layer is:

F _ReLU =max(α×I,I)

This formula indicates that when I is a positive number, no transformation is done, and when I is a negative number, it outputs with a certain probability. The ReLU layer makes the convolution operation nonlinear, so that the model can fit complex actual results;

The principle of the pooling layer is:

The principle of UpSampling layer is:

4 is a schematic diagram of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 40 may be a terminal device with arithmetic functions such as a server, a tablet computer, a portable computer, a desktop computer, and the like.

The electronic device 40 includes a processor 41, a memory 42, a computer program 43, a network interface, and a communication bus.

The electronic device 40 may be a tablet computer, a desktop computer, or a smart phone, but is not limited thereto.

The memory 42 includes at least one type of readable storage medium. The at least one type of readable storage medium may be a non-volatile storage medium such as flash memory, hard disk, multimedia card, card-type memory, and the like. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 40, such as a hard disk of the electronic device 40. In other embodiments, the readable storage medium may also be an external memory of the electronic device 40, such as a plug-in hard disk equipped on the electronic device 40, a smart media card (SMC), a secure digital ( Secure Digital, SD card, Flash Card, etc.

In this embodiment, the readable storage medium of the memory 42 is generally used to store the computer program 43 installed in the electronic device 40 and the like.

The processor 41 may be a central processing unit (CPU), microprocessor or other data processing chip in some embodiments, and is used to run program codes or processing data stored in the memory 42, such as digital pathological image staining. Normalization procedures, etc.

The network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the electronic device 40 and other electronic devices.

The communication bus is used to realize the connection and communication between these components.

FIG. 4 only shows the electronic device 40 with the components 41-43, but it should be understood that it is not required to implement all the illustrated components, and more or fewer components may be implemented instead.

In the device embodiment shown in FIG. 4, the memory 42 as a computer storage medium may include an operating system and a digital pathological image staining normalization program; the processor 41 executes the digital pathological image staining normalization program stored in the memory 42 The following steps are implemented when a program is changed:

S110: Scan the pathological slice into a computer through a slice scanner for storage to generate a digital pathological slice image, perform data analysis on the digital pathological slice image, and generate an RGB image I(x,y),

S120: Perform HSD conversion on the RGB image I(x, y) according to the preset conversion rule, and convert the RGB image into an HSD image;

In addition, the embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium includes digital pathological image staining and normalization. The normalization program based on the digital pathological image staining is executed by the processor to achieve the following operations:

The conversion rule is:

H, S, D are different channels of the image in the HSD space.

The vector γ of posterior probability is extracted through the convolutional neural network, where the vector γ is a k-dimensional vector with a value between (0,1);

Get the pixel point X={x ₁ ,x ₂ ,...,x _p } of the HSD image, and calculate _{the probability that the pixel point x 1} ,x ₂ ,...,x _p in the HSD image is generated by the kth Gaussian distribution:

z _k is the distribution of colors with mean μ=[μ ₁ ,.., μ _k ] and covariance ∑=σ ^{2 I;}

_{Combine probability to solve π k} , ∑ _k , μ _k from the vector γ and HSD image, and continuously train the deep convolution Gaussian mixture model to continuously update π _k , ∑ _k , μ _k using the gradient descent algorithm.

Convolutional neural networks include convolutional layers, batch processing layers, activation layers, pooling layers, and Upsampling layers.

_{In the process of solving π k} , ∑ _k , μ _k from the vector γ and HSD image in combination with probability, the process of repeatedly training the deep convolutional Gaussian mixture model includes:

The deep convolutional Gaussian mixture model obtains gamma through the forward propagation of the network, and solves π _k , ∑ _k , μ _k through gamma.

In the process of continuously updating π _k , ∑ _k , and μ _k , the log-likelihood function is used to continuously update until π _k , ∑ _k , and μ _k are obtained to maximize the log-likelihood function.

The process of staining and normalizing the HE stained digital pathology image of the histopathology to be detected by the deep convolution Gaussian mixture model includes:

Apply the trained deep convolutional Gaussian mixture model on the image to be tested. The deep convolutional Gaussian mixture model automatically calculates the category of each pixel in the image to be detected, and compares the original image and the image to be tested according to the category of each pixel. Perform the conversion of H, S, D for the areas of the same category in the same category to complete the normalization of dyeing;

The conversion of H, S, D includes at least average value, whitening and color conversion.

The specific implementation of the computer-readable storage medium of the present application is substantially the same as the specific implementation of the above-mentioned digital pathological image staining normalization method and electronic device, and will not be repeated here.

It should be noted that in this article, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements not only includes those elements, It also includes other elements not explicitly listed, or elements inherent to the process, device, article, or method. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, device, article, or method that includes the element.

The serial numbers of the foregoing embodiments of the present application are only for description, and do not represent the superiority or inferiority of the embodiments. Through the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disk, optical disk), including a number of instructions to make a terminal device (which may be a computer, a server, or a network device, etc.) execute the method described in each embodiment of the present application.

The above are only the preferred embodiments of the application, and do not limit the scope of the patent for this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of the application, or directly or indirectly applied to other related technical fields , The same reason is included in the scope of patent protection of this application.

Claims

A staining normalization method for digital pathological images, applied to the processing of digital pathological images, includes the following steps:

S110: Perform data analysis on the pre-stored digital pathological slice image to generate an RGB image I(x,y),
Among them, I R is the two-dimensional matrix of the R channel in the RGB image, I G is the two-dimensional matrix of the G channel in the RGB image, and I B is the two-dimensional matrix of the B channel in the RGB image;

S120: Perform HSD conversion on the RGB image I(x, y) according to a preset conversion rule, and convert the RGB image into an HSD image;

S130: Use the HSD image to continuously train a deep convolutional Gaussian mixture model to extract Gaussian mixture models for solving images of different styles until an optimal deep convolutional Gaussian mixture model is obtained;

S140: Perform staining normalization on the HE stained digital pathology image of the histopathology to be detected by the optimal deep convolution Gaussian mixture model.
The staining normalization method for digital pathological images according to claim 1, wherein the conversion rule is:

Wherein, the H, S, D are different channels of the image in the HSD space.
The staining normalization method of digital pathological images according to claim 1, wherein the process of using the HSD image to train a deep convolutional Gaussian mixture model to extract and solve Gaussian mixture models for images of different styles comprises:

Extracting a vector γ of posterior probability through a convolutional neural network; wherein the vector γ is a k-dimensional vector with a value between (0, 1);

Obtain the pixel points X={x 1 , x 2 ,..., x p } of the HSD image, and calculate the pixels x 1 , x 2 ,..., x p in the HSD image generated by the k-th Gaussian distribution Probability:

Said z k is the distribution of colors with mean μ=[μ 1 ,.., μ k ] and covariance ∑=σ 2 I;

Solve π k , Σ k , μ k from the vector γ and the HSD image in combination with the probability, and continuously train the deep convolution Gaussian mixture model to continuously update the π k , Σ k , μ k using a gradient descent algorithm.
The staining normalization method for digital pathological images according to claim 3, wherein the convolutional neural network includes a convolutional layer, a batch processing layer, an activation layer, a pooling layer, and an Upsampling layer.
The staining normalization method of digital pathological images according to claim 3, wherein the process of repetitive training of deep convolutional Gaussian mixture model is to solve π k , Σ k , μ k from the vector γ and the HSD image in combination with the probability Include:

The deep convolutional Gaussian mixture model obtains gamma through the forward propagation of the network, and solves π k , ∑ k , and μ k through gamma.
The staining normalization method of digital pathological images according to claim 3, wherein in the process of continuously updating the π k , Σ k , μ k , the log-likelihood function is used to continuously update until π k , Σ k , μ k maximizes the log likelihood function, and the log likelihood function is:
The method for staining normalization of digital pathology images according to claim 1, wherein the process of staining and normalizing the HE stained digital pathology image for histopathology to be detected through the optimal depth convolution Gaussian mixture model includes:

The optimal deep convolutional Gaussian mixture model is applied to the HE-stained digital pathological image of histopathology to be detected, and the optimal deep convolutional Gaussian mixture model automatically calculates the category to which each pixel in the image to be detected belongs, and according to Each pixel belongs to the same category of the original image and the image to be tested by the conversion of H, S, D to complete the normalization of dyeing.
The staining normalization method of digital pathological images according to claim 7, wherein the conversion of H, S, D includes at least average value, whitening and color conversion.
A staining normalization system for digital pathological images, including:

The RGB unit performs data analysis on the pre-stored digital pathological slice image to generate an RGB image I(x,y),
Among them, I R is the two-dimensional matrix of the R channel in the RGB image, I G is the two-dimensional matrix of the G channel in the RGB image, and I B is the two-dimensional matrix of the B channel in the RGB image;

The HSD unit performs HSD conversion on the RGB image I(x, y) according to a preset conversion rule, and converts the RGB image into an HSD image;

The convolutional neural network unit uses the HSD image to continuously train a deep convolutional Gaussian mixture model to extract Gaussian mixture models for solving different styles of images, until an optimal deep convolutional Gaussian mixture model is obtained;

The staining normalization unit performs staining normalization on the HE stained digital pathology image of the histopathology to be detected through the optimal depth convolution Gaussian mixture model.
An electronic device comprising: a memory, a processor, and a staining normalization program for a digital pathological image stored in the memory; when the staining normalization program for the digital pathological image is executed by the processor Perform the following steps:

S110: Perform data analysis on the pre-stored digital pathological slice image to generate an RGB image I(x,y),
Among them, I R is the two-dimensional matrix of the R channel in the RGB image, I G is the two-dimensional matrix of the G channel in the RGB image, and I B is the two-dimensional matrix of the B channel in the RGB image;

S120: Perform HSD conversion on the RGB image I(x, y) according to a preset conversion rule, and convert the RGB image into an HSD image;

S130: Use the HSD image to continuously train a deep convolutional Gaussian mixture model to extract Gaussian mixture models for solving images of different styles until an optimal deep convolutional Gaussian mixture model is obtained;

S140: Perform staining normalization on the HE stained digital pathology image of the histopathology to be detected by the optimal deep convolution Gaussian mixture model.
The electronic device according to claim 10, wherein the conversion rule is:

Wherein, the H, S, D are different channels of the image in the HSD space.
The electronic device according to claim 10, wherein the process of using the HSD image to train a deep convolutional Gaussian mixture model to extract and solve the Gaussian mixture model for images of different styles comprises:

Extracting a vector γ of posterior probability through a convolutional neural network; wherein the vector γ is a k-dimensional vector with a value between (0, 1);

Obtain the pixel points X={x 1 , x 2 ,..., x p } of the HSD image, and calculate the pixels x 1 , x 2 ,..., x p in the HSD image generated by the kth Gaussian distribution Probability:

Said z k is the distribution of colors with mean μ=[μ 1 ,.., μ k ] and covariance ∑=σ 2 I;

Solve π k , Σ k , μ k from the vector γ and the HSD image in combination with the probability, and continuously train the deep convolution Gaussian mixture model to continuously update the π k , Σ k , μ k using a gradient descent algorithm.
The electronic device according to claim 12, wherein the process of solving π k , Σ k , μ k from the vector γ and the HSD image in combination with the probability, and repeatedly training the deep convolutional Gaussian mixture model comprises:

The deep convolutional Gaussian mixture model obtains gamma through the forward propagation of the network, and solves π k , ∑ k , and μ k through gamma.
The electronic device according to claim 12, wherein in the process of continuously updating the π k , Σ k , and μ k , the log likelihood function is used to continuously update until π k , Σ k , and μ k are obtained such that the logarithm The likelihood function is the largest, and the log likelihood function is:
The electronic device according to claim 10, wherein the process of staining and normalizing the HE stained digital pathology image of the histopathology to be detected by the optimal deep convolution Gaussian mixture model comprises:

The optimal deep convolutional Gaussian mixture model is applied to the HE-stained digital pathological image of histopathology to be detected, and the optimal deep convolutional Gaussian mixture model automatically calculates the category to which each pixel in the image to be detected belongs, and according to Each pixel belongs to the same category of the original image and the image to be tested by the conversion of H, S, D to complete the normalization of dyeing.
A computer-readable storage medium in which a staining normalization program of a digital pathological image is stored, and when the staining normalization program of a digital pathological image is executed by a processor, the following steps are implemented:

S110: Perform data analysis on the pre-stored digital pathological slice image to generate an RGB image I(x,y),
Among them, I R is the two-dimensional matrix of the R channel in the RGB image, I G is the two-dimensional matrix of the G channel in the RGB image, and I B is the two-dimensional matrix of the B channel in the RGB image;

S120: Perform HSD conversion on the RGB image I(x, y) according to a preset conversion rule, and convert the RGB image into an HSD image;

S130: Use the HSD image to continuously train a deep convolutional Gaussian mixture model to extract Gaussian mixture models for solving images of different styles until an optimal deep convolutional Gaussian mixture model is obtained;

S140: Perform staining normalization on the HE stained digital pathology image of the histopathology to be detected by the optimal deep convolution Gaussian mixture model.
The computer-readable storage medium according to claim 16, wherein the conversion rule is:

Wherein, the H, S, D are different channels of the image in the HSD space.
The computer-readable storage medium according to claim 16, wherein the process of using the HSD image to train a deep convolutional Gaussian mixture model to extract and solve the Gaussian mixture model for images of different styles comprises:

Extracting a vector γ of posterior probability through a convolutional neural network; wherein the vector γ is a k-dimensional vector with a value between (0, 1);

Obtain the pixel points X={x 1 , x 2 ,..., x p } of the HSD image, and calculate the pixels x 1 , x 2 ,..., x p in the HSD image generated by the k-th Gaussian distribution Probability:

Said z k is the distribution of colors with mean μ=[μ 1 ,.., μ k ] and covariance ∑=σ 2 I;

Solve π k , Σ k , μ k from the vector γ and the HSD image in combination with the probability, and continuously train the deep convolution Gaussian mixture model to continuously update the π k , Σ k , μ k using a gradient descent algorithm.
The computer-readable storage medium according to claim 18, wherein the process of solving π k , Σ k , μ k from the vector γ and the HSD image in combination with the probability, and repeatedly training the deep convolutional Gaussian mixture model comprises:

The deep convolutional Gaussian mixture model obtains gamma through the forward propagation of the network, and solves π k , ∑ k , and μ k through gamma.
16. The computer-readable storage medium according to claim 16, wherein the process of staining and normalizing the HE stained digital pathology image of the histopathology to be detected by the optimal deep convolution Gaussian mixture model comprises:

The optimal deep convolutional Gaussian mixture model is applied to the HE stained digital pathology image of histopathology to be detected, and the optimal deep convolutional Gaussian mixture model automatically calculates the category to which each pixel in the image to be detected belongs, and according to Each pixel belongs to the same category of the original image and the image to be tested by the conversion of H, S, D to complete the normalization of dyeing.