CN114550169A - Training method, device, equipment and medium for cell classification model - Google Patents

Abstract

The application discloses a training method, a device, equipment and a medium of a cell classification model, and relates to the field of machine learning. The method comprises the following steps: acquiring a sample image and a cell label of the sample image, wherein the sample image comprises at least two cells, and the cell label is used for indicating the type of the cells in the sample image; performing data processing on the sample image through a cell classification model, and outputting a sample prediction hot spot diagram which is used for predicting the type of cells in the sample image; restoring the contour of each cell in the sample image through the contour of each cell nucleus in the sample image to obtain a cell segmentation image; generating a sample classification heat point diagram according to the cell marking and the cell segmentation diagram, wherein the sample classification heat point diagram is used for representing the types of the cells in the cell segmentation diagram; and training the cell classification model according to the loss between the sample prediction hotspot graph and the sample classification hotspot graph. The method and the device can realize weak supervised learning and improve the accuracy of the model.

Description

Training method, device, equipment and medium for cell classification model

technical field

The present application relates to the field of machine learning, and in particular, to a training method, apparatus, device and medium for a cell classification model.

Background technique

Modern medicine's research and understanding of tumors have brought forth new ones, and the means of treating malignant tumors are constantly improving. Among them, immunotherapy is an important way to fight cancer. The implementation of immunotherapy requires the user's tumor cells and mononuclear inflammatory cells to meet the requirements.

The related art requires technicians to pre-mark the cell types. When training the cell classification model, the sample image needs to be divided into several sub-regions, and the images corresponding to the sub-regions are sequentially input into the cell classification model. Calculate the difference between the prediction result output by the cell classification model and the shape and type of the cell, and train the cell classification model in stages according to the difference to obtain the trained cell classification model.

However, the limited amount of information provided by cell annotations in sample images makes the related techniques less accurate.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a training method, device, equipment and medium for a cell classification model, the method can restore the outline of cells, extract more information from sample images, and make the classification results more accurate. The plan is as follows:

According to one aspect of the present application, there is provided a method for training a cell classification model, the method comprising:

acquiring a sample image and a cell annotation of the sample image, the sample image includes at least two types of cells, and the cell annotation is used to represent the type of cells in the sample image;

Data processing is performed on the sample image by the cell classification model, and a sample prediction heat map is output, where the sample prediction heat map is used to predict the type of cells in the sample image;

Through the contour of each cell nucleus in the sample image, the contour of each cell in the sample image is restored to obtain a cell segmentation map;

generating a sample classification heat map according to the cell labeling and the cell segmentation map, where the sample classification heat map is used to represent the types of cells in the cell segmentation map;

The cell classification model is trained according to the loss between the sample prediction heat map and the sample classification heat map.

According to one aspect of the present application, there is provided a training device for a cell classification model, the device comprising:

a sample acquisition module, configured to acquire a sample image and a cell label of the sample image, the sample image includes at least two types of cells, and the cell label is used to indicate the type of cells in the sample image;

a data processing module, configured to perform data processing on the sample image through the cell classification model, and output a sample prediction heat map, where the sample prediction heat map is used to predict the type of cells in the sample image;

The data processing module is further configured to restore the contour of each cell in the sample image by using the contour of each cell nucleus in the sample image to obtain a cell segmentation map;

The data processing module is further configured to generate a sample classification heat map according to the cell labeling and the cell segmentation map, where the sample classification heat map is used to represent the type of cells in the cell segmentation map;

A training module, configured to train the cell classification model according to the loss between the sample prediction heat map and the sample classification heat map.

According to an aspect of the present application, there is provided a cell classification method, the method is performed by a computer device, and the computer device runs the above-mentioned cell classification model, the method comprising:

acquiring an input image, the input image including at least two types of cells;

Data processing is performed on the input image by the cell classification model, and a predicted heat map is output, where the predicted heat map is used to represent the probability that the cell belongs to the target cell type;

Based on the predicted heat map, the type of each cell in the input image is determined.

According to one aspect of the present application, there is provided a cell sorting device, the device running the cell sorting model as described above, the device comprising:

an image acquisition module for acquiring an input image, the input image including at least two types of cells;

a model calling module, configured to perform data processing on the input image through the cell classification model, and output a predicted heat map, where the predicted heat map is used to represent the probability that a cell belongs to a target cell type;

A prediction module, configured to determine the type of each cell in the input image according to the predicted heat map.

According to another aspect of the present application, a computer device is provided, the computer device comprising: a processor and a memory, and the memory stores at least one instruction, at least one program, code set or instruction set, at least one instruction, at least one program , a code set or an instruction set is loaded and executed by the processor to implement the classification method of the cell classification model as described in the above aspect, or, the cell classification method.

According to another aspect of the present application, a computer storage medium is provided, in which at least one program code is stored, the program code is loaded and executed by a processor to implement the classification method of the cell classification model as described in the above aspect , or, cell sorting methods.

According to another aspect of the present application, a computer program product or computer program is provided, and the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the classification method of the cell classification model, or the cell classification method.

The beneficial effects brought by the technical solutions provided in the embodiments of the present application include at least:

When training the cell classification model, the cell labeling of the sample image is converted into a more complex cell segmentation map. The cell segmentation map includes information related to cell morphology. The sample classification heat map is obtained through the cell segmentation map, and the hot spots are classified according to the sample. A sample prediction heatmap of the graph and sample images for the cell classification model. When the cell classification model is trained in the embodiment of the present application, the outline of each cell in the sample image is restored through simple cell labeling, and more information is extracted from the sample image for training the cell classification model, not only It can effectively improve the accuracy of the classification results of the cell classification model. Moreover, since the training of the cell classification model only needs to use the labeling of cell types and does not need to provide the labeling of cell morphology, weakly supervised learning can be implemented and training efficiency can be improved.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a schematic structural diagram of a computer system provided by an exemplary embodiment of the present application;

2 is a schematic diagram of a training method of a cell classification model provided by an exemplary embodiment of the present application;

3 is a schematic flowchart of a training method for a cell classification model provided by an exemplary embodiment of the present application;

FIG. 4 is a schematic diagram of a sample image provided by an exemplary embodiment of the present application;

5 is a schematic diagram of cell labeling provided by an exemplary embodiment of the present application;

6 is a schematic flowchart of a training method for a cell classification model provided by an exemplary embodiment of the present application;

7 is a schematic diagram of a dilation operator provided by an exemplary embodiment of the present application;

8 is a schematic diagram of a dilation operator provided by an exemplary embodiment of the present application;

FIG. 9 is a schematic diagram of an image generation field provided by an exemplary embodiment of the present application;

FIG. 10 is a schematic diagram of the field of generating images provided by an exemplary embodiment of the present application;

11 is a schematic diagram of a cell classification method provided by an exemplary embodiment of the present application;

12 is a schematic flowchart of a cell classification method provided by an exemplary embodiment of the present application;

13 is a schematic diagram of a comparison of classification results provided by an exemplary embodiment of the present application;

14 is a schematic diagram of a comparison of classification results provided by an exemplary embodiment of the present application;

FIG. 15 is a block diagram of a training device for a cell classification model provided by an exemplary embodiment of the present application;

16 is a block diagram of a cell sorting device provided by an exemplary embodiment of the present application;

FIG. 17 is a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

First, the terms involved in the embodiments of the present application are introduced:

Artificial Intelligence (AI): It is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

Computer Vision (CV): Computer vision is a science that studies how to make machines "see". Further, it refers to the use of cameras and computers instead of human eyes to identify and measure objects and other machine vision, and Further graphics processing makes computer processing become images more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, trying to build artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition, Optical Character Recognition), video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual Reality, augmented reality, simultaneous positioning and map construction and other technologies, as well as common biometric identification technologies such as face recognition and fingerprint recognition.

Machine Learning (ML): It is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other subjects. It specializes in how computers simulate or realize human learning behaviors to acquire new knowledge or skills, and to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications are in all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, teaching learning and other techniques.

IHC (immunohistochemistry, immunohistochemistry): Pathologically, specific primary antibodies are used to label cellular proteins in tissue samples, and detection reagents are used to visualize the targets. Protein expression can be assessed using IHC and by chromogenic detection or fluorescence detection.

PD-L1 (Programmed Death-Ligand 1, programmed death-ligand 1): is a protein in the human body. It is a transmembrane protein and has been implicated in the suppression of the immune system.

Pembrolizumab (Pembrolizumab): is a humanized PD-1 (Programmed Death-1, programmed death-1) monoclonal antibody for cancer immunotherapy. For the treatment of non-small cell lung cancer, urothelial carcinoma, esophageal squamous cell carcinoma, triple negative breast cancer and other indications.

CPS (Combined Positive Score): It is an interpretive indicator for PD-L1 detection for judging whether the indication user uses Pembrolizumab for immunotherapy.

If the CPS value calculated according to this formula is greater than 100, the CPS is considered to be equal to 100. CPS is used for PD-L1 detection and patient classification in indications such as head and neck squamous cell carcinoma, gastric or gastroesophageal junction adenocarcinoma, cervical cancer, urothelial carcinoma, esophageal squamous cell carcinoma, and triple-negative breast cancer.

With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart medical care, smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important value.

It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in this application, All are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the images and user information involved in this application are obtained with full authorization.

FIG. 1 shows a schematic structural diagram of a computer system provided by an exemplary embodiment of the present application. The computer system 100 includes: a terminal 120 and a server 140 .

Application programs related to cell classification are installed on the terminal 120 . The application may be a small program in an app (application, application), a special application, or a web client. Exemplarily, when the user is a doctor, the user obtains the result of cell classification of the patient from the terminal 120, and determines the condition of the patient through the result. The terminal 120 is at least one of a smart phone, a tablet computer, an e-book reader, an MP3 player, an MP4 player, a laptop computer, and a desktop computer.

The terminal 120 is connected to the server 140 through a wireless network or a wired network.

The server 140 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, Cloud servers for basic cloud computing services such as middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 140 is used for providing background services for the application program of cell classification, and sending information related to the video and cell classification to the terminal 120 . Optionally, the server 140 undertakes the main computing work, and the terminal 120 undertakes the secondary computing work; or, the server 140 undertakes the secondary computing work, and the terminal 120 undertakes the main computing work; or, both the server 140 and the terminal 120 adopt a distributed computing architecture perform collaborative computing.

FIG. 2 shows a schematic diagram of a training method of a cell classification model provided by an exemplary embodiment of the present application.

The sample channel image 202 is obtained by performing color deconvolution on the sample image 201, wherein the color deconvolution is used to separate the target color in the sample image 201. Exemplarily, the target color is at least one of brownish yellow, blue, and pink. One, specifically, the target color can be adjusted by technicians according to actual needs. The sample image 201 and the sample channel image 202 are input into the cell classification model 200, the cells in the sample image 201 are classified by the cell classification model 200, and the sample prediction heat map 203 is obtained, and the sample prediction heat map 203 is used to predict the sample image 201. A heat map of the types of cells in the . Exemplarily, the cell classification model 200 is a U-Net (U-shaped network), and the cell classification model 200 includes a first feature encoding module and a first upsampling decoding module. The cell classification model 200 upsamples and downsamples the sample image 201 to obtain a sample prediction heat map 203 . The cell classification model 200 may also be other types of classification models.

The cell nucleus is segmented on the sample image 201 , and the outline of the cell nucleus of each cell in the sample image 201 is determined, thereby generating a schematic diagram of the cell nucleus 204 . In the next step, image post-processing is performed on the cell nucleus schematic 204 to generate a cell segmentation map 205 . Image postprocessing is used to restore the contours of the cells from the contours of the nuclei. Taking the cell labeling 206 as a reference, the cell segmentation map 205 is classified to obtain a sample classification heat map 207 . The sample classification heat map 207 is used to represent a heat map of the types of cells in the sample image 201 .

Calculate the first loss 208 between the sample prediction heat map 203 and the sample classification heat map 207, and calculate the second loss 209 between the sample prediction heat map 203, cell annotation 206 and the sample classification heat map 207, the first loss 208 is used In order to represent the difference between the sample prediction heat map and the sample classification heat map, the second loss 209 is used in the training process of the cell classification model 200 to classify the cell classification model according to the position and texture features presented by the nucleus (or chromosome). 200 for optimization and learning. The cell classification model 200 is trained according to the first loss 208 and the second loss 209 .

FIG. 3 shows a schematic flowchart of a training method for a cell classification model provided by an exemplary embodiment of the present application. The method can be performed by the computer system 100 shown in FIG. 1, and the method includes:

Step 302: Obtain a sample image and a cell label of the sample image, the sample image includes at least two types of cells, and the cell label is used to indicate the type of cells in the sample image.

Optionally, the sample image is an electronic image of a stained pathological section. Illustratively, as shown in FIG. 4, a sample image 401 shows stained cells. It should be noted that since the nucleus contains most of the chromosomes in the cell, after staining the cell, the stained part of the cell is the nucleus.

Optionally, the sample image is an RGB color digital image. In an optional design, the pixel physical size of the sample image is not greater than 0.5 μm/pixel (micrometer/pixel), and the pixel physical size is used to represent the actual physical size of a single pixel.

Different types of cells will show different shapes. Since the pathological section itself is transparent, it is necessary to stain the pathological section to observe the shape of the cells, and the dye can also reflect specific types of cells or specific types of substances. , and show different colors. Taking DAB as a dyeing agent as an example, DAB can make positive cells or substances appear brownish-yellow. Specifically, DAB can make diseased cells appear brownish-yellow.

Optionally, the type of cells includes, but is not limited to, at least one of positive tumor cells, negative tumor cells, positive mononuclear inflammatory cells, and negative mononuclear inflammatory cells.

Optionally, the cell annotation belongs to point annotation, that is, a point is used to represent a cell or nucleus in the sample image. Optionally, the cell annotation is an annotation located at the center of the nucleus. Exemplarily, as shown in FIGS. 4 and 5 , in the cell annotation 501 , dots are used to replace the stained cell nuclei in the sample image 401 .

Optionally, label all cells in the sample image, and use different colors to label different types of cells. Exemplarily, label all pixels on the cell. Exemplarily, all pixels on the nucleus are marked. Exemplarily, the center point of the cell is taken, and the center point is marked.

Optionally, the cell label includes the abscissa value, ordinate value and class label value of each point.

Optionally, different colors are used to represent different types of cells. Exemplarily, red is used for positive tumor cells, green is used for negative tumor cells, yellow is used for positive mononuclear inflammatory cells, and blue is used for negative mononuclear inflammatory cells. Optionally, different shapes are used to represent different types of cells. Exemplarily, rectangles are used to represent positive tumor cells and circles are used to represent negative tumor cells.

Step 304 : perform data processing on the sample image through the cell classification model, output a sample prediction heat map, and the sample prediction heat map is used to predict the type of cells in the sample image.

Cell classification models are models used to predict cell types. Optionally, the cell classification model belongs to U-Net (U-shaped network), and the cell classification model includes a first feature encoding module and a first upsampling decoding module. Wherein, the first feature encoding module is used to perform image downsampling on the sample image, and extract image features from the sample image, and the image features are related to the classification of cell types. The first up-sampling decoding module is used to restore the size of the image down-sampling result to the size of the sample image. Exemplarily, the sample image is an image with a size of 32×32, and the first feature encoding module performs image downsampling on the sample image to obtain four 4×4 image features. The first upsampling decoding module is called to perform image upsampling on the four 4×4 image features to obtain four 32×32 image prediction results, wherein the image prediction results are used to represent the prediction results of one cell type.

Optionally, the cell classification model is a model for object detection, or a model for image segmentation. Exemplarily, the cell classification model can also be a Transformer model. The embodiment of the present application does not specifically limit the model type of the cell classification model.

Optionally, the sample prediction heat map is a heat map used to predict a cell type in the sample image, wherein the bth heat map is used to display the bth type of cells, a represents the number of cell types, and b is less than a. positive integer. For example, only cells of cell type A are shown in heat map 1 and only cells of cell type B are shown in heat map 2.

Optionally, the target color of the sample image is separated to obtain a sample channel image, and the sample channel image is an image obtained by the sample image through the expression channel of the target color; the cell classification model is called to perform data processing on the sample image and the sample channel image, and the sample prediction is output. Heat map. The target color is the color corresponding to the target cell type in the sample image. For example, when the target cell type is positive tumor cells, the positive tumor cells will appear brown after staining, and the target color is brown.

In a possible implementation, color decomposition is performed on the Hematoxylin-Eosin-DAB staining space to obtain: a positive expression channel representing DAB (brown), a negative expression channel representing hematoxylin Hematoxylin (blue) counterstaining, and The channel representing Eosin (pink) (which does not appear in PD-L1 staining), the output of the positive expression channel of DAB is taken as the sample channel image, and the sample image of the RGB channel and the sample channel image of the DAB channel are taken as the cells Input to the classification model. Optionally, the sample image of RGB channel and the sample image of hue channel are used as the input of the cell classification model. Alternatively, a sample image of RGB channel and a sample image of gray channel are used as input to the cell classification model.

The number of predicted heatmaps is the same as the number of color channels. For example, after data processing of the input image using RGB channels and DAB channels, 4 sets of predicted heat maps are obtained.

Step 306: Restore the contour of each cell in the sample image by using the contour of each cell nucleus in the sample image to obtain a cell segmentation map.

Optionally, data processing is performed on the sample image through a cell nucleus segmentation model, and the outline of each cell nucleus in the sample image is output. Exemplarily, the cell nucleus segmentation model belongs to the U-Net neural network. The nucleus segmentation model includes a second feature encoding module and a second upsampling decoding module.

Optionally, a morphological dilation operator is used to restore the contour of each cell in the sample image through the contour of each cell nucleus in the sample image.

A cell segmentation map refers to an image corresponding to a sample image including cell outlines. Exemplarily, as shown in the figure, by comparing the graph and the graph, it can be obtained that the cell segmentation graph is a prediction of the contour of each cell in the sample image.

Step 308: Generate a sample classification heat map according to the cell labeling and the cell segmentation map, where the sample classification heat map is used to represent the types of cells in the cell segmentation map.

Optionally, the sample classification heat map is a heat map used to represent a cell type in the sample image, and the c th heat map is used to display the c th cell type. For example, only cells of cell type A are shown in heat map 3 and only cells of cell type B are shown in heat map 4.

Step 310: Train the cell classification model according to the loss between the sample prediction heat map and the sample classification heat map.

Optionally, the loss between the sample prediction heat map and the sample classification heat map includes a first loss and a second loss, the first loss is used to represent the difference between the sample prediction heat map and the sample classification heat map, and the second loss is used to represent the difference between the sample prediction heat map and the sample classification heat map. It is used to optimize and learn the cell classification model according to the position and texture features presented by the nucleus (or chromosome) during the training process of the cell classification model. Exemplarily, the cell classification model is trained according to the first loss between the sample prediction heat map and the sample classification heat map; or, the cell classification model is trained according to the second loss between the sample prediction heat map and the sample classification heat map. The model is trained; or the cell classification model is trained according to the first loss and the second loss between the sample prediction heat map and the sample classification heat map.

Optionally, the mean square error between the sample prediction heat map and the sample classification heat map is calculated to obtain the first loss.

Optionally, according to the cell type provided by the cell label, the second loss is calculated from the sample prediction heat map and the sample classification heat map. The second loss is used to represent the loss term of the dense conditional random field. Conditional Random Field (CRF) is a conditional probability distribution model of output random variables given a set of input random variables. Its characteristic is that the output random variables constitute a Markov random field, which can be used to label or analyze images. . The dense conditional random field is a kind of conditional random field. If the dense conditional random field is used to classify the pixels in the image, the dense conditional random field will associate the target pixel point with other pixels to obtain the target pixel point. Type, other pixels refer to the pixels in the image that are associated with the target pixel. In the embodiment of the present application, the loss term of the dense conditional random field is used as one of the loss functions of the cell classification model to train the cell classification model.

则有：Exemplarily, the loss term (ie the second loss) of the dense conditional random field is denoted as

Then there are:

为细胞类别c对应的预测热点图，W^c为该类别的相似性度量矩阵。W^c在稠密条件随机场中是全连接的高斯，在相应的损失优化中计算其梯度会成为一个双边滤波问题。TC_N表示阳性肿瘤细胞，TC_P表示阴性肿瘤细胞，MIC_N表示阳性单核炎细胞，MIC_P表示阴性单核炎细胞。where M is the total number of pixels in the image per channel,

is the predicted heat map corresponding to cell category c, and W ^c is the similarity measure matrix of this category. W ^c is a fully connected Gaussian in a dense conditional random field, and computing its gradient in the corresponding loss optimization becomes a bilateral filtering problem. TCN indicates positive tumor cells, _{TCP indicates negative tumor cells, MIC N indicates} positive mononuclear inflammatory cells, and MIC _P indicates negative mononuclear inflammatory cells _.

Optionally, according to the loss between the sample prediction heat map and the sample classification heat map, the cell classification model is trained through an error back propagation algorithm.

To sum up, when the cell classification model is trained in this embodiment, the cell labeling of the sample image is converted into a more complex cell segmentation map. The cell segmentation map includes information related to cell morphology, and the sample classification is obtained through the cell segmentation map. Heat map, and model cell classification based on sample classification heat map and sample prediction heat map of sample images. When the cell classification model is trained in the embodiment of the present application, the outline of each cell in the sample image is restored through simple cell labeling, and more information is extracted from the sample image for training the cell classification model, not only It can effectively improve the accuracy of the classification results of the cell classification model. Moreover, since the training of the cell classification model only needs to use the labeling of cell types and does not need to provide the labeling of cell morphology, weakly supervised learning can be implemented and training efficiency can be improved.

In the following embodiment, the process of restoring the contour of the cell according to the contour of the nucleus will be introduced. This embodiment can convert the cell label of the sample image into a stronger pseudo label for cell segmentation, so that in the training of the cell classification model , more information can be obtained from the sample images, which is beneficial to the training of the cell classification model, so that the trained cell classification model can classify cells more accurately.

FIG. 6 shows a schematic flowchart of a training method for a cell classification model provided by an exemplary embodiment of the present application. The method can be performed by the computer system 100 shown in FIG. 1, and the method includes:

Step 601: Use the dilation operator to traverse the contour of the ith cell nucleus.

The dilation operator is used to dilate the occupied area of the i-th nucleus, where i is a positive integer.

The shape of the expansion operator includes, but is not limited to, at least one of a circle, a rectangle, a triangle, a regular hexagon, and a regular octagon. Wherein, the specific shape of the expansion operator can be modified by technicians according to actual needs. For example, when the sample image A is processed, a circular dilation operator is used, and when the sample image B is processed, a rectangular dilation operator is used.

Optionally, the shape of the dilation operator is related to the traversal distance of the dilation operator, and the traversal distance is used to represent the distance that the dilation operator moves during the process of traversing the contour of the i-th cell nucleus. Exemplarily, after the dilation operator is moved by 0.1 microns, the dilation operator is a circle with a radius of 5 microns, and after the dilation operator is moved by 0.2 microns, the dilation operator is a circle with a radius of 5.1 microns. Exemplarily, the size of the dilation operator y=f(x), x represents the traversal distance of the dilation operator, and f(x) is a self-defined function.

Optionally, the shape of the dilation operator is related to the radius of curvature of the center point of the dilation operator. The radius of curvature on the contour. Exemplarily, when the radius of curvature corresponding to the center point of the dilation operator is 4 microns, the dilation operator is a circle with a radius of 5 microns, and when the radius of curvature corresponding to the center point of the dilation operator is 3 microns, the dilation operator is a square with a side length of 4 microns. It should be noted that, the above-mentioned form of the expansion operator may refer to at least one of the radius, diameter, side length, area, and shape of the expansion operator.

Exemplarily, as shown in FIG. 7 , taking the expansion operator 702 as a circle as an example, the center of the expansion operator 702 is placed on the outline of the cell nucleus 701, and the center of the circle is controlled to move on the outline of the cell nucleus 701, so that the expansion operator 702 is moved. Sub 702 traverses the outline of nucleus 701 .

Step 602: Determine the image area corresponding to the ith cell nucleus according to the coverage area of the dilation operator.

The image field of the ith cell nucleus is used to predict the area occupied by the ith cell in the sample image. The i-th cell refers to a cell containing the i-th cell nucleus. In the examples of the present application, only the case where one cell has and only one cell nucleus is considered.

The coverage area of the dilation operator is used to represent the union of the occupied areas of the dilation operator during the traversal of the dilation operator. Exemplarily, as shown in FIG. 8 , in the process of traversing the cell nucleus 701 by the dilation operator 702 , when the dilation operator 702 is at the position A, the dilation operator 702 will occupy the first area. When the dilation operator 702 is at the position B, the dilation operator 702 will occupy the second area, and the union of the first area and the second area is taken as the coverage area of the dilation operator. It should be noted that the movement of the dilation operator is a continuous process. Here, in order to clearly express the traversal process of the dilation operator, two discrete points are used for illustration.

Step 603: Determine the contour of the ith cell according to the image field of the ith cell nucleus.

In practice, the cells in some sample images will exhibit the characteristics of aggregated distribution, for example, when the sample images are used to display cells in tumor cell nests and immune cell nests. Therefore, it is necessary to divide possible overlapping cell neighborhoods.

Optionally, a watershed algorithm is used to determine the contour of the ith cell according to the image field of the ith cell nucleus. Exemplarily, the method includes the following sub-steps:

1. Expand the image field of the i-th cell to obtain the target area.

Among them, the target area is larger than the image area of the ith cell. That is, the image field of the ith cell is located within the target area.

Optionally, the method for expanding the image area of the ith cell may be to set a decision frame in the image area of the ith cell, and the area occupied by the decision frame is marked as the target area. The size of the decision box can be set by technicians according to actual needs. The method of expanding the image field of the ith cell can be realized by using the above-mentioned morphological dilation operator.

2. Determine the pixels in the target area that belong to the gray value range.

Optionally, determine the pixels in the target area that belong to the target gray value. The target gray value is a constant.

Optionally, the gray value interval is dynamically generated. For example, the gray value interval is generated according to the gray value of the pixel points located at the edge of the image area. If the gray value of the pixel at the edge of the image field is in the interval [2, 8], a subset of this interval is taken as the gray value interval, or the target gray value in this interval is taken as the gray value interval.

Optionally, the gray value interval can be set by a technician. For example, set the gray value interval directly to [1, 10].

3. Connect the pixels belonging to the gray value range to form a closed area.

Exemplarily, as shown in FIG. 9, for the image field 901 of the i-th cell, determine the pixel points 902 belonging to the target gray value in the target area (for the convenience of description, only some of the pixels belonging to the target gray value are shown here. point), use a smooth curve to connect the pixel points 902 to obtain a closed area 903.

4. Determine the closed area as the image area of the ith cell.

Step 604: Repeat the above three steps until the contour of each cell is determined, and a cell segmentation map is obtained.

Since the above three steps only provide the outline of one cell, and the sample image includes multiple cells, it is necessary to repeat the above three steps to obtain the outline of each cell, and then generate a cell segmentation map.

Step 605: According to the cell labeling, cells of n cell types are separated from the cell segmentation map to generate n cell type maps.

The n cell type maps correspond one-to-one with n cell types.

Exemplarily, the cell segmentation map includes cells of cell types A, B, C, and D, and the cells of these 4 cell types are separated from the cell segmentation map to obtain a cell type map including only cell type A, including only cells A cell type map of type B, a cell type map that includes only cell type C, and a cell type map that includes only cell type D.

Step 606: For the jth cell type map in the n cell type map, assign a first gray value to the pixel points located outside the cells in the jth cell type map.

The first grayscale value is a constant, optionally, the first grayscale value is 0.

Exemplarily, as shown in FIG. 10 , if the pixel point 1002 is a pixel point located outside the cell 1001 , the gray value of the pixel point 1002 is set to 0.

Step 607: Assign dynamic gray values to the pixels located in the cells in the jth cell type map.

The dynamic gray value is positively correlated with the distance from the pixel in the cell to the edge of the cell. Optionally, the distance from the pixel point to the cell edge refers to the shortest distance from the pixel point to the cell edge. Alternatively, the distance from the pixel to the cell edge refers to the average distance from the pixel to the cell edge.

Exemplarily, pixel 1003 and pixel 1004 are pixels located in cell 1001, then the distance from pixel 1003 to the edge of the cell is greater than the distance from pixel 1004 to the edge of the cell, then the grayscale value of pixel 1003 is 255, The grayscale value of pixel 1004 is 250.

Step 608: Generate the jth sample classification heat map according to the gray value of each pixel in the jth cell type map.

In the jth sample classification heat map, the pixel temperature outside the cell is 0, and the further away from the cell edge, the higher the temperature.

Step 609: Repeat the above three steps until a sample classification heat map is obtained.

Since the above three steps only provide sample classification hotspots for one cell type, and cells include at least two types, the above three steps need to be repeated to obtain sample classification hotspots for each cell type.

To sum up, when the cell classification model is trained in this embodiment, the cell labeling of the sample image is converted into a more complex cell segmentation map. The cell segmentation map includes information related to cell morphology, and the sample classification is obtained through the cell segmentation map. Heat map, and model cell classification based on sample classification heat map and sample prediction heat map of sample images. When the cell classification model is trained in the embodiment of the present application, the outline of each cell in the sample image is restored through simple cell labeling, and more information is extracted from the sample image for training the cell classification model, not only The accuracy of the classification result of the cell classification model can be effectively improved, and weakly supervised learning can be realized to improve the training efficiency.

FIG. 11 shows a schematic diagram of a cell sorting method provided by an exemplary embodiment of the present application.

The channel image 1102 is obtained by performing color deconvolution on the input image 1101 , wherein the color deconvolution is used to separate the target colors in the input image 1101 . The input image 1101 and the channel image 1102 are input into the cell classification model 1100 , and the cells in the input image 1101 are classified by the cell classification model 1100 to obtain a predicted heat map 1103 . After image processing of the predicted heat map 1103, the position and number of cells in the input image 1101 are obtained. Among them, the image processing here is used to count the location and number of different types of cells.

Exemplarily, the location and number of cells include the location and number of negative tumor cells, the location and number of positive tumor cells, the location and number of negative mononuclear inflammatory cells, and the location and number of positive mononuclear inflammatory cells. It should be noted that, the skilled person can determine the location and number of more or less cells according to actual needs. Specifically, after image processing is performed on the predicted heat map 1103, a result image 1104 is obtained.

FIG. 12 shows a schematic flowchart of a cell sorting method provided by an exemplary embodiment of the present application. The method is performed by the cell classification provided in the above embodiment, and the method can be performed by the terminal 120 or the server 140 shown in FIG. 1 . The terminal 120 or the server 140 runs the cell classification model provided in the above embodiment, and the method includes:

Step 1202: Acquire an input image, where the input image includes at least two types of cells.

Optionally, the input image is an electronic image of a stained pathological section.

Optionally, the input image is an RGB color digital image. In an optional design, the pixel physical size of the input image is not greater than 0.5 μm/pixel (microns/pixel).

Step 1204: Perform data processing on the input image through the cell classification model, output a predicted heat map, and the predicted heat map is used to represent the probability that the cells belong to the target cell type.

Cell classification models are models used to predict cell types. Optionally, the cell classification model belongs to U-Net, and the cell classification model includes a first feature encoding module and a first upsampling decoding module. Wherein, the first feature encoding module is used to perform image downsampling on the input image, and extract image features from the input image, and the image features are related to the classification of cell types.

Optionally, the target color of the input image is separated to obtain a channel image, and the channel image is an image obtained through the expression channel of the target color; the cell classification model is called to perform data processing on the input image and the channel image, and a predicted heat map is output. In a possible implementation, color decomposition is performed on the Hematoxylin-Eosin-DAB staining space to obtain: a positive expression channel representing DAB (brown), a negative expression channel representing hematoxylin Hematoxylin (blue) counterstaining, and The channel representing eosin Eosin (pink) (not present in PD-L1 staining), the output of the positive expression channel of DAB was taken as the channel image.

The number of predicted heatmaps is the same as the number of color channels. For example, after data processing of the input image using RGB channels and DAB channels, 4 sets of predicted heat maps are obtained.

Exemplarily, the predicted heatmap includes a set of heatmaps. Each heat map gives the probability that each pixel belongs to the cell type corresponding to the aforementioned heat map. Exemplarily, the first predicted heat map is used to represent the heat map of positive tumor cells, and the second predicted heat map is used to represent the heat map of negative tumor cells.

Step 1206: Determine the type of each cell in the input image according to the predicted heat map.

Optionally, the predicted heatmap is a heatmap representing a cell type in the input image, and the cth heatmap is used to display the cth cell. For example, only cells of cell type A are shown in heat map 3 and only cells of cell type B are shown in heat map 4.

In the embodiment of the present application, after determining the type and number of each cell in the input image, the number of cells of n cell types in the heat map can be predicted statistically, where n is a positive integer greater than 2; according to at least one of the n cell types The number of cells of each cell type, and the composite positivity score was calculated. Recording the composite positive score as CPS, there are:

Among them, N ₀ represents the number of negative tumor cells, N ₁ represents the number of positive tumor cells, and N ₃ represents the number of positive mononuclear inflammatory cells.

Optionally, the embodiments of the present application are applicable to the staining methods of other PD-L1 clone numbers, for example, the detection method of PD-L1 can also be the methods in Ventana SP263, Ventana SP142, Dako 28-8, Cell Signaling TechnologyE1L3N, WuXiDiagnostics WD160. at least one.

Optionally, the number of cells of different cell types is determined by local extreme points in different classification heat maps. Exemplarily, the method for counting the number of cells may include the following steps:

1. Determine the local extreme point in the k-th predicted heat map.

In the embodiment of the present application, since the predicted heat map is used to represent the probability that the cell belongs to the target cell type, the local extreme point in the predicted heat map is located at the edge of the cell.

2. Obtain the center coordinates of the connected region formed by the local extreme points.

Optionally, the coordinates of the center of the connected region represent the coordinates of the center of the cell.

Optionally, the center coordinate of the connected area is determined according to the average value of the coordinates of the edges of the connected area.

3. Determine the number of cells in the kth heat map according to the number of center coordinates.

Optionally, assign the cell type corresponding to the kth heat map to the center point of the center coordinate.

4. Repeat the above three steps until the number of cells of n cell types is obtained.

To sum up, this embodiment restores the outline of each cell in the sample image through simple cell labeling, and then extracts more information from the sample image for the training of the cell classification model, which can not only effectively improve the cell classification model. Moreover, since the training cell classification model only needs to use the labeling of cell types and does not need to provide the labeling of cell morphology, weakly supervised learning can be realized and the training efficiency can be improved.

Optionally, the cell classification model of the present application is applied on the AI pathology cloud platform: in a pathology department or medical center equipped with a digital pathology reading system, the cell classification model of the embodiment of the present application can be integrated into the reading system as an AI algorithm plug-in. in the slice interface. After the doctor opens the digital pathological slice, the classification results of the cell classification model of this method can be superimposed on the digital pathological slice and presented.

Optionally, the cell classification model of the present application is applied to an AI microscope: on a microscope equipped with a digital image acquisition module, the cell classification model of the present application can be implanted in the interpretation process of a pathologist viewing films under a microscope. When the doctor presses the "Start Calculation" button or steps on the foot pedal, the cell classification model will classify the cells in the field of view currently collected (which is also what the doctor is seeing), and return the image superimposed with the classification result to the optical path of the microscope eyepiece in real time. The doctor sees the cell sorting results in the eyepiece.

FIG. 13 and FIG. 14 are schematic diagrams showing the comparison of classification results provided by an exemplary embodiment of the present application.

In FIG. 13 , the interpretation image 1302 is the classification result of performing cell classification on the original image 1301 by using the cell classification method provided in the above embodiment. In FIG. 14 , the interpretation image 1402 is the classification result of the cell classification of the original image 1401 . Positive TCs (red dots), negative TCs (green dots), positive MICs (yellow dots), and negative MICs (blue dots) are marked in interpretation image 1302 and interpretation image 1402, respectively. Among them, the classification results corresponding to the interpretation image 1302 and the interpretation image 1402 are accurate. Therefore, the cell classification model provided in the embodiment of the present application can well complete the cell classification task, and the classification effect is good.

The following are apparatus embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

Please refer to FIG. 15 , which shows a block diagram of an apparatus for training a cell classification model provided by an embodiment of the present application. The above functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The apparatus 1500 includes:

a sample acquisition module 1501, configured to acquire a sample image and a cell label of the sample image, the sample image includes at least two types of cells, and the cell label is used to indicate the type of cells in the sample image;

A data processing module 1502, configured to perform data processing on the sample image through the cell classification model, and output a sample prediction heat map, where the sample prediction heat map is used to predict the type of cells in the sample image;

The data processing module 1502 is further configured to restore the contour of each cell in the sample image through the contour of each cell nucleus in the sample image to obtain a cell segmentation map;

The data processing module 1502 is further configured to generate a sample classification heat map according to the cell labeling and the cell segmentation map, where the sample classification heat map is used to represent the type of cells in the cell segmentation map;

The training module 1503 is configured to train the cell classification model according to the loss between the sample prediction heat map and the sample classification heat map.

In an optional design of the present application, the data processing module 1502 is further configured to use a dilation operator to traverse the contour of the ith cell nucleus, where the dilation operator is used to dilate the occupied area of the ith cell nucleus, i is a positive integer; the image area corresponding to the i-th cell nucleus is determined according to the coverage area of the dilation operator, and the image area of the i-th cell nucleus is used to predict that the i-th cell is in the sample image The area occupied; the contour of the i-th cell is determined according to the image area of the i-th cell nucleus; the above three steps are repeated until the contour of each cell is determined, and the cell segmentation map is obtained.

In an optional design of the present application, the data processing module 1502 is further configured to expand the image field of the i-th cell to obtain a target area; determine the pixels belonging to the gray value range in the target area; Connecting the pixels belonging to the gray value interval to form a closed area; and determining the closed area as the image area of the i-th cell.

In an optional design of the present application, the data processing module 1502 is further configured to separate cells of n cell types from the cell segmentation map according to the cell label, and generate n cell type maps, so that The n cell type maps are in one-to-one correspondence with the n cell types, and n is a positive integer;

In an optional design of the present application, the data processing module 1502 is further configured to, for the jth cell type map in the n cell type map, indicate that the jth cell type map is located outside the cell The first gray value is assigned to the pixel point of , j is a positive integer less than n+1; the pixel point located in the cell in the jth cell type map is assigned a dynamic gray value, and the dynamic gray value is the same as all The distance from the pixel located in the cell to the edge of the cell is positively correlated; according to the gray value of each pixel in the jth cell type map, generate the jth sample classification heat map; Repeat the above three steps until the obtained The sample classification heat map.

In an optional design of the present application, the data processing module 1502 is further configured to separate the target color of the sample image to obtain a sample channel image, where the sample channel image is the result of the sample image passing through the target color. The image obtained by the expression channel; invoking the cell classification model to perform data processing on the sample image and the sample channel image, and outputting the sample prediction heat map.

In an optional design of the present application, the training module 1503 is further configured to calculate the first loss between the sample prediction heat map and the sample classification heat map; calculate the cell labeling, the sample prediction a second loss between the heat map and the sample classification heat map; the cell classification model is trained according to the first loss and the second loss.

In an optional design of the present application, the training module 1503 is further configured to calculate the mean square error between the sample prediction heat map and the sample classification heat map to obtain the first loss.

In an optional design of the present application, the training module 1503 is further configured to calculate the second loss from the sample prediction heat map and the sample classification heat map according to the cell type provided by the cell label, The second loss is used to represent the loss term of the dense conditional random field.

To sum up, when the cell classification model is trained in this embodiment, the cell labeling of the sample image is converted into a more complex cell segmentation map. The cell segmentation map includes information related to cell morphology, and the sample classification is obtained through the cell segmentation map. Heat map, and model cell classification based on sample classification heat map and sample prediction heat map of sample images. When the cell classification model is trained in the embodiment of the present application, the outline of each cell in the sample image is restored through simple cell labeling, and more information is extracted from the sample image for training the cell classification model, not only It can effectively improve the accuracy of the classification results of the cell classification model. Moreover, since the training of the cell classification model only needs to use the labeling of cell types and does not need to provide the labeling of cell morphology, weakly supervised learning can be implemented and training efficiency can be improved.

Please refer to FIG. 16 , which shows a block diagram of a cell sorting apparatus provided by an embodiment of the present application. The above functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The apparatus 1600 runs the cell classification model provided in the above-mentioned embodiment, and the apparatus 1600 includes:

an image acquisition module 1601, configured to acquire an input image, where the input image includes at least two types of cells;

A model calling module 1602, configured to perform data processing on the input image through the cell classification model, and output a predicted heat map, where the predicted heat map is used to represent the probability that a cell belongs to a target cell type;

The prediction module 1603 is configured to determine the type of each cell in the input image according to the predicted heat map.

In an optional design of the present application, the model calling module 1602 is further configured to separate the color of the input image to obtain a channel image, where the channel image is obtained by the input image through the expression channel of the target color The input image and the image are processed by the cell classification model, and the predicted heat map is output.

In an optional design of the present application, the prediction module 1603 is further configured to count the number of cells of n cell types in the prediction heat map, where n is a positive integer greater than 2; A composite positivity score was calculated for the number of cells of at least one cell type.

In an optional design of the present application, the prediction module 1603 is further configured to determine the local extreme point in the kth prediction heat map; obtain the center coordinates of the connected region formed by the local extreme point; The number of the center coordinates determines the number of cells in the kth heat map; the above three steps are repeated until the number of cells of the n cell types is obtained.

To sum up, this embodiment restores the outline of each cell in the sample image through simple cell labeling, and then extracts more information from the sample image for the training of the cell classification model, which can not only effectively improve the cell classification model. Moreover, since the training cell classification model only needs to use the labeling of cell types and does not need to provide the labeling of cell morphology, weakly supervised learning can be realized and the training efficiency can be improved.

Fig. 17 is a schematic structural diagram of a computer device according to an exemplary embodiment. The computer device 1700 includes a Central Processing Unit (CPU) 1701, a system memory 1704 including a Random Access Memory (RAM) 1702 and a Read-Only Memory (ROM) 1703, and A system bus 1705 that connects the system memory 1704 and the central processing unit 1701 . The computer device 1700 also includes a basic input/output system (Input/Output, I/O system) 1706 that facilitates the transfer of information between various devices within the computer device, and is used to store an operating system 1713, application programs 1714, and other programs Mass storage device 1707 for module 1715.

The basic input/output system 1706 includes a display 1708 for displaying information and input devices 1709 such as a mouse, keyboard, etc., for user input of information. The display 1708 and the input device 1709 are both connected to the central processing unit 1701 through the input and output controller 1710 connected to the system bus 1705. The basic input/output system 1706 may also include an input output controller 1710 for receiving and processing input from a number of other devices such as a keyboard, mouse, or electronic stylus. Similarly, input output controller 1710 also provides output to a display screen, printer, or other type of output device.

The mass storage device 1707 is connected to the central processing unit 1701 through a mass storage controller (not shown) connected to the system bus 1705 . The mass storage device 1707 and its associated computer device-readable media provide non-volatile storage for the computer device 1700 . That is, the mass storage device 1707 may include a computer device readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer device readable instructions, data structures, program modules or other data. The storage media of computer equipment include RAM, ROM, Erasable Programmable ReadOnly Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, digital Digital Video Disc (DVD) or other optical storage, cassette, magnetic tape, magnetic disk storage or other magnetic storage device. Of course, those skilled in the art know that the storage medium of the computer device is not limited to the above-mentioned ones. The system memory 1704 and the mass storage device 1707 described above may be collectively referred to as memory.

According to various embodiments of the present disclosure, the computer device 1700 may also operate via a network connection to a remote computer device on a network, such as the Internet. That is, the computer device 1700 can be connected to the network 1711 through the network interface unit 1712 connected to the system bus 1705, or can also use the network interface unit 1712 to connect to other types of networks or remote computer equipment systems (not shown). out).

The memory also includes one or more programs, the one or more programs are stored in the memory, and the central processing unit 1701 implements the above-mentioned training method of the cell classification model by executing the one or more programs, or, cell classification all or part of the steps of the method.

In an exemplary embodiment, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores at least one instruction, at least one piece of program, code set or instruction set, the at least one instruction, the At least one piece of program, the code set or the instruction set is loaded and executed by the processor to implement the training method of the cell classification model provided by each of the above method embodiments, or the cell classification method.

The present application further provides a computer-readable storage medium, where at least one instruction, at least one piece of program, code set or instruction set is stored in the storage medium, the at least one instruction, the at least one piece of program, the code set or The instruction set is loaded and executed by the processor to implement the training method of the cell classification model provided by the above method embodiments, or the cell classification method.

The present application also provides a computer program product or computer program, wherein the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the cell classification model training method provided by the embodiment of the above aspect, or, the cell classification method.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, etc.

The above descriptions are only optional embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection of the present application. within the range.

Claims (17)

1. A method for training a cell classification model, the method comprising:

acquiring a sample image and a cell label of the sample image, wherein the sample image comprises at least two cells, and the cell label is used for representing the type of the cells in the sample image;

performing data processing on the sample image through the cell classification model, and outputting a sample prediction hot spot diagram, wherein the sample prediction hot spot diagram is used for predicting the type of cells in the sample image;

restoring the contour of each cell in the sample image through the contour of each cell nucleus in the sample image to obtain a cell segmentation map;

generating a sample classification heat point diagram according to the cell labeling and the cell segmentation diagram, wherein the sample classification heat point diagram is used for representing the types of the cells in the cell segmentation diagram;

and training the cell classification model according to the loss between the sample prediction hotspot graph and the sample classification hotspot graph.

2. The method of claim 1, wherein the obtaining the cell segmentation map by restoring the contour of each cell in the sample image through the contour of each cell nucleus in the sample image comprises:

traversing the outline of the ith cell nucleus by using a dilation operator, wherein the dilation operator is used for dilating the occupied area of the ith cell nucleus, and i is a positive integer;

determining an image field corresponding to the ith cell nucleus according to the coverage area of the expansion operator, wherein the image field of the ith cell nucleus is used for predicting the area occupied by the ith cell in the sample image;

determining the outline of the ith cell according to the image field of the ith cell nucleus;

and repeating the three steps until the contour of each cell is determined to obtain the cell segmentation graph.

3. The method of claim 2, wherein said determining the contour of the ith cell from the image field of the ith cell comprises:

expanding the image field of the ith cell to obtain a target area;

determining pixel points belonging to a gray value interval in the target area;

connecting the pixel points belonging to the gray value interval to form a closed area;

determining the closed region as an image field of the ith cell.

4. The method according to any one of claims 1 to 3, wherein the generating a sample classification hotspot map from the cell labels and the cell segmentation map comprises:

according to the cell labels, separating the cells of n cell types from the cell segmentation graph to generate n cell type graphs, wherein the n cell type graphs correspond to the n cell types one by one, and n is a positive integer;

for a jth cell type graph in the n cell type graphs, giving a first gray value to a pixel point outside a cell in the jth cell type graph, wherein j is a positive integer smaller than n + 1;

giving a dynamic gray value to a pixel point positioned in a cell in the jth cell type graph, wherein the dynamic gray value is positively correlated with the distance from the pixel point positioned in the cell to the edge of the cell;

generating a jth sample classification hot spot diagram according to the gray value of each pixel point in the jth cell type diagram;

and repeating the three steps until the sample classification heat point diagram is obtained.

5. The method according to any one of claims 1 to 3, wherein the invoking the cell classification model to perform data processing on the sample image and output a sample prediction hotspot graph comprises:

separating the target color of the sample image to obtain a sample channel image, wherein the sample channel image is an image obtained by the sample image through an expression channel of the target color;

and calling the cell classification model to perform data processing on the sample image and the sample channel image, and outputting the sample prediction hotspot graph.

6. The method of any one of claims 1 to 3, wherein the training of the cell classification model based on the loss between the sample prediction hotspot graph and the sample classification hotspot graph comprises:

calculating a first loss between the sample prediction hotspot graph and the sample classification hotspot graph;

calculating a second loss between the cell labeling, the sample prediction hotspot graph, and the sample classification hotspot graph;

training the cell classification model based on the first loss and the second loss.

7. The method of claim 6, wherein the calculating a first penalty between the sample prediction hotspot graph and the sample classification hotspot graph comprises:

and calculating the mean square error between the sample prediction hotspot graph and the sample classification hotspot graph to obtain the first loss.

8. The method of claim 6, wherein said calculating a second loss between said cell labeling, said sample prediction hotspot graph, and said sample classification hotspot graph comprises:

and according to the cell types provided by the cell labels, calculating the second loss by the sample prediction hotspot graph and the sample classification hotspot graph, wherein the second loss is used for representing a loss term of the dense conditional random field.

9. A method of cell sorting, the method being performed by a computer device running a cell sorting model according to any one of claims 1 to 8, the method comprising:

acquiring an input image, the input image comprising at least two types of cells;

performing data processing on the input image through the cell classification model, and outputting a predicted hotspot graph, wherein the predicted hotspot graph is used for representing the probability that the cell belongs to the target cell type;

and determining the type of each cell in the input image according to the predicted hotspot graph.

10. The method according to claim 9, wherein the data processing of the input image by the cell classification model to output a predicted hotspot graph comprises:

separating the colors of the input image to obtain a channel image, wherein the channel image is an image obtained by the input image through an expression channel of the target color;

and performing data processing on the input image and the image through the cell classification model, and outputting the predicted hotspot graph.

11. The method of claim 9, further comprising:

counting the number of the cells of n cell types in the predicted hotspot graph, wherein n is a positive integer greater than 2;

calculating a composite positive score based on the number of cells of at least one of the n cell types.

12. The method of claim 11, wherein said counting the number of cells of n cell types in said predictive heat map comprises:

determining a local extreme point in a kth prediction hotspot graph, wherein k is a positive integer smaller than n + 1;

acquiring the center coordinates of a connected region formed by the local extreme points;

determining the number of cells of the kth hot spot map according to the number of the central coordinates;

repeating the above three steps until the cell number of the n cell types is obtained.

13. An apparatus for training a cell classification model, the apparatus comprising:

the system comprises a sample acquisition module, a cell identification module and a cell identification module, wherein the sample acquisition module is used for acquiring a sample image and a cell identification of the sample image, the sample image comprises at least two cells, and the cell identification is used for indicating the type of the cells in the sample image;

the data processing module is used for carrying out data processing on the sample image through the cell classification model and outputting a sample prediction hot spot diagram, and the sample prediction hot spot diagram is used for predicting the type of the cells in the sample image;

the data processing module is further configured to restore the contour of each cell in the sample image through the contour of each cell nucleus in the sample image to obtain a cell segmentation map;

the data processing module is further used for generating a sample classification heat point diagram according to the cell labels and the cell segmentation diagram, wherein the sample classification heat point diagram is used for representing the types of the cells in the cell segmentation diagram;

and the training module is used for training the cell classification model according to the loss between the sample prediction heat point diagram and the sample classification heat point diagram.

14. A cell sorter apparatus, which runs the cell sorting model according to any one of claims 1 to 8, the apparatus comprising:

an image acquisition module for acquiring an input image, the input image comprising at least two types of cells;

the model calling module is used for carrying out data processing on the input image through the cell classification model and outputting a prediction heat point diagram, and the prediction heat point diagram is used for representing the probability that the cell belongs to the target cell type;

and the prediction module is used for determining the type of each cell in the input image according to the predicted hotspot graph.

15. A computer device, characterized in that the computer device comprises: a processor and a memory having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by the processor to implement a method of training a cell classification model according to any one of claims 1 to 8, or a method of cell classification according to any one of claims 9 to 12.

16. A computer-readable storage medium having stored therein at least one program code, the program code being loaded into and executed by a processor to implement a method of training a cell classification model according to any one of claims 1 to 8, or a method of cell classification according to any one of claims 9 to 12.

17. A computer program product comprising a computer program or instructions which, when executed by a processor, implement a method of training a cell classification model according to any one of claims 1 to 8, or a method of cell classification according to any one of claims 9 to 12.

Images