CN114550169A

CN114550169A - Training method, device, equipment and medium for cell classification model

Info

Publication number: CN114550169A
Application number: CN202210167143.6A
Authority: CN
Inventors: 董培; 韩骁; 刘月平
Original assignee: Tencent Technology Shenzhen Co Ltd; Fourth Hospital of Hebei Medical University Hebei Cancer Hospital
Current assignee: Tencent Technology Shenzhen Co Ltd; Fourth Hospital of Hebei Medical University Hebei Cancer Hospital
Priority date: 2022-02-23
Filing date: 2022-02-23
Publication date: 2022-05-27

Abstract

The application discloses a training method, a device, equipment and a medium of a cell classification model, and relates to the field of machine learning. The method comprises the following steps: acquiring a sample image and a cell label of the sample image, wherein the sample image comprises at least two cells, and the cell label is used for indicating the type of the cells in the sample image; performing data processing on the sample image through a cell classification model, and outputting a sample prediction hot spot diagram which is used for predicting the type of cells in the sample image; restoring the contour of each cell in the sample image through the contour of each cell nucleus in the sample image to obtain a cell segmentation image; generating a sample classification heat point diagram according to the cell marking and the cell segmentation diagram, wherein the sample classification heat point diagram is used for representing the types of the cells in the cell segmentation diagram; and training the cell classification model according to the loss between the sample prediction hotspot graph and the sample classification hotspot graph. The method and the device can realize weak supervised learning and improve the accuracy of the model.

Description

Training method, device, equipment and medium for cell classification model

Technical Field

The present application relates to the field of machine learning, and in particular, to a method, an apparatus, a device, and a medium for training a cell classification model.

Background

Modern medicine researches and understanding on tumors shows up new, and means for treating malignant tumors are continuously improved, wherein immunotherapy is an important mode for resisting cancers, and the implementation of immunotherapy requires that the counts of tumor cells and mononuclear inflammatory cells of users meet requirements.

In the related art, technicians are required to mark cell types in advance, when a cell classification model is trained, a sample image needs to be segmented into a plurality of sub-regions, and images corresponding to the sub-regions are sequentially input into the cell classification model. And calculating the difference between the prediction result output by the cell classification model and the form and the type of the cell, and training the cell classification model in stages according to the difference to obtain the trained cell classification model.

However, the cell labeling of the sample image provides a limited amount of information, which makes the related art less accurate.

Disclosure of Invention

The embodiment of the application provides a training method, a device, equipment and a medium of a cell classification model, the method can restore the outline of a cell, extract more information from a sample image and enable the classification result to be more accurate, and the technical scheme is as follows:

according to an aspect of the present application, there is provided a method of training a cell classification model, the method comprising:

acquiring a sample image and a cell label of the sample image, wherein the sample image comprises at least two cells, and the cell label is used for representing the type of the cells in the sample image;

performing data processing on the sample image through the cell classification model, and outputting a sample prediction hot spot diagram, wherein the sample prediction hot spot diagram is used for predicting the type of cells in the sample image;

restoring the contour of each cell in the sample image through the contour of each cell nucleus in the sample image to obtain a cell segmentation map;

generating a sample classification heat point map according to the cell labels and the cell segmentation map, wherein the sample classification heat point map is used for representing the types of the cells in the cell segmentation map;

and training the cell classification model according to the loss between the sample prediction hotspot graph and the sample classification hotspot graph.

According to an aspect of the present application, there is provided a training apparatus for a cell classification model, the apparatus including:

the system comprises a sample acquisition module, a cell identification module and a cell identification module, wherein the sample acquisition module is used for acquiring a sample image and a cell identification of the sample image, the sample image comprises at least two cells, and the cell identification is used for indicating the type of the cells in the sample image;

the data processing module is used for carrying out data processing on the sample image through the cell classification model and outputting a sample prediction hot spot diagram, and the sample prediction hot spot diagram is used for predicting the type of the cells in the sample image;

the data processing module is further configured to restore the contour of each cell in the sample image through the contour of each cell nucleus in the sample image to obtain a cell segmentation map;

the data processing module is further used for generating a sample classification heat point diagram according to the cell labels and the cell segmentation diagram, wherein the sample classification heat point diagram is used for representing the types of the cells in the cell segmentation diagram;

and the training module is used for training the cell classification model according to the loss between the sample prediction heat point diagram and the sample classification heat point diagram.

According to an aspect of the present application, there is provided a cell classification method, the method being performed by a computer device running a cell classification model as described above, the method comprising:

acquiring an input image, the input image comprising at least two types of cells;

performing data processing on the input image through the cell classification model, and outputting a predicted hotspot graph, wherein the predicted hotspot graph is used for representing the probability that the cell belongs to the target cell type;

and determining the type of each cell in the input image according to the predicted hotspot graph.

According to an aspect of the present application, there is provided a cell sorter apparatus, said apparatus operating with a cell sorting model as described above, the apparatus comprising:

an image acquisition module for acquiring an input image, the input image comprising at least two types of cells;

the model calling module is used for carrying out data processing on the input image through the cell classification model and outputting a predicted heat point diagram, and the predicted heat point diagram is used for representing the probability that the cell belongs to the target cell type;

and the prediction module is used for determining the type of each cell in the input image according to the predicted hotspot graph.

According to another aspect of the present application, there is provided a computer device including: a processor and a memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by the processor to implement a method of classifying a cell classification model, or a method of classifying a cell, as described above.

According to another aspect of the present application, there is provided a computer storage medium having at least one program code stored therein, the program code being loaded into and executed by a processor to implement a method of classifying a cell classification model, or a method of classifying a cell, as described above.

According to another aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and executes the computer instructions to cause the computer device to perform the cell classification model classification method or the cell classification method as described above.

The beneficial effects that technical scheme that this application embodiment brought include at least:

when the cell classification model is trained, the cell labels of the sample image are converted into a more complex cell segmentation map, the cell segmentation map comprises information related to cell morphology, a sample classification heat point diagram is obtained through the cell segmentation map, and the cell classification model is predicted according to the sample classification heat point diagram and the sample prediction heat point diagram of the sample image. When the cell classification model is trained, the outline of each cell in the sample image is restored through simple cell labeling, and then more information is extracted from the sample image and used for training the cell classification model, so that the accuracy of the classification result of the cell classification model can be effectively improved, and the cell classification model can be trained only by using labeling of cell types without providing labeling of cell forms, so that weak supervision learning can be realized, and the training efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic block diagram of a computer system provided in an exemplary embodiment of the present application;

FIG. 2 is a schematic diagram of a method for training a cell classification model provided in an exemplary embodiment of the present application;

FIG. 3 is a schematic flow chart diagram of a method for training a cell classification model according to an exemplary embodiment of the present application;

FIG. 4 is a schematic illustration of a sample image provided by an exemplary embodiment of the present application;

FIG. 5 is a schematic illustration of cell labeling provided by an exemplary embodiment of the present application;

FIG. 6 is a schematic flow chart diagram of a method for training a cell classification model provided in an exemplary embodiment of the present application;

FIG. 7 is a schematic diagram of an inflation operator provided by an exemplary embodiment of the present application;

FIG. 8 is a schematic diagram of an inflation operator provided by an exemplary embodiment of the present application;

FIG. 9 is a schematic diagram of a field of generating an image provided by an exemplary embodiment of the present application;

FIG. 10 is a schematic diagram of a field of generating an image provided by an exemplary embodiment of the present application;

FIG. 11 is a schematic illustration of a cell sorting method provided in an exemplary embodiment of the present application;

FIG. 12 is a schematic flow chart of a method of cell sorting provided in an exemplary embodiment of the present application;

FIG. 13 is a comparison of classification results provided by an exemplary embodiment of the present application;

FIG. 14 is a comparison of classification results provided by an exemplary embodiment of the present application;

FIG. 15 is a block diagram of a training apparatus for a cell classification model provided in an exemplary embodiment of the present application;

FIG. 16 is a block diagram of a cell sorter provided in an exemplary embodiment of the present application;

fig. 17 is a schematic structural diagram of a computer device according to an exemplary embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

First, terms referred to in the embodiments of the present application are described:

artificial Intelligence (AI): the method is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by the digital computer, sensing the environment, acquiring knowledge and obtaining the best result by using the knowledge. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision technology (Computer Vision, CV): computer vision is a science for researching how to make a machine "see", and more specifically, it refers to that a camera and a computer are used to replace human eyes to perform machine vision such as identification and measurement on a target, and further image processing is performed, so that the computer processing becomes an image more suitable for human eyes to observe or is transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. The computer vision technology generally includes technologies such as image processing, image Recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content/behavior Recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning, map construction, and the like, and also includes common biometric technologies such as face Recognition, fingerprint Recognition, and the like.

Machine Learning (ML): the method is a multi-field cross discipline and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

IHC (immunohistochemistry ): the cellular proteins in the tissue sample are pathologically labeled with a specific primary antibody and the target is visualized using a detection reagent. Protein expression can be assessed using IHC and by chromogenic or fluorescent detection.

PD-L1(Programmed Death-Ligand 1, apoptosis-Ligand 1): is a protein in the human body. It is a transmembrane protein and is involved in the suppression of the immune system.

Pembrolizumab (pabolizumab): is a humanized PD-1(Programmed Death-1) monoclonal antibody for cancer immunotherapy. Can be used for treating non-small cell lung cancer, urothelial cancer, esophageal squamous cell carcinoma, triple negative breast cancer, etc.

CPS (Combined Positive Score, composite Positive Score): is an interpretation index of PD-L1 detection for judging whether an indication user adopts Pembrolizumab for immunotherapy.

CPS is considered equal to 100 if the CPS value calculated according to the formula is greater than 100. CPS is used for PD-L1 detection and patient classification of indications such as head and neck squamous carcinoma, adenocarcinoma of stomach or gastroesophageal junction, cervical carcinoma, urothelial carcinoma, esophageal squamous carcinoma, triple negative breast cancer and the like.

With the research and progress of artificial intelligence technology, the artificial intelligence technology is developed and applied in a plurality of fields, such as common intelligent medical treatment, intelligent home, intelligent wearable equipment, virtual assistant, intelligent sound box, intelligent marketing, unmanned driving, automatic driving, unmanned aerial vehicle, robot, intelligent medical treatment, intelligent customer service, and the like.

It should be noted that information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, presented data, etc.), and signals referred to in this application are authorized by the user or sufficiently authorized by various parties, and the collection, use, and processing of the relevant data is required to comply with relevant laws and regulations and standards in relevant countries and regions. For example, the images and user information referred to in this application are obtained with sufficient authorization.

Fig. 1 shows a schematic structural diagram of a computer system provided in an exemplary embodiment of the present application. The computer system 100 includes: a terminal 120 and a server 140.

The terminal 120 has an application program installed thereon related to cell classification. The application program may be an applet in an app (application), may be a special application program, and may also be a web client. For example, in the case where the user is a doctor, the user obtains the result of the cell classification of the patient from the terminal 120, and determines the condition of the patient from the result. The terminal 120 is at least one of a smartphone, a tablet, an e-book reader, an MP3 player, an MP4 player, a laptop portable computer, and a desktop computer.

The terminal 120 is connected to the server 140 through a wireless network or a wired network.

The server 140 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a web service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform. The server 140 is used to provide background services for the application program of the cell classification, and send the information of the video related to the cell classification to the terminal 120. Alternatively, the server 140 undertakes primary computational tasks and the terminal 120 undertakes secondary computational tasks; alternatively, the server 140 undertakes the secondary computing work and the terminal 120 undertakes the primary computing work; alternatively, both the server 140 and the terminal 120 employ a distributed computing architecture for collaborative computing.

Fig. 2 is a schematic diagram illustrating a training method of a cell classification model according to an exemplary embodiment of the present application.

The sample image 201 is subjected to color deconvolution to obtain a sample channel image 202, wherein the color deconvolution is used to separate a target color in the sample image 201, for example, the target color is at least one of brown, blue, and pink, and specifically, the target color can be adjusted by a technician according to actual needs. Inputting the sample image 201 and the sample channel image 202 into the cell classification model 200, classifying the cells in the sample image 201 by the cell classification model 200 to obtain a sample prediction hotspot graph 203, wherein the sample prediction hotspot graph 203 is used for predicting a hotspot graph of the types of the cells in the sample image 201. Illustratively, the cell classification model 200 is a U-Net (U-network), and the cell classification model 200 includes a first feature encoding module and a first upsampling decoding module. The cell classification model 200 performs up-sampling and down-sampling on the sample image 201 to obtain a sample prediction hotspot graph 203. The cell classification model 200 may also be other types of classification models, among others.

The sample image 201 is subjected to cell nucleus segmentation, and the cell nucleus contour of each cell in the sample image 201 is determined, so that the cell nucleus schematic diagram 204 is generated. In the next step, the cell nucleus map 204 is post-processed to generate a cell segmentation map 205. Image post-processing is used to restore the contours of the cells according to the contours of the cell nuclei. With the cell label 206 as a reference, the cell segmentation map 205 is classified to obtain a sample classification hotspot map 207, and the sample classification hotspot map 207 is used for representing the hotspot map of the cell type in the sample image 201.

Calculating a first loss 208 between the sample prediction hotspot graph 203 and the sample classification hotspot graph 207, and calculating a second loss 209 between the sample prediction hotspot graph 203, the cell label 206 and the sample classification hotspot graph 207, wherein the first loss 208 is used for representing a difference value between the sample prediction hotspot graph and the sample classification hotspot graph, and the second loss 209 is used for optimizing and learning the cell classification model 200 according to the position and texture characteristics presented by cell nucleus (or chromosome) in the training process of the cell classification model 200. The cell classification model 200 is trained on the first loss 208 and the second loss 209.

Fig. 3 is a flowchart illustrating a method for training a cell classification model according to an exemplary embodiment of the present application. The method may be performed by the computer system 100 shown in FIG. 1, the method comprising:

step 302: acquiring a sample image and a cell label of the sample image, wherein the sample image comprises at least two cells, and the cell label is used for indicating the type of the cells in the sample image.

Optionally, the sample image is an electronic image of the stained pathological section. Illustratively, as shown in FIG. 4, a sample image 401 shows stained cells. Note that, since the cell nucleus has most of chromosomes in the cell, the stained portion of the cell after staining the cell is the cell nucleus.

Optionally, the sample image belongs to an RGB color digital image. In an alternative design, the physical size of the pixels of the sample image, which are used to represent the actual physical size of a single pixel, is no greater than 0.5 μm/pixel.

Different types of cells show different forms, the pathological section is transparent, the form of the cells can be observed only by staining the pathological section, and the staining agent can reflect the cells of a specific type or substances of the specific type, so that different colors are shown. Taking DAB as an example of a staining agent, DAB can make cells or substances which are positively reflected brown yellow, and in particular DAB can make diseased cells brown yellow.

Optionally, the type of cell includes, but is not limited to, at least one of a positive tumor cell, a negative tumor cell, a positive mononuclear cell, and a negative mononuclear cell.

Optionally, the cell labeling belongs to a point labeling, i.e. a point is used in the sample image to represent a cell or a cell nucleus. Optionally, the cell label is a label located at the center point of the nucleus. Illustratively, as shown in fig. 4 and 5, in the cell labeling 501, a point is used to replace a stained cell nucleus in the sample image 401.

Optionally, all cells in the sample image are labeled, and different types of cells are labeled with different colors. Illustratively, all pixels on a cell are labeled. Illustratively, all pixel points on the nucleus are labeled. Illustratively, the center point of the cell is taken and labeled.

Optionally, the cell label comprises an abscissa value, an ordinate value and a class label value for each point.

Alternatively, different colors are used to represent different types of cells. Illustratively, red is used for positive tumor cells, green is used for negative tumor cells, yellow is used for positive mononuclear inflammatory cells, and blue is used for negative mononuclear inflammatory cells. Alternatively, different shapes are used to represent different types of cells. Illustratively, a rectangle is used to indicate positive tumor cells and a circle is used to indicate negative tumor cells.

Step 304: and performing data processing on the sample image through the cell classification model, and outputting a sample prediction hot spot diagram, wherein the sample prediction hot spot diagram is used for predicting the type of the cells in the sample image.

A cell classification model is a model used to predict cell types. Optionally, the cell classification model belongs to a U-Net (U-network), and the cell classification model includes a first feature encoding module and a first upsampling decoding module. The first feature coding module is used for performing image downsampling on the sample image and extracting image features from the sample image, wherein the image features are related to cell type classification. The first up-sampling decoding module is used for restoring the size of the down-sampling result of the image to the size of the sample image. Illustratively, the sample image is an image of 32 × 32 size, and the sample image is down-sampled by the first feature encoding module to obtain 4 image features of 4 × 4. And calling a first up-sampling decoding module to perform image up-sampling on the 4 × 4 image features to obtain 4 32 × 32 image prediction results, wherein the image prediction results are used for representing the prediction results of one cell type.

Optionally, the cell classification model is a model for object detection, or a model for image segmentation. Illustratively, the cell classification model may also be a Transformer model. The embodiment of the present application does not specifically limit the model type of the cell classification model.

Optionally, the sample prediction hotspot map is a hotspot map for predicting a cell types in the sample image, wherein the b-th hotspot map is used for displaying b-th cells, a represents the number of cell types, and b is a positive integer smaller than a. For example, only cells of cell type A are shown in hotspot FIG. 1, and only cells of cell type B are shown in hotspot FIG. 2.

Optionally, separating a target color of the sample image to obtain a sample channel image, where the sample channel image is an image of the sample image obtained through an expression channel of the target color; and calling a cell classification model to perform data processing on the sample image and the sample channel image, and outputting a sample prediction hotspot graph. The target color is a color corresponding to the target cell type in the sample image, for example, when the target cell type is a positive tumor cell, the positive tumor cell is stained and then displayed as a brown yellow color, and the target color is a brown yellow color.

In one possible implementation, the color decomposition is performed on a Hematoxylin-Eosin-DAB staining space, which can result in: taking an output result of the positive expression channel of DAB as a sample channel image, and taking a sample channel image of an RGB channel and a sample channel image of a DAB channel as input of a cell classification model, wherein the positive expression channel represents DAB (brown yellow), the negative expression channel represents Hematoxylin (blue) counterstain, and the channel represents Eosin (pink). Optionally, the sample images of the RGB channels and the sample images of the hue channels are used as input of the cell classification model. Alternatively, the sample image of the RGB channel and the sample image of the grayscale channel are used as the input of the cell classification model.

The number of predicted hotspot graphs is the same as the number of color channels. For example, after the input image is processed by using the RGB channel and the DAB channel, 4 groups of predicted hotspot graphs are obtained.

Step 306: and restoring the contour of each cell in the sample image through the contour of each cell nucleus in the sample image to obtain a cell segmentation map.

Optionally, the sample image is subjected to data processing by a cell nucleus segmentation model, and the contour of each cell nucleus in the sample image is output. Illustratively, the cell nucleus segmentation model belongs to a U-Net neural network. The cell nucleus segmentation model comprises a second feature coding module and a second up-sampling decoding module.

Optionally, the morphological dilation operator is used to restore the contour of each cell in the sample image by the contour of each cell nucleus in the sample image.

The cell segmentation map refers to an image including cell outlines corresponding to the sample image. Illustratively, as shown, comparing the map to the map, it can be derived that the cell segmentation map is a prediction of the contour of each cell in the sample image.

Step 308: and generating a sample classification heat point diagram according to the cell marking and the cell segmentation diagram, wherein the sample classification heat point diagram is used for representing the types of the cells in the cell segmentation diagram.

Optionally, the sample classification hotspot map is a hotspot map representing a cell types in the sample image, and the c-th hotspot map is used for displaying the c-th cell. For example, only cells of cell type A are shown in hotspot FIG. 3, and only cells of cell type B are shown in hotspot FIG. 4.

Step 310: and training the cell classification model according to the loss between the sample prediction hotspot graph and the sample classification hotspot graph.

Optionally, the loss between the sample prediction hotspot graph and the sample classification hotspot graph includes a first loss and a second loss, the first loss is used for representing a difference value between the sample prediction hotspot graph and the sample classification hotspot graph, and the second loss is used for optimizing and learning the cell classification model according to the position and the texture characteristics presented by the cell nucleus (or the chromosome) in the training process of the cell classification model. Illustratively, the cell classification model is trained on a first loss between the sample prediction hotspot graph and the sample classification hotspot graph; or training the cell classification model according to a second loss between the sample prediction hotspot graph and the sample classification hotspot graph; or training the cell classification model according to the first loss and the second loss between the sample prediction hotspot graph and the sample classification hotspot graph.

Optionally, a mean square error between the sample prediction hotspot graph and the sample classification hotspot graph is calculated, resulting in a first loss.

Optionally, the second loss is calculated from the sample prediction hotspot graph and the sample classification hotspot graph according to the cell type provided by the cell label. The second penalty is used to represent a penalty term for the dense conditional random field. A Conditional Random Field (CRF) is a Conditional probability distribution model of output Random variables given a set of input Random variables, and is characterized in that the output Random variables form a markov Random Field, which can be used for labeling or analyzing images. The dense conditional random field is one of the conditional random fields, and if the dense conditional random field is adopted to classify the pixel points in the image, the dense conditional random field associates the target pixel point with other pixel points to obtain the type of the target pixel point, and the other pixel points refer to the pixel points in the image which are associated with the target pixel point. In the embodiment of the application, the cell classification model is trained by taking the loss term of the dense conditional random field as one of the loss functions of the cell classification model.

Illustratively, the loss term (i.e., the second loss) of the dense conditional random field is noted as

Then there are:

where M is the total number of pixels per channel image,

for the predicted hotspot graph corresponding to cell class c, W^cA similarity metric matrix for the category. W^cIn dense conditional random fields, which are fully connected gaussians, calculating their gradients in the corresponding loss optimization becomes a bilateral filtering problem. TC (tungsten carbide)_NIndicating positive tumor cells, TC_PIndicating negative tumor cells, MIC_NIndicating positive mononuclear inflammatory cells, MIC_PIndicating negative mononuclear cells.

Optionally, the cell classification model is trained by an error back propagation algorithm based on the loss between the sample prediction hotspot graph and the sample classification hotspot graph.

In summary, in this embodiment, when the cell classification model is trained, the cell labels of the sample image are converted into a more complex cell segmentation map, the cell segmentation map includes information related to cell morphology, the sample classification hotspot graph is obtained through the cell segmentation map, and the cell classification model is predicted according to the sample classification hotspot graph and the sample prediction hotspot graph of the sample image. When the cell classification model is trained, the outline of each cell in the sample image is restored through simple cell labeling, and then more information is extracted from the sample image and used for training the cell classification model, so that the accuracy of the classification result of the cell classification model can be effectively improved, and the cell classification model can be trained only by using labeling of cell types without providing labeling of cell forms, so that weak supervision learning can be realized, and the training efficiency is improved.

In the following embodiment, a process of restoring the cell contour according to the contour of the cell nucleus will be described, and the embodiment can convert the cell labeling of the sample image into a stronger pseudo label for cell segmentation, so that more information can be obtained from the sample image in the training of the cell classification model, which is beneficial to the training of the cell classification model, and the trained cell classification model can classify the cell more accurately.

Fig. 6 is a flowchart illustrating a method for training a cell classification model according to an exemplary embodiment of the present application. The method may be performed by the computer system 100 shown in FIG. 1, the method comprising:

step 601: the contour of the ith nucleus is traversed using the dilation operator.

The dilation operator is used to dilate the occupied area of the ith nucleus, i being a positive integer.

The shape of the dilation operator includes, but is not limited to, at least one of a circle, a rectangle, a triangle, a regular hexagon, and a regular octagon. Wherein, the specific shape of the expansion operator can be modified by the technical personnel according to the actual requirement. For example, a circular expansion operator is used when the sample image a is processed, and a rectangular expansion operator is used when the sample image B is processed.

Optionally, the morphology of the dilation operator is related to a traversal distance of the dilation operator, the traversal distance being used to represent a distance the dilation operator moves during traversal of the outline of the ith cell nucleus. Illustratively, the dilation operator is a circle with a radius of 5 microns after a 0.1 micron shift, and the dilation operator is a circle with a radius of 5.1 microns after a 0.2 micron shift. Illustratively, the size y of the dilation operator is f (x), x represents the traversal distance of the dilation operator, and f (x) is a custom function.

Optionally, the morphology of the dilation operator is related to a radius of curvature of a center point of the dilation operator, which refers to a radius of curvature of the center point of the dilation operator on the contour of the ith cell nucleus during traversal of the contour of the ith cell nucleus. Illustratively, when the curvature radius corresponding to the center point of the expansion operator is 4 microns, the expansion operator is a circle with a radius of 5 microns, and when the curvature radius corresponding to the center point of the expansion operator is 3 microns, the expansion operator is a square with a side length of 4 microns. The form of the expansion operator may be at least one of a radius, a diameter, a side length, an area, and a shape of the expansion operator.

For example, as shown in fig. 7, taking the expansion operator 702 as a circle as an example for illustration, the center of the expansion operator 702 is placed on the contour of the cell nucleus 701, and the center of the circle is controlled to move on the contour of the cell nucleus 701, so that the expansion operator 702 traverses the contour of the cell nucleus 701.

Step 602: and determining the image field corresponding to the ith cell nucleus according to the coverage area of the expansion operator.

The image field of the ith cell nucleus is used to predict the area occupied by the ith cell in the sample image. The ith cell is a cell containing the ith nucleus, and in the embodiment of the present application, only one cell having one nucleus and only one nucleus is considered.

The coverage area of the inflation operator is used to represent the union of the occupation areas of the inflation operator over the course of the inflation operator. Illustratively, as shown in FIG. 8, during traversal of the cell nucleus 701 by dilation operator 702, when dilation operator 702 is at position A, dilation operator 702 occupies a first region. When the inflation operator 702 is at position B, the inflation operator 702 will occupy the second region, taking the union of the first region and the second region as the inflation operator's coverage area. It should be noted that the movement of the dilation operator is a continuous process, and here, in order to clearly express the traversal process of the dilation operator, two discrete points are used for illustration.

Step 603: and determining the outline of the ith cell according to the image field of the ith cell nucleus.

In practice, the cells in the partial sample image may exhibit aggregation distribution, for example, when the sample image is used to display the cells in tumor cell nest and immune cell nest. Thus, there is a need to divide the cell neighborhoods that may overlap.

Optionally, the contour of the ith cell is determined according to the image field of the ith cell nucleus by using a watershed algorithm. Illustratively, the method comprises the sub-steps of:

1. and expanding the image field of the ith cell to obtain a target region.

Wherein the target area is larger than the image field of the ith cell. That is, the image field of the ith cell is located within the target region.

Alternatively, the method for expanding the image field of the ith cell may be to provide a decision block in the image field of the ith cell, and an area occupied by the decision block is denoted as a target area. The size of the decision block can be set by the technician according to the actual requirement. The method of expanding the image field of the ith cell can be realized by using the morphological dilation operator.

2. And determining pixel points belonging to the gray value interval in the target area.

Optionally, a pixel point belonging to the target gray value in the target area is determined. The target gray value is a constant.

Optionally, the grey value interval is dynamically generated. For example, a gray value interval is generated according to the gray values of the pixel points located at the edge of the image field. And if the gray value of the pixel point at the edge of the image field is in the interval [2, 8], taking a subset of the interval as a gray value interval, or taking a target gray value in the interval as the gray value interval.

Alternatively, the gray value interval may be set by the technician at his or her discretion. For example, the gradation value interval is directly set to [1, 10 ].

3. And connecting the pixel points belonging to the gray value interval to form a closed area.

For example, as shown in fig. 9, for an image field 901 of an ith cell, pixel points 902 belonging to a target gray value in a target region are determined (for convenience of description, only some of the pixel points belonging to the target gray value are displayed here), and a smooth curve is used to connect the pixel points 902, so as to obtain a closed region 903.

4. The closed area is determined as the image field of the ith cell.

Step 604: and repeating the three steps until the contour of each cell is determined to obtain a cell segmentation map.

Since the three steps only provide the outline of one cell, and the sample image includes a plurality of cells, the three steps need to be repeated to obtain the outline of each cell, so as to generate the cell segmentation map.

Step 605: according to the cell labeling, the cells of n cell types are separated from the cell segmentation map, and n cell type maps are generated.

The n cell type maps correspond one-to-one to the n cell types.

Illustratively, the cell segmentation map includes cells of A, B, C, D cell types, and the cells of these 4 cell types are isolated from the cell segmentation map to yield a cell type map including only cell type A, a cell type map including only cell type B, a cell type map including only cell type C, and a cell type map including only cell type D.

Step 606: and for the jth cell type graph in the n cell type graphs, giving a first gray value to the pixel point outside the cell in the jth cell type graph.

The first gray value is a constant, and optionally, the first gray value is 0.

For example, as shown in fig. 10, if the pixel 1002 is a pixel outside the cell 1001, the gray value of the pixel 1002 is set to 0.

Step 607: and assigning dynamic gray values to pixel points positioned in the cells in the jth cell type graph.

The dynamic gray value is positively correlated with the distance from the pixel point positioned in the cell to the edge of the cell. Optionally, the distance from the pixel point to the cell edge refers to the shortest distance from the pixel point to the cell edge. Alternatively, the distance from a pixel to the edge of a cell refers to the average distance from the pixel to the edge of the cell.

Illustratively, the pixel 1003 and the pixel 1004 are pixels located in the cell 1001, and the distance from the pixel 1003 to the cell edge is greater than the distance from the pixel 1004 to the cell edge, so that the gray value of the pixel 1003 is 255 and the gray value of the pixel 1004 is 250.

Step 608: and generating a jth sample classification hot spot diagram according to the gray value of each pixel point in the jth cell type diagram.

In the j sample classification heat point diagram, the pixel point heat outside the cell is 0, and the farther the cell edge is away from the cell edge, the higher the heat is.

Step 609: and repeating the three steps until a sample classification heat point diagram is obtained.

Since the above three steps provide only one kind of cell type of sample classification hotspot and the cells include at least two kinds, the above three steps need to be repeated to obtain a sample classification hotspot map of each cell type.

In summary, in this embodiment, when the cell classification model is trained, the cell labels of the sample image are converted into a more complex cell segmentation map, the cell segmentation map includes information related to cell morphology, the sample classification hotspot graph is obtained through the cell segmentation map, and the cell classification model is predicted according to the sample classification hotspot graph and the sample prediction hotspot graph of the sample image. When the cell classification model is trained, the outline of each cell in the sample image is restored through simple cell labeling, and then more information is extracted from the sample image and used for training the cell classification model, so that the accuracy of the classification result of the cell classification model can be effectively improved, weak supervision learning can be realized, and the training efficiency is improved.

FIG. 11 shows a schematic diagram of a cell sorting method provided in an exemplary embodiment of the present application.

Color deconvolution is performed on the input image 1101 to obtain a channel image 1102, wherein the color deconvolution is used to separate target colors in the input image 1101. The input image 1101 and the channel image 1102 are input into the cell classification model 1100, and the cell classification model 1100 classifies the cells in the input image 1101 to obtain the predicted hotspot graph 1103. The predicted hotspot graph 1103 is subjected to image processing, and the positions and the number of cells in the input image 1101 are obtained. Where image processing is used to count the location and number of different types of cells.

Exemplary, the location and number of cells includes the location and number of negative tumor cells, the location and number of positive tumor cells, the location and number of negative mononuclear cells, and the location and number of positive mononuclear cells. It should be noted that the skilled person can determine the location and number of more or less cells according to the actual needs. Specifically, the predicted hotspot graph 1103 is subjected to image processing, and a result image 1104 is obtained.

Fig. 12 is a schematic flow chart of a cell sorting method according to an exemplary embodiment of the present application. The method is performed by the cell classification provided by the above embodiment, and the method may be performed by the terminal 120 or the server 140 shown in fig. 1, where the terminal 120 or the server 140 runs the cell classification model provided by the above embodiment, and the method includes:

step 1202: an input image is acquired, the input image including at least two types of cells.

Optionally, the input image is an electronic image of the stained pathological section.

Optionally, the input image belongs to an RGB color digital image. In an alternative design, the physical size of the pixels of the input image is no greater than 0.5 μm/pixel.

Step 1204: and performing data processing on the input image through a cell classification model, outputting a prediction hot spot diagram, wherein the prediction hot spot diagram is used for expressing the probability that the cell belongs to the target cell type.

A cell classification model is a model used to predict cell types. Optionally, the cell classification model belongs to U-Net, and the cell classification model includes a first feature encoding module and a first upsampling decoding module. The first feature coding module is used for performing image down-sampling on an input image and extracting image features from the input image, wherein the image features are related to cell type classification.

Optionally, separating a target color of the input image to obtain a channel image, where the channel image is an image obtained by an expression channel of the target color; and calling a cell classification model to perform data processing on the input image and the channel image and outputting a predicted hotspot graph. In one possible implementation, the color decomposition is performed on a Hematoxylin-Eosin-DAB staining space, which can result in: the positive expression channel representing DAB (tan), the negative expression channel representing Hematoxylin (blue) counterstain, and the channel representing Eosin (pink) (not appearing in the PD-L1 stain) were taken as channel images for the output results of the positive expression channel of DAB.

Illustratively, the predicted hotspot graph comprises a set of hotspot graphs. And each hot spot map respectively gives the probability that each pixel point belongs to the cell type corresponding to the hot spot map. Illustratively, the 1 st predicted hotspot graph is used to represent a hotspot graph of positive tumor cells, and the 2 nd predicted hotspot graph is used to represent a hotspot graph of negative tumor cells.

Step 1206: from the predicted hotspot map, the type of each cell in the input image is determined.

Alternatively, the predicted hotspot map is a hotspot map representing a cell types in the input image, and the c-th hotspot map is used to display the c-th cell. For example, only cells of cell type A are shown in hotspot FIG. 3, and only cells of cell type B are shown in hotspot FIG. 4.

In the embodiment of the present application, after determining the type and number of each cell in the input image, the number of cells of n cell types in the hot spot map may be statistically predicted, where n is a positive integer greater than 2; a composite positive score is calculated based on the number of cells of at least one of the n cell types. The composite positive score was scored as CPS, with:

wherein N is₀Number of negative tumor cells, N₁Number of positive tumor cells, N₃The number of positive mononuclear cells is indicated.

Alternatively, the present embodiments are applicable to other methods of staining PD-L1 clone numbers, for example, the detection method of PD-L1 may also be at least one of Ventana SP263, Ventana SP142, Dako 28-8, Cell Signaling Technology E1L3N, WuXidiagnostics WD 160.

Optionally, the number of cells of different cell types is determined by local extreme points in the differential classification hotspot graph. Illustratively, the method of counting the number of cells may comprise the steps of:

1. and determining a local extreme point in the k-th predicted hotspot graph.

In the embodiment of the present application, since the predicted hotspot graph is used to indicate the probability that the cell belongs to the target cell type, the local extreme point in the predicted hotspot graph is located at the cell edge.

2. And acquiring the center coordinates of the connected region formed by the local extreme points.

Optionally, the coordinates of the center of the connected region represent the coordinates of the center of the cell.

Optionally, the center coordinates of the connected region are determined according to the coordinate average value of the edge of the connected region.

3. The number of cells in the kth heat map is determined from the number of center coordinates.

Optionally, the cell type corresponding to the kth hot spot map is assigned to the center point of the center coordinate.

4. The above three steps are repeated until the cell numbers of n cell types are obtained.

To sum up, the outline of each cell in the sample image is restored through simple cell labeling in the embodiment, and then more information is extracted from the sample image and used for training the cell classification model, so that the accuracy of the classification result of the cell classification model can be effectively improved, and moreover, as the cell classification model is trained only by using the labeling of the cell type, the labeling of the cell form is not required to be provided, therefore, the weak supervised learning can be realized, and the training efficiency is improved.

Optionally, the cell classification model of the present application is applied on an AI pathology cloud platform: in a pathology department or a medical center equipped with a digital pathology radiograph interpretation system, the cell classification model of the embodiment of the application can be integrated in a radiograph interpretation interface as an AI algorithm plug-in. After the digital pathological section is opened by a doctor, the classification result of the cell classification model of the method can be superposed on the digital pathological section to be displayed.

Alternatively, the cell classification model of the present application is applied in AI microscopy: on a microscope equipped with a digital image acquisition module, the cell classification model can be implanted into an interpretation process of a pathologist for watching a film under the microscope. The doctor presses a 'start calculation' button or steps on a foot pedal, the cell classification model performs cell classification on the currently acquired (and seen by the doctor at the same time) visual field, the image with the classification result is returned to the optical path of the microscope eyepiece in real time, and the doctor sees the cell classification result in the eyepiece.

Fig. 13 and 14 are schematic diagrams illustrating comparison of classification results provided by an exemplary embodiment of the present application.

In fig. 13, an interpretation image 1302 is a classification result of classifying cells of an original image 1301 by the cell classification method provided in the above embodiment. In fig. 14, an interpretation image 1402 is a classification result of cell classification of an original image 1401. Positive TC (red dot), negative TC (green dot), positive MIC (yellow dot), and negative MIC (blue dot) are respectively marked in the interpretation image 1302 and the interpretation image 1402. The classification results corresponding to the interpretation image 1302 and the interpretation image 1402 are accurate, so that the cell classification task can be well completed by the cell classification model provided by the embodiment of the application, and the classification effect is good.

The following are embodiments of the apparatus of the present application that may be used to perform embodiments of the method of the present application. For details which are not disclosed in the embodiments of the apparatus of the present application, reference is made to the embodiments of the method of the present application.

Referring to fig. 15, a block diagram of a training apparatus for a cell classification model according to an embodiment of the present application is shown. The above functions may be implemented by hardware, or may be implemented by hardware executing corresponding software. The apparatus 1500 includes:

a sample acquiring module 1501, configured to acquire a sample image and a cell label of the sample image, where the sample image includes at least two cells, and the cell label is used to indicate a type of the cell in the sample image;

a data processing module 1502, configured to perform data processing on the sample image through the cell classification model, and output a sample prediction hot spot map, where the sample prediction hot spot map is used to predict the type of the cells in the sample image;

the data processing module 1502 is further configured to restore the contour of each cell in the sample image according to the contour of each cell nucleus in the sample image, so as to obtain a cell segmentation map;

the data processing module 1502 is further configured to generate a sample classification hotspot graph according to the cell labels and the cell segmentation map, where the sample classification hotspot graph is used to represent types of cells in the cell segmentation map;

a training module 1503, configured to train the cell classification model according to a loss between the sample prediction hotspot graph and the sample classification hotspot graph.

In an alternative design of the present application, the data processing module 1502 is further configured to traverse the outline of the ith cell nucleus using a dilation operator, where the dilation operator is configured to dilate the occupied area of the ith cell nucleus, and i is a positive integer; determining an image field corresponding to the ith cell nucleus according to the coverage area of the expansion operator, wherein the image field of the ith cell nucleus is used for predicting the area occupied by the ith cell in the sample image; determining the outline of the ith cell according to the image field of the ith cell nucleus; and repeating the three steps until the contour of each cell is determined to obtain the cell segmentation graph.

In an optional design of the present application, the data processing module 1502 is further configured to expand an image field of the ith cell to obtain a target region; determining pixel points belonging to a gray value interval in the target area; connecting the pixel points belonging to the gray value interval to form a closed area; determining the closed region as an image field of the ith cell.

In an optional design of the present application, the data processing module 1502 is further configured to separate cells of n cell types from the cell segmentation map according to the cell labels, and generate n cell type maps, where the n cell type maps correspond to the n cell types one by one, and n is a positive integer;

in an optional design of the present application, the data processing module 1502 is further configured to, for a jth cell type map in the n cell type maps, assign a first gray scale value to a pixel point outside a cell in the jth cell type map, where j is a positive integer smaller than n + 1; assigning a dynamic gray value to a pixel point located in a cell in the jth cell type graph, wherein the dynamic gray value is in positive correlation with the distance from the pixel point located in the cell to the edge of the cell; generating a jth sample classification hot spot diagram according to the gray value of each pixel point in the jth cell type diagram; and repeating the three steps until the sample classification heat point diagram is obtained.

In an optional design of the present application, the data processing module 1502 is further configured to separate a target color of the sample image to obtain a sample channel image, where the sample channel image is an image obtained by the sample image through an expression channel of the target color; and calling the cell classification model to perform data processing on the sample image and the sample channel image, and outputting the sample prediction hotspot graph.

In an optional design of the present application, the training module 1503 is further configured to calculate a first loss between the sample prediction hotspot graph and the sample classification hotspot graph; calculating a second loss between the cell labeling, the sample prediction hotspot graph, and the sample classification hotspot graph; training the cell classification model based on the first loss and the second loss.

In an optional design of the present application, the training module 1503 is further configured to calculate a mean square error between the sample predicted heat point diagram and the sample classification heat point diagram, so as to obtain the first loss.

In an alternative design of the present application, the training module 1503 is further configured to calculate the second loss from the sample predicted hotspot graph and the sample classification hotspot graph according to the cell type provided by the cell label, where the second loss is used to represent a loss term of the dense conditional random field.

In summary, in this embodiment, when the cell classification model is trained, the cell labels of the sample image are converted into a more complex cell segmentation map, the cell segmentation map includes information related to cell morphology, the sample classification hotspot graph is obtained through the cell segmentation map, and the cell classification model is predicted according to the sample classification hotspot graph and the sample prediction hotspot graph of the sample image. According to the embodiment of the application, when the cell classification model is trained, the outline of each cell in the sample image is restored through simple cell labeling, and then more information is extracted from the sample image and used for training the cell classification model, so that the accuracy of the classification result of the cell classification model can be effectively improved.

Referring to fig. 16, a block diagram of a cell sorter according to an embodiment of the present application is shown. The above functions may be implemented by hardware, or may be implemented by hardware executing corresponding software. The apparatus 1600 is operated with the cell classification model provided in the above embodiment, the apparatus 1600 includes:

an image acquisition module 1601 for acquiring an input image, the input image comprising at least two types of cells;

a model calling module 1602, configured to perform data processing on the input image through the cell classification model, and output a predicted hotspot graph, where the predicted hotspot graph is used to represent a probability that a cell belongs to a target cell type;

a predicting module 1603 for determining the type of each cell in the input image according to the predicted hotspot graph.

In an optional design of the present application, the model invoking module 1602 is further configured to separate colors of the input image to obtain a channel image, where the channel image is an image obtained by the input image through an expression channel of the target color; and performing data processing on the input image and the image through the cell classification model, and outputting the predicted hotspot graph.

In an optional design of the present application, the predicting module 1603 is further configured to count the number of cells of n cell types in the predicted hotspot graph, wherein n is a positive integer greater than 2; calculating a composite positive score based on the number of cells of at least one of the n cell types.

In an optional design of the present application, the predicting module 1603 is further configured to determine a local extreme point in a kth predicted hotspot graph; acquiring the center coordinates of a connected region formed by the local extreme points; determining the number of cells of the kth hot spot map according to the number of the central coordinates; repeating the above three steps until the cell number of the n cell types is obtained.

FIG. 17 is a block diagram illustrating a computer device according to an example embodiment. The computer apparatus 1700 includes a Central Processing Unit (CPU) 1701, a system Memory 1704 including a Random Access Memory (RAM) 1702 and a Read-Only Memory (ROM) 1703, and a system bus 1705 connecting the system Memory 1704 and the CPU 1701. The computer device 1700 also includes a basic Input/Output system (I/O system) 1706 for facilitating information transfer between various elements within the computer device, and a mass storage device 1707 for storing an operating system 1713, application programs 1714, and other program modules 1715.

The basic input/output system 1706 includes a display 1708 for displaying information and an input device 1709 such as a mouse, keyboard, etc. for a user to input information. Wherein the display 1708 and the input device 1709 are connected to the central processing unit 1701 via an input-output controller 1710 connected to the system bus 1705. The basic input/output system 1706 may also include an input/output controller 1710 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input-output controller 1710 may also provide output to a display screen, a printer, or other type of output device.

The mass storage device 1707 is connected to the central processing unit 1701 through a mass storage controller (not shown) connected to the system bus 1705. The mass storage device 1707 and its associated computer device-readable media provide non-volatile storage for the computer device 1700. That is, the mass storage device 1707 may include a computer device readable medium (not shown) such as a hard disk or Compact Disc-Only Memory (CD-ROM) drive.

Without loss of generality, the computer device readable media may comprise computer device storage media and communication media. Computer device storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media includes RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disk (DVD), or other optical, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer device storage media is not limited to the foregoing. The system memory 1704 and mass storage device 1707 described above may be collectively referred to as memory.

The computer device 1700 may also operate in accordance with various embodiments of the present disclosure by connecting to remote computer devices over a network, such as the internet. That is, the computer device 1700 may be connected to the network 1711 through the network interface unit 1712 connected to the system bus 1705, or may be connected to other types of networks or remote computer device systems (not shown) using the network interface unit 1712.

The memory further includes one or more programs, which are stored in the memory, and the central processor 1701 implements the above-described training method of the cell classification model, or all or part of the steps of the cell classification method, by executing the one or more programs.

In an exemplary embodiment, a computer readable storage medium is further provided, in which at least one instruction, at least one program, code set, or instruction set is stored, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the cell classification model training method, or the cell classification method, provided by the above-mentioned various method embodiments.

The present application further provides a computer-readable storage medium, in which at least one instruction, at least one program, a set of codes, or a set of instructions is stored, and the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded and executed by the processor to implement the method for training a cell classification model, or the method for cell classification provided in the above-mentioned method embodiments.

The present application also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the training method of the cell classification model, or the cell classification method, provided in the above embodiment.

The above-mentioned serial numbers of the embodiments of the present application are merely for description, and do not represent the advantages and disadvantages of the embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for training a cell classification model, the method comprising:

generating a sample classification heat point diagram according to the cell labeling and the cell segmentation diagram, wherein the sample classification heat point diagram is used for representing the types of the cells in the cell segmentation diagram;

2. The method of claim 1, wherein the obtaining the cell segmentation map by restoring the contour of each cell in the sample image through the contour of each cell nucleus in the sample image comprises:

traversing the outline of the ith cell nucleus by using a dilation operator, wherein the dilation operator is used for dilating the occupied area of the ith cell nucleus, and i is a positive integer;

determining an image field corresponding to the ith cell nucleus according to the coverage area of the expansion operator, wherein the image field of the ith cell nucleus is used for predicting the area occupied by the ith cell in the sample image;

determining the outline of the ith cell according to the image field of the ith cell nucleus;

and repeating the three steps until the contour of each cell is determined to obtain the cell segmentation graph.

3. The method of claim 2, wherein said determining the contour of the ith cell from the image field of the ith cell comprises:

expanding the image field of the ith cell to obtain a target area;

determining pixel points belonging to a gray value interval in the target area;

connecting the pixel points belonging to the gray value interval to form a closed area;

determining the closed region as an image field of the ith cell.

4. The method according to any one of claims 1 to 3, wherein the generating a sample classification hotspot map from the cell labels and the cell segmentation map comprises:

according to the cell labels, separating the cells of n cell types from the cell segmentation graph to generate n cell type graphs, wherein the n cell type graphs correspond to the n cell types one by one, and n is a positive integer;

for a jth cell type graph in the n cell type graphs, giving a first gray value to a pixel point outside a cell in the jth cell type graph, wherein j is a positive integer smaller than n + 1;

giving a dynamic gray value to a pixel point positioned in a cell in the jth cell type graph, wherein the dynamic gray value is positively correlated with the distance from the pixel point positioned in the cell to the edge of the cell;

generating a jth sample classification hot spot diagram according to the gray value of each pixel point in the jth cell type diagram;

and repeating the three steps until the sample classification heat point diagram is obtained.

5. The method according to any one of claims 1 to 3, wherein the invoking the cell classification model to perform data processing on the sample image and output a sample prediction hotspot graph comprises:

separating the target color of the sample image to obtain a sample channel image, wherein the sample channel image is an image obtained by the sample image through an expression channel of the target color;

and calling the cell classification model to perform data processing on the sample image and the sample channel image, and outputting the sample prediction hotspot graph.

6. The method of any one of claims 1 to 3, wherein the training of the cell classification model based on the loss between the sample prediction hotspot graph and the sample classification hotspot graph comprises:

calculating a first loss between the sample prediction hotspot graph and the sample classification hotspot graph;

calculating a second loss between the cell labeling, the sample prediction hotspot graph, and the sample classification hotspot graph;

training the cell classification model based on the first loss and the second loss.

7. The method of claim 6, wherein the calculating a first penalty between the sample prediction hotspot graph and the sample classification hotspot graph comprises:

and calculating the mean square error between the sample prediction hotspot graph and the sample classification hotspot graph to obtain the first loss.

8. The method of claim 6, wherein said calculating a second loss between said cell labeling, said sample prediction hotspot graph, and said sample classification hotspot graph comprises:

and according to the cell types provided by the cell labels, calculating the second loss by the sample prediction hotspot graph and the sample classification hotspot graph, wherein the second loss is used for representing a loss term of the dense conditional random field.

9. A method of cell sorting, the method being performed by a computer device running a cell sorting model according to any one of claims 1 to 8, the method comprising:

10. The method according to claim 9, wherein the data processing of the input image by the cell classification model to output a predicted hotspot graph comprises:

separating the colors of the input image to obtain a channel image, wherein the channel image is an image obtained by the input image through an expression channel of the target color;

and performing data processing on the input image and the image through the cell classification model, and outputting the predicted hotspot graph.

11. The method of claim 9, further comprising:

counting the number of the cells of n cell types in the predicted hotspot graph, wherein n is a positive integer greater than 2;

calculating a composite positive score based on the number of cells of at least one of the n cell types.

12. The method of claim 11, wherein said counting the number of cells of n cell types in said predictive heat map comprises:

determining a local extreme point in a kth prediction hotspot graph, wherein k is a positive integer smaller than n + 1;

acquiring the center coordinates of a connected region formed by the local extreme points;

determining the number of cells of the kth hot spot map according to the number of the central coordinates;

repeating the above three steps until the cell number of the n cell types is obtained.

13. An apparatus for training a cell classification model, the apparatus comprising:

14. A cell sorter apparatus, which runs the cell sorting model according to any one of claims 1 to 8, the apparatus comprising:

the model calling module is used for carrying out data processing on the input image through the cell classification model and outputting a prediction heat point diagram, and the prediction heat point diagram is used for representing the probability that the cell belongs to the target cell type;

15. A computer device, characterized in that the computer device comprises: a processor and a memory having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by the processor to implement a method of training a cell classification model according to any one of claims 1 to 8, or a method of cell classification according to any one of claims 9 to 12.

16. A computer-readable storage medium having stored therein at least one program code, the program code being loaded into and executed by a processor to implement a method of training a cell classification model according to any one of claims 1 to 8, or a method of cell classification according to any one of claims 9 to 12.

17. A computer program product comprising a computer program or instructions which, when executed by a processor, implement a method of training a cell classification model according to any one of claims 1 to 8, or a method of cell classification according to any one of claims 9 to 12.