CN116843997B

CN116843997B - Model training method, cell image labeling method, device, equipment and storage medium

Info

Publication number: CN116843997B
Application number: CN202311070500.8A
Authority: CN
Inventors: 请求不公布姓名
Original assignee: Moore Threads Technology Co Ltd
Current assignee: Moore Threads Technology Co Ltd
Priority date: 2023-08-24
Filing date: 2023-08-24
Publication date: 2024-03-19
Anticipated expiration: 2043-08-24
Also published as: CN116843997A

Abstract

The embodiment of the application discloses a model training method, a cell image labeling method, a device, equipment and a storage medium, wherein the model training method comprises the following steps: identifying an unlabeled first cell image by adopting a pre-training model to obtain a first label of the first cell image; training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained first label prediction model; the training process comprises N training periods, wherein the label of the first cell image of the nth training period is obtained by updating the label of the first cell image based on the nth-1 training period, the unlabeled first cell image and a pre-training model after the nth-1 training period; n is a positive integer; n is a positive integer greater than 1 and less than or equal to N.

Description

Model training method, cell image labeling method, device, equipment and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, but is not limited to, and in particular, to a method, apparatus, device, and storage medium for model training and cell image labeling.

Background

In recent years, a neural network has made great progress in aspects of image classification, target detection, strength segmentation and the like, but training a neural network model requires a large amount of labeled training data, and in the field of biological cell images, because cells in the cell images are smaller and denser, a large amount of manpower is consumed when labeling the cell images, and the labeling result is inaccurate.

Disclosure of Invention

In view of this, the embodiments of the present application at least provide a method, apparatus, device, and storage medium for model training and labeling cell images, which can intelligently label cell images, and improve labeling efficiency and labeling accuracy.

The technical scheme of the embodiment of the application is realized as follows:

according to an aspect of the embodiments of the present application, there is provided a model training method, including:

identifying an unlabeled first cell image by adopting a pre-training model to obtain a first label of the first cell image;

training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained first label prediction model;

The training process comprises N training periods, wherein the label of the first cell image of the nth training period is obtained by updating the label of the first cell image based on the nth-1 training period, the unlabeled first cell image and a pre-training model after the nth-1 training period; n is a positive integer; n is a positive integer greater than 1 and less than or equal to N.

In some embodiments, the training the pre-training model based on the first cell image labeled with the first label and the second cell image labeled with the second label to obtain a trained first label prediction model includes:

training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a pre-training model after a first training period;

identifying the first untagged cell image by adopting the pre-training model after the first training period to obtain a label to be updated;

determining a third label of the first cell image of the second training period based on the first label and the label to be updated;

training the pre-training model after the first training period based on the first cell image marked with the third label and the second cell image marked with the second label until the first label prediction model is obtained.

In some embodiments, the training the pre-training model based on the first cell image labeled with the first label and the second cell image labeled with the second label to obtain a pre-training model after the first training period includes:

determining a first loss function, a weight corresponding to the first loss function, a second loss function and a weight corresponding to the second loss function;

determining a target loss function based on the first loss function, the weight corresponding to the first loss function, the second loss function and the weight corresponding to the second loss function;

and training the pre-training model based on the first cell image marked with the first label, the second cell image marked with the second label and the target loss function to obtain the pre-training model after the first training period.

In some embodiments, the method further comprises:

determining a label of a first cell image of an nth training period as a first target label of the first cell image;

or, identifying the first unlabeled cell image by adopting the first label prediction model to obtain a first target label of the first cell image.

In some embodiments, the method further comprises:

and under the condition that the first target label meets the verification condition, fine tuning is carried out on the first target label by adopting image processing software.

In some embodiments, the method further comprises:

under the condition that the first target label does not meet the verification condition, model parameters in the pre-training model are adjusted to obtain an adjusted pre-training model;

training the adjusted pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained second label prediction model and a second target label of the first cell image.

In some embodiments, the method further comprises:

acquiring a plurality of third cell images obtained after shooting cells belonging to different cell categories under a plurality of microscopes;

based on the plurality of third cell images, the unlabeled first cell image and the second cell image labeled with the second label are constructed.

In some embodiments, the constructing the unlabeled first cell image and the second cell image labeled with the second label based on the plurality of third cell images includes:

Preprocessing the plurality of third cell images to obtain a plurality of fourth cell images comprising a first color channel and a second color channel; the first color channel is used for representing the color of the stained cytoplasm, and the second color channel is used for representing the color of the stained nucleus;

based on the plurality of fourth cell images, the unlabeled first cell image and the second cell image labeled with the second label are constructed.

In some embodiments, the preprocessing the plurality of third cell images to obtain a plurality of fourth cell images including a first color channel and a second color channel includes:

performing matrix recombination on each third cell image, and removing a red channel of each third cell image to obtain a plurality of fourth cell images including a green channel and a blue channel; the first color channel is a green channel and the second color channel is a blue channel.

According to an aspect of an embodiment of the present application, there is provided a cell image labeling method, including:

identifying a fifth cell image to be marked by adopting a first label prediction model to obtain a fourth label of the fifth cell image;

Labeling the fifth cell image based on the fourth label to obtain a fifth cell image labeled with the fourth label;

the training process of the first label prediction model is as follows:

training the pre-training model based on a first cell image marked with a first label and a second cell image marked with a second label to obtain a first label prediction model;

the training process comprises N training periods, wherein the label of the first cell image of the nth training period is obtained by updating a pre-training model based on the label of the first cell image of the nth-1 training period, the unlabeled first cell image and the nth-1 training period, and N is a positive integer; n is a positive integer greater than 1 and less than or equal to N.

In some embodiments, the identifying the fifth cell image to be labeled using the first label prediction model to obtain a fourth label of the fifth cell image includes:

identifying cells in the fifth cell image by adopting the first label prediction model to obtain prediction probability corresponding to each pixel in the fifth cell image; the prediction probability is used for representing the probability that the corresponding pixel belongs to the cell;

Determining at least one cell region in the fifth cell image based on the prediction probability corresponding to each pixel in the fifth cell image;

the at least one cell region is taken as the fourth tag.

In some embodiments, the identifying the cells in the fifth cell image using the first label prediction model to obtain the prediction probability corresponding to each pixel in the fifth cell image includes:

adjusting the image size of the fifth cell image based on the target cell diameter to obtain an adjusted fifth cell image; the target cell diameter characterizes the cell diameter of a plurality of different morphologies of cells;

dividing the adjusted fifth cell image to obtain a plurality of sub-images;

identifying each sub-image by adopting the first label prediction model to obtain a prediction probability sub-image corresponding to each sub-image respectively; the prediction probability subgraph comprises the probability that each pixel in the corresponding subgraph belongs to cells;

integrating the predictive probability subgraphs of the plurality of subgraphs to obtain a predictive probability map of the fifth cell image; the predicted probability map includes a probability that each pixel in the fifth cell image belongs to a cell.

In some embodiments, the identifying each sub-image using the first label prediction model to obtain a prediction probability sub-image corresponding to each sub-image includes:

extracting the characteristics of each sub-image by adopting an encoder in the first label prediction model to obtain a first characteristic diagram of each sub-image;

carrying out pooling treatment on the first feature images by adopting a global average pooling module in the first label prediction model to obtain second feature images of each sub-image; wherein the second feature map characterizes the class of cells in each of the sub-images;

adopting a decoder in the first label prediction model to carry out channel recovery processing on the first feature map and the second feature map to obtain a prediction probability subgraph of each sub-image; the encoder and the decoder each include a plurality of feature extraction modules formed by a convolutional network and a residual network.

In some embodiments, the performing, by using a decoder in the first label prediction model, channel recovery processing on the first feature map and the second feature map to obtain a prediction probability subgraph of each sub-image includes:

Determining a first sub-feature map output by a first feature extraction module with the same channel number as that of an ith second feature extraction module in the decoder from a plurality of first feature extraction modules in the encoder; i is a positive integer greater than or equal to 2;

determining a second sub-feature map output by an i-1 th second feature extraction module in the decoder; the feature map output by the first and second feature extraction modules in the decoder is the first feature map output by the last first feature extraction module in the encoder;

adopting an ith second feature extraction module in the decoder to perform channel recovery processing on the second sub-feature map, the first sub-feature map and the second feature map to obtain a feature map output by the ith second feature extraction module in the decoder;

and taking the feature map output by the last second feature extraction module in the decoder as a predictive probability subgraph of each sub-image.

In some embodiments, the performing, by using the ith second feature extraction module in the decoder, a channel recovery process on the second sub-feature map, the first sub-feature map, and the second feature map to obtain a feature map output by the ith second feature extraction module in the decoder, where the method includes:

Summing the second sub-feature map and the second feature map to obtain a third feature map;

performing up-sampling processing on the third feature map to obtain a processed third feature map;

summing the processed third characteristic diagram and the first sub-characteristic diagram to obtain a fourth characteristic diagram;

and carrying out convolution processing on the fourth feature map to obtain a feature map output by an ith second feature extraction module in the decoder.

According to an aspect of the embodiments of the present application, there is provided a model training apparatus, including:

the first identification module is used for identifying the first untagged cell image by adopting a pre-training model to obtain a first label of the first cell image;

the training module is used for training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained first label prediction model;

According to an aspect of an embodiment of the present application, there is provided a cell image labeling apparatus, including:

the second identification module is used for identifying a fifth cell image to be marked by adopting the first label prediction model to obtain a fourth label of the fifth cell image;

the labeling module is used for labeling the fifth cell image based on the fourth label to obtain a fifth cell image labeled with the fourth label;

the training process of the first label prediction model is as follows:

According to an aspect of the embodiment of the present application, there is provided a model training apparatus, including a first memory and a first processor, where the first memory stores a computer program that can be run on the first processor, and the first processor implements the model training method described in the embodiment of the present application when executing the program.

According to an aspect of the embodiments of the present application, there is provided a cell image labeling apparatus, including a second memory and a second processor, where the second memory stores a computer program that can be executed on the second processor, and the second processor implements the cell image labeling method described in the embodiments of the present application when executing the program.

According to an aspect of embodiments of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method as described in embodiments of the present application.

In the embodiment of the application, a first label of a first cell image is obtained by identifying an unlabeled first cell image by adopting a pre-training model; training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained first label prediction model; the label of the first cell image of the nth training period is obtained by updating the label of the first cell image of the nth-1 training period, the unlabeled first cell image and the pre-training model after the nth-1 training period. In this way, in the process of training the pre-training model by adopting the first cell image marked with the first label (pseudo label) and the second cell image marked with the second label, each training period not only carries out iterative training on the pre-training model, but also updates the pseudo label of the first cell image so that the pseudo label used in each training stage is always the current optimal label, thereby improving the recognition accuracy of the trained first label prediction model through the interaction between the update of the pseudo label and the update of the model performance of the pre-training model, further realizing the automatic labeling of the label-free cell image by adopting the first label prediction model, improving the labeling efficiency and guaranteeing the labeling accuracy while improving the labeling efficiency.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the aspects of the present application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the technical aspects of the application.

Fig. 1 is a schematic implementation flow chart of a model training method provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of an implementation flow of another model training method according to an embodiment of the present application;

fig. 3 is a schematic implementation flow chart of a tag determining method according to an embodiment of the present application;

fig. 4 is a schematic implementation flow chart of a cellular image labeling method according to an embodiment of the present application;

fig. 5 is a schematic diagram of a network architecture of a first label prediction model according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a model training device according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a cellular image labeling device according to an embodiment of the present application;

fig. 8 is a schematic hardware entity diagram of a model training device according to an embodiment of the present application;

Fig. 9 is a schematic hardware entity diagram of a cellular image labeling device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application are further elaborated below in conjunction with the accompanying drawings and examples, which should not be construed as limiting the present application, and all other embodiments obtained by those skilled in the art without making inventive efforts are within the scope of protection of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.

The term "first/second/third" is merely to distinguish similar objects and does not represent a specific ordering of objects, it being understood that the "first/second/third" may be interchanged with a specific order or sequence, as permitted, to enable embodiments of the present application described herein to be practiced otherwise than as illustrated or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing the present application only and is not intended to be limiting of the present application.

In order to better understand the model training method provided in the embodiment of the present application, a description will be given below of a scheme in the related art related to the present application.

In recent years, deep neural networks have made significant progress in image classification, object detection, and instance segmentation. However, training deep neural networks requires a large amount of training data, which presents a significant challenge for human labeling. Especially for the labeling data set in the fields such as biological cell images, the labeling data set is very deficient, because cells in the cell images are very dense, part of the cells are hidden and are difficult to identify by naked eyes, so that a large amount of manpower is required for labeling, and meanwhile, the labeling of the biological cell images also needs to have certain expertise.

In the related art, a traditional crowdsourcing labeling method is generally adopted, and the method integrates scattered individuals (including part-time staff) and small labeling teams on the same platform to finish the labeling of a data set of a complete project. The main advantage of this approach is the flexibility. However, the quality of the image is difficult to guarantee, and especially the labeling of the cell image is required to have field expertise. Therefore, the crowdsourcing labeling method is not popular in the field of cell images, and an effective cell image labeling method is needed.

Therefore, the embodiment of the application provides a model training method to realize automatic labeling of cell images. The method may be performed by a processor of a model training apparatus. Fig. 1 is a schematic implementation flow chart of a model training method provided in an embodiment of the present application, as shown in fig. 1, the method includes the following steps 101 to 102:

and step 101, recognizing the unlabeled first cell image by adopting a pre-training model to obtain a first label of the first cell image.

Here, the pre-trained model may be a pre-trained, but not fully trained model; further, the pre-training model may be trained using a supervised learning approach. The first unlabeled cell image may refer to at least one unlabeled cell image; unlabeled, i.e. unlabeled, cell images. The first label is obtained by adopting a pre-training model to identify the unlabeled first cell image. The first tag may be referred to as a pseudo tag; the pseudo tag is not the final tag, and means a tag that also needs to be updated. The tag is used to characterize the nature of the cell in the cell image; illustratively, the tags may include, but are not limited to: at least one cell region in the cell image, cell type, whether the cell is abnormal, and the like.

In some embodiments, the initial model may be supervised trained using the second cell image labeled with the second label to obtain a pre-trained model; wherein the second tag refers to a tag of the second cell image. The second cell image labeled with the second label may refer to at least one labeled cell image. In one possible implementation, the second cell image that is not marked may be marked by using a manual marking method, so as to obtain a second cell image marked with a second label. Specifically, under the condition that the label is a cell area, at least one cell area in the second cell image is marked by adopting an artificial marking mode, so that the second cell image marked with the second label is obtained. The initial model may be a predetermined untrained model; in one possible implementation, the initial model may be a convolutional neural network; illustratively, the initial model may be a full convolutional neural network model (Fully Convolution Network, FCN), a Unet model, or the like.

And step 102, training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained first label prediction model.

Here, the first cell image labeled with the first label may be obtained by labeling an unlabeled first cell image with the first label. Because the first label is obtained by adopting a pre-training model, the accuracy of the first label is low, and therefore, the first label can be called as a pseudo label; the pseudo tag may refer not only to the first tag of the first cell image, but also to the tag after each update in the training process. The first label prediction model may refer to a trained model, and is used for identifying a cell image to be marked (a cell image without a label) so as to obtain a label of the cell image to be marked, thereby realizing automatic marking of the cell image to be marked. The training process of the first label prediction model may include N training periods, N being the total number of training periods; the training period may characterize the update timing of the pseudo tag. In one possible implementation, the training periods may be preset, and each training period may include 100 iterative training; at this time, after training each model 100 times, the pseudo tag is updated once; the pre-training model after the first training period, i.e., the pre-training model after 100 iterative training. The number of iterative training times included in different training periods may be the same or different, which is not limited in this application.

The label of the first cell image of the nth training period is obtained by updating a pre-training model based on the label of the first cell image of the nth-1 training period, the unlabeled first cell image and the nth-1 training period; it means that the label (pseudo label) of the first cell image needs to be updated in each training period, so that the pseudo label used in training is always the current optimal label.

Because the pre-training model is not completely trained, the label (first label) of the first cell image obtained after the recognition by adopting the non-trained pre-training model is not accurate enough, so that the label of the first image needs to be updated in each training period, namely, the pseudo label of the first cell image which is not marked is updated in the training process, so that the pseudo label used for training is always the current optimal label, and the model performance is continuously improved based on the current optimal label; in this way, the first label prediction model with optimal model performance is obtained through the interaction between the update of the pseudo label and the update of the model performance.

In some embodiments, the first cell image labeled with the first label and the second cell image labeled with the second label may be input to a pre-training model for training, so as to obtain a trained first label prediction model. For example, a first cell image marked with a first label and a second cell image marked with a second label can be input into a pre-training model for training, so as to obtain the pre-training model after a first training period; updating to obtain a label of the first cell image of the second training period based on the unlabeled first cell image, the label of the first cell image of the first training period and the pre-training model after the first training period; inputting a first cell image marked with a label after the first updating (a label of a first cell image of a second training period) and a second cell image marked with a second label into a pre-training model after the first training period for training to obtain a pre-training model after the second training period; updating to obtain a label of the first cell image of the third training period based on the unlabeled first cell image, the label of the first cell image of the second training period and the pre-training model after the second training period; and training by continuously using the first cell image marked with the label after the second updating and the second cell image marked with the second label, and circulating until N training periods are all trained, so as to obtain a first label prediction model.

The embodiment of the application provides a model training method which can be executed by a processor of model training equipment. As shown in fig. 2, the method includes the following steps 201 to 204:

step 201, obtaining a plurality of third cell images obtained after shooting cells belonging to different cell types under a plurality of microscopes.

Here, the plurality of microscopes may refer to a plurality of different types of microscopes; illustratively, the plurality of microscopes can include, but are not limited to: bright field microscopy, phase contrast microscopy, fluorescence microscopy, differential interference contrast microscopy, and the like. Different cell categories may refer to cells that are classified into different categories according to different classification criteria; illustratively, cells may be classified into two cell categories, negative cells and positive cells, according to abnormal conditions; alternatively, cells can be classified into three cell types, totipotent cells, pluripotent cells and unipotent cells according to differentiation potential. The plurality of third cell images may be cell images obtained by photographing cells belonging to different cell types under a plurality of microscopes.

In some embodiments, a plurality of microscopes such as a bright field microscope, a phase contrast microscope, a fluorescence microscope, a differential interference contrast microscope, etc. may be used to capture the cells of different cell types to obtain a plurality of third cell images.

Step 202, constructing the unlabeled first cell image and the second cell image labeled with the second label based on the plurality of third cell images.

In some embodiments, the plurality of third cell images may be pre-processed; based on the preprocessed plurality of third cell images, an unlabeled first cell image and a second cell image labeled with a second label are constructed.

And 203, identifying the first untagged cell image by adopting a pre-training model to obtain a first label of the first cell image.

And 204, training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained first label prediction model.

Here, the steps 203 to 204 correspond to the steps 101 to 102, respectively, and reference may be made to the specific embodiments of the steps 101 to 102 when implemented.

In the embodiment of the application, a plurality of third cell images obtained after shooting cells belonging to different cell categories under a plurality of microscopes are adopted to construct an unlabeled first cell image and a second cell image labeled with a second label, which are used for training a first label prediction model. Therefore, the first label prediction model can learn the cell characteristics of the multi-mode (multi-cell type under a multi-microscope) cell image, so that the method is suitable for identifying the multi-mode cell image to acquire the labels of the multi-mode cell image, and further automatic labeling of the multi-mode cell image is realized.

In some embodiments, the step 202 may include the following steps 2021 to 2022:

step 2021, preprocessing the plurality of third cell images to obtain a plurality of fourth cell images including a first color channel and a second color channel; the first color channel is used to characterize the color of the stained cytoplasm and the second color channel is used to characterize the color of the stained nucleus.

Thus, each fourth cell image may be a cell image including the first color channel and the second color channel obtained by preprocessing the corresponding third cell image. The first color channel is different from the second color channel, the first color channel is used for highlighting the color of the stained cytoplasm, and the second color channel is used for highlighting the color of the stained nucleus, namely, the color channel capable of highlighting the cytoplasm is different from the color channel capable of highlighting the nucleus; in this case, the fourth cell image may include both information on cytoplasm and information on nucleus.

The research shows that the main channel (green channel) in the cell image can highlight cytoplasm, but only the main channel corresponding to the cytoplasm is adopted to identify the cell image, so that the identification effect is not ideal; thus, a fourth cell image comprising a first color channel for highlighting the color of the stained cytoplasm and a second color channel for highlighting the color of the stained nucleus is employed here to construct a dataset for use in training the model.

In some embodiments, to highlight the color of the cytoplasm and nucleus, the implementation of obtaining the third cell image may be: staining cytoplasm in the cell with a first dye, and staining nucleus in the cell with a second dye; the stained cells were observed using a microscope and the observed cells were photographed to obtain a third cell image. Illustratively, the first dye may be a green fluorescent dye (bbcelprobe C02), and the second dye may be a blue fluorescent dye (4, 6-diamino-2-phenylindoline (DAPI)); in this way, the cytoplasm in the cells is stained green with a green fluorescent dye, and the nuclei in the cells are stained blue with a blue fluorescent dye. Thus, after the plurality of third cell images are preprocessed, a plurality of fourth cell images including the first color channel (green channel) and the second color channel (blue channel) can be obtained.

In some embodiments, the implementation of determining the plurality of fourth cell images may be: performing matrix recombination on each third cell image, and removing a red channel of each third cell image to obtain a plurality of fourth cell images including a green channel and a blue channel; the first color channel is a green channel and the second color channel is a blue channel.

In this way, matrix recombination is performed on each third cell image, and the red channel of each third cell image is removed, so as to remove interference information irrelevant to cells in the third cell image, and reserve and highlight a green channel corresponding to cytoplasm and a blue channel corresponding to nucleus, which can embody cell characteristics and cell positions.

When the method is implemented, if cytoplasm in the third cell image is stained by adopting green fluorescent dye, a channel capable of highlighting the cytoplasm at the moment is a green channel; if the nuclei in the third cell image are stained with a blue fluorescent dye, then the channel that is capable of highlighting the nuclei is a blue channel.

In one possible implementation manner, when any third cell image is subjected to matrix recombination, the red channel of the third cell image can be directly removed, then the green channel of the third cell image is used as a main channel, the blue channel of the third cell image is used as a second channel, and then the green channel of the third cell image and the blue channel of the third cell image are recombined together according to the weight corresponding to the main channel and the weight corresponding to the second channel, so as to obtain a fourth cell image.

Step 2022, constructing the unlabeled first cell image and the second cell image labeled with the second label based on the plurality of fourth cell images.

In some embodiments, the plurality of fourth cell images may be partitioned to obtain a first set of cell images and a second set of cell images; taking the first cell image in the first cell image set as a second cell image without a label; and labeling the second cell images in the second cell image set to obtain second cell images labeled with second labels.

In the above-described embodiment, the data set used for model training was constructed by employing a fourth cell image including a first color channel for highlighting the color of the stained cytoplasm and a second color channel for highlighting the color of the stained nucleus. Thus, the relevant information of cytoplasm can be obtained and cytoplasm can be positioned through the first color channel, the relevant information of cell nucleus can be obtained and the cell nucleus can be positioned through the second color channel, so that more cell characteristics can be obtained through adding the blue channel corresponding to the extra cell nucleus, and further, the label of the cell image can be accurately determined.

In some embodiments, the step 204 may include the following steps 2041 to 2044:

Step 2041, training the pre-training model based on the first cell image labeled with the first label and the second cell image labeled with the second label, to obtain a pre-training model after the first training period.

In some embodiments, a cell image can be selected from a first cell image marked with a first label and a second cell image marked with a second label as a cell image used for the first iterative training, and is input into a pre-training model for training, so as to obtain a pre-training model after the first iterative training and model output; determining a loss value between the model output after the first iterative training and the label of the used cell image; adjusting model parameters of the pre-training model according to the loss value to obtain a first adjusted pre-training model; continuously selecting a cell image from the first cell image marked with the first label and the second cell image marked with the second label as a cell image used for the second iterative training, inputting the cell image into a pre-training model for training, and obtaining a pre-training model and model output after the second iterative training; determining a loss value between the model output after the second iterative training and the label of the used cell image; continuously adjusting model parameters of the pre-training model according to the loss value to obtain a second adjusted pre-training model; and continuing to select a cell image from the first cell image marked with the first label and the second cell image marked with the second label as a cell image used for the third iterative training, and performing model training until a pre-training model after the first training period is obtained.

And 2042, identifying the first untagged cell image by adopting the pre-training model after the first training period to obtain a label to be updated.

In this way, the label to be updated may be a label obtained by identifying the first unlabeled cell image by using the pre-training model after the first training period. Under the condition that the training period is 100 times, the label to be updated is a label obtained by adopting a pre-training model which is trained for 100 times in an iterative manner to identify the first cell image which is not marked.

Step 2043, determining a third label of the first cell image of the second training cycle based on the first label and the label to be updated.

Thus, the third label may be the label of the first cell image of the second training period, i.e. the third label is the third label updated after the first training period. In the case of 100 training periods, the third label is a pseudo label updated after 100 iterative training.

In some embodiments, if the label refers to a cell area in the cell image, determining the third label to be updated after the first training period based on the first label and the label to be updated may be to superimpose the cell area indicated by the first label with the cell area indicated by the label to be updated to obtain the third label.

In other embodiments, if the tag refers to a cell class in the cell image, the tag to be updated may be determined directly as a third tag; alternatively, the label that occurs most frequently in the first training period may also be determined as the third label.

And 2044, training the pre-training model after the first training period based on the first cell image marked with the third label and the second cell image marked with the second label until the first label prediction model is obtained.

In this way, the pre-training model after the first training period is trained based on the first cell image labeled with the third label and the second cell image labeled with the second label, that is, model training is continued using the label of the updated first cell image.

Training the pre-training model after the first training period based on the first cell image marked with the third label and the second cell image marked with the second label until the first label prediction model is obtained; training the pre-training model after the first training period based on the first cell image marked with the third label and the second cell image marked with the second label to obtain a pre-training model after the second training period (a pre-training model after training for 200 times); identifying the first cell image which is not marked by adopting the pre-training model after the second training period to obtain a label to be updated corresponding to the second training period; determining a pseudo tag updated after the second training period based on the third tag and the tag to be updated corresponding to the second training period; and training by adopting the updated pseudo tag continuously, and circulating until N training periods are trained, so as to obtain a first tag prediction model. The total iteration number corresponding to the N training periods may be 2500 iteration numbers, where the pre-training model after the iteration training is 2500 times is determined as the first label prediction model. Or, other iteration stop conditions, such as the accuracy of model prediction, can be set, and under the condition that the accuracy of model prediction is greater than or equal to the accuracy threshold, the iteration stop conditions are determined to be reached, training is stopped, and a first label prediction model is obtained.

In the above embodiment, the pseudo tag of the first cell image is updated once after each training cycle of training the pre-training model. Therefore, the model is continuously trained through the updated pseudo tag, the recognition accuracy of the trained first tag prediction model can be improved, and further the first tag prediction model is adopted to automatically label the label-free cell image, so that the labeling efficiency is improved, and the labeling accuracy is ensured while the labeling efficiency is improved.

In some embodiments, the step 2041 may include the following steps 2041a to 2041c:

step 2041a, determining a first loss function, a weight corresponding to the first loss function, a second loss function, and a weight corresponding to the second loss function.

Thus, the first loss function may be a loss function corresponding to the tagged cell image (second cell image tagged with the second tag). The second loss function may be a loss function corresponding to an unlabeled cell image (unlabeled first cell image). Because the second cell image marked with the second label is obtained by adopting a manual marking mode, the label accuracy is higher, so that the weight corresponding to the first loss function can be set to be higher than the weight corresponding to the second loss function; for example, the weight corresponding to the first loss function may be set to 0.95, the weight corresponding to the second loss function may be set to 0.05, and the weight corresponding to the first loss function and the weight corresponding to the second loss function may be adjusted according to the actual service scenario, which is not limited in the embodiment of the present application.

In some embodiments, the first loss function and the second loss function may be set to the same loss function or may be set to different loss functions. Illustratively, the loss function may be a square loss function, a cross entropy loss function, an exponential loss function, or the like; both the first loss function and the second loss function may be set as square loss functions; alternatively, the first loss function may be a square loss function and the second loss function may be a cross entropy loss function.

Step 2041b, determining a target loss function based on the first loss function, the weight corresponding to the first loss function, the second loss function, and the weight corresponding to the second loss function.

In some implementations, the first loss function may be multiplied by a weight corresponding to the first loss function; multiplying the second loss function by the weight corresponding to the second loss function; and adding the two multiplied formulas to obtain the target loss function.

And step 2041c, training the pre-training model based on the first cell image marked with the first label, the second cell image marked with the second label and the target loss function to obtain a pre-training model after the first training period.

In some embodiments, when obtaining the pre-trained model and model output after each iteration training, a loss value between the model output after each iteration training and the label of the used cell image may be determined by the objective loss function; model parameters of the pre-trained model are adjusted according to the loss value.

In the embodiment of the application, the target loss function is determined by a first loss function for the labeled cell image, a second loss function for the unlabeled cell image, and a weight corresponding to the first loss function and a weight corresponding to the second loss function. Thus, the model is prone to learn the cell characteristics of the labeled cell image by adjusting the weight, so that the recognition accuracy of the model can be improved.

The embodiment of the application provides a label determining method which can be executed by a processor of model training equipment. As shown in fig. 3, the method includes the following steps 301 to 305:

step 301, obtaining a plurality of third cell images obtained after shooting cells belonging to different cell types under a plurality of microscopes.

Step 302, constructing the unlabeled first cell image and the second cell image labeled with the second label based on the plurality of third cell images.

And 303, identifying the first untagged cell image by adopting a pre-training model to obtain a first label of the first cell image.

And step 304, training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained first label prediction model.

Here, the steps 301 to 304 correspond to the steps 201 to 204, respectively, and reference may be made to the specific embodiments of the steps 201 to 204 when implemented.

And 305, identifying the first unlabeled cell image by adopting the first label prediction model to obtain a first target label of the first cell image.

Thus, the first target tag may be the tag of the first cell image that is ultimately determined. In some embodiments, after the first label prediction model is trained, the first label prediction model may be used to identify the unlabeled first cell image, and the identification result is directly determined as the final label of the first cell image.

In other embodiments, the first target tag of the first cell image may also be determined by: and determining the label of the first cell image of the N training period as the first target label of the first cell image.

Determining the label of the first cell image of the N-th training period as a first target label of the first cell image; determining the label of the last updated cell image as the final label of the first cell image; therefore, in the training process of the first label prediction model, the final label of the unlabeled first cell image is determined, and based on the final label, the label of the unlabeled cell image is determined while the model is trained, so that the labeling efficiency is improved.

In a possible implementation manner, if the total iteration number is 2500 and the training period is 100, the label of the first cell image obtained by updating after 25 training periods can be used as a first target label; alternatively, the label of the first cell image used in the 25 th training period may be used as the first target label.

It should be noted that, because the pre-training model is obtained by training through a supervised learning method, the labeled data set used for training can ensure the accuracy of labeling and the model performance of the pre-training model which is trained subsequently through a manual labeling method; except for the marked data set used by the pre-training model, the rest unmarked data sets can be automatically marked by adopting the trained first label prediction model, so that more than 80% of automatic marking in the cell image can be realized, and the marking efficiency is improved.

In some embodiments, following the step 305, the following step 306, or steps 307 to 308 may be further included.

And 306, fine tuning the first target tag by adopting image processing software under the condition that the first target tag meets the verification condition.

Thus, the first target tag satisfying the verification condition means that the first target tag passes the tag verification. The verification condition is used for verifying the first target label so as to ensure the quality of the first target label. Illustratively, the first target tag satisfying the verification condition may refer to the accuracy rate of the first target tag being greater than an accuracy rate threshold; alternatively, the first target tag satisfying the verification condition may mean that the first target tag passes the manual verification. The accuracy rate threshold may be preset, and the accuracy rate threshold may be set to 0.8. The image processing software is used for fine tuning the first target label so as to ensure the practicability and accuracy of the final label. For example, the image processing software may be ImageJ plug-Labkit.

In some embodiments, F1-Score may be used to determine the accuracy of the first target tag; under the condition that the accuracy rate of the first target label is larger than 0.8, fine tuning is carried out on the first target label by adopting an imageJ plug-Labkit to obtain a final label of the first cell image.

And step 307, adjusting model parameters in the pre-training model to obtain an adjusted pre-training model under the condition that the first target label does not meet the verification condition.

Thus, the first target tag does not satisfy the verification condition, which may mean that the accuracy rate of the first target tag is less than or equal to the accuracy rate threshold; alternatively, the first target tag does not satisfy the verification condition, which may mean that the first target tag fails the manual verification.

In some embodiments, when the accuracy of the first target label is less than or equal to 0.8, the model parameters in the pre-training model are adjusted to obtain an adjusted pre-training model, so that the final label of the first cell image is retrained based on the adjusted pre-training model, and the identification accuracy of the model and the accuracy and the practicability of the determined label are ensured.

Step 308, training the adjusted pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label, to obtain a trained second label prediction model and a second target label of the first cell image.

In the implementation process, the adjusted pre-training model is trained based on the first cell image marked with the first label and the second cell image marked with the second label, the first label is updated in the training process, and the operation of obtaining the second label prediction model and the second target label of the first cell image is similar to the operation of training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label, and the operation of updating the first label in the training process and obtaining the first label prediction model, which is not repeated in the embodiment of the application.

The embodiment of the application provides a cell image labeling method which can be executed by a processor of a cell image labeling device. Fig. 4 is a schematic implementation flow chart of a cellular image labeling method according to an embodiment of the present application, as shown in fig. 4, where the method includes the following steps 401 to 402:

and step 401, identifying a fifth cell image to be marked by adopting a first label prediction model to obtain a fourth label of the fifth cell image.

Thus, the fifth cell image is the cell image to be annotated, i.e. the fifth cell image is the cell image that currently needs to be annotated. The fourth label can be a label obtained after the fifth cell image to be marked is identified by adopting the first label prediction model.

In some embodiments, after obtaining the cell image to be annotated, preprocessing the cell image to be annotated is required to obtain a fifth cell image comprising a first color channel and a second color channel; and inputting the fifth cell image to be marked into the first label prediction model for recognition to obtain a fourth label of the fifth cell image. The implementation manner of obtaining the fifth cell image may be: matrix recombination can be carried out on the cell image to be marked, a red channel of the cell image to be marked is removed, and a fifth cell image comprising a green channel and a blue channel is obtained; the first color channel is a green color channel and the second color channel is a blue color channel.

And step 402, labeling the fifth cell image based on the fourth label, so as to obtain a fifth cell image labeled with the fourth label.

The training process of the first label prediction model is as follows:

In some embodiments, in the case where the tag is a cell region, the outline of the cell region indicated by the fourth tag may be depicted in the fifth cell image, resulting in a fifth cell image labeled with the fourth tag. In another embodiment, when the label is a cell type, the cell type indicated by the fourth label may be labeled in the fifth cell image in the form of a letter, a character, a number, or the like, and the fifth cell image labeled with the fourth label may be obtained.

In the embodiment of the application, the first label pre-storing model is obtained by training the pre-training model based on a first cell image marked with the first label and a second cell image marked with the second label, and updating the first label in the training process. In this way, in the process of training the pre-training model by using the first cell image marked with the first label and the second cell image marked with the second label, not only the first label (pseudo label) identified by the pre-training model is updated, but also the pre-training model is iteratively trained by using the label of the updated first cell image, so that the identification accuracy of the trained first label prediction model is high, the accuracy of the fourth label of the fifth cell image determined by using the first label prediction model is high, and the labeling efficiency is far higher than that of manual labeling because the first label prediction model is directly used for realizing automatic labeling.

In some embodiments, if the tag is a cell region in a cell image, the above step 401 may include the following steps 4011 to 4013:

step 4011, identifying cells in the fifth cell image by using the first label prediction model to obtain a prediction probability corresponding to each pixel in the fifth cell image; the predicted probabilities are used to characterize the probability that the corresponding pixel belongs to the cell.

It can be understood that, since the first color channel is used for highlighting the color of the stained cytoplasm and the second color channel is used for highlighting the color of the stained nucleus, the fifth cell image highlights the relevant information of the cytoplasm and the relevant information of the nucleus, so that the first label prediction model can learn the cell characteristics well and accurately locate the cells based on the prior knowledge in the fields (the information highlighted by the first color channel and the second color channel), the recognition accuracy of the first label prediction model is improved, the unlabeled fifth cell image is processed by adopting the first label prediction model, and the prediction probability that each pixel in the determined fifth cell image belongs to the cells can be more accurate.

Step 4012, determining at least one cell region in the fifth cell image based on the respective prediction probabilities for each pixel in the fifth cell image.

Therefore, based on the prediction probability that each pixel in the fifth cell image corresponds to the cell, the pixels in the fifth cell image belong to the cell and the pixels do not belong to the cell, and a plurality of cell areas can be determined according to the pixels belonging to the cell and the positions of the pixels.

In some embodiments, the implementation of step 4012 may be: determining a mask map of the fifth cell image based on the predictive probability map of the fifth cell image; the prediction probability map comprises a prediction probability corresponding to each pixel in the fifth cell image; the prediction probability is used for representing the probability that the corresponding pixel belongs to the cell; and performing segmentation processing on cells in the fifth cell image based on the mask image of the fifth cell image by adopting thermal diffusion simulation to obtain at least one cell region in the fifth cell image.

It will be appreciated that the mask of the fifth cell image may be used to divide cells, but in order to obtain a more accurate cell division result, after the mask of the fifth cell image is obtained, a thermal diffusion simulation may also be used to divide cells in the fifth cell image based on the mask of the fifth cell image to achieve a more accurate cell division.

In one possible implementation, a masking operator may be used to process the predicted probability map of the fifth cell image to obtain a masking map of the fifth cell image; then, determining a plurality of gradient vector fields based on the mask map of the fifth cell image using thermal diffusion simulation; constructing a plurality of power systems with fixed points based on the plurality of gradient vector fields; dividing pixels converged to the same fixed point in the fifth cell image into a group based on a plurality of fixed points to obtain a plurality of pixel groups; and determining the image area formed by each pixel group as a cell area, and obtaining at least one cell area in the fifth cell image.

Step 4013, using at least one cell region as a fourth tag.

In some embodiments, the step 4011 described above may include the following steps 4011a through 4011d:

step 4011a, adjusting the image size of the fifth cell image based on the target cell diameter to obtain an adjusted fifth cell image; the target cell diameter characterizes the cell diameter of a plurality of different morphologies of cells.

Thus, the target cell diameter is used to characterize the cell diameter of cells of a variety of different morphologies. The target cell diameter is learned in the training process of the target neural network model; specifically, when the cell image obtained by shooting the cells in various different forms is adopted for model training, the target cell diameter can be continuously updated according to the cell diameters of the cells in various different forms in the cell image, so that the target cell diameter continuously tends to be the average value of the cell diameters of the cells in various different forms, and the target cell diameter can reflect the cell diameters of the cells in various different forms. The target cell diameter is a known value when identifying the fifth cell image to be annotated. The cells in biology not only exhibit diversity in category, but also exhibit different morphological characteristics including size, shape, color, internal structure, function, etc. of the cells, and the image size of the fifth cell image is adjusted using the target cell diameter in order to fuse a large amount of information about the cell morphology.

In some embodiments, the implementation of step 4011a may be: determining an adaptive correspondence between cell diameter and cell image size; the self-adaptive corresponding relation can be learned in the training process of the model; determining a first image size based on the adaptive cell diameter and the adaptive correspondence; and adjusting the image size of the fifth cell image to the first image size to obtain an adjusted fifth cell image.

Wherein the target correspondence is a correspondence between cell diameter and cell image size. The target correspondence may characterize the relationship between cell morphology and cell characteristics such as the number of cells in the cell image; since the cell morphology has a plurality of forms such as dendrite, bar, circle, ellipse, etc., and the number of cells and the characteristics of cells included in the cell image in different forms are different, the cell morphology and the cell characteristics can be represented by the target correspondence. The target corresponding relation can be obtained through learning in the training process of the first label prediction model, or can be obtained through learning in the training process of the pre-training model; specifically, since the cells in the same form can obtain the cell images with different sizes under the shooting of different cameras, the cells in different forms can also obtain the cell images with different sizes under the same camera, when the cell images after the cells in different forms are shot by adopting a plurality of cameras are used for model training, a corresponding relation can be established between the image size of the cell image and the cell diameter of the cells in the cell image, and the target corresponding relation in the model is continuously updated according to the corresponding relation, so that the target corresponding relation can represent the corresponding relation between the cell diameters of the cells in a plurality of forms and the cell sizes, and the target corresponding relation can be fused with the cell characteristics. When the fifth cell image to be marked is identified, the target corresponding relation corresponds to a known value.

For example, if the target cell diameter is 48, the cell image size corresponding to the cell diameter 48 isThen the first image size may be +.>。

Step 4011b, segmenting the adjusted fifth cell image to obtain a plurality of sub-images.

In some embodiments, the adjusted fifth cell image may be segmented based on a preset image size to obtain a plurality of sub-images.

Thus, the preset image size is a preset image size; for example, the preset image size may be set toThe setting may be specifically performed according to actual service requirements, which is not limited in the embodiment of the present application. The plurality of sub-images are obtained by dividing the adjusted fifth cell image based on a preset image size.

It can be understood that the number of cells contained in the cell image is large, and the cell information is relatively fine, if the fifth cell image to be marked is identified by directly adopting the first label prediction model, all the cell information in the fifth cell image cannot be considered during identification, and the critical cell information is ignored, so that the accuracy of identification is reduced; after the fifth cell image is divided into the plurality of sub-images, the cell information in the fifth cell image can be dispersed in the plurality of sub-images, and the cell information included in each sub-image is less than the cell information included in the fifth cell image, so that the first label prediction model fully learns the cell characteristics in the fifth cell image.

In other embodiments, in the case that the ratio between the image size of the adjusted fifth cell image and the preset image size is an integer, dividing the adjusted fifth cell image based on the preset image size to obtain a plurality of sub-images; determining a second image size based on the preset image size and the adjusted image size of the fifth cell image, in the case that the ratio between the adjusted image size of the fifth cell image and the preset image size is not an integer; the ratio between the second image size and the preset image size is an integer; filling the adjusted second cell image based on the target pixel value to obtain a sixth cell image with the image size being the second image size; and dividing the sixth cell image based on the preset image size to obtain a plurality of sub-images.

Thus, the second image size is determined based on the preset image size and the adjusted image size of the fifth cell image, and the ratio between the second image size and the preset image size is an integer. The sixth cell image is a cell image obtained by performing a filling process on the adjusted fifth cell image. The target pixel value is used to fill the cell image, and the target pixel value is a pixel value that does not affect the cell information contained in the cell image, and may be 0, for example.

It can be understood that the ratio between the image size of the adjusted fifth cell image and the preset image size is an integer, which means that the adjusted fifth cell image can be completely divided into a plurality of sub-images with the image size being the preset image size, and no missing pixels exist in each sub-image, which are all complete image areas, i.e. if the preset image size isThen the image size of each sub-image is also +.>。

It can be understood that the ratio between the image size of the adjusted fifth cell image and the preset image size is not an integer, which means that the adjusted fifth cell image cannot be divided into a plurality of sub-images with the image size being the preset image size; in this case, the second image size may be determined based on the preset image size and the adjusted image size of the fifth cell image, and the adjusted fifth cell image may be filled in with the cell image (sixth cell image) having the image size of the second image size, and at this time, since the ratio of the second image size to the preset image size is an integer, the sixth cell image may be completely divided into a plurality of sub-images having the image size of the preset image size.

For example, if the preset image size is The adjusted fifth cell image has an image size ofThen the second image size may be +.>At this time, the image size of the fifth cell image can be adjusted from +.>Fill to->A sixth cell image was obtained. It can be seen that the filling is not arbitrary, and that the complexity of the filling is kept to a minimum, i.e. the image size after filling is the image size closest to the fifth cell image, but a multiple of the preset image size.

Step 4011c, identifying each sub-image by using the first label prediction model to obtain a prediction probability sub-image corresponding to each sub-image respectively; the predicted probability subgraph comprises the probability that each pixel in the corresponding subgraph belongs to cells.

Thus, the predicted probability subgraph for any one sub-image includes the probability that each pixel in that sub-image belongs to a cell.

As the cell information in the fifth cell image is dispersed in the plurality of sub-images, when the first label prediction model is adopted to identify each sub-image, the cell information in each sub-image can be identified more finely, the identification accuracy is improved, the obtained prediction probability subgraph of each sub-image is more accurate, and the prediction probability graph of the first cell image obtained based on the prediction probability subgraphs of the plurality of sub-images is more accurate, so that the cell region (label) in the cell image can be accurately determined.

Step 4011d, performing an integration process on the predicted probability subgraphs of the plurality of subgraphs to obtain a predicted probability map of the fifth cell image; the predicted probability map includes a probability that each pixel in the fifth cell image belongs to a cell.

In some embodiments, the implementation of step 4011d may be: integrating the predictive probability subgraphs of the plurality of subgraphs according to the positions of the plurality of subgraphs in the fifth cell image to obtain a first predictive probability map; under the condition that filling processing is carried out on the fifth cell image based on the target pixel value, removing the prediction probability corresponding to the filled target pixel value from the first prediction probability map to obtain a second prediction probability map; and adjusting the image size of the second predictive probability map to the initial image size of the fifth cell image to obtain the predictive probability map of the fifth cell image.

The first pre-stored probability map is obtained by integrating the predictive probability subgraphs of the plurality of subgraphs according to the positions of the plurality of subgraphs in the fifth cell image; and integrating the predicted probability subgraphs of the plurality of sub-images according to the positions of the plurality of sub-images in the fifth cell image, namely integrating the predicted probability subgraphs of the plurality of sub-images according to the positions of the fifth cell image before the plurality of sub-images are segmented. The second predictive probability map is obtained by removing predictive probabilities corresponding to the filled target pixel values from the first predictive probability map.

Since the fifth cell image is subjected to the image size adjustment, the filling process, and the segmentation process before, after the prediction probability subgraph of each sub-image is obtained, the integration process, the filling removal, and the image size adjustment are required to obtain the prediction probability map of the fifth cell image.

In some embodiments, step 4011C described above may comprise steps a through C as follows:

and step A, extracting the characteristics of each sub-image by adopting an encoder in the first label prediction model to obtain a first characteristic diagram of each sub-image.

As such, the first tag prediction model may include an encoder, a decoder, and a global average pooling module. The encoder is used for carrying out feature extraction and downsampling on the cell image, the decoder is used for carrying out jump linking and upsampling, and the global average pooling module is used for obtaining a feature map representing cell categories. The first feature map of each sub-image is a feature map obtained by extracting features of the first sub-image by an encoder adopting a first label prediction model.

Step B, carrying out pooling treatment on the first feature images by adopting a global average pooling module in the first label prediction model to obtain second feature images of each sub-image; wherein the second feature map characterizes the class of cells in each of the sub-images.

Thus, the second feature map is a feature map obtained by processing the first feature map by the global average pooling module of the first label prediction model, and is used for representing the category of the cells in each sub-image.

Step C, adopting a decoder in the first label prediction model to carry out channel recovery processing on the first feature map and the second feature map to obtain a prediction probability subgraph of each sub-image; the encoder and the decoder each include a plurality of feature extraction modules formed by a convolutional network and a residual network.

Fig. 5 is a schematic network structure diagram of a first label prediction model according to an embodiment of the present application. As shown in fig. 5, the network structure of the first tag prediction model includes an encoder, a decoder, and a global averaging pooling module (not shown), and the encoder may include 4 first feature extraction modules, each of which includes 2 feature extraction layers composed of a convolutional network and a residual network; similarly, the decoder may include 4 second feature extraction modules, each of which includes 2 feature extraction layers composed of a convolutional network and a residual network. The number on each feature extraction layer (e.g., 32 labeled on the first feature extraction layer) is the number of channels; it can be seen that the encoder and decoder share a feature extraction module with 256 channels. All the convolution networks are adopted Is a convolution kernel of (a). Arrows between each first feature extraction module in the encoder represent downsampling for reducing the dimension; the encoder mainly performs convolution operation and downsampling operation. Arrows between each second feature extraction module in the decoder represent upsampling for dimension improvement; the decoder mainly performs up-sampling operation and skip-linking operation. After the encoder acquires the feature map of the cell image, the decoder takes the feature map gray scale as the original dimension (original resolution). In addition, a global averaging pooling module is added after the feature extraction module with 256 channels to obtain a tyle representation of the cell image, wherein the tyle representation characterizes cell types, and the tyle representation is fed in an up-sampling stage; feeding the tyle representation means that before convolution, the tyle representation is added to the input of the current feature extraction module in the decoder after a linear transformation, and the added result enters the convolution.

When the method is realized, after each sub-image is input into a first label prediction model, an encoder is adopted to conduct feature extraction on each sub-image to obtain a first feature image of each sub-image, then a global average pooling module is adopted to conduct pooling processing on the first feature image to obtain a second feature image of each sub-image, and then an encoder is adopted to conduct channel gray scale processing on the first feature image and the second feature image to obtain a prediction probability sub-image of each sub-image.

In the above embodiment, the residual network is added to the feature extraction module, so that the first label prediction model focuses more on the cell information in the cell image, and the feature extraction capability is improved.

In some embodiments, the step C may include the following steps C1 to C4:

step C1, determining a first sub-feature map output by a first feature extraction module with the same channel number as that of an ith second feature extraction module in the decoder from a plurality of first feature extraction modules in the encoder; i is a positive integer greater than or equal to 2.

For example, referring to fig. 5, the plurality of first feature extraction modules in the encoder may include: a first feature extraction module with 32 channels, a first feature extraction module with 64 channels, a first feature extraction module with 128 channels, and a first feature extraction module with 256 channels. The plurality of second feature extraction modules in the decoder may include: a second feature extraction module with 256 channels, a second feature extraction module with 128 channels, a second feature extraction module with 64 channels, and a second feature extraction module with 32 channels. The number of channels of a first feature extraction module in the encoder has a one-to-one correspondence with the number of channels of a second feature extraction module in the decoder.

Step C2, determining a second sub-feature map output by an i-1 th second feature extraction module in the decoder; the feature map output by the first and second feature extraction modules in the decoder is the first feature map output by the last first feature extraction module in the encoder.

Thus, the first feature map is a feature map output by the first feature extraction module with 256 channels in fig. 5. For the decoder, the first feature extraction module with 256 channels is also the first second feature extraction module in the decoder, and then the first feature map output by the first feature extraction module with 256 channels is also the first sub-feature map output by the first second feature extraction module in the decoder.

And step C3, carrying out channel recovery processing on the second sub-feature map, the first sub-feature map and the second feature map by adopting an ith second feature extraction module in the decoder to obtain a feature map output by the ith second feature extraction module in the decoder.

It can be understood that after the first feature map (the first sub-feature map output by the first second feature extraction module) is obtained, for the second feature extraction module (the second feature extraction module with the number of channels being 128) in the decoder, the channel recovery processing needs to be performed on the first feature map (the second sub-feature map output by the first second feature extraction module in the decoder), the second feature map, and the first sub-feature map output by the first feature extraction module with the number of channels being 128 in the decoder; for the third feature module (the second feature extraction module with the channel number of 64) in the encoder, at this time, the second sub-feature map output by the second feature extraction module (the second feature extraction module with the channel number of 128), the second feature map, and the first sub-feature map output by the first feature extraction module with the channel number of 64 in the decoder need to be subjected to channel recovery processing; for the fourth feature module (the second feature extraction module with the number of channels being 32) in the encoder, the second sub-feature map outputted by the third second feature extraction module (the second feature extraction module with the number of channels being 64), the second feature map, and the first sub-feature map outputted by the first feature extraction module with the number of channels being 32 in the decoder need to be subjected to channel recovery processing.

In some embodiments, the implementation of step C3 may be: summing the second sub-feature map and the second feature map to obtain a third feature map; performing up-sampling processing on the third feature map to obtain a processed third feature map; summing the processed third characteristic diagram and the first sub-characteristic diagram to obtain a fourth characteristic diagram; and carrying out convolution processing on the fourth feature map to obtain a feature map output by an ith second feature extraction module in the decoder.

The third feature map is obtained by summing the second sub-feature map and the second feature map. The up-sampling process is performed on the third feature map in order to perform channel recovery to recover the original resolution of the cell image. The fourth feature map is convolved to extract the cell features.

As can be seen from fig. 5, after the feature map output by each second feature extraction module in the decoder is summed with the second feature map (cell type) to obtain a third feature map, the third feature map is up-sampled, and then enters the next second feature extraction module, before the next second feature extraction module processes, the processed third feature map and the feature map output by the corresponding first feature extraction module need to be summed to obtain a fourth feature map, and at this time, the fourth feature map is the input of the next second feature extraction module.

It should be noted that, the two feature graphs are processed by adopting a summation mode, rather than the two feature graphs are processed by adopting a feature stitching mode, so as to reduce the number of features and improve the rate of model processing. In the process of adopting the decoder to carry out channel recovery processing, a type representation (second characteristic diagram) is added, so that the recognition accuracy of the model can be improved.

And C4, taking the feature map output by the last second feature extraction module in the decoder as a predictive probability subgraph of each sub-image.

The second feature extraction module with the channel number of 32 in fig. 5 is a feature map output by the last second feature extraction module in the decoder, and the feature map output by the second feature extraction module is used as a prediction probability subgraph of each sub-image.

The application of the cell image labeling method provided by the embodiment of the application in an actual scene is described below.

The embodiment of the application provides a semiautomatic labeling method of a multi-mode cell image, which is used for improving the labeling time efficiency of a biological cell image, and mainly labeling a cell region in the cell image. The method comprises the following steps 1) to 7):

1) And acquiring a plurality of cell images obtained by shooting cells of different cell types under a plurality of microscopes, and constructing a labeling cell image (the number can be 2040) and a cell image to be labeled (the number can be 1712) based on the plurality of cell images.

2) Training is carried out on the marked cell image to obtain a pre-training model.

3) And reasoning the cell image to be marked by adopting a pre-training model to obtain the pseudo tag.

4) The method comprises the steps of inputting a labeled cell image and a pseudo-label cell image into a pre-training model for further training, and carrying out time integration accumulation on the pseudo-label in the training process, namely, carrying out reasoning by adopting the obtained pre-training model after each training period, carrying out superposition updating on the pseudo-label, and outputting a final label of the cell image to be labeled after N training periods. Each training period may be set to 100 iterative training and N may be set to 25.

5) After the final model is obtained, the final model is adopted to identify the cell image to be marked, and the final label of the cell image to be marked is output.

6) Performing label verification on a final label of the cell image to be marked; specifically, the accuracy of the final tag may be obtained using the evaluation index F1-score, and in the case where the accuracy of the final tag is greater than the accuracy threshold (the accuracy threshold may be set to 0.8), step 7) is performed; and under the condition that the accuracy rate of the final label is smaller than or equal to the accuracy rate threshold value, retraining is carried out by adjusting model parameters of the pre-trained model, and the final label of the cell image to be marked is obtained again.

7) And (3) manually fine-tuning the final label of the cell image to be marked so as to meet the practical requirement.

In order to improve the label quality of model output, a semi-supervised learning method of a time integrated model is adopted in the method, and prior knowledge in the cell structure field such as a nuclear auxiliary segmentation channel and a self-adaptive diameter is increased. The concrete steps are as follows:

firstly, training a pre-training model on the existing labeling data set, and reasoning the cell images to be labeled by adopting the pre-training model to obtain the pseudo labels of the cell images to be labeled. And then, further training the pre-training model by using the labeled and pseudo-label data set, performing time integration accumulation on the pseudo-labels in the training process, performing superposition updating on each T iteration times, and outputting the final label of the cell image to be labeled after n iteration times. During training, the target loss function may be weighted by the annotated loss function (first loss function) and the pseudo tag loss function (second loss function).

Secondly, domain priori knowledge is introduced into the model in the training process, and the nuclear assisted segmentation channels and the self-adaptive cell diameters (target cell diameters) are increased. The cell image typically comprises a primary channel corresponding to the cytoplasmic marker and an optional secondary channel corresponding to the nucleus, typically stained with 4, 6-diamino-2-phenylindoline (DAPI), which is blue in color, which can provide cell localization information, and additional nuclear channels are used herein to express relative positional information and cellular features in order to take advantage of the cellular structure. In addition, cells in biology exhibit not only diversity in type, but also different morphology and characteristics. These characteristics include cell size, shape, color, internal structure and function. To fuse a large amount of information about cell morphology, the present application assigns an adaptive cell diameter to each cell image and saves it to the model. In the process of reasoning and superposing the pseudo labels of the cell images to be marked, parameters are required to be set to be self-adaptive cell diameters, and the size of the cell images is guided to be adjusted so as to match the pre-stored average diameters of the models.

Finally, except for the marked data set used by the pre-training model, other non-marked cell images can obtain the labels in the model training process, semi-automatic marking is realized, and marking efficiency is greatly improved.

The embodiment of the application further provides a semi-automatic labeling method of the multi-mode cell image, which comprises the following steps of 1 to 6:

in step 1, 2040 annotated cell images, 1712 unlabeled cell images, and 100 annotated test images were collected in this example.

Step 2, 2040 labeled cell images include cell images obtained by photographing multiple cell types under multiple microscopes. The multiple microscopes may include bright field microscope images, phase contrast microscope images, differential interference contrast microscope images, and fluorescence images. Cell classes may include stained cells, morphologically branched cells, fluorescently labeled tissue cells, bacterial cells, and the like. In the training process, the embodiment utilizes the nuclear assisted segmentation channel and the cell self-adaptive diameter to improve the model performance.

Step 3, in the test image reasoning process, the embodiment determines an adaptive cell diameter parameter (diameter). Diameter may be set to 40 pixels (pixels), which may be the optimal value obtained through repeated experiments.

And 4, simultaneously inputting the labeling data set (2040 labeled cell images) and the pseudo labeling data set (712 unlabeled cell images) into a pre-training model for further training, and simultaneously superposing and updating the pseudo label of the cell image to be labeled (unlabeled cell image). Specifically, the label pseudo label of the cell image to be labeled is subjected to superposition update once after each training period T, and the update formula can be expressed as follows:the method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>The pseudo tag after the kth update; />The pseudo tag after the k-1 time updating is the pseudo tag; />The model output after the kth training iteration. And outputting a final model and a final label of the cell image to be marked after N training periods. In the application, the total iteration number is 2500 (N is 25, and the training period N comprises 100 iterative training) with the best effect (obtained by multiple experiments); the step can also use the model with the best output effect to identify the cell image to be marked and output the final label of the cell image to be marked.

In step 5, in the label verification process, the higher the F1-Score threshold of the label is, the better the label quality is, and in this embodiment, the F1-Score threshold can be more than 0.8.

And 6, after obtaining the final label of the cell image to be marked, evaluating and fine-tuning the final label of the cell image to be marked by a field expert, wherein a fine-tuning and fine-tuning tool can select an imageJ plug-Labkit, open a cell original image and import the final label, and perform comparative evaluation and fine tuning.

In this embodiment, the model structure adopts an optimized network structure of the internet containing the residual error, as shown in fig. 5, the standard network does not contain the residual error, but in this embodiment, the residual error is added into the network of the internet, so that the effect of adding the residual error is to add attention, so that the model is more focused on cell information, and the capability of extracting the characteristics is improved; in addition, in jump linking, in order to reduce the number of parameters of the model, the present embodiment performs fusion by direct summation instead of using the conventional feature stitching method.

The application provides an effective biological cell image semiautomatic labeling method which can be applied to labeling of various cell microscope images, effectively improves the labeling time efficiency of biological cell images, and is hopeful to rapidly apply growing cell images to deep learning.

Based on the foregoing embodiments, the embodiments of the present application provide a model training apparatus, where the apparatus includes units included, and modules included in the units may be implemented by a processor in a model training device; of course, the method can also be realized by a specific logic circuit; in practice, the processor may be a central processing unit (Central Processing Unit, CPU), microprocessor (Microprocessor Unit, MPU), digital signal processor (Digital Signal Processor, DSP) or field programmable gate array (Field Programmable Gate Array, FPGA), etc.

Fig. 6 is a schematic structural diagram of a model training device according to an embodiment of the present application, and as shown in fig. 6, a model training device 60 includes: a first recognition module 601 and a training module 602, wherein:

the first recognition module 601 is configured to recognize an unlabeled first cell image by using a pre-training model, so as to obtain a first label of the first cell image;

the training module 602 is configured to train the pre-training model based on the first cell image labeled with the first tag and the second cell image labeled with the second tag, to obtain a trained first tag prediction model;

In some embodiments, training module 602 is further to: training the pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a pre-training model after a first training period; identifying the first untagged cell image by adopting the pre-training model after the first training period to obtain a label to be updated; determining a third label of the first cell image of the second training period based on the first label and the label to be updated; and training the pre-training model after the first training period based on the first cell image marked with the third label and the second cell image marked with the second label until the first label prediction model is obtained.

In some embodiments, training module 602 is further to: determining a first loss function, a weight corresponding to the first loss function, a second loss function and a weight corresponding to the second loss function; determining a target loss function based on the first loss function, the weight corresponding to the first loss function, the second loss function and the weight corresponding to the second loss function; and training the pre-training model based on the first cell image marked with the first label, the second cell image marked with the second label and the target loss function to obtain the pre-training model after the first training period.

In some embodiments, training module 602 is further to: determining a label of a first cell image of an nth training period as a first target label of the first cell image; or, identifying the first unlabeled cell image by adopting the first label prediction model to obtain a first target label of the first cell image.

In some embodiments, training module 602 is further to: and under the condition that the first target label meets the verification condition, adopting image processing software to finely tune the first target label.

In some embodiments, training module 602 is further to: under the condition that the first target label does not meet the verification condition, model parameters in the pre-training model are adjusted to obtain an adjusted pre-training model; training the adjusted pre-training model based on the first cell image marked with the first label and the second cell image marked with the second label to obtain a trained second label prediction model and a second target label of the first cell image.

In some embodiments, training module 602 is further to: acquiring a plurality of third cell images obtained after shooting cells belonging to different cell categories under a plurality of microscopes; based on the plurality of third cell images, the unlabeled first cell image and the second cell image labeled with the second label are constructed.

In some embodiments, training module 602 is further to: preprocessing the plurality of third cell images to obtain a plurality of fourth cell images comprising a first color channel and a second color channel; the first color channel is used for representing the color of the stained cytoplasm, and the second color channel is used for representing the color of the stained nucleus; based on the plurality of fourth cell images, the unlabeled first cell image and the second cell image labeled with the second label are constructed.

In some embodiments, training module 602 is further to: performing matrix recombination on each third cell image, and removing a red channel of each third cell image to obtain a plurality of fourth cell images including a green channel and a blue channel; the first color channel is a green channel and the second color channel is a blue channel.

Based on the foregoing embodiments, the embodiments of the present application provide a cell image labeling apparatus, where the apparatus includes units included, and modules included in the units may be implemented by a processor in a cell image labeling device; of course, the method can also be realized by a specific logic circuit; in practice, the processor may be a central processing unit, a microprocessor, a digital signal processor, a field programmable gate array, or the like.

Fig. 7 is a schematic structural diagram of a cellular image labeling device according to an embodiment of the present application, and as shown in fig. 7, the cellular image labeling device 70 includes: a second recognition module 701 and an annotation module 702, wherein:

the second identifying module 701 is configured to identify a fifth cell image to be labeled by using the first label prediction model, so as to obtain a fourth label of the fifth cell image;

The labeling module 702 is configured to label the fifth cell image based on the fourth label, so as to obtain a fifth cell image labeled with the fourth label;

the training process of the first label prediction model is as follows:

In some embodiments, the second identification module 701 is further configured to: identifying cells in the fifth cell image by adopting the first label prediction model to obtain prediction probability corresponding to each pixel in the fifth cell image; the prediction probability is used for representing the probability that the corresponding pixel belongs to the cell; determining at least one cell region in the fifth cell image based on the prediction probability corresponding to each pixel in the fifth cell image; the at least one cell region is taken as the fourth tag.

In some embodiments, the second identification module 701 is further configured to: adjusting the image size of the fifth cell image based on the target cell diameter to obtain an adjusted fifth cell image; the target cell diameter characterizes the cell diameter of a plurality of different morphologies of cells; dividing the adjusted fifth cell image to obtain a plurality of sub-images; identifying each sub-image by adopting the first label prediction model to obtain a prediction probability sub-image corresponding to each sub-image respectively; the prediction probability subgraph comprises the probability that each pixel in the corresponding subgraph belongs to cells; integrating the predictive probability subgraphs of the plurality of subgraphs to obtain a predictive probability map of the fifth cell image; the predicted probability map includes a probability that each pixel in the fifth cell image belongs to a cell.

In some embodiments, the second identification module 701 is further configured to: extracting the characteristics of each sub-image by adopting an encoder in the first label prediction model to obtain a first characteristic diagram of each sub-image; carrying out pooling treatment on the first feature images by adopting a global average pooling module in the first label prediction model to obtain second feature images of each sub-image; wherein the second feature map characterizes the class of cells in each of the sub-images; adopting a decoder in the first label prediction model to carry out channel recovery processing on the first feature map and the second feature map to obtain a prediction probability subgraph of each sub-image; the encoder and the decoder each include a plurality of feature extraction modules formed by a convolutional network and a residual network.

In some embodiments, the second identification module 701 is further configured to: determining a first sub-feature map output by a first feature extraction module with the same channel number as that of an ith second feature extraction module in the decoder from a plurality of first feature extraction modules in the encoder; i is a positive integer greater than or equal to 2; determining a second sub-feature map output by an i-1 th second feature extraction module in the decoder; the feature map output by the first and second feature extraction modules in the decoder is the first feature map output by the last first feature extraction module in the encoder; adopting an ith second feature extraction module in the decoder to perform channel recovery processing on the second sub-feature map, the first sub-feature map and the second feature map to obtain a feature map output by the ith second feature extraction module in the decoder; and taking the feature map output by the last second feature extraction module in the decoder as a predictive probability subgraph of each sub-image.

In some embodiments, the second identification module 701 is further configured to: summing the second sub-feature map and the second feature map to obtain a third feature map; performing up-sampling processing on the third feature map to obtain a processed third feature map; summing the processed third characteristic diagram and the first sub-characteristic diagram to obtain a fourth characteristic diagram; and carrying out convolution processing on the fourth feature map to obtain a feature map output by an ith second feature extraction module in the decoder.

The description of the apparatus embodiments above is similar to that of the method embodiments above, with similar advantageous effects as the method embodiments. In some embodiments, functions or modules included in the apparatus provided in the embodiments of the present application may be used to perform the methods described in the embodiments of the methods, and for technical details that are not disclosed in the embodiments of the apparatus of the present application, please refer to the description of the embodiments of the methods of the present application for understanding.

It should be noted that, in the embodiment of the present application, if the method is implemented in the form of a software functional module, and sold or used as a separate product, the method may also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or portions contributing to the related art, and the software product may be stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes. Thus, embodiments of the present application are not limited to any specific hardware, software, or firmware, or to any combination of hardware, software, and firmware.

An embodiment of the present application provides a model training device, fig. 8 is a schematic hardware entity diagram of the model training device provided in the embodiment of the present application, and as shown in fig. 8, the model training device 80 includes a first memory 801 and a first processor 802, where the first memory 801 stores a computer program that can be run on the first processor 802, and the first processor 802 implements the model training method described in the embodiment of the present application when executing the program.

It should be noted that, the first memory 801 is configured to store instructions and applications executable by the first processor 802, and may also be cached in the first processor 802 and the data (for example, image data, audio data, voice communication data, and video communication data) to be processed or already processed by each module in the model training device 80, and may be implemented by a FLASH memory (FLASH) or a random access memory (Random Access Memory, RAM).

In the embodiment of the present application, the model training device 80 may be various types of devices with information processing capability in implementing the model training method described in the embodiment of the present application, for example, the model training device 80 may include a tablet computer, a desktop computer, a notebook computer, a host computer, and the like.

An embodiment of the present application provides a cellular image labeling device, fig. 9 is a schematic hardware entity diagram of the cellular image labeling device provided in the embodiment of the present application, as shown in fig. 9, the cellular image labeling device 90 includes a second memory 901 and a second processor 902, where the second memory 901 stores a computer program that can be run on the second processor 902, and the cellular image labeling method described in the embodiment of the present application is implemented when the second processor 902 executes the program.

It should be noted that, the second memory 901 is configured to store instructions and applications executable by the second processor 902, and may also be cached in the second processor 902 and the cell image labeling device 90, where data (for example, image data, audio data, voice communication data, and video communication data) to be processed or already processed by each module may be implemented by a flash memory or a random access memory.

In the embodiment of the present application, the cell image labeling device 90 may be various types of devices with information processing capability in implementing the cell image labeling method described in the embodiment of the present application, for example, the cell image labeling device 90 may include a tablet computer, a desktop computer, a notebook computer, a host computer, and the like.

The present embodiment provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of the method provided in the above embodiment.

The present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the steps of the method provided by the method embodiments described above.

It should be noted here that: the above description of the storage medium, chip embodiments and device embodiments is similar to that of the method embodiments described above, with similar advantageous effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" or "some embodiments" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in some embodiments" in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application. The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments. The foregoing description of various embodiments is intended to highlight differences between the various embodiments, which may be the same or similar to each other by reference, and is not repeated herein for the sake of brevity.

The term "and/or" is herein merely an association relation describing associated objects, meaning that there may be three relations, e.g. object a and/or object B, may represent: there are three cases where object a alone exists, object a and object B together, and object B alone exists.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments are merely illustrative, and the division of the modules is merely a logical function division, and other divisions may be implemented in practice, such as: multiple modules or components may be combined, or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or modules, whether electrically, mechanically, or otherwise.

The modules described above as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules; can be located in one place or distributed to a plurality of network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated in one processing unit, or each module may be separately used as one unit, or two or more modules may be integrated in one unit; the integrated modules may be implemented in hardware or in hardware plus software functional units.

Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium, where the program, when executed, performs steps including the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a Read Only Memory (ROM), a magnetic disk or an optical disk, or the like, which can store program codes.

Alternatively, the integrated units described above may be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or part contributing to the related art, and the computer software product may be stored in a storage medium, including several instructions for causing an electronic device to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

The methods disclosed in the several method embodiments provided in the present application may be arbitrarily combined without collision to obtain a new method embodiment.

The features disclosed in the several product embodiments provided in the present application may be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or apparatus embodiments provided in the present application may be arbitrarily combined without conflict to obtain new method embodiments or apparatus embodiments.

The foregoing is merely an embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered in the protection scope of the present application.

Claims

1. A method for labeling a cell image, the method comprising:

the training process of the first label prediction model is as follows:

the training process comprises N training periods, wherein the label of the first cell image of the nth training period is obtained by updating the label based on the first cell image of the nth-1 training period, the unlabeled first cell image and a pre-training model after the nth-1 training period; n is a positive integer; n is a positive integer greater than 1 and less than or equal to N;

The identifying the cells in the fifth cell image by using the first label prediction model to obtain a fourth label of the fifth cell image comprises:

adjusting the image size of the fifth cell image based on the target cell diameter and the target corresponding relation to obtain an adjusted fifth cell image; the target cell diameter characterizes the cell diameter of a plurality of different morphologies of cells; the target correspondence characterizes the relationship between cell morphology and cell characteristics in the cell image;

identifying cells in the adjusted fifth cell image by adopting the first label prediction model to obtain prediction probability corresponding to each pixel in the fifth cell image; the prediction probability is used for representing the probability that the corresponding pixel belongs to the cell;

constructing a plurality of power systems with fixed points based on the prediction probability corresponding to each pixel in the fifth cell image;

determining an image area formed by pixels converged at the same fixed point in the fifth cell image as a cell area, and obtaining at least one cell area in the fifth cell image;

a fourth tag for the fifth cell image that is the at least one cell region;

The network architecture of the first tag prediction model comprises an encoder, a decoder and a global average pooling module; the global average pooling module is used for obtaining a characteristic diagram representing cell types, and the encoder and the decoder both comprise a plurality of characteristic extraction modules consisting of a convolution network and a residual network.

2. The method of claim 1, wherein identifying cells in the adjusted fifth cell image using the first label prediction model to obtain a prediction probability for each pixel in the fifth cell image, comprises:

dividing the adjusted fifth cell image to obtain a plurality of sub-images;

3. The method according to claim 2, wherein said identifying each of said sub-images using said first label prediction model to obtain a prediction probability sub-image for each of said sub-images, respectively, comprises:

and carrying out channel recovery processing on the first feature map and the second feature map by adopting a decoder in the first label prediction model to obtain a prediction probability subgraph of each sub-image.

4. A method according to claim 3, wherein said performing a channel recovery process on said first feature map and said second feature map using a decoder in said first label prediction model to obtain a predicted probability subgraph for each sub-image comprises:

5. The method of claim 4, wherein the performing channel recovery processing on the second sub-feature map, the first sub-feature map, and the second feature map by using an i-th second feature extraction module in the decoder to obtain a feature map output by the i-th second feature extraction module in the decoder comprises:

6. The method of claim 1, wherein training the pre-training model based on the first cell image labeled with the first label and the second cell image labeled with the second label results in a trained first label prediction model, comprising:

7. The method of claim 6, wherein training the pre-training model based on the first cell image labeled with the first label and the second cell image labeled with the second label results in a pre-training model after a first training period, comprising:

8. The method according to any one of claims 1 or 7, further comprising:

determining a label of a first cell image of an nth training period as a first target label of the first cell image; or,

And identifying the first unlabeled cell image by adopting the first label prediction model to obtain a first target label of the first cell image.

9. The method of claim 8, wherein the method further comprises:

10. The method of claim 8, wherein the method further comprises:

11. The method according to claim 1, wherein the method further comprises:

12. The method of claim 11, wherein constructing the unlabeled first cell image and the second cell image labeled with the second label based on the plurality of third cell images comprises:

13. The method of claim 12, wherein the preprocessing the plurality of third cell images to obtain a plurality of fourth cell images including a first color channel and a second color channel comprises:

14. A cell image labeling apparatus, the apparatus comprising:

the training process of the first label prediction model is as follows:

The second identification module is specifically configured to: adjusting the image size of the fifth cell image based on the target cell diameter and the target corresponding relation to obtain an adjusted fifth cell image; the target cell diameter characterizes the cell diameter of a plurality of different morphologies of cells; the target correspondence characterizes the relationship between cell morphology and cell characteristics in the cell image; identifying cells in the adjusted fifth cell image by adopting the first label prediction model to obtain prediction probability corresponding to each pixel in the fifth cell image; the prediction probability is used for representing the probability that the corresponding pixel belongs to the cell; constructing a plurality of power systems with fixed points based on the prediction probability corresponding to each pixel in the fifth cell image; determining an image area formed by pixels converged at the same fixed point in the fifth cell image as a cell area, and obtaining at least one cell area in the fifth cell image; a fourth tag for the fifth cell image that is the at least one cell region;

15. A cellular image labeling apparatus comprising a second memory and a second processor, said second memory storing a computer program executable on the second processor, wherein the second processor implements the method of any of claims 1 to 13 when said program is executed.

16. A computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method according to any one of claims 1 to 13.