CN116883325A

CN116883325A - Immunofluorescence image analysis method and device

Info

Publication number: CN116883325A
Application number: CN202310739039.4A
Authority: CN
Inventors: 王晓梅; 蔡博君
Original assignee: Hangzhou Yice Technology Co ltd
Current assignee: Hangzhou Yice Technology Co ltd
Priority date: 2023-06-21
Filing date: 2023-06-21
Publication date: 2023-10-13
Anticipated expiration: 2043-06-21
Also published as: CN116883325B

Abstract

The invention discloses an immunofluorescence image analysis method and device, wherein the method comprises the following steps: acquiring a plurality of immunofluorescence images of a target patient in different areas, and inputting each immunofluorescence image into a convolution recurrent neural network to obtain an image analysis result; the convolutional recurrent neural network comprises a pre-trained encoder and at least one pre-trained decoder, wherein the encoder is obtained by training an ANA image sample and a corresponding classification label, and the decoder is obtained by training fluorescent image samples of different areas of the same patient. In this way, the scheme utilizes the weak supervision CRNN model based on the deep learning attention mechanism, the CNN extracts feature images of each sampling area of the same patient, abstracts Cheng Gaowei feature information of the image, and generalizes and fuses all features by combining the RNN with the attention mechanism, so that the accuracy of image analysis can be improved, and further, an ANA conclusion of a case level can be provided.

Description

Immunofluorescence image analysis method and device

Technical Field

The invention relates to the technical field of artificial intelligence, in particular to an immunofluorescence image analysis method and device.

Background

Antinuclear antibodies (ANA), which broadly refer to antibodies against various nuclear components, are a widely occurring class of autoantibodies. Because of the complexity of the nuclear components, the antigenicity of the different components also varies, and thus there are a number of different ANAs. The method commonly used at present for detecting serum total ANA by using an immunofluorescence method is a fluorescence immunohistochemical method, a multi-purpose mouse liver slice or a printing sheet is used as a cell nucleus matrix, the result is relatively stable and reliable, and the cell nucleus found under a fluorescence microscope has fluorescence staining as a positive reaction.

In the prior art, an immunofluorescence image is processed by using a deep learning image detection algorithm, a plurality of cavity convolution layers are used for respectively carrying out feature extraction on the immunofluorescence image to be detected to obtain a plurality of cavity convolution feature images, then feature fusion is carried out to obtain a multi-scale feature image, a residual error network is input to obtain a deep feature image, the deep feature image is subjected to weight distribution by using the convolution layers, finally a feature area with a weighted pixel value higher than a specified threshold value is determined as a target frame, the area in the target frame is an area of interest, wherein the cavity coefficients of the plurality of cavity convolution layers are different, and the features extracted by the cavity convolution layers with different cavity coefficients can be simultaneously extracted to carry out multi-scale fusion, so that more complete semantic information is obtained, and the detected area of interest has more pertinence.

However, for a single immunofluorescent sample, the entire slide sample needs to be manually observed under a microscope, and then an ANA conclusion is made. The digital microscan is used to obtain the sampling images of multiple regions at 40 times magnification of the sample, but the sampling images of multiple regions are not necessarily all able to obtain unified ANA positive conclusion, so the accuracy of image analysis by using the cavity convolution provided in the prior art is poor, and the ANA conclusion of case level cannot be provided.

Disclosure of Invention

Therefore, the embodiment of the invention provides an immunofluorescence image analysis method and device, so that the accuracy of image analysis can be improved, and further an ANA conclusion of a case level can be provided.

In order to achieve the above object, the embodiment of the present invention provides the following technical solutions:

the invention provides an immunofluorescence image analysis method, which comprises the following steps:

acquiring a plurality of immunofluorescence images of a target patient in different areas;

inputting each immunofluorescence image into a convolution recurrent neural network to obtain an image analysis result;

the convolutional recurrent neural network comprises a pre-trained encoder and at least one pre-trained decoder, wherein the encoder is obtained by training an ANA image sample and a corresponding classification label, and the decoder is obtained by training fluorescent image samples of different areas of the same patient.

In some embodiments, inputting the immunofluorescence image into a convolutional recurrent neural network to obtain an image analysis result, specifically including:

under the condition of processing single task requirements, inputting each immunofluorescence image into an encoder to obtain an encoding result;

and synchronously inputting the coding result into a plurality of parallel decoders, and respectively obtaining case type results through the decoders.

under the condition of processing the multi-task requirement, inputting each immunofluorescence image into an encoder to obtain an encoding result;

inputting the encoding result into a decoder, and obtaining case category results through the decoder, wherein the number of the decoders is one.

In some embodiments, training the encoder using the ANA image samples and corresponding classification labels specifically includes:

taking a plurality of ANA pathological images of the same patient as an ANA image sample, and taking an ANA karyotype result of the patient as a classification label;

inputting the ANA image sample into an encoder for training to obtain an image category result;

the classification labels and the image classification results are input to an optimizer to update parameters of the encoder with a counter-propagating gradient.

In some embodiments, the decoder is obtained by training fluorescence image samples of different regions of the same patient, and specifically includes:

taking a plurality of ANA pathological images of different areas of the same patient as fluorescent image samples;

freezing encoder parameters so that the encoder parameters are not updated during the decoder training process, and only providing feature extraction functions for the encoder;

extracting features from the encoder, taking the extracted features as input of a decoder, outputting the ANA core type of the patient by the decoder, and calculating a loss function by an optimizer according to the ANA core type output by the decoder and the real ANA core type of the patient;

and updating parameters of the decoder according to the loss function to complete decoder training.

In some embodiments, in the convolutional recurrent neural network, the CNN acts as an encoder and the RNN acts as a decoder.

In some embodiments, the convolutional recurrent neural network uses a sigmoid activation function in the case of handling multi-tag classification problems;

in the case of handling the single tag classification problem, the convolutional recurrent neural network uses a softmax activation function.

The invention also provides an immunofluorescence image analysis device, which comprises:

an image acquisition unit for acquiring a plurality of immunofluorescence images of a target patient under different regions;

the image analysis unit is used for inputting each immunofluorescence image into the convolution recurrent neural network to obtain an image analysis result;

The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method as described above when executing the program.

The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method as described above.

According to the immunofluorescence image analysis method and device provided by the invention, a plurality of immunofluorescence images of a target patient in different areas are obtained, and each immunofluorescence image is input into a convolution recurrent neural network so as to obtain an image analysis result; the convolutional recurrent neural network comprises a pre-trained encoder and at least one pre-trained decoder, wherein the encoder is obtained by training an ANA image sample and a corresponding classification label, and the decoder is obtained by training fluorescent image samples of different areas of the same patient. In this way, the scheme utilizes a weakly supervised CRNN model based on a deep learning attention mechanism, an algorithm comprises a convolutional neural network CNN and recurrent neural networks RNN, the CNN extracts feature images of each sampling area of the same patient, the image is abstracted Cheng Gaowei feature information, all features are integrated by the RNN in a summary way in combination with the attention mechanism, the accuracy of image analysis can be improved, and then an ANA conclusion of a case level can be provided.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It will be apparent to those of ordinary skill in the art that the drawings in the following description are exemplary only and that other implementations can be obtained from the extensions of the drawings provided without inventive effort.

The structures, proportions, sizes, etc. shown in the present specification are shown only for the purposes of illustration and description, and are not intended to limit the scope of the invention, which is defined by the claims, so that any structural modifications, changes in proportions, or adjustments of sizes, which do not affect the efficacy or the achievement of the present invention, should fall within the ambit of the technical disclosure.

FIG. 1 is a flow chart of an immunofluorescence image analysis method according to the present invention;

FIG. 2 is a second flowchart of an immunofluorescence image analysis method according to the present invention;

FIG. 3 is a third flowchart of an immunofluorescence image analysis method according to the present invention;

FIG. 4 is a training flow chart of an encoder provided by the present invention;

FIG. 5 is a training flow chart of a decoder according to the present invention;

FIG. 6 is a block diagram showing the construction of an immunofluorescence image analyzer according to the present invention;

fig. 7 is a block diagram of a computer device according to the present invention.

Detailed Description

Other advantages and advantages of the present invention will become apparent to those skilled in the art from the following detailed description, which, by way of illustration, is to be read in connection with certain specific embodiments, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Referring to fig. 1, fig. 1 is a flowchart of an immunofluorescence image analysis method according to the present invention.

In one embodiment, the present invention provides an immunofluorescence image analysis method comprising the steps of:

s110: acquiring a plurality of immunofluorescence images of a target patient in different areas;

s120: inputting each immunofluorescence image into a convolution recurrent neural network to obtain an image analysis result;

For example, the prepared immunofluorescence slide is scanned by a dark field scanner to obtain digital pathology images, the same patient can have a plurality of pathology images, each image is a certain area of the immunofluorescence slide, a plurality of ANA pathology images of the same patient are used as input of the algorithm, and the output is the ANA karyotype classification result of the patient.

In an actual usage scenario, the task requirement for performing the picture analysis may be a single task requirement or a multi-task requirement.

As shown in fig. 2, a plurality of fluorescence images of different regions of the same patient sample are input to the encoder, the encoding results (for example, extracted features) obtained by the encoder are synchronously input to a plurality of decoders, that is, simultaneously input to the decoders 1,2, … …, n, so as to obtain a plurality of case category results, that is, the decoder 1 obtains the case category 1, the decoder 2 obtains the case category 2, and so on.

As shown in fig. 3, a plurality of fluorescence images of different regions of the same patient sample are input to an encoder, and the encoding result (for example, the extracted feature) obtained by the encoder is input to a decoder, so as to obtain the case category corresponding to the decoder.

When model training is carried out, firstly, the encoder is trained, at the moment, the decoder does not participate in training, a plurality of ANA pathological images of the same patient are input, ANA core type results of the patient are used as classification labels, and the optimizer updates parameters of the encoder aiming at counter-propagating gradients in the training process. And secondly, training the decoder, wherein the input is a plurality of ANA pathological images of the same patient, then the parameters of the decoder are frozen, the updating is not performed at this stage, the feature extraction function is provided for the decoder, the extracted features (features) are used as the input of the decoder, the decoder outputs the ANA kernel type of the patient, and the optimizer calculates loss according to the output of the decoder and the real ANA kernel type of the patient so as to update the parameters of the decoder, thereby achieving the training purpose.

Specifically, as shown in fig. 4, training the encoder by using the ANA image sample and the corresponding classification label includes the following steps:

As shown in fig. 5, the decoder is obtained by training using fluorescence image samples of different regions of the same patient, and specifically comprises the following steps:

In some embodiments, in the convolutional recurrent neural network, the CNN acts as an encoder and the RNN acts as a decoder. The weak supervision CRNN (Convolutional Recurrent Neural Network) adopted by the invention structurally comprises CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) parts, specifically, CNN is used as an encoder, RNN is used as a decoder, the CNN structure such as resnet, resnext, efficientnet, inceptionnet which is common at present can be used, ANA images and categories thereof are used as training labels to pretrain the encoder, the convergence rate of the subsequent decoders can be improved, and the training time is reduced. The structure of the Decoder is an RNN layer or an LSTM, and self-attribute is added in the embodiment, so that key information in an image can be captured better, and the classification accuracy is improved. After the end of the pre-training, the gradient of the CNN is frozen, and the feature output before the CNN full-connection layer is used as the input of the decoder, wherein the size of the feature is unified to 2048 elements.

In some embodiments, the convolutional recurrent neural network uses a sigmoid activation function in the case of handling multi-tag classification problems; in the case of handling the single tag classification problem, the convolutional recurrent neural network uses a softmax activation function.

That is, while training the entire CRNN, the gradient of the CNN is frozen, and the feature is output as input to the decoder, the gradient within the decoder is updateable. Depending on the problem of decoder processing, different activation functions, such as multi-tag classification, may be used, with sigmoid activation functions, and single tag classification using softmax activation functions. If the multi-task project needs to be processed, the decoders can be added according to task requirements, namely one decoder can correspond to a plurality of different decoders. The data labels required in the CRNN training are the ANA categories of the patient, the input data are ANA fluorescent images of the patient, and the training uses a weak supervision training method, so that the data labels do not need to correspond to each image in a patient case sample, even if any characteristics of the labels are not contained in some image areas.

In the above specific embodiment, according to the immunofluorescence image analysis method provided by the present invention, a plurality of immunofluorescence images of a target patient in different regions are obtained, and each immunofluorescence image is input into a convolutional recurrent neural network, so as to obtain an image analysis result; the convolutional recurrent neural network comprises a pre-trained encoder and at least one pre-trained decoder, wherein the encoder is obtained by training an ANA image sample and a corresponding classification label, and the decoder is obtained by training fluorescent image samples of different areas of the same patient. In this way, the scheme utilizes a weakly supervised CRNN model based on a deep learning attention mechanism, an algorithm comprises a convolutional neural network CNN and recurrent neural networks RNN, the CNN extracts feature images of each sampling area of the same patient, the image is abstracted Cheng Gaowei feature information, all features are integrated by the RNN in a summary way in combination with the attention mechanism, the accuracy of image analysis can be improved, and then an ANA conclusion of a case level can be provided.

In addition to the above method, as shown in fig. 6, the present invention also provides an immunofluorescence image analysis apparatus, the apparatus comprising:

an image acquisition unit 610 for acquiring a plurality of immunofluorescence images of a target patient under different regions;

an image analysis unit 620, configured to input each of the immunofluorescence images into a convolutional recurrent neural network to obtain an image analysis result;

In the above specific embodiment, according to the immunofluorescence image analysis device provided by the present invention, a plurality of immunofluorescence images of a target patient in different regions are obtained, and each immunofluorescence image is input into a convolutional recurrent neural network, so as to obtain an image analysis result; the convolutional recurrent neural network comprises a pre-trained encoder and at least one pre-trained decoder, wherein the encoder is obtained by training an ANA image sample and a corresponding classification label, and the decoder is obtained by training fluorescent image samples of different areas of the same patient. In this way, the scheme utilizes a weakly supervised CRNN model based on a deep learning attention mechanism, an algorithm comprises a convolutional neural network CNN and recurrent neural networks RNN, the CNN extracts feature images of each sampling area of the same patient, the image is abstracted Cheng Gaowei feature information, all features are integrated by the RNN in a summary way in combination with the attention mechanism, the accuracy of image analysis can be improved, and then an ANA conclusion of a case level can be provided.

In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and model predictions. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The model predictions of the computer device are used to store static information and dynamic information data. The network interface of the computer device is used for communicating with an external terminal through a network connection. Which computer program, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

It will be appreciated by those skilled in the art that the structure shown in FIG. 7 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

Corresponding to the above embodiments, the present invention further provides a computer storage medium, which contains one or more program instructions. Wherein the one or more program instructions are for being executed with the method as described above.

The present invention also provides a computer program product comprising a computer program storable on a non-transitory computer readable storage medium, the computer program being capable of performing the above method when being executed by a processor.

In the embodiment of the invention, the processor may be an integrated circuit chip with signal processing capability. The processor may be a general purpose processor, a digital signal processor (Digital Signal Processor, DSP for short), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), a field programmable gate array (Field Programmable Gate Array, FPGA for short), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.

The disclosed methods, steps, and logic blocks in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The processor reads the information in the storage medium and, in combination with its hardware, performs the steps of the above method.

The storage medium may be memory, for example, may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory.

The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable ROM (Electrically EPROM, EEPROM), or a flash Memory.

The volatile memory may be a random access memory (Random Access Memory, RAM for short) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (Double Data Rate SDRAM), enhanced SDRAM (ESDRAM), synchronous DRAM (SLDRAM), and direct memory bus RAM (Direct Rambus RAM, DRRAM).

The storage media described in embodiments of the present invention are intended to comprise, without being limited to, these and any other suitable types of memory.

Those skilled in the art will appreciate that in one or more of the examples described above, the functions described in the present invention may be implemented in a combination of hardware and software. When the software is applied, the corresponding functions may be stored in a computer-readable medium or transmitted as one or more instructions or code on the computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

The foregoing detailed description of the invention has been presented for purposes of illustration and description, and it should be understood that the foregoing is by way of illustration and description only, and is not intended to limit the scope of the invention.

Claims

1. A method of immunofluorescence image analysis, the method comprising:

2. The immunofluorescence image analysis method according to claim 1, wherein inputting the immunofluorescence image into a convolutional recurrent neural network to obtain an image analysis result, specifically comprising:

3. The immunofluorescence image analysis method according to claim 1, wherein inputting the immunofluorescence image into a convolutional recurrent neural network to obtain an image analysis result, specifically comprising:

4. The immunofluorescence image analysis method of claim 1, wherein training the encoder with the ANA image sample and the corresponding classification tag specifically comprises:

5. The immunofluorescence image analysis method according to claim 1, wherein the decoder is trained using fluorescence image samples of different regions of the same patient, comprising:

6. The immunofluorescence image analysis method according to any one of claims 1-5, wherein in the convolutional recurrent neural network, CNN acts as an encoder and RNN acts as a decoder.

7. The immunofluorescence image analysis method of claim 1, wherein in the case of handling a multi-label classification problem, the convolutional recurrent neural network uses a sigmoid activation function;

8. An immunofluorescence image analysis apparatus, the apparatus comprising:

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1-7 when the program is executed.

10. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any of claims 1-7.