CN114882284A

CN114882284A - Color fundus image classification system and method and electronic equipment

Info

Publication number: CN114882284A
Application number: CN202210561912.0A
Authority: CN
Inventors: 皮喜田; 孙凯; 刘洪英; 徐尧; 吴沁莹; 贺梦嘉
Original assignee: Chongqing University
Current assignee: Chongqing University
Priority date: 2022-05-23
Filing date: 2022-05-23
Publication date: 2022-08-09

Abstract

The invention provides a color fundus image classification system, a color fundus image classification method and electronic equipment, wherein the system comprises the following steps: the image acquisition equipment comprises a camera and a first communication module; the intelligent terminal comprises a computer or a mobile phone, wherein the computer or the mobile phone comprises an image processing module and a second communication module; the cloud server comprises an image processing module and a third communication module; the system has two working modes, namely: after the image acquisition equipment acquires the fundus image, the acquired fundus image is sent to the intelligent terminal, the acquired fundus image is processed by the image processing module, and the image classification result is displayed on the intelligent terminal; the second method comprises the following steps: after the image acquisition equipment acquires the fundus images, the acquired fundus images are sent to the cloud server, the acquired fundus images are processed by the image processing module, and the image classification result is sent to the intelligent terminal and displayed by the third communication module through the second communication module.

Description

Color fundus image classification system and method and electronic equipment

[ technical field ] A method for producing a semiconductor device

The invention relates to the technical field of color fundus image classification, in particular to a color fundus image classification system and method and electronic equipment.

[ background of the invention ]

According to the statistics of the world health organization, the total number of people with impaired vision in 2019 is not less than 22 hundred million, wherein the vision disorder suffered by more than 10 hundred million patients can be prevented and even cured. Therefore, the regular eye health examination and the prevention of ophthalmic diseases have great significance, but the basic eye health screening is difficult to implement and guarantee due to the imbalance between the huge population and the small medical resources in China. The color fundus images are images formed by shooting the inner wall of an eyeball at different angles by using a fundus camera, and can be used for discovering various eye diseases such as glaucoma, macular degeneration and the like as soon as possible, thereby being convenient for timely and effective treatment. The eye ground images are processed through the computer training model to help doctors to assist in classification, so that the classification efficiency of the doctors can be greatly improved, the medical pressure is relieved, the medical cost of patients is reduced, and the wide ophthalmologic general investigation work is facilitated to be carried out.

The cataract image classification method and the classification device based on the combined classifier disclosed in the patent application publication No. CN104881683 extract characteristics through wavelet transformation, round-assistant method and texture analysis, and then carry out prediction classification through a support vector machine and a BP neural network, so as to obtain higher classification accuracy. The invention can only help doctors to assist in classifying cataract diseases.

In the method for identifying the multi-label eye fundus image based on the GACNN disclosed in the patent application publication No. CN202110075947, the model establishment is completed by establishing and constructing the GACNN model to train the image with the label and paying attention to the relation of each label in network processing, but the method has poor image feature extraction capability, cannot well solve the problem of relevance among the labels and causes low classification result accuracy.

In a fundus image retinal vessel segmentation method based on a classification regression tree and AdaBoost disclosed in patent application publication No. CN104809480, a 36-dimensional feature vector including local features, morphological features and vector field divergence features of pixels is constructed for each pixel point to judge whether the point belongs to a vessel. The method has good effect on the blood vessel trunk part, but can only provide help for artificial classification of ophthalmic related diseases, and cannot further complete disease classification.

The method for identifying glaucoma disclosed in patent application publication No. CN113011450 is used for performing feature extraction and training classification on optic discs and optic cup areas of fundus images based on an artificial neural network for deep learning, and completing disease prediction on glaucoma. However, the module only realizes two classifications of glaucoma, has a small application range and is not suitable for large-scale popularization.

According to the method for constructing the classification model for macular degeneration region segmentation of the fundus image disclosed in patent application publication No. CN107437252, new features are learned by combining supervised learning and image bottom layer features, and a low-dimensional and high-distinguishability feature descriptor can be extracted by combining the classification model established by means of a generalized low-rank approximation method, a manifold regularization item construction target function and the like, so that the accuracy of macular degeneration region segmentation is improved.

A classification method for retinopathy of prematurity plus lesion disclosed in patent application publication No. CN109635862 is characterized in that a U-Net model is used for carrying out blood vessel segmentation on an eyeground image, a classification model capable of carrying out plus lesion classification on a blood vessel map is constructed, classification efficiency of retinopathy of prematurity plus lesion is improved, and the classification method has important significance in timely screening of children blinding causes.

As can be seen from the above, most existing eye disease classification models only classify a certain disease twice, i.e., the application range and the screening content are small, and the practical application efficiency is low. The problem of high computing resource consumption exists when a plurality of binary classification networks are directly integrated, and the wide popularization is difficult. Therefore, it is of great significance to develop a single-model color fundus image classification system, method and electronic device with a small calculation amount.

[ summary of the invention ]

The invention aims to overcome the defects of the prior art and provides a method for multi-label disease classification of color fundus images.

In order to achieve the purpose, the invention adopts the following technical scheme:

a color fundus image classification system, comprising:

the image acquisition equipment comprises a camera and a first communication module;

the intelligent terminal comprises a computer or a mobile phone, wherein the computer or the mobile phone comprises an image processing module and a second communication module;

the cloud server comprises an image processing module and a third communication module;

the system has two working modes, namely: after the image acquisition equipment acquires the fundus image, the first communication module sends the acquired fundus image to the intelligent terminal through the second communication module, the image processing module processes the acquired fundus image, and an image classification result is displayed on the intelligent terminal;

the second method comprises the following steps: after the image acquisition equipment acquires the fundus images, the first communication module sends the acquired fundus images to the cloud server through the third communication module, the image processing module processes the acquired fundus images, and the third communication module sends image classification results to the intelligent terminal through the second communication module and displays the image classification results.

Further, the image processing module includes:

an image acquisition module: acquiring a color fundus image to be classified;

a preprocessing module: preprocessing a color fundus image to be classified;

a feature extraction module: performing characteristic extraction on the color fundus images to be classified to obtain an image characteristic matrix;

a model processing module: inputting the image characteristic matrix into a pre-trained eye disease classification model, and outputting a classification result;

and a result optimization output module: and optimizing the classification result of the model processing module and outputting the color fundus image classification result.

Further, the model processing module comprises a pre-trained eye disease classification model, the pre-trained eye disease classification model is an SL _ EfficientNet network model, and the training method of the model comprises the following steps:

A1. creating a color fundus image dataset; the color fundus image data set comprises a normal fundus image and a pathological fundus image with various labels;

A2. performing expansion processing on the color fundus image data set based on the multiple labels to obtain an expanded color fundus image data set;

A3. training the expanded color fundus image data set through an EfficientNet-B4 network structure based on a spatial attention and loss function to obtain parameters of the trained EfficientNet-B4 network structure;

A4. inputting the obtained parameters of the EfficientNet-B4 network structure into numpy, adopting numpy, and adopting numpy to construct an EfficientNet-B4 network model based on space attention and loss correction to obtain a trained SL _ EfficientNet network model;

the model processing module comprises the following working steps:

B1. inputting image characteristic information of a color fundus image;

B2. carrying out a series of processing such as convolution, spatial attention, average pooling, loss function and the like on the image characteristic information to obtain a plurality of predicted values;

B3. classifying the plurality of predicted values through a sigmoid activation function to obtain a plurality of probability values;

B4. threshold processing is carried out on the plurality of probability values, and a plurality of final label values are output

Further, the result output by the image processing module is multi-label, and is divided into a valid label and an invalid label;

further, the space attention processing comprises the following specific steps:

C1. inputting the processed color fundus image feature matrix;

C2. performing maximum pooling and average pooling on the input image feature matrix;

C3. sending the image characteristic matrix after the maximum pooling and the image characteristic matrix after the average pooling into a neural network multilayer perceptron (MLP) to respectively obtain a maximum characteristic matrix and an average characteristic matrix;

C4. summing the maximum feature matrix and the average feature matrix;

C5. obtaining a value T between 0 and 1 through a sigmiod function;

C6. multiplying the value T with the image feature matrix;

C7. performing maximum pooling and average pooling to obtain a fusion characteristic matrix;

C8. performing parallel convolution calculation on the fusion characteristic matrix;

C9. outputting a weight matrix S with the same dimension as the input color fundus image characteristic matrix;

C10. multiplying the weight matrix S by the image characteristic matrix;

C11. outputting a feature matrix with spatial attention;

further, the parallel convolution calculation specifically includes four types of convolution:

C81. performing convolution processing on the fusion feature matrix and the convolution kernel of 1 x 1;

C82. performing maximum pooling on the fusion feature matrix and performing convolution processing on the fusion feature matrix and a convolution kernel of 1 x 1;

C83. performing convolution processing on the fusion feature matrix and the convolution kernel of 1 × 1, and performing convolution processing on the fusion feature matrix and the convolution kernel of 3 × 3;

C84. and (3) performing convolution processing on the fusion feature matrix and the convolution kernel of 1 × 1, and then performing convolution processing on the fusion feature matrix and the convolution kernel of 3 × 3, namely performing convolution processing on the fusion feature matrix and the convolution kernel of 3 × 3.

Further, in the result optimization output module, the optimization output result means:

(1) if the output labels are all invalid, setting the label corresponding to the maximum value in the probability values as valid;

(2) if a plurality of labels are output, and the number of invalid labels in the rest labels is more than 1 under the condition that the normal labels are valid, setting the value of the normal labels as invalid;

(3) if the number of the tags with the effective numerical values in the output tags is larger than the preset effective tag number, only the first tags with larger probability values are taken as effective tags, and the rest tags are taken as ineffective tags;

a color fundus image classification method, characterized in that the method comprises:

s1, acquiring a color fundus image to be classified;

s2, preprocessing a color fundus image to be classified;

s3, performing feature extraction on the color fundus images to be classified to obtain an image feature matrix;

s4, inputting the image feature matrix into a pre-trained multi-label eye disease classification model;

s5, outputting an image processing result by a pre-trained multi-label eye disease classification model;

s6, optimizing an output result;

and S7, determining zero or more diseases corresponding to the color fundus image according to the optimized output result.

A cloud server comprising a communication module, an analysis module, and a storage module, for executing the fundus image analysis method according to claim 10. The communication module is used for acquiring fundus images to be classified and outputting classification results; the analysis module is used for processing the fundus image; the storage module is used for storing data related to fundus image analysis.

An electronic device, comprising:

[ description of the drawings ]

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Wherein:

FIG. 1 is a schematic view of two working modes of a color fundus image classification system according to an embodiment of the present invention

FIG. 2 is a diagram illustrating the training steps of the pre-trained eye disease classification model according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of the working steps of a model processing module according to an embodiment of the present invention;

FIG. 4 is a flowchart illustrating a spatial attention process according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of the principle of parallel convolution computation according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of the steps of a color fundus image classification method according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of the result output of the color fundus image classification system according to the embodiment of the present invention.

[ detailed description ] embodiments

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example one

The present embodiment provides a color fundus image classification system characterized by comprising:

the system has two working modes, as shown in fig. 1, the first mode is: after the image acquisition equipment acquires the fundus image, the first communication module sends the acquired fundus image to the intelligent terminal through the second communication module, the image processing module processes the acquired fundus image, and an image classification result is displayed on the intelligent terminal;

The image processing module comprises:

an image acquisition module: acquiring a color fundus image to be classified;

a preprocessing module: preprocessing a color fundus image to be classified;

The model processing module comprises a pre-trained eye disease classification model, the pre-trained eye disease classification model is an SL _ EfficientNet network model, and the training method of the model comprises the following steps as shown in FIG. 2:

A2. expanding the color fundus image data set based on the multiple labels to obtain an expanded color fundus image data set;

the model processing module comprises the following working steps as shown in fig. 3:

B1. inputting image characteristic information of a color fundus image;

B4. performing threshold processing on the plurality of probability values, and outputting a plurality of final label values which are respectively represented by 0 and 1; 0 indicates that the image feature does not correspond to the label, and 1 indicates that the image feature corresponds to the label.

The result output by the image processing module is multi-label and is divided into an effective label and an ineffective label; if the image characteristics are consistent with the normal label, the output result is as follows: the normal label is 1, and the labels for the remaining 7 diseases (diabetic retinopathy (DR), age-related macular degeneration (AMD), corneal haze (MH)), Drusen (DR)), Myopia (MYA), leopard-like fundus tesselation (tsln), optic disc depression (ODC)) are 0; if the image features match one or more disease labels, the output result is: the normal label is 0 and the corresponding one or more disease labels are 1.

The specific steps of the space attention processing are as shown in fig. 4:

C1. inputting the processed color fundus image feature matrix;

C4. summing the maximum feature matrix and the average feature matrix;

C5. obtaining a value T between 0 and 1 through a sigmiod function;

C6. multiplying the value T with the image feature matrix;

C10. multiplying the weight matrix S by the image characteristic matrix;

C11. outputting a feature matrix with spatial attention;

as shown in fig. 5, the parallel convolution calculation specifically includes four types of convolution:

In the result optimization output module, the optimization output result means:

(3) if the number of the output labels with the effective numerical values is larger than the preset effective label number, only the first labels with larger probability values are taken as effective labels, and the rest labels are taken as ineffective labels;

a color fundus image classification method, as shown in fig. 6, characterized in that it comprises:

s1, acquiring a color eye fundus image to be classified;

s2, preprocessing a color fundus image to be classified;

s6, optimizing an output result;

An electronic device, comprising:

Example two

The embodiment provides a method for multi-label disease classification based on EfficientNet color fundus images in the attention mechanism.

The classification effect of the network before and after adding the spatial attention and loss function is compared through an ablation experiment. As can be seen from table 1, all the evaluation indexes of the model after the spatial attention and loss function is adopted are improved, and are independent of the main CNN model, i.e. it is effective to increase the spatial attention and loss function.

TABLE 1 results of the melting experiment on the test set

As can be seen from the contents in table 1, the results of introducing spatial attention in the basic convolutional neural network are superior to the original model. The F1 improvement of the model is over 0.5, so the spatial attention model has a certain role in the final multi-label classification model.

EXAMPLE III

In this embodiment, the size of the image size of the most suitable input model is explored by testing the performance of the multi-label classification model when images of different sizes are input.

In this embodiment, images with the sizes of 100 × 100, 224 × 224, 300 × 300, and 380 × 380 are input into the model, and the comparison results are shown in table 2.

TABLE 2 statistical results of Effeicient-B4 at different input scales

As can be seen from table 2, the performance of the input 380 x 380 sized image network is relatively highest. The accuracy of the test set differed greatly compared to the input size of 100 x 100 and 224 x 224, but the increase in accuracy gradually saturates as the input pixels increase. The difference in accuracy between the 380 x 380 input and the 300 x 300 input is less than 1%, but the required storage capacity will increase significantly. The accuracy of 100 x 100 input can reach 84.52%, and the training storage requirement is only 6.8 g. If there is insufficient memory, one may choose to sacrifice accuracy for multi-label training using an input size of 100 x 100. Of course, for 380 x 380 input images, increasing the focal length loss and spatial attention on this basis can achieve higher accuracy.

In medical imaging, higher performance is crucial and should be pursued as much as possible. Finally, the invention uses 380 x 380 input images and EfficientNet-B4 embedded with spatial attention to train, and establishes a multi-label classification model of fundus diseases.

Example four

A large amount of sparse data often exists in multi-label data, only a small number of labels can work in most cases, and the situation that sick labels and non-sick labels appear simultaneously exists, and prediction time can be shortened by adopting some strategies when the maximum number of labels and data distribution are calculated. The embodiment is a test result of the model before and after the error correction strategy is added to different networks, and is shown in table 3.

The error correction strategy comprises the following steps: firstly, a normal fundus picture and a diseased fundus picture cannot appear at the same time; and secondly, the maximum number of the labels is not more than 3. When they occur simultaneously, they may be deleted or reclassified.

TABLE 3 comparison of model Performance under different strategies

It can be seen from table 3 that the final results of the application policy test set outputted by different networks are improved to some extent. This is because the characteristic dimension model obtained by multi-label classification may have some errors, but the proportion of the errors is small, and the influence on the model is not obvious.

EXAMPLE five

The present embodiment provides a system for multi-label eye disease classification of a color fundus picture, the system including:

the preprocessing module is used for acquiring a color fundus image, and performing data preprocessing on the acquired color fundus image to acquire an image characteristic matrix;

the fusion module is used for inputting the image characteristic matrix into a pre-trained multi-label eye disease classification model to complete the prediction classification of the color fundus images;

the optimization module is used for optimizing the prediction result output by the multi-label eye disease classification model;

and the evaluation module is used for outputting results to determine zero or more diseases corresponding to the color fundus images.

Example six

The present embodiments also provide a computer readable storage medium having stored therein a plurality of instructions, which are loadable by a processor, causing the processor to execute the method for multi-label eye disease classification of color fundus images according to any of the embodiments of the present invention. Specifically, a system or an apparatus equipped with a storage medium on which software program codes that realize the functions of any of the above-described embodiments are stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program codes stored in the storage medium.

In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

Examples of the storage medium for supplying the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD + RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer via a communications network.

Further, it should be clear that the functions of any one of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform a part or all of the actual operations based on instructions of the program code.

Further, it is to be understood that the program code read out from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then causes a CPU or the like mounted on the expansion board or the expansion unit to perform part or all of the actual operations based on instructions of the program code, thereby realizing the functions of any of the above-described embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A color fundus image classification system, comprising:

2. The system of claim 1, wherein the image processing module comprises:

an image acquisition module: acquiring a color fundus image to be classified;

a preprocessing module: preprocessing a color fundus image to be classified;

3. The system of claim 2, wherein the model processing module comprises a pre-trained eye disease classification model, the pre-trained eye disease classification model is a SL _ EfficientNet network model, and the training method of the model comprises the following steps:

the model processing module comprises the following working steps:

B1. inputting image characteristic information of a color fundus image;

B4. and performing threshold processing on the plurality of probability values, and outputting a plurality of final label values.

4. The system of claim 1, wherein the result output by the image processing module is multi-labeled, divided into a valid label and an invalid label.

5. The system of claim 3, wherein the spatial attention processing comprises the following specific steps:

C1. inputting the processed color fundus image feature matrix;

C4. summing the maximum feature matrix and the average feature matrix;

C5. obtaining a value T between 0 and 1 through a sigmiod function;

C6. multiplying the value T with the image feature matrix;

C10. multiplying the weight matrix S by the image characteristic matrix;

C11. and outputting the feature matrix with the spatial attention.

6. The system according to claim 5, characterized in that said parallel convolution calculation comprises in particular four convolutions:

7. The system according to claim 2, wherein in the result optimization output module, the optimization output result is:

(3) if the number of the tags with the effective numerical values in the output tags is larger than the preset effective tag number, only the first tags with larger probability values are taken as effective tags, and the rest tags are taken as ineffective tags.

8. A color fundus image classification method according to any one of claims 1 to 7, characterized in that said method comprises:

s1, acquiring a color fundus image to be classified;

s2, preprocessing a color fundus image to be classified;

s6, optimizing an output result;

9. A cloud server, comprising a communication module, an analysis module and a storage module, for executing the fundus image analysis method according to claim 8, wherein the communication module is responsible for acquiring fundus images to be classified and outputting the classification results; the analysis module is used for processing the fundus image; the storage module is used for storing data related to fundus image analysis.

10. An electronic device, comprising: