CN116824116A - Super wide angle fundus image identification method, device, equipment and storage medium - Google Patents

Abstract

This application discloses an ultra-wide-angle fundus image recognition method, device, equipment and storage medium, which relates to the field of image processing technology and the field of deep learning technology, including: performing an image pre-processing operation on the original ultra-wide-angle fundus image to obtain a pre-processed image; Use the pre-trained feature extraction model to extract features from the pre-processed image to obtain the target encoding information; decode the target encoding information through the bilinear decoder model to obtain the output probability value information set; select the category with the largest probability value in the probability value information set Determined as the final prediction category corresponding to the original ultra-wide-angle fundus image. This application extracts feature vectors from pre-processed image features through a pre-trained feature extraction model, uses a bilinear decoder model to perform multiple complex algebraic operations, outputs multiple probability values, and determines the final recognition result by comparing the probability values. It reduces the probability of misidentification caused by manual reading and improves the accuracy of ultra-wide-angle fundus image recognition.

Description

An ultra-wide-angle fundus image recognition method, device, equipment and storage medium

Technical field

The present invention relates to the field of image processing technology and the field of deep learning technology, and in particular to an ultra-wide-angle fundus image recognition method, device, equipment and storage medium.

Background technique

As the aging of the population continues to accelerate, various fundus diseases will spread more rapidly if timely intervention is not carried out. The current number of ophthalmologists and medical equipment are not enough to meet the needs of such a large group of eye disease patients.

Most of the existing fundus disease image recognition systems are based on ordinary fundus color ultrasound images. The visible range of the retinal fundus in such images is narrow, usually only 30°-75°. A relatively narrow visual area cannot provide complete fundus information, which will increase the probability of misidentification. Ultra-wide-angle fundus images are less difficult to obtain, and the fundus visual range can reach 200°. The judgment of fundus diseases can be based on more information and the judgment accuracy is higher. However, current fundus image recognition based on ultra-wide angles uses manual reading, which is greatly limited by the number of clinicians and clinical experience. The probability of image recognition errors during manual reading is high and the accuracy of image recognition is low.

Contents of the invention

In view of this, the purpose of the present invention is to provide an ultra-wide-angle fundus image recognition method, device, equipment and storage medium, which can reduce the probability of false recognition caused by manual reading and improve the accuracy of ultra-wide-angle fundus image recognition. The specific plan is as follows:

In the first aspect, this application discloses an ultra-wide-angle fundus image recognition method, including:

Perform image preprocessing operations on the original ultra-wide-angle fundus image to obtain the preprocessed image;

Using a pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain target encoding information;

Decode the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model;

The category with the largest probability value in the probability value information set is determined as the final predicted category corresponding to the original ultra-wide-angle fundus image.

Optionally, the image preprocessing operation is performed on the original ultra-wide-angle fundus image to obtain the preprocessed image, including:

Extract a target area from the original ultra-wide-angle fundus image based on an adaptive ROI area rough extraction component; wherein the target area is an area composed of pixels that meet preset pixel conditions;

Determine whether the target area meets the preset area quality requirements;

If the target area meets the preset area quality requirements, adjust the brightness of the target area based on the preset brightness adjustment rules to obtain the preprocessed image;

If the target area does not meet the preset area quality requirements or the target area is not extracted, then perform maximum ellipse approximate fitting to obtain the fitting area;

Determine the fitting area as the target area, and re-enter the step of adjusting the brightness of the target area based on a preset brightness adjustment rule to obtain the preprocessed image.

Optionally, adjusting the brightness of the target area based on preset brightness adjustment rules to obtain the preprocessed image includes:

Obtain the current brightness of the target area, and determine the current brightness interval corresponding to the current brightness;

Determine the current brightness adjustment strategy corresponding to the current brightness interval based on the preset brightness adjustment strategy determination rules;

The current brightness adjustment strategy is used to adjust the brightness of the target area to obtain the preprocessed image.

Optionally, performing maximum ellipse approximation fitting to obtain the fitting area includes:

Calculate the corresponding long axis length information and short axis length information according to the length and width of the original ultra-wide-angle fundus image;

Determine the center point position information based on the reference point information;

An elliptical area is generated based on the center point position information, the major axis length information, and the minor axis length information, and the elliptical area is determined as the fitting area.

Optionally, before using the pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain the target encoding information, it also includes:

Use open source data sets to train the original feature extraction model to obtain an initial feature extraction model with initial weights;

Using the ultra-wide-angle fundus image data set to train the initial feature extraction model to obtain the pre-trained feature extraction model with target weights;

Correspondingly, using the pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain target encoding information includes:

Using the pre-trained feature extraction model with the target weight to extract feature information that meets the preset feature requirements in the pre-processed image;

The feature information is encoded in the form of a vector to obtain the target encoding information.

Optionally, decoding the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model includes:

Input the target encoding information to the first channel and the second channel of the bilinear decoder model;

Obtain the first output information output by the first channel and the second output information output by the second channel;

Perform spatial dimension feature stacking on the first output information and the second output information to obtain a stacked feature layer;

A preset prediction operation is performed through the stacked feature layer to output the probability value information set.

Optionally, inputting the target encoding information to the first channel and the second channel of the bilinear decoder model includes:

Input the target encoding information into the first channel of the bilinear decoder model, and perform same-scale convolution according to the convolution kernel in the first channel to obtain post-convolution information;

Calculate the first attention weight of the current feature layer based on the convolved information and the target encoding information;

If the current feature layer is the last layer, the first output information output by the first channel is calculated based on the target encoding information and the first attention weight of the previous layer; wherein, the first One output information is the weighted attention feature;

Input the target encoding information to the second channel of the bilinear decoder model, and calculate the average value of the target encoding information to obtain a second attention weight;

The second output information output by the second channel is calculated based on the second attention weight and the target encoding information.

In the second aspect, this application discloses an ultra-wide-angle fundus image recognition device, including:

The image preprocessing module is used to perform image preprocessing operations on the original ultra-wide-angle fundus image to obtain the preprocessed image;

A feature extraction module, used to perform feature extraction on the preprocessed image using a pre-trained feature extraction model to obtain target encoding information;

A decoding module configured to decode the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model;

A prediction category determination module, configured to determine the category with the largest probability value in the probability value information set as the final prediction category corresponding to the original ultra-wide-angle fundus image.

In a third aspect, this application discloses an electronic device, including:

Memory, used to hold computer programs;

A processor, configured to execute the computer program to implement the steps of the ultra-wide-angle fundus image recognition method disclosed above.

In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein when the computer program is executed by a processor, the ultra-wide-angle fundus image recognition method disclosed above is implemented.

It can be seen that this application provides an ultra-wide-angle fundus image recognition method, which includes: performing an image pre-processing operation on the original ultra-wide-angle fundus image to obtain a pre-processed image; using a pre-trained feature extraction model to characterize the pre-processed image. Extract to obtain target encoding information; decode the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model; obtain the maximum probability value in the probability value information set The category of is determined as the final predicted category corresponding to the original ultra-wide-angle fundus image. It can be seen that this application uses a pre-trained feature extraction model to extract features from the pre-processed image to obtain a feature vector, and then uses a bilinear decoder model to perform multiple complex algebraic operations on the feature vector to output multiple probability values. The final recognition result is determined by comparing the probability values, which reduces the probability of misrecognition caused by manual reading and improves the accuracy of ultra-wide-angle fundus image recognition.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without exerting creative efforts.

Figure 1 is a flow chart of an ultra-wide-angle fundus image recognition method disclosed in this application;

Figure 2 is a flow chart of a specific ultra-wide-angle fundus image recognition method disclosed in this application;

Figure 3 is a schematic flowchart of the ultra-wide-angle fundus image preprocessing process disclosed in this application;

Figure 4 is a flow chart of a specific ultra-wide-angle fundus image recognition method disclosed in this application;

Figure 5 is a flow chart of bilinear feature decoding and prediction disclosed in this application;

Figure 6 is a schematic structural diagram of the ultra-wide-angle fundus image recognition device provided by this application;

Figure 7 is a structural diagram of an electronic device provided by this application.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

At present, most of the existing fundus disease image recognition systems are based on ordinary fundus color ultrasound ultra-wide-angle fundus images. The visible range of the retinal fundus in such ultra-wide-angle fundus images is narrow, usually only 30°-75°. A relatively narrow visual area cannot provide complete fundus information, which will increase the probability of misidentification. Ultra-wide-angle fundus images are less difficult to obtain, and the visual range of the fundus can reach 200°. The judgment of fundus diseases can be based on more information and the judgment accuracy is higher. However, current ultra-wide-angle fundus image recognition based on ultra-wide-angle fundus images uses manual reading, which is greatly limited by the number of clinicians and clinical experience. The probability of image recognition errors during manual reading is high and ultra-wide-angle fundus image recognition The accuracy is lower. To this end, this application provides an ultra-wide-angle fundus image recognition method, which can reduce the probability of misidentification caused by manual reading and improve the accuracy of ultra-wide-angle fundus image recognition.

An embodiment of the present invention discloses an ultra-wide-angle fundus image recognition method, as shown in Figure 1. The method includes:

Step S11: Perform an image preprocessing operation on the original ultra-wide-angle fundus image to obtain a preprocessed image.

In this embodiment, an image preprocessing operation is performed on the original ultra-wide-angle fundus image to obtain a preprocessed image. Specifically, the target area is extracted from the original ultra-wide-angle fundus image based on the adaptive ROI area rough extraction component; wherein the target area is an area composed of pixels that meet preset pixel conditions; it is determined whether the target area Reach the preset area quality requirements; if the target area reaches the preset area quality requirements, adjust the brightness of the target area based on the preset brightness adjustment rules to obtain the preprocessed image; if the target area does not If the preset area quality requirements are reached or the target area is not extracted, then maximum ellipse approximate fitting is performed to obtain the fitting area; the fitting area is determined as the target area, and re-enters the based on The step of adjusting the brightness of the target area by a preset brightness adjustment rule to obtain the preprocessed image. It can be understood that the target area is an area composed of pixels that meet preset pixel conditions. For example, if the characteristics of the ultra-wide-angle fundus image itself are pixel values, then the pixels that are greater than the preset pixel threshold are determined to be sufficient for the preset pixels. Set the pixel point of the pixel condition.

It can be understood that adjusting the brightness of the target area based on a preset brightness adjustment rule to obtain the preprocessed image includes: obtaining the current brightness of the target area and determining the current brightness interval corresponding to the current brightness; The current brightness adjustment strategy corresponding to the current brightness interval is determined based on the preset brightness adjustment strategy determination rule; the brightness of the target area is adjusted using the current brightness adjustment strategy to obtain the preprocessed image. Perform maximum ellipse approximate fitting to obtain the fitting area, including: calculating the corresponding major axis length information and minor axis length information based on the length and width of the original ultra-wide-angle fundus image; determining the center point position information based on the reference point information; An elliptical area is generated based on the center point position information, the major axis length information, and the minor axis length information, and the elliptical area is determined as the fitting area.

As shown in Figure 2, the present invention is divided into three main modules: image preprocessing module (including adaptive ROI area rough extraction, maximum inscribed ellipse approximate fitting and adaptive brightness adjustment); feature extraction and encoding module; feature decoding and disease prediction module. Before the image preprocessing module, the original ultra-wide-angle fundus effect is obtained, a 6-category training set and a test set are constructed, and data balancing operations are performed to ensure that the amount of data for each category is close to 1:1. In the feature extraction and encoding module, ResNet50 is used for feature extraction. It is first pre-trained on the ImageNet open source data set to obtain the initialization weights, and then the initialization weights are adjusted during training on the ultra-wide-angle data set. In the feature decoding and disease prediction module, the feature vector encoded by ResNet50 is received, the features are decoded through the dual-branch model and the prediction results are output. The modules are closely connected and decoupled from each other, that is, the output of the previous module is the input of the next module, but each module is independent of each other when performing the corresponding function. The function of the entire system is to input an ultra-wide-angle fundus ultra-wide-angle After a series of processing of the fundus image, the fundus disease corresponding to the highest probability value of the ultra-wide-angle fundus image is output.

In the image preprocessing module, a module is constructed that can automatically locate and segment the fundus area of ultra-wide-angle fundus images. For individual ultra-wide-angle fundus images where the brightness of the original image is too low or the distortion is severe, an inscribed elliptical fitting module is added, with a maximum It can perform fundus area fitting on these ultra-wide-angle fundus images that are difficult to position correctly to ensure strong usability of the overall data. This module contains 3 components: adaptive ROI area rough extraction component, maximum inscribed ellipse approximate fitting, and adaptive brightness adjustment. It can be understood that the ultra-wide-angle fundus image enters the adaptive ROI area rough extraction component. Based on machine learning and traditional image processing methods, a KMENAS clustering model is first built and its n_clusters hyperparameter is set to 2, indicating two-cluster clustering. Task, KMEANS is an unsupervised clustering algorithm, based on the characteristics of the ultra-wide-angle fundus image itself (that is, the pixel values in the fundus area are relatively close, while the pixel values in other areas have relatively large changes), and then automatically capture this characteristic through KMEANS , divide the fundus area and other areas into two different clusters. For most ultra-wide-angle fundus images with normal shooting quality, fundus and background segmentation can be achieved normally, thus achieving rough extraction of the ROI area. Then decide whether to perform maximum ellipse approximate fitting based on the effect of ROI region extraction in the previous step. As shown in Figure 3, if the quality of the ROI region extracted in the first step is poor (for example, no ROI is extracted, the area or position of the ROI region does not meet expectations ) is further processed using the second component, which first calculates the long axis length l and short axis length s of the ultra-wide-angle fundus image based on the length and width of the original image (for example, take half the length of the original image as the long axis length , take half the width of the original image as the short axis length), and obtain the center point position (x, y) (usually the preset reference point position, such as (0, 0)), and generate a maximum inscribed of the fundus image For the elliptical mask, the corresponding mask area in the original image is segmented as an approximate fitting, and the approximate fitting area is determined as the roughly extracted ROI area (i.e., the target area). Finally, based on the brightness adjustment algorithm provided by opencv, first calculate the brightness of the input image, that is, the target area, and then determine which order the brightness value is in (the order refers to the brightness threshold, which is known through multiple manual measurements). For different levels of brightness Different degrees of brightness improvement strategies are adopted until the brightness of the final output image is basically at the same level. The main purpose of this component is to light up the dim ultra-wide-angle fundus images, while reducing the brightness of the ultra-wide-angle fundus images that are too bright, and ultimately improve the prediction accuracy of subsequent models.

As shown in Figure 3 above, if the target area meets the preset area quality requirements, or the approximate fitting area is obtained after performing the maximum inscribed ellipse approximate fitting operation, the target is adjusted based on the preset brightness adjustment rules. The brightness of the area.

Step S12: Use the pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain target coding information.

In this embodiment, after performing an image preprocessing operation on the original ultra-wide-angle fundus image to obtain a preprocessed image, a pretrained feature extraction model is used to perform feature extraction on the preprocessed image to obtain target encoding information. Specifically, the pre-trained feature extraction model with the target weight is used to extract feature information that meets the preset feature requirements in the pre-processed image; the feature information is encoded in the form of a vector to obtain the Target encodes information.

It can be understood that before using the pre-trained feature extraction model to perform feature extraction on the pre-processed image, the open source data set is used to train the original feature extraction model to obtain an initial feature extraction model with initial weights; the ultra-wide-angle fundus image is used The initial feature extraction model is trained on the data set to obtain the pre-trained feature extraction model with target weights. For example, the ResNet50 feature extraction module uses the open source model ResNet50. The model is first trained on the pre-training data set (i.e., the open source data set) ImageNet, because this data set contains more than 14 million images and covers 1,000 categories. ResNet50 trained with a large amount of data has better initial weights and can show much better performance in other personalized tasks than ResNet50 initialized with random weights. The initial feature extraction model with initial weights is used to train on the current ultra-wide-angle fundus image data set to obtain the pre-trained feature extraction model with target weights. The above-mentioned pre-trained feature extraction model can efficiently extract fundus features, which is convenient for Subsequent modules decode these features.

It should be pointed out that, as shown in Figure 2 above, when using the initial feature extraction model with initial weights to train on the current ultra-wide-angle fundus image data set, the current ultra-wide-angle fundus image data set is obtained and a 6-category training set is constructed. and test set, perform data balancing operations at the same time, so that the amount of data of each type is close to 1:1.

Step S13: Decode the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model.

In this embodiment, a pre-trained feature extraction model is used to perform feature extraction on the pre-processed image to obtain target encoding information, and the target encoding information is decoded through a bilinear decoder model to obtain the bilinear decoding The set of probability value information output by the controller model. Specifically, the target encoding information is input to the first channel and the second channel of the bilinear decoder model; the first output information output by the first channel and the second output information output by the second channel are obtained. Output information; perform spatial dimension feature stacking on the first output information and the second output information to obtain a stacked feature layer; perform a preset prediction operation through the stacked feature layer to output the probability value information gather.

Step S14: Determine the category with the largest probability value in the probability value information set as the final predicted category corresponding to the original ultra-wide-angle fundus image.

In this embodiment, after decoding the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model, the category with the largest probability value in the probability value information set is The final prediction category corresponding to the original ultra-wide-angle fundus image is determined. It can be understood that each image predicts 6 probability values, corresponding to the probabilities of the six categories of vitreous opacity, macular degeneration, diabetic retinopathy, glaucoma, other fundus diseases, and normal. The category with the largest probability value is selected as the final model. output category. The number of predicted probability values and categories for each image can be set according to different situations.

The real clinical data experimental performance of the present invention is good. The 6-classification model has achieved 86.2% Acc, 86.1% specificity, 97.3% sensitivity and 86% f1 value on tens of thousands of Oubao ultra-wide-angle fundus image data. Currently, Even ophthalmologists with rich clinical experience must combine the patient's age, gender, medical history and other information to identify ultra-wide-angle fundus images, and determine the disease category based on the recognition results, and there are inconsistent ultra-wide-angle fundus image recognition results among multiple experts. In this case, it is even more difficult to accurately identify directly through manual reading of the film. Currently, there is no multi-eye disease auxiliary AI system based on ultra-wide-angle fundus images. Most of the existing technologies can only perform two-category tasks, such as glaucoma/non-glaucoma. The reality is that the patient may be suffering from macular degeneration. In such a two-category category, cannot be detected in the classification model, making most existing technologies unable to be truly used clinically to assist doctors in ultra-wide-angle fundus image recognition. The present invention establishes a research paradigm for an ultra-wide-angle fundus image auxiliary system. This model establishes a complete set of multi-fundus diseases. Assisting the ultra-wide-angle fundus image recognition research paradigm, it can further enrich the categories of identifiable eye diseases, improve the model's ultra-wide-angle fundus image recognition accuracy, and build a bilinear decoding network based on the attention mechanism. This model not only achieves high test accuracy on current ultra-wide-angle fundus imaging data, but also achieves good results on the glucose reticular classification task based on fundus color ultrasound, proving that the model can be further fine-tuned and expanded to richer Image recognition and classification scenarios.

This invention performs data preprocessing on the original ultra-wide-angle fundus image. Its main purpose is to extract the area containing only the fundus from the original image containing part of the eyelids, eyelashes and background information. The method is to build an automatic fundus recognition system based on traditional image processing technology. ROI area and extract the image preprocessing module of this area. After passing the original image through this module, a relatively pure fundus image will be obtained. Secondly, the ResNet50 deep learning model is used to encode image features, and the more detailed pixel-level features in the fundus images are extracted and encoded in the form of vectors. Finally, the encoded features are decoded, and a bilinear decoder model is constructed to perform multiple complex algebraic operations on the feature vector, finally outputting 6 probability values, which correspond to 5 different eye diseases and normal categories respectively. Probability, take the category with the largest probability value as the output category, that is, the ultra-wide-angle fundus image recognition result. The multi-fundus disease auxiliary image recognition system based on ultra-wide-angle fundus images can automatically identify multiple fundus diseases with high recognition accuracy. It can be used in clinical early-stage fundus disease screening and provides professional ophthalmologists with reliable ultra-wide-angle fundus images. The image recognition reference relieves the pressure on current front-line clinicians, reduces the probability of misrecognition caused by manual reading, and improves the efficiency of reading and the accuracy of ultra-wide-angle fundus image recognition.

It can be seen that this application provides an ultra-wide-angle fundus image recognition method, which includes: performing an image pre-processing operation on the original ultra-wide-angle fundus image to obtain a pre-processed image; using a pre-trained feature extraction model to characterize the pre-processed image. Extract to obtain target encoding information; decode the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model; obtain the maximum probability value in the probability value information set The category of is determined as the final predicted category corresponding to the original ultra-wide-angle fundus image. It can be seen that this application uses a pre-trained feature extraction model to extract features from the pre-processed image to obtain a feature vector, and then uses a bilinear decoder model to perform multiple complex algebraic operations on the feature vector to output multiple probability values. The final recognition result is determined by comparing the probability values, which reduces the probability of misrecognition caused by manual reading and improves the accuracy of ultra-wide-angle fundus image recognition.

As shown in Figure 4, an embodiment of the present invention discloses an ultra-wide-angle fundus image recognition method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

Step S21: Perform an image preprocessing operation on the original ultra-wide-angle fundus image to obtain a preprocessed image.

Step S22: Use the pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain target encoding information.

Step S23: Input the target encoding information to the first channel and the second channel of the bilinear decoder model.

In this embodiment, the target encoding information is input to the first channel and the second channel of the bilinear decoder model. Specifically, the target encoding information is input to the first channel of the bilinear decoder model, and the same-scale convolution is performed according to the convolution kernel in the first channel to obtain the convolved information; The first attention weight of the current feature layer is calculated based on the post-convolution information and the target coding information; if the current feature layer is the last layer, the first attention weight of the current feature layer is calculated based on the target coding information and the first attention weight of the previous layer. An attention weight calculates the first output information output by the first channel; wherein the first output information is a weighted attention feature; inputting the target encoding information to the bilinear decoder model The second channel calculates the average value of the target encoding information to obtain a second attention weight; calculates the second output of the second channel based on the second attention weight and the target encoding information. Output information.

It can be understood that, as shown in Figure 5, the bilinear feature decoding prediction module uses two parallel channel attention branches: AttentionNet1 is based on the convolution branch, and AttentionNet2 is based on the pooling branch, using different attention generation paradigms and Initial weights are used to find important features to the greatest extent, and different channel weights are given to features with different degrees of importance, so that the model can focus on training and learning. Finally, the weighted features are fused in the spatial dimension to ensure the integrity of the features and a subsequent series of convolution, pooling, and fully connected layers are processed to obtain the final category score vector. For each category, a corresponding score will be given. After the softmax operation, the score is output in the form of a probability value to obtain the probability of each category, and the category with the largest probability value is taken as the final prediction result.

The AttentionNet1 unit (i.e. the first channel) implements the channel attention mechanism based on convolution. A 7×7 convolution kernel is used to perform same-scale convolution on the features extracted by the previous module ResNet50. Each feature layer will output a floating point number of size 1×1, which is determined as the attention of the current feature layer. Weight, the (B, N, 7, 7) output by the original ResNet50 is output as (B, N, 1, 1) after the same scale convolution. B represents the Batch size, N represents the number of feature channels, and W and H represent the image respectively. Width and height, the attention weight of each feature layer will be multiplied with the original output feature in the next stage to obtain the weighted attention feature.

I∈(B,N,W,H);

K ₁ =W×H;

Attention ₁ =I*K ₁ ∈ (B, N, 1, 1);

Out ₁ =I×Attention ₁ ∈ (B, N, W, H);

I is the target encoding information, Attention ₁ is the first attention weight, and Out ₁ is the first output information.

The AttentionNet2 unit (i.e., the second channel) implements the channel attention mechanism based on global average pooling. The average value of the 7×7 feature matrix is used as the attention weight of the feature, because most detailed features in the important feature layer will have higher pixel values, because the weight value obtained after averaging is also larger, which can be It is believed that greater attention will be paid in the subsequent training process. Similarly, most pixel values of background and other features tend to be 0, and the average value is also small, and their influence will be gradually ignored in the subsequent training process.

I∈(B,N,W,H);Attention2=GlobalAvgPool(I)∈(B,N,1,1);

Out ₂ =I×Attention ₂ ∈ (B, N, W, H);

I is the target encoding information, Attention ₂ is the second attention weight, and Out ₂ is the second output information.

Step S24: Obtain the first output information output by the first channel and the second output information output by the second channel, and perform spatial dimension feature stacking on the first output information and the second output information to obtain Stacked feature layers.

In this embodiment, after inputting the target encoding information into the first channel and the second channel of the bilinear decoder model, the first output information output by the first channel and the second channel output are obtained. The second output information is stacked with spatial dimension features on the first output information and the second output information to obtain a stacked feature layer. It can be understood that the FeatureFusion unit uses spatial dimension feature stacking. When the feature layers of Out ₁ and Out ₂ are stacked together, it will become 2N feature layers.

Step S25: Perform a preset prediction operation through the stacked feature layer to output the probability value information set.

In this embodiment, after the first output information and the second output information are stacked with spatial dimension features to obtain a stacked feature layer, a preset prediction operation is performed through the stacked feature layer to output the A collection of probability value information. It can be understood that stacking the feature layers of Out ₁ and Out ₂ together will become 2N feature layers, retaining all weighted feature layers and performing a subsequent series of convolution, pooling, and fully connected layers. Get the final class probability.

F=Concatenate(Out ₁ , Out ₂ )∈(B, 2N, W, H);

Score=FC(AvgPool(ConvBlock(F)))∈(B, 6, 1, 1);

Out=SoftMax(Score)∈(B, 6, 1, 1).

The final Out indicates that for B input images, each image predicts 6 probability values, and the 6 probability values correspond to the probabilities of the six categories of vitreous opacity, macular degeneration, diabetic retinopathy, glaucoma, other fundus diseases, and normal. The category with the largest probability value is taken as the final output category of the model.

Step S26: Determine the category with the largest probability value in the probability value information set as the final predicted category corresponding to the original ultra-wide-angle fundus image.

Regarding the specific content of the above steps S21, S22, and S26, reference may be made to the corresponding content disclosed in the foregoing embodiments and will not be described again here.

It can be seen that the embodiment of the present application performs image preprocessing on the original ultra-wide-angle fundus image to obtain a preprocessed image; uses a pretrained feature extraction model to perform feature extraction on the preprocessed image to obtain target encoding information; The target encoding information is input to the first channel and the second channel of the bilinear decoder model; the first output information output by the first channel and the second output information output by the second channel are obtained; The first output information and the second output information are stacked with spatial dimension features to obtain a stacked feature layer; a preset prediction operation is performed through the stacked feature layer to output the probability value information set; The category with the largest probability value in the probability value information set is determined as the final predicted category corresponding to the original ultra-wide-angle fundus image, which reduces the probability of misrecognition caused by manual reading and improves the accuracy of ultra-wide-angle fundus image recognition.

Referring to Figure 6, the embodiment of the present application also discloses an ultra-wide-angle fundus image recognition device, which includes:

The image preprocessing module 11 is used to perform image preprocessing operations on the original ultra-wide-angle fundus image to obtain a preprocessed image;

The feature extraction module 12 is used to perform feature extraction on the pre-processed image using a pre-trained feature extraction model to obtain target encoding information;

Decoding module 13, configured to decode the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model;

The prediction category determination module 14 is configured to determine the category with the largest probability value in the probability value information set as the final prediction category corresponding to the original ultra-wide-angle fundus image.

It can be seen that this application includes: performing image pre-processing operations on the original ultra-wide-angle fundus image to obtain a pre-processed image; using a pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain target encoding information; through double lines The linear decoder model decodes the target encoding information to obtain a probability value information set output by the bilinear decoder model; determines the category with the largest probability value in the probability value information set as the original ultra-wide-angle fundus image The corresponding final prediction category. It can be seen that this application uses a pre-trained feature extraction model to extract features from the pre-processed image to obtain a feature vector, and then uses a bilinear decoder model to perform multiple complex algebraic operations on the feature vector to output multiple probability values. The final recognition result is determined by comparing the probability values, which reduces the probability of misrecognition caused by manual reading and improves the accuracy of ultra-wide-angle fundus image recognition.

In some specific embodiments, the image preprocessing module 11 specifically includes:

A target area extraction unit, configured to extract a target area from the original ultra-wide-angle fundus image based on an adaptive ROI area rough extraction component; wherein the target area is an area composed of pixels that meet preset pixel conditions;

A target area quality judgment unit, used to judge whether the target area meets the preset area quality requirements;

A current brightness acquisition unit, configured to acquire the current brightness of the target area if the target area meets the preset area quality requirements;

A brightness interval determination unit, used to determine the current brightness interval corresponding to the current brightness;

A current brightness adjustment strategy determination unit, configured to determine the current brightness adjustment strategy corresponding to the current brightness interval based on a preset brightness adjustment strategy determination rule;

A brightness adjustment unit configured to adjust the brightness of the target area using the current brightness adjustment strategy to obtain the preprocessed image;

A long axis and short axis determination unit, configured to calculate the corresponding long axis based on the length and width of the original ultra-wide-angle fundus image if the target area does not meet the preset area quality requirements or the target area is not extracted. long information as well as short axis length information;

The center point determination unit is used to determine the center point position information based on the reference point information;

An elliptical region generating unit, configured to generate an elliptical region based on the center point position information, the major axis length information, and the minor axis length information, and determine the elliptical region as the fitting region;

A target area determination unit, configured to determine the fitting area as the target area, and re-enter the step of adjusting the brightness of the target area based on a preset brightness adjustment rule to obtain the preprocessed image.

In some specific embodiments, the feature extraction module 12 specifically includes:

The initial feature extraction model acquisition unit is used to train the original feature extraction model using open source data sets to obtain an initial feature extraction model with initial weights;

A pre-trained feature extraction model acquisition unit is used to train the initial feature extraction model using an ultra-wide-angle fundus image data set to obtain the pre-trained feature extraction model with target weights;

A feature information acquisition unit, configured to use the pre-trained feature extraction model with the target weight to extract feature information that meets preset feature requirements in the pre-processed image;

An encoding unit is used to encode the feature information in the form of a vector to obtain the target encoding information.

In some specific embodiments, the decoding module 13 specifically includes:

A convolution unit, configured to input the target encoding information to the first channel of the bilinear decoder model, and perform same-scale convolution according to the convolution kernel in the first channel to obtain convolution post information;

A first attention weight calculation unit configured to calculate the first attention weight of the current feature layer based on the convolved information and the target encoding information;

A first output information generation unit configured to, if the current feature layer is the last layer, calculate the first output of the first channel based on the target encoding information and the first attention weight of the previous layer. An output information; wherein the first output information is a weighted attention feature;

A second attention weight calculation unit is used to input the target encoding information to the second channel of the bilinear decoder model, and calculate the average value of the target encoding information to obtain a second attention weight. ;

A second output information generation unit configured to calculate the second output information output by the second channel based on the second attention weight and the target encoding information;

An output information acquisition unit, configured to acquire the first output information output by the first channel and the second output information output by the second channel;

An output information stacking unit is used to stack spatial dimension features on the first output information and the second output information to obtain a stacked feature layer;

A probability value information set output unit is configured to perform a preset prediction operation through the stacked feature layer to output the probability value information set.

In some specific embodiments, the prediction category determination module 14 specifically includes:

A prediction category determination unit is configured to determine the category with the largest probability value in the probability value information set as the final prediction category corresponding to the original ultra-wide-angle fundus image.

Further, the embodiment of the present application also provides an electronic device. FIG. 7 is a structural diagram of the electronic device 20 according to an exemplary embodiment. The content in the figure cannot be considered to limit the scope of the present application.

FIG. 7 is a schematic structural diagram of an electronic device 20 provided by an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input-output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the ultra-wide-angle fundus image recognition method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be applicable Any communication protocol of the technical solution of this application is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the external world, and its specific interface type can be selected according to specific application needs. Here No specific limitation is made.

In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be short-term storage or permanent storage. .

Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the ultra-wide-angle fundus image recognition method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

Further, embodiments of the present application also disclose a storage medium, which stores a computer program. When the computer program is loaded and executed by a processor, the ultra-wide-angle fundus image recognition disclosed in any of the foregoing embodiments is realized. Method steps.

Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on its differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For relevant details, please refer to the description in the method section.

Finally, it should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or any such actual relationship or sequence between operations. Furthermore, the terms "comprises," "comprises," or any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes the stated element.

The above is a detailed introduction to the ultra-wide-angle fundus image recognition method, device, equipment and storage medium provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only It is used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the field, there will be changes in the specific implementation and application scope according to the idea of the present invention. In summary, this The content of the description should not be construed as limiting the invention.

Claims (10)

1. An ultra-wide-angle fundus image recognition method, which is characterized by including: Perform image preprocessing operations on the original ultra-wide-angle fundus image to obtain the preprocessed image; Using a pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain target encoding information; Decode the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model; The category with the largest probability value in the probability value information set is determined as the final predicted category corresponding to the original ultra-wide-angle fundus image. 2. The ultra-wide-angle fundus image recognition method according to claim 1, characterized in that the image pre-processing operation is performed on the original ultra-wide-angle fundus image to obtain the pre-processed image, including: Extract a target area from the original ultra-wide-angle fundus image based on an adaptive ROI area rough extraction component; wherein the target area is an area composed of pixels that meet preset pixel conditions; Determine whether the target area meets the preset area quality requirements; If the target area meets the preset area quality requirements, adjust the brightness of the target area based on the preset brightness adjustment rules to obtain the preprocessed image; If the target area does not meet the preset area quality requirements or the target area is not extracted, then perform maximum ellipse approximate fitting to obtain the fitting area; Determine the fitting area as the target area, and re-enter the step of adjusting the brightness of the target area based on a preset brightness adjustment rule to obtain the preprocessed image. 3. The ultra-wide-angle fundus image recognition method according to claim 2, wherein the adjusting the brightness of the target area based on a preset brightness adjustment rule to obtain the pre-processed image includes: Obtain the current brightness of the target area, and determine the current brightness interval corresponding to the current brightness; Determine the current brightness adjustment strategy corresponding to the current brightness interval based on the preset brightness adjustment strategy determination rules; The current brightness adjustment strategy is used to adjust the brightness of the target area to obtain the preprocessed image. 4. The ultra-wide-angle fundus image recognition method according to claim 2, wherein the maximum ellipse approximate fitting is performed to obtain the fitting area, including: Calculate the corresponding long axis length information and short axis length information according to the length and width of the original ultra-wide-angle fundus image; Determine the center point position information based on the reference point information; An elliptical area is generated based on the center point position information, the major axis length information, and the minor axis length information, and the elliptical area is determined as the fitting area. 5. The ultra-wide-angle fundus image recognition method according to claim 1, characterized in that, before using a pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain the target encoding information, it also includes: Use open source data sets to train the original feature extraction model to obtain an initial feature extraction model with initial weights; Using the ultra-wide-angle fundus image data set to train the initial feature extraction model to obtain the pre-trained feature extraction model with target weights; Correspondingly, using the pre-trained feature extraction model to perform feature extraction on the pre-processed image to obtain target encoding information includes: Using the pre-trained feature extraction model with the target weight to extract feature information that meets the preset feature requirements in the pre-processed image; The feature information is encoded in the form of a vector to obtain the target encoding information. 6. The ultra-wide-angle fundus image recognition method according to claim 1, characterized in that the target encoding information is decoded through a bilinear decoder model to obtain the probability value output by the bilinear decoder model. Collection of information including: Input the target encoding information to the first channel and the second channel of the bilinear decoder model; Obtain the first output information output by the first channel and the second output information output by the second channel; Perform spatial dimension feature stacking on the first output information and the second output information to obtain a stacked feature layer; A preset prediction operation is performed through the stacked feature layer to output the probability value information set. 7. The ultra-wide-angle fundus image recognition method according to claim 6, wherein the input of the target encoding information to the first channel and the second channel of the bilinear decoder model includes: Input the target encoding information into the first channel of the bilinear decoder model, and perform same-scale convolution according to the convolution kernel in the first channel to obtain post-convolution information; Calculate the first attention weight of the current feature layer based on the convolved information and the target encoding information; If the current feature layer is the last layer, the first output information output by the first channel is calculated based on the target encoding information and the first attention weight of the previous layer; wherein, the first One output information is the weighted attention feature; Input the target encoding information to the second channel of the bilinear decoder model, and calculate the average value of the target encoding information to obtain a second attention weight; The second output information output by the second channel is calculated based on the second attention weight and the target encoding information. 8. An ultra-wide-angle fundus image recognition device, characterized by including: The image preprocessing module is used to perform image preprocessing operations on the original ultra-wide-angle fundus image to obtain the preprocessed image; A feature extraction module, used to perform feature extraction on the preprocessed image using a pre-trained feature extraction model to obtain target encoding information; A decoding module configured to decode the target encoding information through a bilinear decoder model to obtain a probability value information set output by the bilinear decoder model; A prediction category determination module, configured to determine the category with the largest probability value in the probability value information set as the final prediction category corresponding to the original ultra-wide-angle fundus image. 9. An electronic device, characterized in that it includes: Memory, used to hold computer programs; A processor, configured to execute the computer program to implement the steps of the ultra-wide-angle fundus image recognition method according to any one of claims 1 to 7. 10. A computer-readable storage medium, characterized in that it is used to store a computer program; wherein when the computer program is executed by a processor, the ultra-wide-angle fundus image recognition method according to any one of claims 1 to 7 is implemented. .