CN113240655A - Method, storage medium and device for automatically detecting type of fundus image - Google Patents

Method, storage medium and device for automatically detecting type of fundus image Download PDF

Info

Publication number
CN113240655A
CN113240655A CN202110559562.XA CN202110559562A CN113240655A CN 113240655 A CN113240655 A CN 113240655A CN 202110559562 A CN202110559562 A CN 202110559562A CN 113240655 A CN113240655 A CN 113240655A
Authority
CN
China
Prior art keywords
fundus
classifier
fundus image
automatically detecting
type
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110559562.XA
Other languages
Chinese (zh)
Other versions
CN113240655B (en
Inventor
雷柏英
陈少滨
谢海
杜曰山一
赵金凤
张汝钢
汪天富
张国明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen University
Original Assignee
Shenzhen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen University filed Critical Shenzhen University
Priority to CN202110559562.XA priority Critical patent/CN113240655B/en
Publication of CN113240655A publication Critical patent/CN113240655A/en
Application granted granted Critical
Publication of CN113240655B publication Critical patent/CN113240655B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G06F18/2132—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on discrimination criteria, e.g. discriminant analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30041—Eye; Retina; Ophthalmic

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • Evolutionary Biology (AREA)
  • Computing Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biomedical Technology (AREA)
  • Quality & Reliability (AREA)
  • Probability & Statistics with Applications (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Eye Examination Apparatus (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a method, a storage medium and a device for automatically detecting the type of an eyeground image, wherein the method comprises the following steps: obtaining two different versions of unmarked fundus images by two different operations of a weak intensifier and a strong intensifier, respectively, first, generating a pseudo label by the weak intensifier, the pseudo label serving as a supervision training label of the strong-intensified version of the unmarked fundus image; to improve the classification performance of the classifier, the E-GAN is integrated into a network that generates high quality labeled fundus images using compressed versions of the labeled fundus images; finally, the self-attention module, the spatial attention module and the channel attention module are combined with the classifier model to enhance the feature extraction capability. The experimental result shows that the method provided by the invention can obtain better category identification accuracy under the condition of less marked fundus images.

Description

Method, storage medium and device for automatically detecting type of fundus image
Technical Field
The invention relates to the field of deep learning algorithm application, in particular to a method, a storage medium and a device for automatically detecting the type of an eyeground image.
Background
Retinopathy of prematurity (ROP) is a common retinal disease in low birth weight infants and is also a major cause of blindness in children. AP-ROP is a special ROP characterized by a vascular proliferative disease in retinal vascular development that, if left untimely, can lead to irreversible visual impairment. The ROP, AP-ROP and normal fundus image are shown in fig. 1, and as can be seen from fig. 1, the ROP and AP-ROP are very similar in appearance, which is a major obstacle for accurate and rapid recognition by an ophthalmologist. Therefore, computer-aided identification is particularly important in objective assessment of ROP and AP-ROP.
The deep learning algorithm is applied to ROP disease detection and automatic screening, and Brownian et al utilize a CNN network to realize secondary diagnosis of ROP positive diseases. There are studies to develop an automated ROP screening system with wide-angle retinal images using AlexNet, VGG-16 and GoogleNet. Since there are many labeled datasets, deep neural networks can achieve powerful performance through supervised learning. However, the tagged data is generally annotated by an expert with sufficient expertise and time, which results in a deficiency of the tagged data such that the accuracy of the identification of the AP-ROP image is reduced.
Accordingly, the prior art is yet to be improved and developed.
Disclosure of Invention
The invention aims to solve the technical problem that the accuracy of identifying the type of an eye fundus image by adopting a deep learning method is low due to the defect of labeled AP-ROP image data in the prior art.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
a method of automatically detecting a type of a fundus image, comprising the steps of:
inputting the unmarked fundus image into a classifier after being processed by a weak intensifier to generate a pseudo label;
inputting the unmarked fundus image into a classifier after being processed by a strong intensifier, and calculating to obtain the supervision loss of the unlabeled data by taking the pseudo label as a supervision training label;
compressing the marked fundus images and inputting the compressed marked fundus images into an E-GAN network to generate high-quality marked fundus images, wherein the marked fundus images comprise a normal fundus image, an ROP fundus image and an AP-ROP fundus image;
inputting the high-quality marked fundus image into a classifier, and calculating the pseudo-supervision loss of a generated sample;
inputting the marked fundus images into a classifier after weak enhancement processing, and calculating the supervision loss of labeled data;
calculating to obtain a total loss function of the classifier according to the supervision loss of the non-labeled data, the pseudo supervision loss of the generated sample and the supervision loss of the labeled data;
adjusting the classifier parameters according to the total loss function of the classifier to obtain a trained classifier;
and inputting the fundus sample to be detected into the trained classifier to obtain the image type of the fundus sample to be detected.
The method for automatically detecting the type of the fundus image comprises the step of carrying out weak enhancement processing on the fundus image, wherein the weak enhancement processing is one or more of overturning enhancement, translation enhancement and cropping enhancement.
The method for automatically detecting the type of the fundus image comprises the step of carrying out strong enhancement processing on one or more of adjusting brightness, maximizing contrast, adjusting definition and adjusting color balance of the image.
The method for automatically detecting the type of the fundus image comprises the following steps of inputting an unmarked fundus image into a classifier after being processed by a strong intensifier, taking the pseudo label as a supervision training label, and calculating the supervision loss of unlabelled data, wherein the steps comprise:
generating a set of probability values p for each label generated for an unlabeled fundus imagej＝fclassifier(F(uj) C) thenj＝argmax(pj) As a pseudo tag;
before calculating the loss function of unmarked fundus images, p is usedjGenerating a confidence coefficient εj＝T(max(pj) Not less than theta), when max (p)j) When the value of (a) is greater than theta, epsilonjIs 1, otherwise the value is 0, θ is the scalar hyperparameter;
based on the confidence coefficient, defining the supervision loss of the unlabeled data as:
Figure BDA0003078431030000021
wherein f isclassifierIs a classifier.
The method for automatically detecting the type of the fundus image, wherein the E-GAN network comprises a generator network and a discriminator network, wherein the generator network comprises 18 dense residual blocks and the discriminator network comprises 3 discriminator blocks.
The method for automatically detecting the type of the fundus image, wherein the total loss function of the generator network is
LGe＝β1Ladv+β2Lcontent+LperceptualWherein, β1And beta2Are coefficients that balance the different loss terms,
Figure BDA0003078431030000022
wherein x islFor marked fundus images, xg＝G(x′l)，x′lRepresenting the marked fundus image subjected to the compression processing; l isperceptualAre respectively from x using VGG networkslAnd xgExtracting features from the raw materials, and then comparing the average absolute errors of the extracted features; l iscontentBy directly calculating xlAnd xgThe L1 norm.
The method for automatically detecting the type of the fundus image comprises a modified ResNet50, wherein the modified ResNet50 comprises 4 block groups, namely [ group1, group2, group3 and group4], wherein 9 spatial (3 x 3) convolutions in the group3 and the group4 are replaced by self-attention modules, and a channel attention module and a spatial attention module are integrated in the group1 and the group 2.
The method for automatically detecting the type of the fundus image, wherein in the self-attention module, the relative position code is divided into group1 and group2 height Wrand([1,d,1,h])And width Wrand([1,d,w,1])(ii) a The W isrand([1,d,1,h])And Wrand([1,d,w,1])The elements in between are added to obtain a position code p; by passing the input features to three sets of 1 × 1 convolutions, respectively, the query (q), key (k) and value (v) are obtained, respectively, and the resulting output of the self-attention module is defined as:
Z＝softmax(qPT+qkT)。
a storage medium, wherein the storage medium stores one or more programs executable by one or more processors to implement steps in a method of automatically detecting a type of a fundus image of the present invention.
An apparatus for automatically detecting a type of a fundus image, comprising a processor adapted to implement instructions; and a storage medium adapted to store a plurality of instructions adapted to be loaded by the processor and to perform the steps of the method of automatically detecting a type of a fundus image according to the present invention.
Has the advantages that: the invention provides a method for automatically detecting the type of an eyeground image, which obtains two unmarked eyeground images of different versions through two different operations of a weak intensifier and a strong intensifier respectively, firstly, a pseudo label is generated by the weak intensifier, and the pseudo label is used as a supervision training label of the unmarked eyeground image of the strong intensified version; to improve the classification performance of the classifier, the E-GAN is integrated into a network that generates high quality labeled fundus images using compressed versions of the labeled fundus images; finally, the self-attention module, the spatial attention module and the channel attention module are combined with the classifier model to enhance the feature extraction capability. The experimental result shows that the method provided by the invention can obtain better category identification accuracy under the condition of less marked fundus images.
Drawings
Fig. 1 is a display diagram of a conventional ROP, an AP-ROP, and a normal fundus image.
FIG. 2 is a flow chart of a preferred embodiment of the method for automatically detecting the type of fundus image according to the present invention.
Fig. 3 is a diagram of the architecture of a method for automatically detecting the type of fundus image according to the present invention.
Fig. 4 is a schematic structural composition diagram of the E-GAN network.
Fig. 5 is a schematic structural diagram of the classifier.
Fig. 6 is a schematic structural diagram of a self-attention module.
Fig. 7 is a schematic structural diagram of a channel attention module and a space attention module.
Fig. 8 is a schematic block diagram of an apparatus for automatically detecting the type of fundus image according to the present invention.
Detailed Description
The present invention provides a method, a storage medium and a device for automatically detecting fundus image types, and in order to make the purpose, technical scheme and effect of the present invention clearer and clearer, the present invention will be further described in detail below by referring to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The invention will be further explained by the description of the embodiments with reference to the drawings.
The labeled fundus image data is generally annotated by an expert having sufficient expertise and time, which results in a deficiency of the labeled fundus image data, so that the prior art is liable to have a problem of low recognition accuracy when recognizing the category of the fundus image by the depth learning method.
Based on this, the present invention provides a method for automatically detecting the type of fundus image, as shown in fig. 2, comprising the steps of:
s10, inputting the unmarked fundus image into a classifier after being processed by a weak intensifier, and generating a pseudo label;
s20, inputting the unmarked fundus image into a classifier after being processed by a strong intensifier, and calculating to obtain the supervision loss of the unlabeled data by taking the pseudo label as a supervision training label;
s30, inputting the marked fundus images into the E-GAN network after compression, and generating high-quality marked fundus images, wherein the marked fundus images comprise a normal fundus image, an ROP fundus image and an AP-ROP fundus image;
s40, inputting the high-quality marked fundus image into a classifier, and calculating the pseudo-supervision loss of a generated sample;
s50, inputting the marked fundus image into a classifier after weak enhancement processing, and calculating the supervision loss of the labeled data;
s60, calculating to obtain a total loss function of the classifier according to the supervision loss of the unlabeled data, the pseudo supervision loss of the generated sample and the supervision loss of the labeled data;
s70, adjusting the classifier parameters according to the total loss function of the classifier to obtain a trained classifier;
and S80, inputting the fundus sample to be tested into the trained classifier, and obtaining the image type of the fundus sample to be tested.
The present embodiment acquires 4228 fundus image examination samples including unmarked fundus images and marked fundus images each including a normal fundus image, an ROP fundus image, and an AP-ROP fundus image from an ophthalmology hospital in shenzhen city. As shown in fig. 3, the present embodiment obtains two different versions of unmarked fundus images by two different operations of a weak enhancer and a strong enhancer, respectively, first, a pseudo tag serving as a supervised training tag of the unmarked fundus image processed by the strong enhancer is generated with the weak enhancer; to improve the classification performance of the classifier, the E-GAN is integrated into a network that generates high quality labeled fundus images using compressed versions of the labeled fundus images; finally, the self-attention module, the spatial attention module and the channel attention module are combined with the classifier model to enhance the feature extraction capability. The experimental result shows that the method provided by the invention can obtain better category identification accuracy under the condition of less marked fundus images.
For multi-class classification problems, let XL＝{(xi,yi) I e (1,2, …, K) } is a collection of K labeled data pairs, where xiIs a training sample, yiIs a one bit efficient encoding. Let XU＝{ujI j e (1,2, …, ak) is a batch of ak unlabeled samples, where a is a hyperparameter that determines the ratio of the number of labeled data to unlabeled data in the training process. As an example, the α is set to 2. For unlabeled samples x, this embodiment employs two different data enhancement methods: weak and strong enhancement, denoted f (x) and i (x), respectively. For f (x), the present embodiments may use one or more of flip enhancement, panning enhancement, and cropping enhancement, by way of example, when a standard flip enhancement strategy is used, i.e., flipping it at a given probability level. Based on f (x), the embodiment adds four other enhancement methods to the method i (x), including adjusting brightness, maximizing contrast, adjusting sharpness, and adjusting color balance of the image.
In some embodiments, the loss function of the proposed method of this embodiment consists of three cross-entropy loss terms: tagged data XLSupervision loss L ofLData X without labelUSupervision loss L ofUAnd generating a pseudo-supervised loss L of the sampleGWherein L isLAnd LGRespectively, the standard cross-entropy loss for the marked fundus image and the high quality marked fundus image processed by the weak enhancer. For unlabeled samples, a set of probability values p is generated for each label generated for the unlabeled fundus imagej＝fclassifier(F(uj) C) thenj＝argmax(pj) As a pseudo tag; before calculating the loss function of unmarked fundus images, p is usedjGenerating a confidence coefficient εj＝T(max(pj) ≧ theta) which ensures that the classifier can provide a more stable label for the unlabeled sample, experimental studies show that the method can significantly improve the classifierPerformance; when max (p)j) When the value of (a) is greater than theta, epsilonjIs 1, otherwise is 0, theta is a scalar hyperparameter that represents the threshold at which we retain the pseudo-label, i.e. the dashed portion in fig. 3; based on the confidence coefficient, defining the supervision loss of the unlabeled data as:
Figure BDA0003078431030000051
wherein f isclassifierIs a classifier. By way of example, the optimal scalar hyperparameter θ for obtaining a pseudo tag may be set to 0.90.
In some embodiments, to improve classification performance, the present embodiments propose an E-GAN (enhanced generation countermeasure network) that includes a generator network and a discriminator network based on which compressed low resolution labeled fundus images can be generated into high quality training samples, i.e., high quality labeled fundus images. The present embodiment uses dense residual blocks to design the generator and combines perceptual and content loss during the training process to optimize the quality of the generated image. By way of example, the marked fundus image may be compressed by a factor of 4 as an input to the generator to generate a high quality marked fundus image. Inspired by the generation of a countermeasure network by relativistic averaging, the function of the discriminator of the present embodiment is no longer merely to distinguish between the true and false input images, but to compare whether the image generated by the generator is closer to a true image. Therefore, the generator proposed by the present embodiment can generate more real and reliable training samples. Fig. 4 is a schematic structural diagram of the E-GAN network proposed in this embodiment, and as shown in the figure, the generator network includes 18 dense residual blocks, and the discriminator network includes 3 discriminator blocks.
Specifically, the discriminant loss function is defined as follows:
Figure BDA0003078431030000061
the total loss function of the generator network is
LGe＝β1Ladv+β2Lcontent+Lperceptual，
Wherein, beta1And beta2Are coefficients that balance the different loss terms,
Figure BDA0003078431030000062
wherein x islFor marked fundus images, xg＝G(x′l)，x′lRepresenting the marked fundus image subjected to the compression processing; l isperceptualAre respectively from x using VGG networkslAnd xgExtracting features from the raw materials, and then comparing the average absolute errors of the extracted features; l iscontentBy directly calculating xlAnd xgThe L1 norm.
In some embodiments, fig. 5 is a schematic structural diagram of the classifier of this embodiment, as shown in the figure, the classifier is a modified resenet 50, the modified resenet 50 includes 4 block groups [ group1, group2, group3, group4], the 4 block groups correspond to 3, 4, 6, 3 bottleneck blocks respectively, wherein 9 spatial (3 × 3) convolutions in the group3 and group4 are replaced by self-attention modules, and image features extracted from the network may not be enough to predict a final classification result due to a reduction in the number of convolutions, so a channel attention module and a spatial attention (CASA) module are integrated into the group1 and group2 to improve classification performance. Notably, the first bottleneck block in group3 and group4 of ResNet50 is spatially convolved by step 2, and accordingly we use a 2 × 2 average pooling layer instead of this operation on top of the self attention module (SeA).
In this embodiment, self-attention is a computational primitive that enables paired entity interaction through a content-based addressing mechanism, such that rich association features can be learned in long sequences. It has become a standard tool in the form of a transform block and has found widespread use in natural language processing. For visual tasks, a multi-headed self-attention layer is used in place of the spatial convolution layer to enable self-attention applications. In the SeA module of the present embodimentWe split the relative position code into heights W, respectivelyrand([1，d，1，h])And width Wrand([1，d，w，1])They are random numbers from a normal distribution with both 0 and a variance of 1. W is to berand([1，d，1，h])And Wrand([1,d,w,1])The elements in between are added, we can obtain the position code p. By passing the input features to three sets of 1 × 1 convolutions, respectively, the query (q), key (k), and value (v) can be obtained. The structure of the SeA module is shown in fig. 6. Finally, the output of our SeA module is defined as:
Z＝softmax(qPT+qkT)。
in some embodiments, the block diagrams of the channel attention module and the spatial attention module are shown in FIG. 7, and the channel attention module mainly explores "what" is more meaningful to the input image. Based on the relationship between different channel features, we generate a channel attention map Mc∈RC×1×1. Then using McGenerating a channel profile FC∈RC×H×WIt is specifically defined as follows:
Figure BDA0003078431030000071
Figure BDA0003078431030000072
where a represents a function of the sigmoid type,
Figure BDA0003078431030000073
it is indicated that the multiplication is element-by-element,
Figure BDA0003078431030000074
and
Figure BDA0003078431030000075
representing global average pooling and global maximum pooling, respectively. W1And W2Representing the weight of two shared fully connected layers. In contrast, spatial attention moduleIs a supplement to the channel attention and its main purpose is to solve the "where target" problem. We use the channel profile FCRelationships between spatial features are explored. We define the generated spatial attention as MS∈R1×H×W. Similarly, we use MSGenerating a spatial feature map FS∈RC×H×WThe implementation can be defined as follows:
Figure BDA0003078431030000076
Figure BDA0003078431030000077
wherein f is3×3Representing a convolution operation, the filter kernel size is 3 x 3.
Figure BDA0003078431030000078
And
Figure BDA0003078431030000079
mean pooling and maximum pooling, respectively.
In some embodiments, the total loss function of the classifier is calculated according to the supervision loss of the unlabeled data, the pseudo-supervision loss of the generated samples, and the supervision loss of the labeled data as follows:
LT＝λLLL+λULU+λGLG，
wherein λ isL,λG,λUWeights for marking loss, unmarked loss and forgery supervision loss, respectively, as an example, the parameter λL,λU,λGSet to 0.6, 0.3 and 0.1, respectively; adjusting the classifier parameters according to the total loss function of the classifier to obtain a trained classifier; and inputting the fundus sample to be detected into the trained classifier to obtain the image type of the fundus sample to be detected. The present embodiment will provide a self-attention module, a spatial attention module and a channelThe attention module is combined with a classifier model to enhance feature extraction capabilities. The experimental result shows that the method provided by the embodiment can obtain better category identification accuracy under the condition that the marked fundus images are fewer.
In some embodiments, the present embodiment may use precision (Pre), recall (recall, recc), and F1-score as the evaluation index of the model performance, and the calculation method is as follows:
Figure BDA00030784310300000710
Figure BDA00030784310300000711
Figure BDA00030784310300000712
in the above formula, TP (true positive), TN (true negative), FP (false positive) and FN (false negative) are the number of true positive, true negative, false positive and false negative samples, respectively.
In some embodiments, there is also provided a storage medium storing one or more programs executable by one or more processors to implement the steps in the method of automatically detecting a type of a fundus image of the present invention.
An apparatus for automatically detecting a type of a fundus image, as shown in fig. 8, includes at least one processor (processor) 20; a display screen 21; and a memory (memory)22, and may further include a communication Interface (Communications Interface)23 and a bus 24. The processor 20, the display 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a user guidance interface preset in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may call logic instructions in the memory 22 to perform the methods in the embodiments described above.
Furthermore, the logic instructions in the memory 22 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product.
The memory 22, which is a computer-readable storage medium, may be configured to store a software program, a computer-executable program, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the functional application and data processing, i.e. implements the method in the above-described embodiments, by executing the software program, instructions or modules stored in the memory 22.
The memory 22 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. Further, the memory 22 may include a high speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.
In addition, the specific processes loaded and executed by the storage medium and the instruction processors in the terminal device are described in detail in the method, and are not stated herein.
Comprises a processor, which is suitable for realizing each instruction; and a storage medium adapted to store a plurality of instructions adapted to be loaded by the processor and to perform the steps of the method of automatically detecting a type of a fundus image according to the present invention.
In summary, the present invention provides a method for automatically detecting the type of fundus image, which obtains two different versions of unmarked fundus images by two different operations of a weak enhancer and a strong enhancer, respectively, first, a pseudo label is generated by the weak enhancer, and the pseudo label is used as a supervision training label for the strong enhanced version of unmarked fundus image; to improve the classification performance of the classifier, the E-GAN is integrated into a network that generates high quality labeled fundus images using compressed versions of the labeled fundus images; finally, the self-attention module, the spatial attention module and the channel attention module are combined with the classifier model to enhance the feature extraction capability. The experimental result shows that the identification category accuracy rate can reach 99.53% by using 300 labeled fundus images. The invention helps a user to quickly and accurately identify the AP-ROP from the fundus image sample on the premise of marking a small amount of data.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A method of automatically detecting the type of fundus image, comprising the steps of:
inputting the unmarked fundus image into a classifier after being processed by a weak intensifier to generate a pseudo label;
inputting the unmarked fundus image into a classifier after being processed by a strong intensifier, and calculating to obtain the supervision loss of the unlabeled data by taking the pseudo label as a supervision training label;
compressing the marked fundus images and inputting the compressed marked fundus images into an E-GAN network to generate high-quality marked fundus images, wherein the marked fundus images comprise a normal fundus image, an ROP fundus image and an AP-ROP fundus image;
inputting the high-quality marked fundus image into a classifier, and calculating the pseudo-supervision loss of a generated sample;
inputting the marked fundus images into a classifier after weak enhancement processing, and calculating the supervision loss of labeled data;
calculating to obtain a total loss function of the classifier according to the supervision loss of the non-labeled data, the pseudo supervision loss of the generated sample and the supervision loss of the labeled data;
adjusting the classifier parameters according to the total loss function of the classifier to obtain a trained classifier;
and inputting the fundus sample to be detected into the trained classifier to obtain the image type of the fundus sample to be detected.
2. The method of automatically detecting a fundus image type according to claim 1, wherein said weak enhancement processing is one or more of flip enhancement, shift enhancement, and cropping enhancement.
3. The method of automatically detecting a type of a fundus image according to claim 1, wherein the strong enhancement processing is one or more of adjusting brightness, maximizing contrast, adjusting sharpness, and adjusting color balance of an image.
4. A method for automatically detecting the type of a fundus image according to claim 1, wherein the step of inputting the unmarked fundus image after being processed by the emphasizer into the classifier, and calculating the surveillance loss of the unlabeled data by using said pseudo label as the surveillance training label comprises:
generating a set of probability values p for each label generated for an unlabeled fundus imagej＝fclassifier(F(uj) C) thenj＝argmax(pj) As a pseudo tag;
before calculating the loss function of unmarked fundus images, p is usedjGenerating a confidence coefficient εj＝T(max(pj) Not less than theta), when max (p)j) When the value of (a) is greater than theta, epsilonjIs 1, otherwise the value is 0, θ is the scalar hyperparameter;
based on the confidence coefficient, defining the supervision loss of the unlabeled data as:
Figure FDA0003078431020000011
wherein f isclassifierIs a classifier.
5. A method of automatically detecting a type of a fundus image according to claim 1, wherein said E-GAN network comprises a generator network comprising 18 dense residual blocks and a discriminator network comprising 3 discriminator blocks.
6. A method of automatically detecting a fundus image type according to claim 5, wherein the total loss function of the generator network is
LGe＝β1Ladv+β2Lcontent+Lperceptual，
Wherein, beta1And beta2Are coefficients that balance the different loss terms,
Figure FDA0003078431020000021
wherein x islFor marked fundus images, xg＝G(x′l)，x′lRepresenting the marked fundus image subjected to the compression processing; l ispercep ualAre respectively from x using VGG networkslAnd xgExtracting features from the raw materials, and then comparing the average absolute errors of the extracted features; l iscontentBy directly calculating xlAnd xgThe L1 norm.
7. A method of automatically detecting a type of an eyeground image as claimed in claim 1, characterized in that said classifier is a modified resenet 50, said modified resenet 50 includes 4 block groups [ group1, group2, group3, group4], respectively, wherein 9 spatial (3 × 3) convolutions in said group3 and group4 are replaced by self attention modules, and a channel attention module and a space attention module are integrated in said group1 and group 2.
8. The method of automatically detecting a type of a fundus image according to claim 7, wherein in the self-attention module, the relative position encoding is split into a group1, a group2 height Wrand([1，d，1，h])And width Wrand([1，d，w，1])(ii) a The W israndd([1，d，1，h])And Wrand([1，d，w，1])The elements in between are added to obtain a position code p; by passing the input features to three sets of 1 × 1 convolutions, respectively, the query (q), key (k) and value (v) are obtained, respectively, and the resulting output of the self-attention module is defined as:
Z＝softmax(qpT+qkT)。
9. a storage medium storing one or more programs, which are executable by one or more processors to implement the steps in the method for automatically detecting a type of a fundus image according to any one of claims 1 to 8.
10. An apparatus for automatically detecting the type of an image of a fundus, comprising a processor adapted to implement instructions; and a storage medium adapted to store a plurality of instructions adapted to be loaded by a processor and to perform the steps of the method of automatically detecting a type of a fundus image according to any one of claims 1 to 8.
CN202110559562.XA 2021-05-21 2021-05-21 Method, storage medium and device for automatically detecting type of fundus image Active CN113240655B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110559562.XA CN113240655B (en) 2021-05-21 2021-05-21 Method, storage medium and device for automatically detecting type of fundus image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110559562.XA CN113240655B (en) 2021-05-21 2021-05-21 Method, storage medium and device for automatically detecting type of fundus image

Publications (2)

Publication Number Publication Date
CN113240655A true CN113240655A (en) 2021-08-10
CN113240655B CN113240655B (en) 2023-04-07

Family

ID=77138143

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110559562.XA Active CN113240655B (en) 2021-05-21 2021-05-21 Method, storage medium and device for automatically detecting type of fundus image

Country Status (1)

Country Link
CN (1) CN113240655B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114155365A (en) * 2022-02-07 2022-03-08 北京航空航天大学杭州创新研究院 Model training method, image processing method and related device
CN115130599A (en) * 2022-07-05 2022-09-30 中国矿业大学 Semi-supervision method for strip mine card state recognition under time series GAN data enhancement
WO2024016945A1 (en) * 2022-07-19 2024-01-25 马上消费金融股份有限公司 Training method for image classification model, image classification method, and related device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180336471A1 (en) * 2017-05-19 2018-11-22 Mehdi Rezagholizadeh Semi-supervised regression with generative adversarial networks
CN109948660A (en) * 2019-02-26 2019-06-28 长沙理工大学 A kind of image classification method improving subsidiary classification device GAN
CN110689086A (en) * 2019-10-08 2020-01-14 郑州轻工业学院 Semi-supervised high-resolution remote sensing image scene classification method based on generating countermeasure network
CN111046900A (en) * 2019-10-25 2020-04-21 重庆邮电大学 Semi-supervised generation confrontation network image classification method based on local manifold regularization
CN112132149A (en) * 2020-09-10 2020-12-25 武汉汉达瑞科技有限公司 Semantic segmentation method and device for remote sensing image
WO2021051519A1 (en) * 2019-09-18 2021-03-25 平安科技（深圳）有限公司 Recognition model training method and apparatus, fundus feature recognition method and apparatus, device and medium
US20210150281A1 (en) * 2019-11-14 2021-05-20 Nec Laboratories America, Inc. Domain adaptation for semantic segmentation via exploiting weak labels

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180336471A1 (en) * 2017-05-19 2018-11-22 Mehdi Rezagholizadeh Semi-supervised regression with generative adversarial networks
CN109948660A (en) * 2019-02-26 2019-06-28 长沙理工大学 A kind of image classification method improving subsidiary classification device GAN
WO2021051519A1 (en) * 2019-09-18 2021-03-25 平安科技（深圳）有限公司 Recognition model training method and apparatus, fundus feature recognition method and apparatus, device and medium
CN110689086A (en) * 2019-10-08 2020-01-14 郑州轻工业学院 Semi-supervised high-resolution remote sensing image scene classification method based on generating countermeasure network
CN111046900A (en) * 2019-10-25 2020-04-21 重庆邮电大学 Semi-supervised generation confrontation network image classification method based on local manifold regularization
US20210150281A1 (en) * 2019-11-14 2021-05-20 Nec Laboratories America, Inc. Domain adaptation for semantic segmentation via exploiting weak labels
CN112132149A (en) * 2020-09-10 2020-12-25 武汉汉达瑞科技有限公司 Semantic segmentation method and device for remote sensing image

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
CHAOYUE WANG ET AL: "Evolutionary Generative Adversarial Networks", 《ARXIV》 *
KIHYUK SOHN ET AL: "FixMatch: Simplifying Semi-Supervised Learning", 《34TH CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS (NEURIPS 2020)》 *
SHUQIANG WANG ET AL: "Diabetic Retinopathy Diagnosis Using Multichannel Generative Adversarial Network With Semisupervision" *
TIM SALIMANS ET AL: "Improved Techniques for Training GANs", 《ARXIV》 *
XINTAO WANG ET AL: "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks" *
吕昊远 等: "半监督深度学习图像分类方法研究综述", 《计算机科学与探索》 *
马力 等: "嵌入自注意力机制的美学特征图像生成方法", 《计算机科学与探索》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114155365A (en) * 2022-02-07 2022-03-08 北京航空航天大学杭州创新研究院 Model training method, image processing method and related device
CN115130599A (en) * 2022-07-05 2022-09-30 中国矿业大学 Semi-supervision method for strip mine card state recognition under time series GAN data enhancement
CN115130599B (en) * 2022-07-05 2024-03-26 中国矿业大学 Semi-supervision method for identifying open pit mine card state under time sequence GAN data enhancement
WO2024016945A1 (en) * 2022-07-19 2024-01-25 马上消费金融股份有限公司 Training method for image classification model, image classification method, and related device

Also Published As

Publication number Publication date
CN113240655B (en) 2023-04-07

Similar Documents

Publication Publication Date Title
US10482603B1 (en) Medical image segmentation using an integrated edge guidance module and object segmentation network
CN110399929B (en) Fundus image classification method, fundus image classification apparatus, and computer-readable storage medium
Chakrabarty A deep learning method for the detection of diabetic retinopathy
Yu et al. Image quality classification for DR screening using deep learning
US20190191988A1 (en) Screening method for automated detection of vision-degenerative diseases from color fundus images
CN113240655B (en) Method, storage medium and device for automatically detecting type of fundus image
dos Santos Ferreira et al. Convolutional neural network and texture descriptor-based automatic detection and diagnosis of glaucoma
CN113906472A (en) Method and system for selecting embryos
CN114846507A (en) Method and system for non-invasive gene detection using Artificial Intelligence (AI) models
CN111368672A (en) Construction method and device for genetic disease facial recognition model
JP2020535529A (en) Biological detection methods and devices, electronic devices, and computer-readable storage media
CN113128271A (en) Counterfeit detection of face images
CN114287878A (en) Diabetic retinopathy focus image identification method based on attention model
CN113793345B (en) Medical image segmentation method and device based on improved attention module
Saleh et al. Transfer learning‐based platform for detecting multi‐classification retinal disorders using optical coherence tomography images
CN113012163A (en) Retina blood vessel segmentation method, equipment and storage medium based on multi-scale attention network
CN113011450B (en) Training method, training device, recognition method and recognition system for glaucoma recognition
CN116364292A (en) Method, device, equipment and storage medium for predicting prognosis of thyroid eye disease
US20220301153A1 (en) Diabetic retinopathy detection using machine learning
Tian et al. Learning discriminative representations for fine-grained diabetic retinopathy grading
CN112016592A (en) Domain adaptive semantic segmentation method and device based on cross domain category perception
CN115689981A (en) Lung image detection method and device based on information fusion and storage medium
CN114140381A (en) Vitreous opacity grading screening method and device based on MDP-net
CN111651626B (en) Image classification method, device and readable storage medium
Deepa et al. Pre-Trained Convolutional Neural Network for Automated Grading of Diabetic Retinopathy

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant