CN110335254B - Fundus image regionalization deep learning method, device and equipment and storage medium - Google Patents
Fundus image regionalization deep learning method, device and equipment and storage medium Download PDFInfo
- Publication number
- CN110335254B CN110335254B CN201910498239.9A CN201910498239A CN110335254B CN 110335254 B CN110335254 B CN 110335254B CN 201910498239 A CN201910498239 A CN 201910498239A CN 110335254 B CN110335254 B CN 110335254B
- Authority
- CN
- China
- Prior art keywords
- fundus image
- image
- region
- fundus
- template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30041—Eye; Retina; Ophthalmic
Abstract
The application relates to a fundus image regionalization deep learning method, a fundus image regionalization deep learning device, fundus image regionalization deep learning equipment and a storage medium, wherein the method comprises the following steps: acquiring a fundus image, and constructing a corresponding image template based on the acquired fundus image; wherein the different image modes represent different regions in the fundus image; selecting a corresponding current required template from the image templates, and processing the fundus image based on the selected current required template to obtain a processed fundus image; and inputting the processed fundus images into a pre-constructed neural network model, and training the neural network model. By constructing the image template, the fundus image used in the training of the neural network model is not the whole fundus image, but the fundus image is processed based on the selected current required template. Therefore, unnecessary interference factors are effectively eliminated, and the robustness and the generalization of the trained neural network model are finally improved.
Description
Technical Field
The present disclosure relates to the field of medical image processing technologies, and in particular, to a method, an apparatus, and a device for eye fundus image regionalization deep learning, and a storage medium.
Background
Deep learning is a data-driven technique, and is currently the preferred solution in the field of imaging. Among the medical images, the color fundus image has the characteristics of low complexity, convenient acquisition and easy diagnosis, and the characteristics enable the fundus image to obtain better classification and detection effects by using a deep learning technology. The existing disease classification and detection technology based on color fundus images basically adopts a deep learning technology and strongly depends on image quality, image quality and marking accuracy. In the related art, when the fundus image is classified and detected by adopting a deep learning technology, the whole image is mostly used as learning data for model learning, so that the interference in the model learning process is more, and the robustness and the generalization of the model are influenced.
Disclosure of Invention
In view of this, the present disclosure provides a method, an apparatus, a device, and a storage medium for eye fundus image regionalization deep learning, which can effectively avoid unnecessary interference in a model learning process and improve robustness and generalization of a model.
According to an aspect of the present disclosure, there is provided a fundus image regionalization deep learning method, including:
acquiring a fundus image, and constructing a corresponding image template based on the acquired fundus image;
wherein different ones of the image modes represent different regions in the fundus image;
selecting a corresponding current required template from the image templates, and processing the fundus image based on the selected current required template to obtain a processed fundus image;
and inputting the processed fundus images into a pre-constructed neural network model, and training the neural network model.
In one possible implementation, constructing a corresponding image template based on the acquired fundus images includes:
positioning and detecting the optic disc region and the macular region in the fundus image by adopting a target detection algorithm to obtain corresponding target detection results;
and performing structural analysis on the fundus image according to the target detection result to obtain the corresponding image template.
In one possible implementation, performing a structured analysis on the fundus image according to the target detection result includes:
when the target detection result shows that the optic disc region and the macular region are both visible, respectively moving the contour lines of the fundus images along the directions of two ends of a central connecting line of the center of the optic disc region and the macular region to obtain a corresponding first contour and a corresponding second contour;
wherein the first and second profiles are each tangent to an edge of the view area;
and generating the corresponding image template according to the intersection condition of the first contour and the second contour with the fundus image respectively.
In one possible implementation, performing a structured analysis on the fundus image according to the target detection result includes:
when the target detection result is that the optic disc region is visible and the yellow spot region is invisible, respectively moving the contour lines of the fundus images along the directions of two ends of a connecting line between the center of the optic disc region and the center of the fundus image to obtain a corresponding third contour and a corresponding fourth contour;
wherein the third and fourth profiles are each tangent to an edge of the view area;
and generating the corresponding image template according to the intersection condition of the third contour and the fourth contour with the fundus image respectively.
In one possible implementation, performing a structured analysis on the fundus image based on the target detection result includes:
and when the target detection result is that the macular region is visible and the optic disc region is invisible, performing region division on the fundus image according to the located macular region to generate the corresponding image template.
In one possible implementation, processing the fundus image based on the selected currently desired template includes:
and keeping the pixel value of the selected area of the fundus image corresponding to the current required template as the pixel value of the original image area, and setting the pixel value of the fundus image corresponding to the unselected image template to be zero.
In one possible implementation, the pre-constructed neural network model includes a base network and a plurality of sub-networks;
wherein the output end of the basic network is the input end of each sub-network;
the basic network comprises a plurality of convolution combination layers which are sequentially cascaded, and each sub-network comprises a cascaded convolution layer and a full connection layer.
According to another aspect of the present disclosure, there is also provided a fundus image regionalization deep learning apparatus including:
the image template construction module is configured to acquire a fundus image and construct a corresponding image template based on the acquired fundus image; wherein different ones of the image templates represent different regions in the fundus image;
a fundus image processing module configured to select a corresponding current required template from the image templates, and process the fundus image based on the selected current required template to obtain a processed fundus image;
and the network model training module is configured to input the processed fundus images to a pre-constructed neural network model and train the neural network model.
According to an aspect of the present disclosure, there is also provided a fundus image regionalization deep learning apparatus including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to implement any of the above-described fundus image regionalized depth learning methods when executing the executable instructions.
According to another aspect of the present disclosure, there is also provided a non-transitory computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the fundus image regionalized depth learning method of any one of the preceding.
Image templates representing different areas in the fundus images are constructed based on the acquired fundus images, then corresponding current required templates are selected from the image templates, and the fundus images are processed based on the selected current required templates to obtain processed fundus images. And finally, inputting the processed fundus image to a pre-constructed neural network model, and training the neural network model, so that the fundus image used in the training of the neural network model is not the whole fundus image, but the fundus image processed based on the selected current required template. Therefore, unnecessary interference factors are effectively eliminated, and the robustness and the generalization of the trained neural network model are finally improved.
Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
Fig. 1 illustrates a flowchart of a fundus image regionalized depth learning method of an embodiment of the present disclosure;
fig. 2 shows a flow design schematic diagram of a fundus image regionalization depth learning method according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram illustrating a rule of constructing an image template when a target detection result is that both the optic disc region and the macular region are visible in the fundus image regionalization depth learning method according to the embodiment of the present disclosure;
fig. 4 is a schematic diagram illustrating a rule of constructing an image template when a target detection result is that a optic disc region is visible but a macular region is invisible in a fundus image regionalization depth learning method according to an embodiment of the present disclosure;
fig. 5 is a schematic diagram illustrating a rule of constructing an image template when a target detection result is that a macular region is visible and a optic disc region is invisible in a fundus image regionalization depth learning method according to an embodiment of the present disclosure;
fig. 6 is a schematic network structure diagram of a base network in a neural network model constructed in the fundus image regionalization deep learning method according to the embodiment of the present disclosure;
fig. 7 is a schematic diagram showing a network structure of a sub-network in a neural network model constructed in the fundus image regionalization depth learning method according to the embodiment of the present disclosure;
fig. 8 shows a block diagram of a fundus image regionalization depth learning apparatus of an embodiment of the present disclosure;
fig. 9 illustrates a block diagram of a fundus image regionalization depth learning apparatus of an embodiment of the present disclosure.
Detailed Description
Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.
Fig. 1 illustrates a flowchart of a fundus image regionalization depth learning method of an embodiment of the present disclosure. Fig. 2 is a schematic flow design diagram illustrating a fundus image regionalization depth learning method according to an embodiment of the present disclosure. Referring to fig. 1 and 2, a fundus image regionalization depth learning method according to an embodiment of the present disclosure includes: step S100, fundus images are acquired, and corresponding image templates are constructed on the basis of the acquired fundus images. Here, it should be noted that the number of image templates may be plural, and different image templates represent different regions in the fundus image.
Such as: the image template may include at least one of a disc region template, a macular region template, an upper vascular arch region template, a lower vascular arch region template, a peri-macular vascular region template, and a paradiscal paranasal region template. Wherein the optic disc region template corresponds to an optic disc region in the fundus image. The macular region template corresponds to a macular region in the fundus image. The upper vascular arch region template corresponds to a partial region in the fundus image at a position above the optic disc region. The lower vascular arch area template corresponds to a partial area of the fundus image at a position below the optic disc area. The macular peripheral blood vessel region template corresponds to a partial region on the peripheral side of the macular region in the fundus image. The paradisk nose side area template corresponds to a partial area of the fundus image on the side of the paradisk area far from the macula lutea area.
Meanwhile, it should also be noted that the number and type of image templates constructed may differ from one fundus image to another, and are not particularly limited herein.
After the step S100 is executed and the corresponding image template is constructed, the step S200 may be executed, the corresponding current required template is selected from the image templates, and the fundus image is processed based on the selected current required template to obtain the processed fundus image. And step S300 is further executed, the processed fundus images are input into a pre-constructed neural network model, and the neural network model is trained.
Therefore, according to the fundus image regionalization depth learning method disclosed by the embodiment of the disclosure, the image templates representing different regions in the fundus image are constructed based on the acquired fundus image, then the corresponding current required template is selected from the image templates, and the fundus image is processed based on the selected current required template to obtain the processed fundus image. And finally, inputting the processed fundus image to a pre-constructed neural network model, and training the neural network model, so that the fundus image used in the training of the neural network model is not the whole fundus image, but the fundus image processed based on the selected current required template. Therefore, unnecessary interference factors are effectively eliminated, and the robustness and the generalization of the trained neural network model are finally improved.
In one possible implementation, in step S100, constructing a corresponding image template based on the acquired fundus images may be implemented in the following manner.
Firstly, a target detection algorithm is adopted to carry out positioning detection on a optic disc region and a macula lutea region in a fundus image, and a corresponding target detection result is obtained. Here, it should be noted that, in general, the optic disc region and the macular region in the fundus image are obvious, and in order to ensure the accuracy of target detection, when the optic disc region and the macular region in the fundus image are detected in a positioning manner by using a target detection algorithm, the optic disc region and the macular region may be detected by using a fast Rcnn model based on deep learning. The fast Rcnn model based on deep learning is a commonly used image processing network model in the field, and is not described herein again.
And after the corresponding target detection result is obtained, carrying out structural analysis on the fundus image according to the target detection result to obtain a corresponding image template.
Here, it should be noted that, when the target detection algorithm is used to perform the positioning detection on different fundus images, the obtained target detection result may be different. Therefore, when the structured analysis is performed on the fundus image based on different target detection results, the obtained image template may also be different.
For example, the target detection result may include four types, that is, both the disc region and the macular region are visible (i.e., both the disc region and the macular region are localized and detected), the disc region is visible but the macular region is invisible (i.e., the disc region in the fundus image is localized and detected and the macular region is not localized and detected), the disc region is invisible but the macular region is visible (i.e., the macular region in the fundus image is localized and detected and the disc region is not localized and detected), and neither the disc region nor the macular region is visible (i.e., neither the disc region nor the macular region in the fundus image is localized and detected).
When the target detection result indicates that neither the optic disc region nor the macular region is visible, the currently acquired fundus image is invalid. That is, the currently acquired fundus image cannot be used as training sample data for training learning of the neural network model. Therefore, the result that neither the optic disc area nor the macular area is visible can be directly output, the eye fundus image is discarded, and the next eye fundus image is directly obtained for positioning detection.
When the target detection result shows that both the optic disc region and the macular region are visible, the currently acquired fundus image is effective, and the fundus image can be used as sample data to train a neural network model. Therefore, the fundus image can be subjected to structural analysis according to the target detection result so as to obtain a corresponding image template.
When the target detection result is that both the optic disc region and the macular region are visible, the structured analysis of the fundus image can be realized in the following manner.
And when the target detection result shows that the optic disc area and the macular area are visible, respectively moving the contour lines of the fundus images along the directions of two ends of a central connecting line of the center of the optic disc area and the center of the macular area so as to obtain a first contour and a second contour. Wherein the first and second profiles are each tangent to an edge of the view area.
And then, generating a corresponding image template according to the intersection condition of the first contour and the second contour and the fundus image respectively.
Here, it should be noted that the outline of the fundus image may be circular or rectangular. And are not limited herein. Accordingly, the shape of the edge of the optic disc region and the macular region may be either circular or rectangular. When the optic disc region and the macula lutea region are located by performing the location detection on the fundus image by using the aforementioned target detection algorithm (e.g., fast Rcnn model), the edges of the optic disc region and the macula lutea region are usually defined by rectangular frames. In the fundus image regionalization depth learning method according to the embodiment of the present disclosure, for convenience of calculation, a rectangular frame may be fitted to a circle by a fitting manner, so that the edge (i.e., the contour) of the optic disc region and the macular region detected by positioning is circular.
To more clearly describe the rule of constructing the image template when the target detection result indicates that both the optic disc region and the macular region are visible, the following description will take an example in which the contour line of the fundus image is a circle and the edges of the detected optic disc region and macular region are positioned to form a rectangular frame.
Referring to fig. 3, when the target detection result is that both the optic disc region and the macular region are visible, with both ends of the connecting line between the center of the optic disc region and the center of the macular region as the moving direction, the contour lines of the fundus image are moved to both ends of the connecting line, respectively, to move out two dotted outlines (i.e., a first outline and a second outline). Wherein, the moved distance is respectively moved until the two sides of the edge of the optic disc area (rectangle or circle) are tangent left and right. That is, the first profile is tangent to one side of the edge of the view area and the second profile is tangent to the other side of the edge of the view area.
And then, generating a corresponding image template according to the intersection condition of the first contour and the second contour and the fundus image respectively. That is, referring to fig. 3, a corresponding optic disc region template (i.e., region 1) is generated corresponding to the optic disc region in the fundus image; generating a corresponding macular region template (i.e., region 2) corresponding to the macular region in the fundus image; corresponding to the first contour, the second contour and the contour of the fundus image, generating a corresponding upper vascular arch area template (namely, area 3) in an area above the optic disc area; corresponding to the area enclosed by the first contour, the second contour and the contour of the fundus image, a corresponding lower vascular arch area template (namely, area 4) is generated in the area below the optic disc area; in the region corresponding to the intersection part of the first contour and the fundus image, the other partial region except the macular region generates a corresponding macular peripheral vascular region template (i.e., region 5); a corresponding paradiscal nose-side region template (i.e., region 6) is generated corresponding to a region of the intersection of the second contour and the fundus image.
Further, when the target detection result is that the optic disc region is visible and the macular region is invisible, the structured analysis is performed on the fundus image at the time, and the method can be realized by moving the contour lines of the fundus image along the directions of the two ends of the connecting line between the center of the optic disc region and the center of the fundus image to obtain the corresponding third contour and fourth contour. Wherein the third and fourth profiles are each tangent to an edge of the view area. And then, after a third contour and a fourth contour are obtained by moving the contour of the fundus image, generating a corresponding image template according to the intersection condition of the third contour and the fourth contour with the fundus image respectively.
Here, it should be noted that the shape of the contour line of the fundus image and the edge shape of the visual field may also be set according to the actual situation. Such as: the shape of the outline of the fundus image may be circular or rectangular. The edge shape of the view area may be a circle or a rectangular frame. None of them is specifically limited.
For example, the outline of the fundus image is a circle, and the edge of the fundus region is a rectangular frame. When the target detection result is that the optic disc region is visible and the macular region is invisible, referring to fig. 4, the contour lines of the fundus image are respectively moved along the two ends of the connecting line between the center of the optic disc region and the center of the fundus image until the third contour and the fourth contour obtained by movement are respectively tangent to the edge of the optic disc region.
And generating a corresponding image template according to the intersection condition of the third contour and the fourth contour with the fundus image respectively. That is, referring to fig. 4, a corresponding optic disc region template (i.e., region 1') is generated corresponding to the optic disc region in the fundus image; corresponding to the first contour, the second contour and the region surrounded by the contour of the eyeground image, generating a corresponding upper vascular arch region template (namely, region 3') in the region above the optic disc region; corresponding to the area enclosed by the first contour, the second contour and the contour of the fundus image, a corresponding lower vascular arch area template (i.e., area 4') is generated in the area below the optic disc area; corresponding to the area of the intersection part of the first contour and the fundus image, generating a corresponding template (namely, an area 5') of the macular area and the peripheral blood vessel area of the macula; a corresponding paradiscal nasal region template (i.e., region 6') is generated corresponding to the region of intersection of the second contour with the fundus image.
Further, when the target detection result is that the macular region is visible and the optic disc region is invisible, the fundus image can be divided into regions according to the located frontal macular region, and a corresponding image template is generated.
That is, referring to fig. 5, when the target detection result is that the macular region is visible and the optic disc region is not visible, two kinds of templates may be generated at this time. That is, the macular regions are directly corresponded to generate the respective macular region templates (i.e., region 2'), and the remaining regions (i.e., the regions other than the macular regions in the fundus image) are corresponded to generate the remaining region templates (i.e., region 7).
Here, it should be noted that after the corresponding image templates are constructed and generated based on different target detection results, when the number of the image templates is multiple, the multiple image templates may be recorded and stored in a form of a template combination list, so that a currently required template can be found more quickly when the image templates are subsequently selected, and the currently required template is selected more efficiently.
After the corresponding image template is constructed in any of the above manners, in step S200, the currently required template is selected from the constructed image templates, and the fundus image is processed based on the selected currently required template.
Here, it should be noted that when the currently required template is selected from the constructed image templates, the selection may be performed according to the current detection requirement. That is, in general, different fundus disease types require attention to different regions in a fundus image. Such as: the glaucoma disease needs to be judged by combining the optic disc area, the upper vascular arch area above the optic disc area and the lower vascular arch area below the optic disc area. Age-related macular degeneration needs to be judged according to the macular region. The diabetic retinopathy type needs to be determined in combination with other regions than the optic disc region in the fundus image.
Therefore, according to the current detection requirement, the current required template is selected from the constructed image templates, and the fundus image is processed based on the selected current required template, so that the fundus image used in the final training of the neural network model has more pertinence.
Meanwhile, it should be noted that the current detection requirement may be determined according to the function of the currently trained neural network model. Such as: when the function of the currently trained neural network model is to detect the glaucoma disease species, the current detection requirement can be determined to be the detection requirement of the glaucoma disease species, and therefore, according to the determined detection requirement of the glaucoma disease species, the corresponding combination of the three templates, namely the optic disc area template, the upper vascular arch area template and the lower vascular arch area template, is selected from the constructed and generated image template. And then, processing the fundus images based on the selected template combination, so that the processed fundus images are used as training samples to train and learn the neural network model.
In one possible implementation, the processing of the fundus image based on the selected currently desired template may be implemented in the following manner.
That is, the pixel value of the region of the fundus image corresponding to the selected currently required template is held as the pixel value of the original image region, and the pixel values of the fundus images corresponding to the unselected image templates are set to zero.
For example, and again using the above-described glaucoma disease detection as an example, the combination of templates selected at this time includes a optic disc region template, an upper vascular arch region template, and a lower vascular arch region template. Thus, when the fundus image is processed based on the three selected image templates, the processed fundus image can be obtained by setting all the pixel values of the other regions except the optic disc region, the upper vascular arch region, and the lower vascular arch region to zero while keeping the pixel values of the optic disc region, the upper vascular arch region, and the lower vascular arch region in the fundus image unchanged. And then, inputting the processed fundus image serving as training sample data into a pre-constructed neural network model for training.
It should also be noted that, in the fundus image regionalization depth learning method of the embodiment of the present disclosure, the neural network model constructed in advance may include a base network and a plurality of sub-networks. Referring to fig. 2, the base network serves as a shared network of a plurality of sub-networks, and the output terminals of the base network are input terminals of the respective sub-networks. That is, the outputs of the base network are all inputs to each sub-network. It should be noted that, in a possible implementation manner, the number of the base networks is one, and the number of the sub-networks can be flexibly set according to actual requirements (for example, the number of disease types to be detected). Each subnetwork is used to detect one type of disease species.
Referring to fig. 6, in a possible implementation, the base network includes a plurality of convolution combination layers cascaded in sequence, and is mainly responsible for feature extraction of an image. In the basic network of the embodiment of the present disclosure, the convolution combination layer may include a convolution layer, a pooling layer, a batch normalization layer, a residual connection, and other mature network connection modes. And will not be described in detail herein.
Referring to fig. 7, in a possible implementation manner, the sub-networks have a shallow depth, and each sub-network may include a convolutional layer and a fully-connected layer in cascade, and is mainly used for final discrimination after the basic network feature extraction. The number of categories of the discrimination layer may be different based on different types of disease detection (discrimination).
It should be noted that, in the fundus image regionalization depth learning method of the embodiment of the present disclosure, training of the entire neural network model is mainly divided into two stages:
training the basic network:
initial parameters of the base network were obtained from a fine training of the public dataset ImageNet and fine-tuned using fundus images. Such as: fine-tuning can be performed using multi-classification detection of fundus images of diabetic retinopathy. Based on the transfer learning theory, the network parameters of the base network after fine tuning are used for classification tasks of different fundus images, and the network parameters of the base network are determined and cannot be changed along with the training of the following sub-networks.
Training of each subnetwork:
the base network is connected in series with each sub-network, one shared by all sub-networks, and each sub-network is trained with a corresponding fundus image (i.e., a fundus image processed based on the selected current desired template). And during training, only updating the network parameters of the sub-networks, and finally determining the network parameters of each sub-network. In addition, when training the sub-networks, each sub-network corresponds to a respective image template combination, which can be obtained from the template combination list.
Therefore, according to the fundus image regionalization depth learning method disclosed by the embodiment of the disclosure, the image template is constructed, and the model learning is combined with the current detection requirement, so that the interference of a non-interested region in the model learning process is avoided, and the robustness and the generalization of the model are enhanced. In addition, in the fundus image regional deep learning method according to the embodiment of the disclosure, by constructing a neural network model formed by combining a base network and a plurality of sub-networks, the corresponding network model is determined and customized for each disease type detection, so that not only can the model learning efficiency be effectively improved, but also the regional interpretation effect of the fundus image can be realized. Furthermore, the detection, the discrimination and the classification of multiple disease types of the fundus images are achieved by sharing one basic network by a plurality of sub-networks, the function of multi-task output is realized, the complexity and the redundancy of a network model are greatly reduced, and the condition that the network structure is too large is avoided.
Correspondingly, based on any one of the fundus image regionalization depth learning methods, the disclosure also provides a fundus image regionalization depth learning device. Since the working principle of the fundus image regionalization depth learning device of the embodiment of the present disclosure is the same as or similar to that of the fundus image regionalization depth learning method of the embodiment of the present disclosure, repeated descriptions are omitted.
Referring to fig. 8, the fundus image regionalization depth learning apparatus 100 of the embodiment of the present disclosure may include an image template construction module 110, a fundus image processing module 120, and a network model training module 130.
The image template construction module 110 is configured to acquire a fundus image and construct a corresponding image template based on the acquired fundus image; wherein the different image templates represent different regions in the fundus image.
And the fundus image processing module 120 is configured to select a corresponding current required template from the image templates, and process the fundus image based on the selected current required template to obtain a processed fundus image.
And a network model training module 130 configured to input the processed fundus image to a pre-constructed neural network model and train the neural network model.
In one possible implementation, the image template building module 110 may include a localization detection sub-module and a structure analysis sub-module (not shown in the figures). The positioning detection submodule is configured to perform positioning detection on an optic disc region and a macular region in the fundus image by adopting a target detection algorithm to obtain a corresponding target detection result; and the structure analysis submodule is configured to perform structural analysis on the fundus image according to the target detection result to obtain a corresponding image template.
In one possible implementation, the structure analysis submodule may include a first moving unit and a first generating unit (not shown in the figure). And the first moving unit is configured to move the contour lines of the fundus image along the directions of two ends of a central connecting line of the center of the optic disc region and the central of the macular region respectively to obtain a corresponding first contour and a corresponding second contour when the target detection result shows that the optic disc region and the macular region are both visible. Wherein the first and second profiles are each tangent to an edge of the view area. And a first generating unit configured to generate a corresponding image template according to the intersection of the first contour and the second contour with the fundus image, respectively.
In a possible implementation manner, the structure analysis submodule may further include a second moving unit and a second generating unit (not shown in the figure). And the second moving unit is configured to move the contour lines of the fundus image along the directions of two ends of a connecting line between the center of the optic disc area and the center of the fundus image respectively to obtain a corresponding third contour and a corresponding fourth contour when the target detection result shows that the optic disc area is visible and the macular area is invisible. Wherein the third and fourth profiles are each tangent to an edge of the view area. And a second generating unit configured to generate a corresponding image template according to the intersection of the third contour and the fourth contour with the fundus image, respectively.
In a possible implementation manner, the structure analysis sub-module may further include a region dividing unit (not shown in the figure). And the region dividing unit is configured to perform region division on the fundus image according to the located macular region to generate a corresponding image template when the target detection result is that the macular region is visible and the optic disc region is invisible.
In one possible implementation, the fundus image processing module 120 includes a pixel value setting sub-module (not shown in the drawings). And the pixel value setting submodule is configured to keep the pixel value of the area of the fundus image corresponding to the selected current required template as the pixel value of the original image area, and set the pixel value of the fundus image corresponding to the unselected image template to be zero.
In a possible implementation manner, a network model building module (not shown in the figure) may be further included. Wherein the network model construction module is configured to construct a neural network model. Wherein the constructed neural network model comprises a basic network and a plurality of sub-networks; the output end of the basic network is the input end of each sub-network; . The basic network comprises a plurality of convolution combination layers which are sequentially cascaded, and each sub-network comprises a cascaded convolution layer and a full connection layer.
Further, according to another aspect of the present disclosure, there is also provided a fundus image regionalization deep learning apparatus. Referring to fig. 9, the fundus image regionalization depth learning apparatus 200 according to the embodiment of the present disclosure includes a processor 210 and a memory 220 for storing instructions executable by the processor 210. Wherein the processor 210 is configured to implement any of the fundus image regionalized depth learning methods described above when executing the executable instructions.
Here, it should be noted that the processor 210 may be a general-purpose processor, such as: the CPU (Central Processing Unit/Processor) can also be an artificial intelligence Processor. Wherein, an artificial intelligence processor refers to a processor (IPU) for performing artificial intelligence operations, such as: the system comprises one or a combination of a GPU (Graphics Processing Unit), a Neural-Network Processing Unit (NPU), a Digital Signal Processing (DSP) and a Field Programmable Gate Array (FPGA) chip. The present disclosure is not limited to a particular type of artificial intelligence processor.
It should also be noted that the number of processors 210 may be one or more. Meanwhile, in the fundus image processing apparatus 200 of the embodiment of the present disclosure, an input device 230 and an output device 240 may also be included. The processor 210, the memory 220, the input device 230, and the output device 240 may be connected via a bus, or may be connected via other methods, which is not limited in detail herein.
The memory 220, which is a computer-readable storage medium, may be used to store software programs, computer-executable programs, and various modules, such as: the fundus image regionalization depth learning method according to the embodiment of the present disclosure corresponds to a program or a module. The processor 210 executes various functional applications and data processing of the fundus image processing apparatus 200 by executing software programs or modules stored in the memory 220.
The input device 230 may be used to receive an input number or signal. Wherein the signal may be a key signal generated in connection with user settings and function control of the device/terminal/server. The output device 240 may include a display device such as a display screen.
According to another aspect of the present disclosure, there is also provided a non-transitory computer readable storage medium having stored thereon computer program instructions which, when executed by the processor 210, implement the fundus image regionalized depth learning method as described in any one of the preceding.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (6)
1. A fundus image regionalization deep learning method is characterized by comprising the following steps:
acquiring a fundus image, and constructing a corresponding image template based on the acquired fundus image;
wherein different ones of the image templates represent different regions in the fundus image;
selecting a corresponding current required template from the image templates, and processing the fundus image based on the selected current required template to obtain a processed fundus image;
inputting the processed fundus images into a pre-constructed neural network model, and training the neural network model;
constructing a corresponding image template based on the acquired fundus images, comprising:
positioning and detecting the optic disc region and the macular region in the fundus image by adopting a target detection algorithm to obtain corresponding target detection results;
according to the target detection result, performing structural analysis on the fundus image to obtain a corresponding image template;
wherein, according to the target detection result, performing structural analysis on the fundus image comprises:
when the target detection result shows that the optic disc region and the macular region are both visible, respectively moving the contour lines of the fundus images along the directions of two ends of a central connecting line of the center of the optic disc region and the macular region to obtain a corresponding first contour and a corresponding second contour;
wherein the first and second profiles are each tangent to an edge of the view area;
generating a corresponding image template according to the intersection condition of the first contour and the second contour with the fundus image respectively;
when the target detection result is that the optic disc region is visible and the yellow spot region is invisible, respectively moving the contour lines of the fundus images along the directions of two ends of a connecting line between the center of the optic disc region and the center of the fundus image to obtain a corresponding third contour and a corresponding fourth contour;
wherein the third and fourth profiles are each tangent to an edge of the view area;
generating a corresponding image template according to the intersection condition of the third contour and the fourth contour with the fundus image respectively;
and when the target detection result is that the macular region is visible and the optic disc region is invisible, performing region division on the fundus image according to the located macular region to generate the corresponding image template.
2. The method of claim 1, wherein processing the fundus image based on the selected current desired template comprises:
and keeping the pixel value of the selected area of the fundus image corresponding to the current required template as the pixel value of the original image area, and setting the pixel value of the fundus image corresponding to the unselected image template to be zero.
3. The method of claim 1, wherein the pre-constructed neural network model comprises a base network and a plurality of sub-networks;
wherein the output end of the basic network is the input end of each sub-network;
the basic network comprises a plurality of convolution combination layers which are sequentially cascaded, and each sub-network comprises a cascaded convolution layer and a full connection layer.
4. An eye fundus image regionalization deep learning device, comprising:
the image template construction module is configured to acquire a fundus image and construct a corresponding image template based on the acquired fundus image; wherein different ones of the image templates represent different regions in the fundus image;
a fundus image processing module configured to select a corresponding current required template from the image templates, and process the fundus image based on the selected current required template to obtain a processed fundus image;
the network model training module is configured to input the processed fundus images to a pre-constructed neural network model and train the neural network model;
wherein the image template construction module, when configured to acquire a fundus image and construct a corresponding image template based on the acquired fundus image, comprises:
positioning and detecting the optic disc region and the macular region in the fundus image by adopting a target detection algorithm to obtain corresponding target detection results;
according to the target detection result, performing structural analysis on the fundus image to obtain a corresponding image template;
wherein, according to the target detection result, performing structural analysis on the fundus image comprises:
when the target detection result shows that the optic disc region and the macular region are both visible, respectively moving the contour lines of the fundus images along the directions of two ends of a central connecting line of the center of the optic disc region and the macular region to obtain a corresponding first contour and a corresponding second contour;
wherein the first and second profiles are each tangent to an edge of the view area;
generating a corresponding image template according to the intersection condition of the first contour and the second contour with the fundus image respectively;
when the target detection result is that the optic disc region is visible and the yellow spot region is invisible, respectively moving the contour lines of the fundus images along the directions of two ends of a connecting line between the center of the optic disc region and the center of the fundus image to obtain a corresponding third contour and a corresponding fourth contour;
wherein the third and fourth profiles are each tangent to an edge of the view area;
generating a corresponding image template according to the intersection condition of the third contour and the fourth contour with the fundus image respectively;
and when the target detection result is that the macular region is visible and the optic disc region is invisible, performing region division on the fundus image according to the located macular region to generate the corresponding image template.
5. An fundus image regionalization deep learning apparatus, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to carry out the method of any one of claims 1 to 3 when executing the executable instructions.
6. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, implement the method of any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910498239.9A CN110335254B (en) | 2019-06-10 | 2019-06-10 | Fundus image regionalization deep learning method, device and equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910498239.9A CN110335254B (en) | 2019-06-10 | 2019-06-10 | Fundus image regionalization deep learning method, device and equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110335254A CN110335254A (en) | 2019-10-15 |
| CN110335254B true CN110335254B (en) | 2021-07-27 |
Family
ID=68140886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910498239.9A Active CN110335254B (en) | 2019-06-10 | 2019-06-10 | Fundus image regionalization deep learning method, device and equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110335254B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111292296A (en) * | 2020-01-20 | 2020-06-16 | 京东方科技集团股份有限公司 | Training set acquisition method and device based on eye recognition model |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105243669A (en) * | 2015-10-15 | 2016-01-13 | 四川和生视界医药技术开发有限公司 | Method for automatically identifying and distinguishing eye fundus images |
| CN107209933A (en) * | 2014-08-25 | 2017-09-26 | 新加坡科技研究局 | For assessing retinal images and the method and system of information being obtained from retinal images |
| CN107256410A (en) * | 2017-05-26 | 2017-10-17 | 北京郁金香伙伴科技有限公司 | To the method and device of class mirror image image classification |
| CN108520522A (en) * | 2017-12-31 | 2018-09-11 | 南京航空航天大学 | Retinal fundus images dividing method based on the full convolutional neural networks of depth |
| CN108596895A (en) * | 2018-04-26 | 2018-09-28 | 上海鹰瞳医疗科技有限公司 | Eye fundus image detection method based on machine learning, apparatus and system |
| CN108717696A (en) * | 2018-05-16 | 2018-10-30 | 上海鹰瞳医疗科技有限公司 | Macula lutea image detection method and equipment |
| CN109199322A (en) * | 2018-08-31 | 2019-01-15 | 福州依影健康科技有限公司 | A kind of macula lutea detection method and a kind of storage equipment |
| CN109325942A (en) * | 2018-09-07 | 2019-02-12 | 电子科技大学 | Eye fundus image Structural Techniques based on full convolutional neural networks |
| CN109662686A (en) * | 2019-02-01 | 2019-04-23 | 北京致远慧图科技有限公司 | A kind of fundus flavimaculatus localization method, device, system and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5582772B2 (en) * | 2009-12-08 | 2014-09-03 | キヤノン株式会社 | Image processing apparatus and image processing method |
-
2019
- 2019-06-10 CN CN201910498239.9A patent/CN110335254B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107209933A (en) * | 2014-08-25 | 2017-09-26 | 新加坡科技研究局 | For assessing retinal images and the method and system of information being obtained from retinal images |
| CN105243669A (en) * | 2015-10-15 | 2016-01-13 | 四川和生视界医药技术开发有限公司 | Method for automatically identifying and distinguishing eye fundus images |
| CN107256410A (en) * | 2017-05-26 | 2017-10-17 | 北京郁金香伙伴科技有限公司 | To the method and device of class mirror image image classification |
| CN108520522A (en) * | 2017-12-31 | 2018-09-11 | 南京航空航天大学 | Retinal fundus images dividing method based on the full convolutional neural networks of depth |
| CN109598733A (en) * | 2017-12-31 | 2019-04-09 | 南京航空航天大学 | Retinal fundus images dividing method based on the full convolutional neural networks of depth |
| CN108596895A (en) * | 2018-04-26 | 2018-09-28 | 上海鹰瞳医疗科技有限公司 | Eye fundus image detection method based on machine learning, apparatus and system |
| CN108717696A (en) * | 2018-05-16 | 2018-10-30 | 上海鹰瞳医疗科技有限公司 | Macula lutea image detection method and equipment |
| CN109199322A (en) * | 2018-08-31 | 2019-01-15 | 福州依影健康科技有限公司 | A kind of macula lutea detection method and a kind of storage equipment |
| CN109325942A (en) * | 2018-09-07 | 2019-02-12 | 电子科技大学 | Eye fundus image Structural Techniques based on full convolutional neural networks |
| CN109662686A (en) * | 2019-02-01 | 2019-04-23 | 北京致远慧图科技有限公司 | A kind of fundus flavimaculatus localization method, device, system and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110335254A (en) | 2019-10-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Al-Bander et al. | Multiscale sequential convolutional neural networks for simultaneous detection of fovea and optic disc | |
| CN108198184B (en) | Method and system for vessel segmentation in contrast images | |
| CN111046717B (en) | Fundus image macula lutea center positioning method, fundus image macula lutea center positioning device, electronic equipment and storage medium | |
| CN110349147B (en) | Model training method, fundus macular region lesion recognition method, device and equipment | |
| CN108022251B (en) | Method and system for extracting central line of tubular structure | |
| Alhussein et al. | An unsupervised retinal vessel segmentation using hessian and intensity based approach | |
| CN108198185B (en) | Segmentation method and device for fundus focus image, storage medium and processor | |
| CN108416776A (en) | Image-recognizing method, pattern recognition device, computer product and readable storage medium storing program for executing | |
| JP7083037B2 (en) | Learning device and learning method | |
| CN112132817A (en) | Retina blood vessel segmentation method for fundus image based on mixed attention mechanism | |
| CN110428421A (en) | Macula lutea image region segmentation method and apparatus | |
| CN110263755B (en) | Eye ground image recognition model training method, eye ground image recognition method and eye ground image recognition device | |
| CN111882566B (en) | Blood vessel segmentation method, device, equipment and storage medium for retina image | |
| CN111242933B (en) | Retinal image artery and vein classification device, apparatus, and storage medium | |
| CN110363760B (en) | Computer system for recognizing medical images | |
| JP2019192215A (en) | 3d quantitative analysis of retinal layers with deep learning | |
| Sedai et al. | Multi-stage segmentation of the fovea in retinal fundus images using fully convolutional neural networks | |
| US20220044404A1 (en) | Deriving target data from selected brain data | |
| CN109919915A (en) | Retinal fundus images abnormal area detection method and equipment based on deep learning | |
| CN115953393B (en) | Intracranial aneurysm detection system, device and storage medium based on multitask learning | |
| CN108876770B (en) | Fundus multispectral image joint registration method and system | |
| CN110796135A (en) | Target positioning method and device, computer equipment and computer storage medium | |
| Kumar et al. | Image processing in diabetic related causes | |
| CN107146231A (en) | Retinal image bleeding area segmentation method and device and computing equipment | |
| CN110335254B (en) | Fundus image regionalization deep learning method, device and equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |