CN114372985A - Diabetic retinopathy focus segmentation method and system adapting to multi-center image - Google Patents

Abstract

The invention provides a diabetic retinopathy focus segmentation method adapting to a multi-center image, which comprises the steps of constructing a multi-center data set; training a cycleGAN model according to the multi-center data set to generate a corresponding cycleGAN model; establishing simulation data sets corresponding to DR focus segmentation data sets of different target centers by using a trained CycleGAN model; constructing a DR focus segmentation model for training to obtain a DR focus segmentation model which is suitable for the eye fundus color photograph image style of a plurality of target center DR focus segmentation data sets, and detecting and evaluating the DR focus segmentation model to obtain a DR focus segmentation model meeting the requirements; and inputting the fundus color photograph image to be predicted into the trained corresponding DR focus segmentation model, and outputting a final segmentation result. The invention also provides a lesion segmentation system, which only uses a single-center data set containing detailed lesion labels and a multi-center data set without any label to complete the training of a DR lesion segmentation model, so that the DR lesion segmentation model has a good lesion segmentation function on fundus color photograph images of a target center.

Description

Diabetic retinopathy focus segmentation method and system adapting to multi-center image

Technical Field

The invention relates to the technical field of medical image segmentation, in particular to a diabetic retinopathy focus segmentation method and system suitable for a multi-center image.

Background

Diabetic Retinopathy (hereinafter, referred to as DR) is a common blindness-causing eye disease, is one of the most common chronic complications of Diabetic patients, is the most rapidly growing cause of blindness, and approximately 4.15 hundred million Diabetic patients worldwide have blindness risks. DR has no obvious clinical symptoms in the early stage, and if fundus examination can be performed regularly at the initial stage of onset, the risk of blindness due to DR can be significantly reduced. In the clinical diagnosis of DR, the severity of DR is examined and graded by observing the presence of one or more lesions, such as microaneurysms, hemorrhage, hard exudates, soft exudates, etc., on the fundus image. The DR focus segmentation system based on the deep learning method can display the type and position information of the focus in the fundus image, and is beneficial to improving the efficiency of DR clinical diagnosis.

However, in clinical applications, the DR lesion segmentation system based on the deep learning method is often trained on data of one central hospital (single center) and then deployed to multiple community hospitals (multiple centers) for use. As shown in fig. 1, images in different centers often have different appearances due to the operation habits of the capturing device and the image capturing person, and the like. References [1] Rundo L, Han C, Nagano Y, et al, USE-Net: associating space-and-interaction blocks inter U-Net for promoting regional MRI data sets [ J ]. neuro-compression, 2019,365:31-43, describe that images of different central datasets have different visual appearances, and authors train a medical image segmentation model using one central image dataset and evaluate the segmentation model using the other central datasets. The experimental result shows that when other central images are faced, the segmentation performance of the medical image segmentation model trained by the single-central image is reduced to a certain extent. Therefore, when a DR lesion segmentation model trained using a single-center fundus image faces a multi-center fundus image, a degradation in segmentation performance may result.

The prior art discloses a method for automatically segmenting diabetic retinopathy regions based on a loop-adaptive multi-target weighting network, which comprises the following steps: (1) acquiring a sample fundus color photograph image; (2) the diabetic retinopathy regional segmentation model is trained according to a sample fundus color-photographed image and comprises a circulation self-adaptive multi-target weighting network, the circulation self-adaptive multi-target weighting network is used for self-adaptively distributing weights to different targets and enhancing the stability of the network, the different targets comprise at least one of background, bleeding, hard exudation, microangioma, optic disc and cotton velvet spot in the sample fundus color-photographed image, and the trained retinopathy segmentation model is used for segmenting the target fundus color-photographed image. However, the technology still has certain limitations, and the performance of the proposed DR lesion segmentation method on DR lesion segmentation of a multi-center image is not verified, which causes that the method has certain uncertainty in an actual clinical multi-center application scene. The fundus images of multiple centers have different visual appearances due to various reasons (different acquisition equipment, different parameter settings, use habits of operators and the like), and lack of unified standards, so that the difficulty of popularizing the DR focus segmentation model trained by adopting single-center data to other centers is objectively improved. A multi-channel method for automatically calibrating a left vertical function from MR images, a multi-channel, multi-center student [ J ] Radiology,2019,290(1), 81-88, discloses that the generalization of a deep learning model to multi-channel images can be improved. However, this involves a large time and labor cost, and it is difficult to collect all central images and corresponding lesion markings. In the case where a large number of multicenter images with detailed DR lesion marking information cannot be acquired, no method is disclosed at present.

Disclosure of Invention

In order to solve at least one technical defect, the invention provides a diabetic retinopathy focus segmentation method and system suitable for a multi-center image, and the method and system only adopt a single-center eyeground color photograph image containing detailed DR focus labels and a multi-center eyeground color photograph image without any labels to train a DR focus segmentation model, so that the DR focus segmentation model has good focus segmentation performance on other multi-center images.

In order to solve the technical problems, the technical scheme of the invention is as follows:

the diabetic retinopathy focus segmentation method adapting to the multi-center image comprises the following steps:

s1: collecting fundus color photograph images from a plurality of centers to construct a multi-center data set; the multi-center dataset comprises a source center dataset and a plurality of target center datasets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set;

s2: for a certain target center, preprocessing the fundus color photograph image in the source center fundus color photograph data set and the target center fundus color photograph data set, and training a cycleGAN model by adopting the preprocessed fundus color photograph image to generate the cycleGAN model of the target center; repeatedly executing the step S2 until the CycleGAN models of all the target centers are obtained;

s3: processing fundus color photography images in the DR focus segmentation data set at the source center by using a trained cycleGAN model, and establishing a simulation target center image style simulation data set by using one cycleGAN model;

s4: preprocessing a source center DR lesion segmentation data set and each simulation data set;

s5: constructing a DR focus segmentation model and training by utilizing a preprocessed source center DR focus segmentation data set and a simulation data set to obtain the DR focus segmentation model which is suitable for the eye fundus color photograph image style of a target center focus segmentation data set; repeatedly executing the step S5 until a plurality of DR focus segmentation models which are suitable for all target centers are obtained;

s6: respectively testing the focus segmentation performance of all DR focus segmentation models on the fundus color-photographed image of the DR focus segmentation data set at the source center by using the DR focus segmentation data set at the source center, and simultaneously evaluating the DR focus segmentation performance of the DR focus segmentation model corresponding to the target center on the fundus color-photographed image of the DR focus segmentation data set at the target center by using the DR focus segmentation data set at the target center; if the DR focus segmentation performance of the source center and the DR focus segmentation performance of the target center of the DR focus segmentation model both meet the requirements, outputting the trained DR focus segmentation model, and executing step S7; otherwise, returning to execute the step S5;

s7: and inputting the target center to which the fundus color image to be predicted belongs into a trained DR focus segmentation model corresponding to the target center, outputting a final segmentation result, and completing the segmentation of the DR focus.

In the scheme, the DR focus segmentation model of each target center is trained under the condition that only a source center data set containing detailed focus marks and a multi-center data set without any marks are used, the DR focus segmentation model has a good focus segmentation function on fundus color images of the target center, the requirement for a large number of multi-center images containing focus marks is avoided, and the cost for training the DR focus model suitable for the multi-center fundus color images is greatly reduced.

Wherein, the step S1 specifically includes the following steps:

s11: acquiring a source center database and a plurality of target center databases;

s12: collecting fundus color-photograph images without DR focus or with DR focus and with clear retina structure in DR screening from a source central database and a plurality of target central databases respectively;

s13: extracting a small amount of fundus color-photograph images with DR focuses from each central database to carry out manual labeling;

s14: using the eyeground color photograph image subjected to manual marking as a focus segmentation data set, and using the residual unmarked eyeground color photograph image as an eyeground color photograph data set without a focus label to obtain a source center data set and a plurality of target center data sets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set; dividing a source center DR focus segmentation data set into a training set and a test set, wherein the training set is used for training a DR focus segmentation model in the step S5, the test set is used in the step S6, and the source center DR focus segmentation performance of the fundus color photograph image of the source center DR focus segmentation data set is respectively tested for all DR focus segmentation models; the target center DR lesion segmentation data set is used in step S6 for evaluation of target center DR lesion segmentation performance of a fundus color photograph image of the target center DR lesion segmentation data set for a DR lesion segmentation model of the data set corresponding to the target center.

In step S2, the CycleGAN model is a style transition model based on a generation countermeasure network, and the process of training the CycleGAN model using the preprocessed source central fundus oculi color photograph data set a and one target central fundus oculi color photograph data set B specifically includes: defining a source central fundus color photograph data set A as an X domain and a target central fundus color photograph data set B as a Y domain; the cycleGAN model comprises a first image generator, a first discriminator, a second image generator and a second discriminator; wherein: a first image generator for generating a pseudo fundus color photographic image having a target central fundus color photographic data set B style from a fundus color photographic image of the X domain; the first discriminator is used for distinguishing the eyeground color photograph image of the Y domain from the false fundus color photograph image of the target central eyeground color photograph data set B; a second image generator for generating a pseudo-fundus color image having a source-centric fundus color data set A-style from the fundus color image of the Y domain; the second discriminator is used for distinguishing the fundus color photograph image of the X domain from the pseudo fundus color photograph image of the source center fundus color photograph data set A; wherein, the confrontation loss of the first image generator and the first discriminator is as follows:

the countermeasure loss of the second image generator and the second discriminator is:

the cycle consistency loss is:

the integrity loss function is:

L(G,F,D_X,D_Y)＝L_GAN(G,D_Y,X,Y)+L_GAN(F,D_Y,Y,X)+λL_cyc(G,F)

wherein the parameter G represents the first image generator; ex to pdata (x), Ey to pdata (y) each represent a loss function of L1; x in g (X) represents an X-domain fundus color photograph image input to the first image generator; parameter D_YRepresents a first discriminator; d_YY in (Y) represents a Y-domain fundus color-photograph image input to the first discriminator; the parameter F represents the second image generator; y in f (Y) represents a Y-domain fundus color-photograph image input to the second image generator; parameter D_XRepresents a second discriminator; d_X(x) X in (2) represents an X-domain fundus color photograph image input to the second discriminator; the parameter lambda represents a weight coefficient of the cyclic consistency loss, and the default value is 10; in the process of training a cycleGAN model by adopting fundus color images preprocessed by a source central fundus color image dataset A and a target central fundus color image dataset B, using an Adam optimizer to alternately update a first image generator and a second image generator in the cycleGAN model and update parameters of a first discriminator and a second discriminator so as to reduce a complete loss function of the model, and finishing training when the network reaches a Nash balance state, namely the discriminator cannot distinguish whether the fundus color images are true or false, so as to obtain a cycleGAN AB model; at this time, the first image generator serves as a style conversion generator capable of converting the fundus color-photograph image of the source central fundus color-photograph dataset a into a fundus color-photograph image having a style of the target central fundus color-photograph dataset B; and generating a cycleGAN model corresponding to the source central eyeground color photograph data set and the other target central eyeground color photograph data sets according to the same training method.

In the foregoing solution, the process of preprocessing the fundus color-photograph images in the source central fundus color-photograph data set and the target central fundus color-photograph data set in step S2 specifically includes: performing image preprocessing on all central fundus color-photograph images, removing black areas of the images because all the central fundus color-photograph images contain meaningless information such as black frames and the like, reserving effective areas of the fundus color-photograph images, and then performing size adjustment to adjust each fundus color-photograph image to be the same size of 1280x 1280; and normalizing the image to be in a range of-1 to 1. Meanwhile, in order to enhance the robustness of the model and reduce the overfitting of the model, the fundus image is processed by using data enhancement means such as random horizontal inversion, random vertical inversion, random rotation (the rotation degree range is between-180 degrees and 180 degrees), brightness adjustment, contrast adjustment and the like.

In step S3, the fundus color photograph image in the training set of the source center DR lesion segmentation data set is processed by using the CycleGAN model corresponding to the target center fundus color photograph data set obtained in step S2, so as to obtain a target center DR lesion segmentation data set capable of simulating a corresponding target center style, thereby establishing a simulated data set corresponding to the target center DR lesion segmentation data set; the simulation data set corresponding to all target center DR focus segmentation data sets has the same focus label as the training set of the source center DR focus segmentation data set because the important anatomical structure of the CycleGAN model is kept unchanged when the fundus color photograph image is converted.

In step S4, the preprocessing process of the simulation data sets corresponding to the DR lesion segmentation data sets at the source center and different target centers specifically includes: since the fundus color-photograph image of each data set contains meaningless information such as black frames, the black area of the image is removed, the effective area in the fundus image is retained, and then the size adjustment is performed to adjust each fundus image to the same size of 1280 × 1280. Meanwhile, in order to enhance the robustness of the model and reduce the overfitting of the model, the fundus image is processed by using data enhancement means such as random horizontal inversion, random vertical inversion, random rotation (the rotation degree range is between-180 degrees and 180 degrees), brightness adjustment, contrast adjustment and the like.

Wherein, in the step S5, the constructed DR lesion segmentation model includes a lesion attention module and a DR lesion segmentation module; wherein: the focus attention module consists of a seven-layer convolutional neural network and is used for predicting the probability that a certain local area in the fundus color photographic image is normal or has a certain focus; a convolutional neural network model with an encoder-decoder structure is used in a DR focus segmentation module; for an eyeground color photograph image to be predicted, firstly, the eyeground color photograph image is cut into a plurality of slices in a gridding cutting mode, then, a focus attention module is utilized to predict the slices one by one and generate corresponding probability vectors, and the generated probability vectors correspond to the probabilities of normal focuses and various focuses; then, combining the generated probability vectors into an initial attention map according to the prediction sequence of the focus attention module; finally, expanding the initialized attention diagram through three deconvolution layers to obtain an attention diagram AM; the generated attention map AM highlights potential focus areas on the fundus color-photographed image to be predicted, and then guides a DR focus segmentation module to identify DR focuses in the fundus color-photographed image so as to obtain prediction results of various DR focuses; in addition, the initial attention diagram generated by the focus attention module is merged with the characteristic diagram output by the DR focus segmentation module encoder, and the merged initial attention diagram and the characteristic diagram are input into a decoder of the DR focus segmentation module, loss information is transmitted back to the focus attention module in the training process, and the problem of gradient disappearance is avoided; the loss function of the DR focus segmentation model is a binary cross entropy loss function, which is specifically represented as:

wherein, N represents the number of training samples, and i represents the ith training sample; y is a binary label of 0 or 1, p (y) is the probability that the output belongs to the y label; the binary label is pixel-level DR focus mark of the input eye ground color image, the mark is a two-dimensional matrix with the same size as the input eye ground color image, if a certain pixel point of the input eye ground color image is marked as a focus, the value of the corresponding position of the mark matrix is 1, otherwise, the value is 0; based on the process, the DR focus segmentation model is trained in a supervised learning mode by adopting a preprocessed source center DR focus segmentation data set training set and simulation data sets corresponding to different target center DR focus segmentation data sets, wherein parameters of the DR focus segmentation model are updated by adopting an Adam optimizer, so that loss function values are reduced, and finally the DR focus segmentation model which is suitable for the fundus color image styles of a plurality of target center DR focus segmentation data sets is obtained.

In the above scheme, the evaluation indexes of the DR lesion segmentation model in step S6 are: AUC _ ROC and AUC _ PR, wherein AUC _ ROC is the area under the ROC curve and is used for evaluating the misdiagnosis condition of the DR lesion segmentation model on the lesion; AUC _ PR is the area under the PR curve and is used for evaluating the comprehensive condition of the DR lesion segmentation model on lesion misdiagnosis and missed diagnosis.

The scheme also provides a diabetic retinopathy focus segmentation system based on the multi-center image, which is used for realizing the diabetic retinopathy focus segmentation method and specifically comprises a multi-center data set construction module, a first preprocessing module, a CycleGAN model training module, a simulation data set construction module, a second preprocessing module, a DR focus segmentation model training module, a test evaluation module and a focus segmentation module; wherein: the multi-center data set construction module is used for collecting fundus color photograph images from a plurality of centers and constructing a multi-center data set; the multi-center dataset comprises a source center dataset and a plurality of target center datasets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set; the first preprocessing module is used for preprocessing the eyeground color photograph images in the source central eyeground color photograph data set and the target central eyeground color photograph data set; the cycleGAN model training module is used for training the cycleGAN model by adopting the pretreated fundus color photographic image to generate the cycleGAN model; aiming at a certain target center, generating a CycleGAN model of the target center until the CycleGAN models of all the target centers are generated; the simulation data set establishing module is used for processing fundus color images in the DR focus segmentation data set of the source center by utilizing a trained cycleGAN model, and one cycleGAN model establishes a simulation target center image style simulation data set; the second preprocessing module is used for preprocessing the DR focus segmentation data set of the source center and each simulation data set; the DR focus segmentation model training module is used for constructing a DR focus segmentation model and training the DR focus segmentation model with each simulation data set by utilizing the preprocessed source center DR focus segmentation data set to obtain a DR focus segmentation model which is suitable for the eye fundus color photograph image style of each target center focus segmentation data set; the testing and evaluating module is used for respectively testing the focus segmentation performance of all DR focus segmentation models on the fundus color-photographed image of the DR focus segmentation data set at the source center by utilizing the DR focus segmentation data set at the source center, and simultaneously evaluating the DR focus segmentation performance of the DR focus segmentation model corresponding to the target center on the fundus color-photographed image of the DR focus segmentation data set at the target center by utilizing the DR focus segmentation data set at the target center; if the source center DR focus segmentation performance of the source center of the DR focus segmentation model and the target center DR focus segmentation performance meet the requirements, outputting the trained DR focus segmentation model to a focus segmentation module; otherwise, retraining the DR focus segmentation model by a DR focus segmentation model training module; the focus segmentation module is used for inputting the eyeground color photograph image to be predicted into a DR focus segmentation model corresponding to the trained target center to which the eyeground color photograph image belongs, outputting a final segmentation result and completing the segmentation of the diabetic retinopathy focus.

The multi-center data set construction module comprises a data acquisition unit, a data screening unit, a manual labeling unit and a data classification unit; the data acquisition unit is used for acquiring a source center database and a plurality of target center databases; the data screening unit is used for respectively screening DR from the source central database and the target central databases and collecting fundus color photography images without DR focuses or with DR focuses and with clear retina structures; the manual labeling unit is used for extracting a small amount of fundus color photographic images with DR focuses from the fundus color photographic images of each central database to perform manual labeling; the data classification unit is used for classifying all fundus color photograph images, taking the fundus color photograph images subjected to manual labeling as a focus segmentation data set, and taking the remaining unmarked fundus color photograph images as fundus color photograph data sets without focus labels to obtain a source center data set and a plurality of target center data sets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set; dividing a source center DR focus segmentation data set into a training set and a test set, wherein the training set is used for training a DR focus segmentation model in a DR focus segmentation model training module, and the test set is used in a test evaluation module for respectively testing the source center DR focus segmentation performance of all DR focus segmentation models; the target center DR lesion segmentation data set is used in a test evaluation module to evaluate the target center DR lesion segmentation performance of the target center DR lesion segmentation model corresponding to the target center.

In the CycleGAN model training module, a CycleGAN model of the CycleGAN model training module is a style migration model based on a generation countermeasure network, and comprises a first image generator, a first discriminator, a second image generator and a second discriminator; the method comprises the following steps of (1) training a cycleGAN model by using a preprocessed source central eye fundus color photograph data set A and a preprocessed target central eye fundus color photograph data set B, wherein the training process comprises the following steps: defining a source central fundus color photograph data set A as an X domain and a target central fundus color photograph data set B as a Y domain; a first image generator generating a pseudo fundus color photographic image having a target central fundus color photographic data set B style from a fundus color photographic image of the X domain; the first discriminator is used for distinguishing the eyeground color photograph image of the Y domain from the false fundus color photograph image of the target central eyeground color photograph data set B; a second image generator for generating a pseudo-fundus color image having a source-centric fundus color data set A-style from the fundus color image of the Y domain; the second discriminator is used for distinguishing the fundus color photograph image of the X domain from the pseudo fundus color photograph image of the source center fundus color photograph data set A; wherein:

the countermeasure loss of the first image generator and the first discriminator is:

the countermeasure loss of the second image generator and the second discriminator is:

the cycle consistency loss is:

the integrity loss function is:

L(G,F,D_X,D_Y)＝L_GAN(G,D_Y,X,Y)+L_GAN(F,D_Y,Y,X)+λL_cyc(G,F)

wherein the parameter G represents the first image generator; ex to pdata (x), Ey to pdata (y) each represent a loss function of L1; x in g (X) represents an X-domain fundus color photograph image input to the first image generator; parameter D_YRepresents a first discriminator; d_YY in (Y) represents a Y-domain fundus color-photograph image input to the first discriminator; the parameter F represents the second image generator; y in f (Y) represents a Y-domain fundus color-photograph image input to the second image generator; parameter D_XRepresents a second discriminator; d_X(x) X in (2) represents an X-domain fundus color photograph image input to the second discriminator; the parameter λ represents the weight coefficient of the cyclic consistency loss; in the process of training a cycleGAN model by adopting fundus color images preprocessed by a source central fundus color image dataset A and a target central fundus color image dataset B, using an Adam optimizer to alternately update a first image generator and a second image generator in the cycleGAN model and update parameters of a first discriminator and a second discriminator so as to reduce a complete loss function of the model, and finishing training when a network reaches a Nash balance state, namely the discriminator cannot distinguish whether the fundus color images are true or false, so as to obtain a cycleGAN AB model; at this time, the first image generator is capable of converting the fundus color of the source center fundus color photograph data set A as a style conversion generatorThe images are converted into fundus color photograph images with a style B of a target central fundus color photograph data set; according to the same training method, the CycleGAN model training module generates a CycleGAN model corresponding to the source center eyeground color photograph data set and the other target center eyeground color photograph data sets through training.

In the simulation data set establishing module, a CycleGAN model which is obtained by a CycleGAN model training module and corresponds to a target center eyeground color photograph data set is used for processing eyeground color photograph images in a source center DR focus segmentation data set training set to obtain a target center DR focus segmentation data set which can simulate a corresponding target center style, so that a simulation data set corresponding to the target center DR focus segmentation data set is established; the simulation data set corresponding to all target center DR focus segmentation data sets has the same focus label as the training set of the source center DR focus segmentation data set because the important anatomical structure of the CycleGAN model is kept unchanged when the fundus color photograph image is converted.

The DR focus segmentation model training module is loaded with a DR focus segmentation model, and comprises a focus attention module and a DR focus segmentation module; wherein: the focus attention module consists of a seven-layer convolutional neural network and is used for predicting the probability that a certain local area in the fundus color photographic image is normal or has a certain focus; a convolutional neural network model with an encoder-decoder structure is used in a DR focus segmentation module; for an eyeground color photograph image to be predicted, firstly, the eyeground color photograph image is cut into a plurality of slices in a gridding cutting mode, then, a focus attention module is utilized to predict the slices one by one and generate corresponding probability vectors, and the generated probability vectors correspond to the probabilities of normal focuses and various focuses; then, combining the generated probability vectors into an initial attention map according to the prediction sequence of the focus attention module; finally, expanding the initialized attention diagram through three deconvolution layers to obtain an attention diagram AM; the generated attention map AM highlights potential focus areas on the fundus color-photographed image to be predicted, and then guides a DR focus segmentation module to identify DR focuses in the fundus color-photographed image so as to obtain prediction results of various DR focuses; in addition, the initial attention diagram generated by the focus attention module is merged with the characteristic diagram output by the DR focus segmentation module encoder, and the merged initial attention diagram and the characteristic diagram are input into a decoder of the DR focus segmentation module, loss information is transmitted back to the focus attention module in the training process, and the problem of gradient disappearance is avoided; the loss function of the DR focus segmentation model is a binary cross entropy loss function, which is specifically represented as:

wherein, N represents the number of training samples, and i represents the ith training sample; y is a binary label of 0 or 1, p (y) is the probability that the output belongs to the y label; the binary label is pixel-level DR focus mark of the input eye ground color image, the mark is a two-dimensional matrix with the same size as the input eye ground color image, if a certain pixel point of the input eye ground color image is marked as a focus, the value of the corresponding position of the mark matrix is 1, otherwise, the value is 0;

based on the above process, the DR focus segmentation model training module trains the DR focus segmentation model in a supervised learning mode by adopting a preprocessed source center DR focus segmentation data set training set and a simulation data set corresponding to a target center DR focus segmentation data set, wherein an Adam optimizer is adopted to update parameters of the DR focus segmentation model, so that loss function values are reduced, and finally a plurality of DR focus segmentation models which are suitable for the eye fundus color image style of the DR focus segmentation data set to which the target center belongs are obtained.

In the above scheme, the calculation process of the prediction results of various DR lesions specifically includes: adjusting the size of the complete fundus color image to be measured, and performing dot product calculation with the thermodynamic diagram one by one to obtain a weighted fundus color image, wherein the calculation formula is as follows:

X^AM＝(AM_i+1)⊙X

wherein, AM_iThe ith channel, representing a thermodynamic diagram, indicates a pixel-by-pixel matrix multiplication; and inputting the weighted fundus map into a DR focus segmentation module to obtain the prediction results of the four DR focuses.

According to the diabetic retinopathy focus segmentation system suitable for the multi-center image, for an input fundus color photographic image, a focus attention module of the system automatically gives higher weight to a potential focus area in the fundus color photographic image through learning, and inhibits irrelevant background information, so that the DR focus segmentation module further focuses on the focus area to generate characteristic representation with higher resolution, the focus segmentation performance of a focus segmentation model is improved, and misdiagnosis and missed diagnosis of the DR focus are reduced.

Compared with the prior art, the technical scheme of the invention has the beneficial effects that:

the invention provides a diabetic retinopathy focus segmentation method and system suitable for a multi-center image, which can complete the training of a DR focus segmentation model of each target center under the condition of only using a source center data set containing detailed focus marks and a multi-center data set without any marks, wherein each DR focus segmentation model has a good focus segmentation function on an eyeground color image of the target center, thereby avoiding the requirement on a large number of multi-center images containing focus marks and greatly reducing the cost for training the DR focus model suitable for the multi-center eyeground color image.

Drawings

FIG. 1 is a schematic view of fundus color photographs from different centers (three lines of images from three centers, respectively);

FIG. 2 is a schematic flow diagram of the process of the present invention;

FIG. 3 is a flowchart illustrating a step S1 of the method according to the present invention;

FIG. 4 is a schematic diagram of an image generated by the cycleGAN model according to an embodiment of the present invention (the left image is a real fundus color photograph image, and the right image is a cycleGAN generated image);

FIG. 5 is a schematic structural diagram of the cycleGAN model according to an embodiment of the present invention;

FIG. 6 is a structural diagram of the DR focus segmentation model according to an embodiment of the present invention;

fig. 7 is a schematic diagram of the module connection of the system of the present invention.

Detailed Description

The drawings are for illustrative purposes only and are not to be construed as limiting the patent;

example 1:

the embodiment is a complete use example and has rich content

For the purpose of better illustrating the embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product;

it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.

The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.

As shown in fig. 2, the method for segmenting diabetic retinopathy focus adapting to a multi-center image includes the following steps:

s1: collecting fundus color photograph images from a plurality of centers to construct a multi-center data set; the multi-center dataset comprises a source center dataset and a plurality of target center datasets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set;

s2: for a certain target center, preprocessing the fundus color photograph image in the source center fundus color photograph data set and the target center fundus color photograph data set, and training a cycleGAN model by adopting the preprocessed fundus color photograph image to generate the cycleGAN model of the target center; repeatedly executing the step S2 until the CycleGAN models of all the target centers are obtained;

s3: processing fundus color photography images in the DR focus segmentation data set at the source center by using a trained cycleGAN model, and establishing a simulation target center image style simulation data set by using one cycleGAN model;

s4: preprocessing a source center DR lesion segmentation data set and each simulation data set;

s5: constructing a DR focus segmentation model and training by utilizing a preprocessed source center DR focus segmentation data set and a simulation data set to obtain the DR focus segmentation model which is suitable for the eye fundus color photograph image style of a target center focus segmentation data set; repeatedly executing the step S5 until a plurality of DR focus segmentation models which are suitable for all target centers are obtained;

s6: respectively testing the focus segmentation performance of all DR focus segmentation models on the fundus color-photographed image of the DR focus segmentation data set at the source center by using the DR focus segmentation data set at the source center, and simultaneously evaluating the DR focus segmentation performance of the DR focus segmentation model corresponding to the target center on the fundus color-photographed image of the DR focus segmentation data set at the target center by using the DR focus segmentation data set at the target center; if the DR focus segmentation performance of the source center and the DR focus segmentation performance of the target center of the DR focus segmentation model both meet the requirements, outputting the trained DR focus segmentation model, and executing step S7; otherwise, returning to execute the step S5;

s7: and inputting the target center to which the fundus color image to be predicted belongs into a trained DR focus segmentation model corresponding to the target center, outputting a final segmentation result, and completing the segmentation of the DR focus.

In the specific implementation process, the method completes the training of the DR focus segmentation model of each target center under the condition of only using the source center data set containing detailed focus marks and the multi-center data set without any marks, each DR focus segmentation model has a good focus segmentation function on the fundus color-photographed image of the target center, the requirement on a large number of multi-center images containing focus marks is avoided, and the cost for training the DR focus model suitable for the multi-center fundus color-photographed image is greatly reduced.

More specifically, as shown in fig. 3, the step S1 specifically includes the following steps:

s11: acquiring a source center database and a plurality of target center databases;

s12: collecting fundus color-photograph images without DR focus or with DR focus and with clear retina structure in DR screening from a source central database and a plurality of target central databases respectively;

s13: extracting a small amount of fundus color-photograph images with DR focuses from each central database to carry out manual labeling;

s14: using the eyeground color photograph image subjected to manual marking as a focus segmentation data set, and using the residual unmarked eyeground color photograph image as an eyeground color photograph data set without a focus label to obtain a source center data set and a plurality of target center data sets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set; dividing a source center DR focus segmentation data set into a training set and a test set, wherein the training set is used for training a DR focus segmentation model in the step S5, the test set is used in the step S6, and the source center DR focus segmentation performance of the fundus color photograph image of the source center DR focus segmentation data set is respectively tested for all DR focus segmentation models; the target center DR lesion segmentation data set is used in step S6 for evaluation of target center DR lesion segmentation performance of a fundus color photograph image of the target center DR lesion segmentation data set for a DR lesion segmentation model of the data set corresponding to the target center.

In the specific implementation process, in the manual labeling process, a professional ophthalmologist marks hemangioma, hemorrhage, soft exudation and hard exudation on the fundus color-photograph image in a sketching way, and performs secondary check on the labeling information to ensure that the labeling is correct. After the labeling work is finished, the labeled fundus color-photograph images are combined into a focus segmentation data set by a professional ophthalmologist, and the residual unmarked fundus color-photograph images are used as the fundus color-photograph image data set without focus labels.

More specifically, in step S2, the CycleGAN model is a style transition model based on a generation countermeasure network, and the generated image is as shown in fig. 4, which can convert the image from the image style of the current center to the style of the target image without a matching image and keep the important anatomical structures such as the lesion, blood vessel, optic disc, and the like in the fundus color-photographed image unchanged.

In a specific implementation process, the process of training the CycleGAN model by using the preprocessed source central fundus color photograph data set a and one target central fundus color photograph data set B specifically comprises the following steps: defining a source central fundus color photograph data set A as an X domain and a target central fundus color photograph data set B as a Y domain; the CycleGAN model structure is shown in fig. 5 and comprises a first image generator, a first discriminator, a second image generator and a second discriminator; wherein: a first image generator for generating a pseudo fundus color photographic image having a target central fundus color photographic data set B style from a fundus color photographic image of the X domain; the first discriminator is used for distinguishing the eyeground color photograph image of the Y domain from the false fundus color photograph image of the target central eyeground color photograph data set B; a second image generator for generating a pseudo-fundus color image having a source-centric fundus color data set A-style from the fundus color image of the Y domain; the second discriminator is used for distinguishing the fundus color photograph image of the X domain from the pseudo fundus color photograph image of the source center fundus color photograph data set A; wherein, the confrontation loss of the first image generator and the first discriminator is as follows:

the countermeasure loss of the second image generator and the second discriminator is:

the cycle consistency loss is:

the integrity loss function is:

L(G,F,D_X,D_Y)＝L_GAN(G,D_Y,X,Y)+L_GAN(F,D_Y,Y,X)+λL_cyc(G,F)

wherein the parameter G represents the first image generator; ex to pdata (x), Ey to pdata (y) each represent a loss function of L1; x in g (X) represents an X-domain fundus color photograph image input to the first image generator; parameter D_YRepresents a first discriminator; d_YY in (y) represents a first judgment of inputY-domain fundus color-photograph images of the discriminator; the parameter F represents the second image generator; y in f (Y) represents a Y-domain fundus color-photograph image input to the second image generator; parameter D_XRepresents a second discriminator; d_X(x) X in (2) represents an X-domain fundus color photograph image input to the second discriminator; the parameter lambda represents a weight coefficient of the cyclic consistency loss, and the default value is 10; in the process of training a cycleGAN model by adopting fundus color images preprocessed by a source central fundus color image dataset A and a target central fundus color image dataset B, using an Adam optimizer to alternately update a first image generator and a second image generator in the cycleGAN model and update parameters of a first discriminator and a second discriminator so as to reduce a complete loss function of the model, and finishing training when the network reaches a Nash balance state, namely the discriminator cannot distinguish whether the fundus color images are true or false, so as to obtain a cycleGAN AB model; at this time, the first image generator serves as a style conversion generator capable of converting the fundus color-photograph image of the source central fundus color-photograph dataset a into a fundus color-photograph image having a style of the target central fundus color-photograph dataset B; and generating a cycleGAN model corresponding to the source central eyeground color photograph data set and the other target central eyeground color photograph data sets according to the same training method.

The process of preprocessing the fundus color-photograph images in the source central fundus color-photograph data set and the target central fundus color-photograph data set in the step S2 is specifically as follows: performing image preprocessing on all central fundus color-photograph images, removing black areas of the images because all the central fundus color-photograph images contain meaningless information such as black frames and the like, reserving effective areas of the fundus color-photograph images, and then performing size adjustment to adjust each fundus color-photograph image to be the same size of 1280x 1280; and normalizing the image to be in a range of-1 to 1. Meanwhile, in order to enhance the robustness of the model and reduce the overfitting of the model, the fundus image is processed by using data enhancement means such as random horizontal inversion, random vertical inversion, random rotation (the rotation degree range is between-180 degrees and 180 degrees), brightness adjustment, contrast adjustment and the like.

More specifically, in step S3, the fundus color photograph image in the training set of the source center DR lesion segmentation data set is processed by using the CycleGAN model corresponding to the target center fundus color photograph data set obtained in step S2, so as to obtain a target center DR lesion segmentation data set capable of simulating the corresponding target center style, thereby establishing a simulated data set corresponding to the target center DR lesion segmentation data set; the simulation data set corresponding to all target center DR focus segmentation data sets has the same focus label as the training set of the source center DR focus segmentation data set because the important anatomical structure of the CycleGAN model is kept unchanged when the fundus color photograph image is converted.

In the specific implementation process, the simulation data set corresponding to the target center DR lesion segmentation data set is established as follows: if a CycleGAN AB model is adopted to process the fundus color photograph images in the source center DR focus segmentation data set training set, a DR focus segmentation training set simulating a center B style can be generated; similarly, the same is true for other CycleGAN models, and finally, a simulation data set corresponding to the target central DR lesion segmentation data set can be obtained.

More specifically, in step S4, the preprocessing process performed on the simulation dataset corresponding to the source-center DR lesion segmentation dataset and the target-center DR lesion segmentation dataset specifically includes: since the fundus color-photograph image of each data set contains meaningless information such as black frames, the black area of the image is removed, the effective area in the fundus image is retained, and then the size adjustment is performed to adjust each fundus image to the same size of 1280 × 1280. Meanwhile, in order to enhance the robustness of the model and reduce the overfitting of the model, the fundus image is processed by using data enhancement means such as random horizontal inversion, random vertical inversion, random rotation (the rotation degree range is between-180 degrees and 180 degrees), brightness adjustment, contrast adjustment and the like.

More specifically, in the step S5, as shown in fig. 6, the constructed DR lesion segmentation model includes a lesion attention module and a DR lesion segmentation module; wherein: the focus attention module consists of a seven-layer convolutional neural network and is used for predicting the probability that a certain local area in the fundus color photographic image is normal or has a certain focus; a convolutional neural network model with an encoder-decoder structure is used in a DR focus segmentation module; for a fundus color photograph image to be predicted with the size of 1280x1280, cutting the fundus color photograph image into 400 slices with the size of 64 x 64 in a gridding cutting mode, predicting the slices one by utilizing a focus attention module and generating corresponding probability vectors, wherein the generated probability vectors correspond to the probabilities of normal focuses and 4 focuses; then, combining the generated probability vectors into a 20 × 20 initial attention diagram according to the prediction sequence of the focus attention module; finally, expanding the initialized attention map through three deconvolution layers to obtain an attention map AM of 640 x 640; the generated attention map AM can highlight a potential focus area on the fundus color-photographed image to be predicted, so as to guide a DR focus segmentation module to identify DR focuses in the fundus color-photographed image, reduce misdiagnosis and missed diagnosis of the DR focuses and finally obtain prediction results of various DR focuses; in addition, the initial attention diagram generated by the focus attention module is merged with the characteristic diagram output by the DR focus segmentation module encoder, and the merged initial attention diagram and the characteristic diagram are input into a decoder of the DR focus segmentation module, loss information is transmitted back to the focus attention module in the training process, and the problem of gradient disappearance is avoided; the loss function of the DR focus segmentation model is a binary cross entropy loss function, which is specifically represented as:

wherein, N represents the number of training samples, and i represents the ith training sample; y is a binary label of 0 or 1, p (y) is the probability that the output belongs to the y label; the binary label is pixel-level DR focus mark of the input eye ground color image, the mark is a two-dimensional matrix with the same size as the input eye ground color image, if a certain pixel point of the input eye ground color image is marked as a focus, the value of the corresponding position of the mark matrix is 1, otherwise, the value is 0; based on the process, the DR focus segmentation model is trained in a supervised learning mode by adopting a preprocessed source center DR focus segmentation data set training set and simulation data sets corresponding to different target center DR focus segmentation data sets, wherein parameters of the DR focus segmentation model are updated by adopting an Adam optimizer, so that loss function values are reduced, and finally the DR focus segmentation model which is suitable for the fundus color image styles of a plurality of target center DR focus segmentation data sets is obtained.

In the specific implementation process, the evaluation indexes of the DR lesion segmentation model in step S6 are as follows: AUC _ ROC and AUC _ PR, wherein AUC _ ROC is the area under the ROC curve and is used for evaluating the misdiagnosis condition of the DR lesion segmentation model on the lesion; AUC _ PR is the area under the PR curve and is used for evaluating the comprehensive condition of the DR lesion segmentation model on lesion misdiagnosis and missed diagnosis.

Example 2

More specifically, on the basis of embodiment 1, as shown in fig. 7, a diabetic retinopathy lesion segmentation system based on a multi-center image is provided, and is used for implementing the above diabetic retinopathy lesion segmentation method, and specifically includes a multi-center dataset construction module, a first preprocessing module, a CycleGAN model training module, a simulation dataset construction module, a second preprocessing module, a DR lesion segmentation model training module, a test evaluation module, and a lesion segmentation module; wherein: the multi-center data set construction module is used for collecting fundus color photograph images from a plurality of centers and constructing a multi-center data set; the multi-center dataset comprises a source center dataset and a plurality of target center datasets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set; the first preprocessing module is used for preprocessing the eyeground color photograph images in the source central eyeground color photograph data set and the target central eyeground color photograph data set; the cycleGAN model training module is used for training the cycleGAN model by adopting the pretreated fundus color photographic image to generate the cycleGAN model; aiming at a certain target center, generating a CycleGAN model of the target center until the CycleGAN models of all the target centers are generated; the simulation data set establishing module is used for processing fundus color images in the DR focus segmentation data set of the source center by utilizing a trained cycleGAN model, and one cycleGAN model establishes a simulation target center image style simulation data set; the second preprocessing module is used for preprocessing the DR focus segmentation data set of the source center and each simulation data set; the DR focus segmentation model training module is used for constructing a DR focus segmentation model and training the DR focus segmentation model with each simulation data set by utilizing the preprocessed source center DR focus segmentation data set to obtain a DR focus segmentation model which is suitable for the eye fundus color photograph image style of each target center focus segmentation data set; the testing and evaluating module is used for respectively testing the focus segmentation performance of all DR focus segmentation models on the fundus color-photographed image of the DR focus segmentation data set at the source center by utilizing the DR focus segmentation data set at the source center, and simultaneously evaluating the DR focus segmentation performance of the DR focus segmentation model corresponding to the target center on the fundus color-photographed image of the DR focus segmentation data set at the target center by utilizing the DR focus segmentation data set at the target center; if the DR focus segmentation performance of the source center DR focus segmentation model and the DR focus segmentation performance of the target center meet the requirements, outputting the trained DR focus segmentation model to a focus segmentation module; otherwise, retraining the DR focus segmentation model by a DR focus segmentation model training module; the focus segmentation module is used for inputting the eyeground color photograph image to be predicted into a DR focus segmentation model corresponding to the trained target center to which the eyeground color photograph image belongs, outputting a final segmentation result and completing the segmentation of the diabetic retinopathy focus.

More specifically, the multi-center data set construction module comprises a data acquisition unit, a data screening unit, a manual labeling unit and a data classification unit; wherein: the data acquisition unit is used for acquiring a source center database and a plurality of target center databases; the data screening unit is used for respectively screening DR from the source central database and the target central databases and collecting fundus color photography images without DR focuses or with DR focuses and with clear retina structures; the manual labeling unit is used for extracting a small amount of fundus color photographic images with DR focuses from the fundus color photographic images of each central database to perform manual labeling; the data classification unit is used for classifying all fundus color photograph images, taking the fundus color photograph images subjected to manual labeling as a focus segmentation data set, and taking the remaining unmarked fundus color photograph images as fundus color photograph data sets without focus labels to obtain a source center data set and a plurality of target center data sets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set; dividing a source center DR focus segmentation data set into a training set and a test set, wherein the training set is used for training a DR focus segmentation model in a DR focus segmentation model training module, and the test set is used in a test evaluation module for respectively testing the source center DR focus segmentation performance of all DR focus segmentation models; the target center DR lesion segmentation data set is used in a test evaluation module to evaluate the target center DR lesion segmentation performance of the target center DR lesion segmentation model corresponding to the target center.

In the CycleGAN model training module, a CycleGAN model of the CycleGAN model training module is a style migration model based on a generation countermeasure network, and comprises a first image generator, a first discriminator, a second image generator and a second discriminator; the method comprises the following steps of (1) training a cycleGAN model by using a preprocessed source central eye fundus color photograph data set A and a preprocessed target central eye fundus color photograph data set B, wherein the training process comprises the following steps: defining a source central fundus color photograph data set A as an X domain and a target central fundus color photograph data set B as a Y domain; a first image generator generating a pseudo fundus color photographic image having a target central fundus color photographic data set B style from a fundus color photographic image of the X domain; the first discriminator is used for distinguishing the eyeground color photograph image of the Y domain from the false fundus color photograph image of the target central eyeground color photograph data set B; a second image generator for generating a pseudo-fundus color image having a source-centric fundus color data set A-style from the fundus color image of the Y domain; the second discriminator is used for distinguishing the fundus color photograph image of the X domain from the pseudo fundus color photograph image of the source center fundus color photograph data set A; wherein:

the countermeasure loss of the first image generator and the first discriminator is:

the countermeasure loss of the second image generator and the second discriminator is:

the cycle consistency loss is:

the integrity loss function is:

L(G,F,D_X,D_Y)＝L_GAN(G,D_Y,X,Y)+L_GAN(F,D_Y,Y,X)+λL_cyc(G,F)

wherein the parameter G represents the first image generator; ex to pdata (x), Ey to pdata (y) each represent a loss function of L1; x in g (X) represents an X-domain fundus color photograph image input to the first image generator; parameter D_YRepresents a first discriminator; d_YY in (Y) represents a Y-domain fundus color-photograph image input to the first discriminator; the parameter F represents the second image generator; y in f (Y) represents a Y-domain fundus color-photograph image input to the second image generator; parameter D_XRepresents a second discriminator; d_X(x) X in (2) represents an X-domain fundus color photograph image input to the second discriminator; the parameter λ represents the weight coefficient of the cyclic consistency loss; in the process of training a cycleGAN model by adopting fundus color images preprocessed by a source central fundus color image dataset A and a target central fundus color image dataset B, using an Adam optimizer to alternately update a first image generator and a second image generator in the cycleGAN model and update parameters of a first discriminator and a second discriminator so as to reduce a complete loss function of the model, and finishing training when a network reaches a Nash balance state, namely the discriminator cannot distinguish whether the fundus color images are true or false, so as to obtain a cycleGAN AB model; at this time, the first image generator serves as a style conversion generator capable of converting the fundus color-photograph image of the source central fundus color-photograph dataset a into a fundus color-photograph image having a style of the target central fundus color-photograph dataset B; according to the same training method, the CycleGAN model training module generates a source center eyeground color photo data set and other target center eyeground color photo data set pairs through trainingThe corresponding CycleGAN model.

More specifically, in the simulation data set establishing module, the cyclic GAN model corresponding to the target central eye fundus color photograph data set obtained by the cyclic GAN model training module is used for processing eye fundus color photograph images in the source central DR focus segmentation data set training set to obtain a target central DR focus segmentation data set capable of simulating a corresponding target central style, so that a simulation data set corresponding to the target central DR focus segmentation data set is established; the simulation data set corresponding to all target center DR focus segmentation data sets has the same focus label as the training set of the source center DR focus segmentation data set because the important anatomical structure of the CycleGAN model is kept unchanged when the fundus color photograph image is converted.

More specifically, a DR focus segmentation model is loaded in a DR focus segmentation model training module, and comprises a focus attention module and a DR focus segmentation module; wherein: the focus attention module consists of a seven-layer convolutional neural network and is used for predicting the probability that a certain local area in the fundus color photographic image is normal or has a certain focus; a convolutional neural network model with an encoder-decoder structure is used in a DR focus segmentation module; for an eyeground color photograph image to be predicted, firstly, the eyeground color photograph image is cut into a plurality of slices in a gridding cutting mode, then, a focus attention module is utilized to predict the slices one by one and generate corresponding probability vectors, and the generated probability vectors correspond to the probabilities of normal focuses and various focuses; then, combining the generated probability vectors into an initial attention map according to the prediction sequence of the focus attention module; finally, expanding the initialized attention diagram through three deconvolution layers to obtain an attention diagram AM; the generated attention map AM highlights potential focus areas on the fundus color-photographed image to be predicted, and then guides a DR focus segmentation module to identify DR focuses in the fundus color-photographed image so as to obtain prediction results of various DR focuses; in addition, the initial attention diagram generated by the focus attention module is merged with the characteristic diagram output by the DR focus segmentation module encoder, and the merged initial attention diagram and the characteristic diagram are input into a decoder of the DR focus segmentation module, loss information is transmitted back to the focus attention module in the training process, and the problem of gradient disappearance is avoided; the loss function of the DR focus segmentation model is a binary cross entropy loss function, which is specifically represented as:

wherein, N represents the number of training samples, and i represents the ith training sample; y is a binary label of 0 or 1, p (y) is the probability that the output belongs to the y label; the binary label is pixel-level DR focus mark of the input eye ground color image, the mark is a two-dimensional matrix with the same size as the input eye ground color image, if a certain pixel point of the input eye ground color image is marked as a focus, the value of the corresponding position of the mark matrix is 1, otherwise, the value is 0;

based on the above process, the DR focus segmentation model training module trains the DR focus segmentation model in a supervised learning mode by adopting a preprocessed source center DR focus segmentation data set training set and a simulation data set corresponding to a target center DR focus segmentation data set, wherein an Adam optimizer is adopted to update parameters of the DR focus segmentation model, so that loss function values are reduced, and finally a plurality of DR focus segmentation models which are suitable for the eye fundus color image style of the DR focus segmentation data set to which the target center belongs are obtained.

Example 3

The embodiment provides a specific calculation process of prediction results of various DR lesions, which specifically comprises the following steps: adjusting the size of the complete fundus color image to be measured, and performing dot product calculation with the thermodynamic diagram one by one to obtain a weighted fundus color image, wherein the calculation formula is as follows:

X^AM＝(AM_i+1)⊙X

wherein, AM_iThe ith channel, representing a thermodynamic diagram, indicates a pixel-by-pixel matrix multiplication; and inputting the weighted fundus map into a DR focus segmentation module to obtain the prediction results of the four DR focuses.

According to the diabetic retinopathy focus segmentation system suitable for the multi-center image, for an input fundus color photographic image, a focus attention module of the system automatically gives higher weight to a potential focus area in the fundus color photographic image through learning, and inhibits irrelevant background information, so that the DR focus segmentation module further focuses on the focus area to generate characteristic representation with higher resolution, the focus segmentation performance of a focus segmentation model is improved, and misdiagnosis and missed diagnosis of the DR focus are reduced.

In a specific implementation process, the scheme provides a method and a flow for establishing a simulation data set capable of simulating the appearance style of other central images based on a single-center eyeground color photograph image data set and an eyeground color photograph data set of a target center, training a DR focus segmentation model by adopting the simulation data set, and improving the DR focus segmentation performance of the DR focus segmentation model on other central eyeground images DR focus; meanwhile, a DR focus segmentation system is also provided, wherein a focus attention module gives higher weight to a potential focus region in a picture to be predicted and inhibits background information, so that the DR focus segmentation performance is improved, and misdiagnosis and missed diagnosis are reduced.

It should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.

Claims (10)

1. The diabetic retinopathy focus segmentation method adapting to the multi-center image is characterized by comprising the following steps of:

s1: collecting fundus color photograph images from a plurality of centers to construct a multi-center data set; the multi-center dataset comprises a source center dataset and a plurality of target center datasets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set;

s2: for a certain target center, preprocessing the fundus color photograph image in the source center fundus color photograph data set and the target center fundus color photograph data set, and training a cycleGAN model by adopting the preprocessed fundus color photograph image to generate the cycleGAN model of the target center; repeatedly executing the step S2 until the CycleGAN models of all the target centers are obtained;

s3: processing fundus color photography images in the DR focus segmentation data set at the source center by using a trained cycleGAN model, and establishing a simulation target center image style simulation data set by using one cycleGAN model;

s4: preprocessing a source center DR lesion segmentation data set and each simulation data set;

s5: constructing a DR focus segmentation model and training by utilizing a preprocessed source center DR focus segmentation data set and a simulation data set to obtain the DR focus segmentation model which is suitable for the eye fundus color photograph image style of a target center focus segmentation data set; repeatedly executing the step S5 until a plurality of DR focus segmentation models which are suitable for all target centers are obtained;

s6: respectively testing the focus segmentation performance of all DR focus segmentation models on the fundus color-photographed image of the DR focus segmentation data set at the source center by using the DR focus segmentation data set at the source center, and simultaneously evaluating the DR focus segmentation performance of the DR focus segmentation model corresponding to the target center on the fundus color-photographed image of the DR focus segmentation data set at the target center by using the DR focus segmentation data set at the target center; if the DR focus segmentation performance of the source center and the DR focus segmentation performance of the target center of the DR focus segmentation model both meet the requirements, outputting the trained DR focus segmentation model, and executing step S7; otherwise, returning to execute the step S5;

s7: and inputting the target center to which the fundus color image to be predicted belongs into a trained DR focus segmentation model corresponding to the target center, outputting a final segmentation result, and completing the segmentation of the DR focus.

2. The method for segmenting diabetic retinopathy focus according to the claim 1, characterized in that the step S1 specifically includes the following steps:

s11: acquiring a source center database and a plurality of target center databases;

s12: collecting fundus color-photograph images without DR focus or with DR focus and with clear retina structure in DR screening from a source central database and a plurality of target central databases respectively;

s13: extracting a small amount of fundus color-photograph images with DR focuses from each central database to carry out manual labeling;

s14: using the eyeground color photograph image subjected to manual marking as a focus segmentation data set, and using the residual unmarked eyeground color photograph image as an eyeground color photograph data set without a focus label to obtain a source center data set and a plurality of target center data sets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set; dividing a source center DR focus segmentation data set into a training set and a test set, wherein the training set is used for training a DR focus segmentation model in the step S5, the test set is used in the step S6, and the source center DR focus segmentation performance of the fundus color photograph image of the source center DR focus segmentation data set is respectively tested for all DR focus segmentation models; the target center DR lesion segmentation data set is used in step S6 for evaluation of target center DR lesion segmentation performance of a fundus color photograph image of the target center DR lesion segmentation data set for a DR lesion segmentation model of the data set corresponding to the target center.

3. The method for segmenting diabetic retinopathy focus according to claim 2 adapted to multiple central images, wherein in the step S2, the CycleGAN model is a style transition model based on a generation countermeasure network, and the training process of the CycleGAN model by using the preprocessed source central fundus oculi color photograph data set a and one target central fundus oculi color photograph data set B specifically comprises:

defining a source central fundus color photograph data set A as an X domain and a target central fundus color photograph data set B as a Y domain;

the cycleGAN model comprises a first image generator, a first discriminator, a second image generator and a second discriminator; wherein:

a first image generator for generating a pseudo fundus color photographic image having a target central fundus color photographic data set B style from a fundus color photographic image of the X domain;

the first discriminator is used for distinguishing the eyeground color photograph image of the Y domain from the false fundus color photograph image of the target central eyeground color photograph data set B;

a second image generator for generating a pseudo-fundus color image having a source-centric fundus color data set A-style from the fundus color image of the Y domain;

the second discriminator is used for distinguishing the fundus color photograph image of the X domain from the pseudo fundus color photograph image of the source center fundus color photograph data set A;

wherein, the confrontation loss of the first image generator and the first discriminator is as follows:

the countermeasure loss of the second image generator and the second discriminator is:

the cycle consistency loss is:

the integrity loss function is:

L(G,F,D_X,D_Y)＝L_GAN(G,D_Y,X,Y)+L_GAN(F,D_Y,Y,X)+λL_cyc(G,F)

wherein the parameter G represents the first image generator; ex to pdata (x), Ey to pdata (y) each represent a loss function of L1; x in g (X) represents an X-domain fundus color photograph image input to the first image generator; ginseng radix (Panax ginseng C.A. Meyer)Number D_YRepresents a first discriminator; d_YY in (Y) represents a Y-domain fundus color-photograph image input to the first discriminator; the parameter F represents the second image generator; y in f (Y) represents a Y-domain fundus color-photograph image input to the second image generator; parameter D_XRepresents a second discriminator; d_X(x) X in (2) represents an X-domain fundus color photograph image input to the second discriminator; the parameter λ represents the weight coefficient of the cyclic consistency loss;

in the process of training a cycleGAN model by adopting fundus color images preprocessed by a source central fundus color image dataset A and a target central fundus color image dataset B, using an Adam optimizer to alternately update a first image generator and a second image generator in the cycleGAN model and update parameters of a first discriminator and a second discriminator so as to reduce a complete loss function of the model, and finishing training when the network reaches a Nash balance state, namely the discriminator cannot distinguish whether the fundus color images are true or false, so as to obtain a cycleGAN AB model; at this time, the first image generator serves as a style conversion generator capable of converting the fundus color-photograph image of the source central fundus color-photograph dataset a into a fundus color-photograph image having a style of the target central fundus color-photograph dataset B;

and generating a cycleGAN model corresponding to the source central eyeground color photograph data set and the other target central eyeground color photograph data sets according to the same training method.

4. The method for diabetic retinopathy lesion focus segmentation adaptive to multiple central images as claimed in claim 3, wherein in the step S3, the fundus oculi color photograph image in the training set of source central DR lesion focus segmentation data set is processed by using the cycleGAN model corresponding to the target central fundus color photograph data set obtained in the step S2 to obtain a target central DR lesion focus segmentation data set capable of simulating the corresponding target central style, so as to establish a simulation data set corresponding to the target central DR lesion focus segmentation data set; the simulation data set corresponding to all target center DR focus segmentation data sets has the same focus label as the training set of the source center DR focus segmentation data set because the important anatomical structure of the CycleGAN model is kept unchanged when the fundus color photograph image is converted.

5. The method for diabetic retinopathy lesion segmentation adaptive to multi-center image according to claim 4, wherein in the step S5, the constructed DR lesion segmentation model includes a lesion attention module and a DR lesion segmentation module; wherein: the focus attention module consists of a seven-layer convolutional neural network and is used for predicting the probability that a certain local area in the fundus color photographic image is normal or has a certain focus; a convolutional neural network model with an encoder-decoder structure is used in a DR focus segmentation module; for an eyeground color photograph image to be predicted, firstly, the eyeground color photograph image is cut into a plurality of slices in a gridding cutting mode, then, a focus attention module is utilized to predict the slices one by one and generate corresponding probability vectors, and the generated probability vectors correspond to the probabilities of normal focuses and various focuses; then, combining the generated probability vectors into an initial attention map according to the prediction sequence of the focus attention module; finally, expanding the initialized attention diagram through three deconvolution layers to obtain an attention diagram AM; the generated attention map AM highlights potential focus areas on the fundus color-photographed image to be predicted, and then guides a DR focus segmentation module to identify DR focuses in the fundus color-photographed image so as to obtain prediction results of various DR focuses; in addition, the initial attention diagram generated by the focus attention module is merged with the characteristic diagram output by the DR focus segmentation module encoder, and the merged initial attention diagram and the characteristic diagram are input into a decoder of the DR focus segmentation module, loss information is transmitted back to the focus attention module in the training process, and the problem of gradient disappearance is avoided; the loss function of the DR focus segmentation model is a binary cross entropy loss function, which is specifically represented as:

wherein, N represents the number of training samples, and i represents the ith training sample; y is a binary label of 0 or 1, p (y) is the probability that the output belongs to the y label; the binary label is pixel-level DR focus mark of the input eye ground color image, the mark is a two-dimensional matrix with the same size as the input eye ground color image, if a certain pixel point of the input eye ground color image is marked as a focus, the pixel value corresponding to the mark matrix is 1, otherwise, the pixel value is 0;

based on the process, the DR focus segmentation model is trained in a supervised learning mode by adopting a preprocessed source center DR focus segmentation data set training set and a simulation data set corresponding to a target center DR focus segmentation data set, wherein an Adam optimizer is adopted to update parameters of the DR focus segmentation model, so that loss function values are reduced, and finally the DR focus segmentation model suitable for the eye fundus color image style of a plurality of target center DR focus segmentation data sets is obtained.

6. The diabetic retinopathy focus segmentation system adapting to the multi-center image is characterized by comprising a multi-center data set construction module, a first preprocessing module, a CycleGAN model training module, a simulation data set establishment module, a second preprocessing module, a DR focus segmentation model training module, a test evaluation module and a focus segmentation module; wherein:

the multi-center data set construction module is used for collecting fundus color photograph images from a plurality of centers and constructing a multi-center data set; the multi-center dataset comprises a source center dataset and a plurality of target center datasets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set;

the first preprocessing module is used for preprocessing the eyeground color photograph images in the source central eyeground color photograph data set and the target central eyeground color photograph data set;

the cycleGAN model training module is used for training the cycleGAN model by adopting the pretreated fundus color photographic image to generate the cycleGAN model; aiming at a certain target center, generating a CycleGAN model of the target center until the CycleGAN models of all the target centers are generated;

the simulation data set establishing module is used for processing fundus color images in the DR focus segmentation data set of the source center by utilizing a trained cycleGAN model, and one cycleGAN model establishes a simulation target center image style simulation data set;

the second preprocessing module is used for preprocessing the DR focus segmentation data set of the source center and each simulation data set;

the DR focus segmentation model training module is used for constructing a DR focus segmentation model and training the DR focus segmentation model with each simulation data set by utilizing the preprocessed source center DR focus segmentation data set to obtain a DR focus segmentation model which is suitable for the eye fundus color photograph image style of each target center focus segmentation data set;

the testing and evaluating module is used for respectively testing the source center DR focus segmentation performance of all DR focus segmentation models on the fundus color-photographed image of the source center DR focus segmentation data set by using the source center DR focus segmentation data set, and simultaneously evaluating the target center DR focus segmentation performance of the DR focus segmentation model corresponding to the target center on the fundus color-photographed image of the target center DR focus segmentation data set by using the target center DR focus segmentation data set; if the DR focus segmentation performance of the source center DR focus segmentation model and the DR focus segmentation performance of the target center meet the requirements, outputting the trained DR focus segmentation model to a focus segmentation module; otherwise, retraining the DR focus segmentation model by a DR focus segmentation model training module;

the focus segmentation module is used for inputting the eyeground color photograph image to be predicted into a DR focus segmentation model corresponding to the trained target center to which the eyeground color photograph image belongs, outputting a final segmentation result and completing the segmentation of the DR focus.

7. The system for diabetic retinopathy lesion segmentation adapted to the multicenter image according to claim 6, wherein the multicenter data set construction module includes a data acquisition unit, a data screening unit, an artificial labeling unit and a data classification unit; wherein:

the data acquisition unit is used for acquiring a source center database and a plurality of target center databases;

the data screening unit is used for respectively screening DR from the source central database and the target central databases and collecting fundus color photography images without DR focuses or with DR focuses and with clear retina structures;

the manual labeling unit is used for extracting a small amount of fundus color photographic images with DR focuses from the fundus color photographic images of each central database to perform manual labeling;

the data classification unit is used for classifying all fundus color photograph images, taking the fundus color photograph images subjected to manual labeling as a focus segmentation data set, and taking the remaining unmarked fundus color photograph images as fundus color photograph data sets without focus labels to obtain a source center data set and a plurality of target center data sets; wherein the source central data set comprises a source central DR lesion segmentation data set and a source central fundus color photograph data set; the target central data set comprises a target central DR focus segmentation data set and a target central eyeground color photograph data set; dividing a source center DR focus segmentation data set into a training set and a test set, wherein the training set is used for training a DR focus segmentation model in a DR focus segmentation model training module, and the test set is used in a test evaluation module for respectively testing the source center DR focus segmentation performance of all DR focus segmentation models; the target center DR lesion segmentation data set is used for testing and evaluating the target center DR lesion segmentation performance of a DR lesion segmentation model corresponding to the target center in an evaluation module.

8. The system of claim 7, wherein in the cyclic gan model training module, the cyclic gan model is a style migration model based on generation of an antagonistic network, and comprises a first image generator, a first discriminator, a second image generator and a second discriminator; the method comprises the following steps of (1) training a cycleGAN model by using a preprocessed source central eye fundus color photograph data set A and a preprocessed target central eye fundus color photograph data set B, wherein the training process comprises the following steps:

defining a source central fundus color photograph data set A as an X domain and a target central fundus color photograph data set B as a Y domain;

a first image generator generating a pseudo fundus color photographic image having a target central fundus color photographic data set B style from a fundus color photographic image of the X domain; the first discriminator is used for distinguishing the eyeground color photograph image of the Y domain from the false fundus color photograph image of the target central eyeground color photograph data set B; a second image generator for generating a pseudo-fundus color image having a source-centric fundus color data set A-style from the fundus color image of the Y domain; the second discriminator is used for distinguishing the fundus color photograph image of the X domain from the pseudo fundus color photograph image of the source center fundus color photograph data set A; wherein:

the countermeasure loss of the first image generator and the first discriminator is:

the countermeasure loss of the second image generator and the second discriminator is:

the cycle consistency loss is:

the integrity loss function is:

L(G,F,D_X,D_Y)＝L_GAN(G,D_Y,X,Y)+L_GAN(F,D_Y,Y,X)+λL_cyc(G,F)

wherein the parameter G represents the first image generator; ex to pdata (x), Ey to pdata (y) each represent a loss function of L1; x in g (X) represents an X-domain fundus color photograph image input to the first image generator; parameter D_YRepresents a first discriminator; d_YY in (Y) represents a Y-domain fundus color-photograph image input to the first discriminator; the parameter F represents the second image generator; y in f (Y) represents a Y-domain fundus color-photograph image input to the second image generator; parameter D_XRepresents a second discriminator; d_X(x) X in (2) represents an X-domain fundus color photograph image input to the second discriminator; parameter(s)λ represents a weight coefficient of the cyclic consistency loss;

in the process of training a cycleGAN model by adopting fundus color images preprocessed by a source central fundus color image dataset A and a target central fundus color image dataset B, using an Adam optimizer to alternately update a first image generator and a second image generator in the cycleGAN model and update parameters of a first discriminator and a second discriminator so as to reduce a complete loss function of the model, and finishing training when a network reaches a Nash balance state, namely the discriminator cannot distinguish whether the fundus color images are true or false, so as to obtain a cycleGAN AB model; at this time, the first image generator serves as a style conversion generator capable of converting the fundus color-photograph image of the source central fundus color-photograph dataset a into a fundus color-photograph image having a style of the target central fundus color-photograph dataset B;

according to the same training method, the CycleGAN model training module generates a CycleGAN model corresponding to the source center eyeground color photograph data set and the other target center eyeground color photograph data sets through training.

9. The system of claim 8, wherein in the simulation dataset creating module, the cyclic gan model corresponding to the target central fundus oculi color photograph dataset obtained by the cyclic gan model training module is used to process the fundus oculi color photograph images in the source central DR lesion segmentation dataset training set to obtain a target central DR lesion segmentation dataset capable of simulating the corresponding target central style, thereby creating a simulation dataset corresponding to the target central DR lesion segmentation dataset; the simulation data set corresponding to all target center DR focus segmentation data sets has the same focus label as the training set of the source center DR focus segmentation data set because the important anatomical structure of the CycleGAN model is kept unchanged when the fundus color photograph image is converted.

10. The system of claim 9, wherein the DR lesion segmentation model training module includes a lesion attention module and a DR lesion segmentation module; wherein: the focus attention module consists of a seven-layer convolutional neural network and is used for predicting the probability that a certain local area in the fundus color photographic image is normal or has a certain focus; a convolutional neural network model with an encoder-decoder structure is used in a DR focus segmentation module; for an eyeground color photograph image to be predicted, firstly, the eyeground color photograph image is cut into a plurality of slices in a gridding cutting mode, then, a focus attention module is utilized to predict the slices one by one and generate corresponding probability vectors, and the generated probability vectors correspond to the probabilities of normal focuses and various focuses; then, combining the generated probability vectors into an initial attention map according to the prediction sequence of the focus attention module; finally, expanding the initialized attention diagram through three deconvolution layers to obtain an attention diagram AM; the generated attention map AM highlights potential focus areas on the fundus color-photographed image to be predicted, and then guides a DR focus segmentation module to identify DR focuses in the fundus color-photographed image so as to obtain prediction results of various DR focuses; in addition, the initial attention diagram generated by the focus attention module is merged with the characteristic diagram output by the DR focus segmentation module encoder, and the merged initial attention diagram and the characteristic diagram are input into a decoder of the DR focus segmentation module, loss information is transmitted back to the focus attention module in the training process, and the problem of gradient disappearance is avoided; the loss function of the DR focus segmentation model is a binary cross entropy loss function, which is specifically represented as:

wherein, N represents the number of training samples, and i represents the ith training sample; y is a binary label of 0 or 1, p (y) is the probability that the output belongs to the y label; the binary label is pixel-level DR focus mark of the input eye ground color image, the mark is a two-dimensional matrix with the same size as the input eye ground color image, if a certain pixel point of the input eye ground color image is marked as a focus, the value of the corresponding position of the mark matrix is 1, otherwise, the value is 0;

based on the above process, the DR focus segmentation model training module trains the DR focus segmentation model in a supervised learning mode by adopting a preprocessed source center DR focus segmentation data set training set and a simulation data set corresponding to a target center DR focus segmentation data set, wherein an Adam optimizer is adopted to update parameters of the DR focus segmentation model, so that loss function values are reduced, and finally a plurality of DR focus segmentation models which are suitable for the eye fundus color image style of the DR focus segmentation data set to which the target center belongs are obtained.

Images