CN114298206B - Fundus image domain conversion method and fundus image domain conversion system - Google Patents
Fundus image domain conversion method and fundus image domain conversion system Download PDFInfo
- Publication number
- CN114298206B CN114298206B CN202111600197.9A CN202111600197A CN114298206B CN 114298206 B CN114298206 B CN 114298206B CN 202111600197 A CN202111600197 A CN 202111600197A CN 114298206 B CN114298206 B CN 114298206B
- Authority
- CN
- China
- Prior art keywords
- image
- fundus
- images
- color
- fundus image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012549 training Methods 0.000 claims abstract description 37
- 238000012360 testing method Methods 0.000 claims abstract description 14
- 210000001525 retina Anatomy 0.000 claims abstract description 13
- 239000000284 extract Substances 0.000 claims abstract description 3
- 230000006870 function Effects 0.000 claims description 26
- 238000012937 correction Methods 0.000 claims description 9
- 230000002194 synthesizing effect Effects 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 7
- 238000013135 deep learning Methods 0.000 claims description 6
- 230000002708 enhancing effect Effects 0.000 claims description 6
- 238000011426 transformation method Methods 0.000 claims description 5
- 230000015572 biosynthetic process Effects 0.000 claims description 4
- 238000003786 synthesis reaction Methods 0.000 claims description 4
- 238000013528 artificial neural network Methods 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 claims description 2
- 238000005070 sampling Methods 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 4
- 238000003759 clinical diagnosis Methods 0.000 abstract description 3
- 238000004364 calculation method Methods 0.000 abstract description 2
- 230000004913 activation Effects 0.000 description 5
- 230000008485 antagonism Effects 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 2
- 210000004204 blood vessel Anatomy 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000001575 pathological effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Image Processing (AREA)
- Eye Examination Apparatus (AREA)
Abstract
The invention relates to a fundus image domain conversion method and a fundus image domain conversion system, and belongs to the technical field of medical image processing. The method extracts multispectral fundus images and generates three-channel pseudo-color fundus images according to detailed information characteristics of fundus images in different wave bands; extracting an ROI mask of the retina area; the three-band single-channel fundus image is enhanced and synthesized into a three-channel pseudo-color image; forming a training set by relying on true color and pseudo color fundus images; forming a test set by using the pseudo-color fundus image; inputting the training set to generate an countermeasure network, outputting a discrimination result, calculating loss based on a loss function, and continuously optimizing parameters of a generator and a discriminator to obtain a trained generated countermeasure network; outputting fundus images subjected to style conversion based on the trained generation countermeasure network; the system completes implementation of the method, reserves lines and contours of fundus images, solves the problem of scarcity of fundus images, is convenient for clinical diagnosis, consumes less calculation resources and has faster training speed.
Description
Technical Field
The invention relates to a fundus image domain conversion method and a fundus image domain conversion system, and belongs to the technical field of medical image processing.
Background
The multispectral fundus camera can obtain single-spectrum images under the irradiation of light with various wavelengths, so that focus information which cannot be observed by naked eyes or displayed in a true color fundus image is obtained. The image under the single spectrum is a gray level image, and three single spectrum fundus images containing important information are combined together, so that an artificially synthesized multispectral pseudo-color fundus image can be obtained, and a visual effect similar to that of a common fundus image is obtained. The technology can enable the single Zhang Yande image to contain more pathological information, reduce information redundancy caused by complementary wave band imaging, facilitate the diagnosis of focus of medical staff, and also more accord with the film reading habit of the medical staff.
Although pseudo-color fundus images synthesized by single-band images of a plurality of channels reduce factors such as uneven illumination through a filter, a statistical histogram and the like, certain problems still exist, such as black shadows are commonly existing on the periphery of a video disc, global color anomalies of the image, grid-like noise exists in the image, the overall contrast of the image is low and the like, and quality degradation from the factors is unfavorable for medical staff to diagnose diseases, so that the possibility of missed diagnosis and misdiagnosis is increased.
Generating the countermeasure network (GAN, generative Adversarial Networks) is a leading edge technology in the field of image domain conversion. Wherein, as in the prior CycleGAN, the image style can be converted on the premise of preserving the content using the contrast loss function and the loop consistency loss function. For the task of converting a pseudo-color fundus image into a true color style in one direction, the loss function used in the training of the existing image domain conversion algorithm introduces excessive limitation, so that a large amount of computing resources are wasted on an auxiliary network; in addition, since the fundus image data is often not disclosed due to the personal information and privacy concerns, pseudo-color and true-color fundus images are sometimes difficult to acquire, resulting in scarcity of training data. Therefore, it is an object of the present application to convert a multispectral fundus image into a true color fundus image so that the image contains more pathological information and conforms to the film reading habit of medical staff.
Disclosure of Invention
The invention aims at solving the problems that the pseudo-color and true-color fundus images of fundus image data are difficult to acquire sometimes, training data are scarce, and certain quality degradation exists in a synthesized image when the pseudo-color fundus image is synthesized for a single-band fundus image.
In order to achieve the above purpose, the invention adopts the following technical scheme:
the fundus image domain conversion method comprises two parts of multispectral image enhancement and synthesis of a pseudo-color fundus image and conversion of the pseudo-color fundus image style into a true color style based on deep learning;
the multispectral image is enhanced and synthesized into a pseudo-color fundus image, and the method comprises the following steps:
Step 1: extracting multispectral fundus images and selecting three wave band images to be used as red, green and blue channels for generating pseudo-color fundus images according to detailed information characteristics of fundus images of different wave bands;
Wherein, multispectral fundus images, namely fundus images shot in different wave bands;
Step 2: extracting ROI masks of retina areas in the red, green and blue channel images generated in the step 1;
step 3: enhancing the red, green and blue channel images and synthesizing a pseudo-color fundus image, comprising the sub-steps of:
Step 3.1: brightness correction is carried out on fundus images of red, green and blue channels, and fundus images of the red, green and blue channels after brightness correction are generated;
the brightness correction is based on Gao Sidi-pass filtering to remove the gray level non-uniformity of the image;
step 3.2: sharpening the red, green and blue channel images subjected to the brightness correction in the step 3.1 by using an anti-mask to generate sharpened red, green and blue channel fundus images;
Step 3.3: performing brightness equalization on the sharpened red, green and blue channel fundus images to generate red, green and blue channel fundus images with balanced brightness;
step 3.4: performing CLAHE histogram equalization on the red, green and blue channel fundus images with balanced brightness to generate red, green and blue channel fundus images with balanced histograms;
step 3.5: gray balance is carried out on the red, green and blue channel fundus images with balanced histograms, and red, green and blue channel fundus images with balanced gray scale are generated;
Step 3.6: multiplying the ROI mask of the retina area extracted in the step2 with the red, green and blue channel fundus images with balanced gray scale generated in the step 3.5 respectively and combining the images into a pseudo-color fundus image;
so far, from step 1 to step 3.6, multispectral image enhancement and pseudo-color fundus image synthesis are completed;
The neural network model based on deep learning for converting the pseudo-color fundus image style into true color style relies on generates an antagonism network, and the antagonism network comprises a generator and a discriminator;
the generator includes several layers using a jump connection, including an encoder and a decoder;
The encoder comprises a plurality of convolution layers, wherein the convolution layers are used for downsampling an input image and acquiring high-dimensional image characteristics; the decoder is opposite to the encoder in structure, and the obtained high-dimensional image features are subjected to up-sampling processing to obtain an output image;
The input of the generator is a pseudo-color fundus image synthesized in the steps 1to 3.6, the generator extracts different layers of characteristic data, carries out negative sample extraction and contrast learning on the different layers of characteristic data, adjusts style information of the characteristic data, and splices and outputs the characteristic data to form an enhanced image;
The discriminator converts the output image of the generator into a one-dimensional vector, discriminates whether the output image of the generator is a real fundus image or not based on the one-dimensional vector, and specifically discriminates by training the discriminator and optimizing parameters of the generator and the discriminator according to a loss function;
the Loss functions include a contrast Loss function GAN Loss and a multi-image block contrast Loss function PatchNCE Loss;
The GAN Loss is used for enabling the generated image to be similar to the image of the target domain as much as possible; matching the image blocks at the corresponding positions of the input and the output, taking the image blocks at other positions of the same image as a negative sample, and marking the loss as PatchNCE Loss, wherein the effect is to ensure that the input and the output images have the same structure;
The deep learning-based method for converting the pseudo-color fundus image style into the true color style comprises fundus image style conversion model training and fundus image style conversion model implementation;
the fundus image style conversion model training comprises the following steps:
Step 4, shooting a true color fundus image by using a fundus camera and a pseudo color fundus image synthesized from the multispectral fundus image, and forming a training set together; a test set is constructed using pseudo-color fundus images synthesized from multispectral fundus images,
The pseudo-color fundus images in the training set are not required to be matched with the true-color fundus images one by one; the image in the test set is a pseudo-color fundus image and an achromatic color fundus image;
step 5, inputting the training set into a generated countermeasure network, inputting pseudo-color fundus images, sequentially passing through an encoder and a decoder of the generator, outputting a judging result by the generated output images, calculating loss based on a loss function, and continuously optimizing parameters of the generator and the judging device to obtain a trained generated countermeasure network;
the fundus image style conversion model implementation comprises the following steps:
Step 6, inputting the test set into a trained generated countermeasure network, enabling pseudo-color fundus images in the test set to pass through an encoder and a decoder of a generator in sequence, enabling the generated output images to pass through a discriminator and outputting discrimination results, and obtaining the trained generated countermeasure network;
Step 7, obtaining a trained fundus image which is output by a generated countermeasure network and is subjected to style conversion;
A fundus image domain transformation system, comprising: a processor, a memory communicatively coupled to the processor, and instructions stored in the memory and executable on the processor; when the instruction is executed, the multi-spectral image enhancement and the synthesis of pseudo-color fundus images, the fundus image style conversion model training and the fundus image domain conversion implementation can be completed;
The fundus image domain conversion system firstly synthesizes a pseudo-color fundus image from a multispectral fundus image; inputting the synthesized pseudo-color fundus image and the true-color fundus image shot by the common fundus camera into a neural network for training to obtain a trained generation countermeasure network, synthesizing the pseudo-color fundus image from the multispectral fundus image and taking the pseudo-color fundus image as input data of the trained generation countermeasure network, thereby obtaining a fundus image with a true color style, and learning how to convert the color attribute of the pseudo-color fundus image into the color attribute of the true-color fundus image by the generation countermeasure network and reserving all lines and contours in the fundus, wherein the generation countermeasure network effectively solves the problem that a large amount of computation resources are consumed in the training process by the existing conversion method; the fundus image with the true color style keeps lines and contours of the input fundus image, accords with the habit of reading the film by medical workers, and is convenient for clinical diagnosis.
Advantageous effects
Compared with the prior art, the fundus image domain conversion method and system have the following beneficial effects:
1. The fundus image domain conversion method and the fundus image domain conversion system have the advantages that the multispectral image comprises not only visible light wave band images, but also some non-visible light, so that information which cannot be obtained in the visible light wave band can be obtained, the visible light wave band and the non-visible light wave band are combined, more abundant information can be obtained through pseudo-color processing, a doctor can conveniently identify retina focus areas, and a fundus image conforming to the habit of medical staff in reading can be obtained through a method of converting pseudo-color fundus image styles into true color styles based on deep learning;
2. The generator of the fundus image domain conversion support uses contrast learning based on image blocks to extract negative samples from a single image for training learning, so that the image conversion on the single image can be realized, and the problem of scarcity of fundus images is solved;
3. According to the method, a fundus image data training enhancement network can be used, a pseudo-color fundus image is directly subjected to style conversion enhancement by using a trained generator, compared with the existing generation countermeasure network fundus image enhancement method, the paired fundus image data training enhancement network is not needed, a comparison learning method is used, the training consumption calculation resources are less, the training speed is faster, the memory occupation of 2/3 of that of a CycleGAN model can be reduced under the same training parameter setting, the time required by training is shortened by 1/5, and the image style conversion efficiency is improved;
4. According to the method, the generator is obtained through training the countermeasure network, the enhanced image can be directly generated after the pseudo-color fundus image is input, and the complex prior model is prevented from being designed;
5. the method well restores the details and color styles of the retina fundus images through contrast learning, reserves the lines and the outlines of the input fundus images, accords with the film reading habit of medical workers, and is convenient for clinical diagnosis.
Drawings
FIG. 1 is a schematic flow chart of enhancing and synthesizing pseudo-color fundus images in a fundus image domain transformation method of the present invention;
fig. 2 is a schematic diagram of an countermeasure network structure on which the fundus image domain transformation method of the present invention relies;
Fig. 3 is a schematic flow chart of a fundus image domain conversion method according to an embodiment of the present invention.
Detailed Description
The fundus image domain converting method and system according to the present invention will be further described in detail with reference to the accompanying drawings and examples.
Example 1
The embodiment illustrates the specific implementation of the fundus image domain conversion method.
Fig. 1 is a schematic flow chart of enhancing and synthesizing a pseudo-color fundus image by using a multispectral image in a fundus image domain conversion method, which specifically comprises the following steps:
Step A: extracting fundus images shot at 550nm and 620nm, using the 550nm images as green and blue channels for generating pseudo-color fundus images and using the 620nm images as red channels for generating the pseudo-color fundus images according to the detailed information characteristics reserved by the fundus images at different wave bands;
And (B) step (B): extracting ROI masks of retina areas in the red, green and blue channel images generated in the step A:
setting the pixel brightness of the red, green and blue channel images with the brightness greater than 10 to be 1, otherwise setting the pixel brightness to be 0, converting the images into 8-bit images, and performing binary closed operation on the images to obtain an ROI mask of the fundus region of the images;
step C: enhancing the red, green and blue channel images and synthesizing a pseudo-color fundus image, comprising the sub-steps of;
Substep c.1: brightness correction operation is used for fundus images of red, green, and blue channels, and gray-scale unevenness of an image is removed based on gaussian low-pass filtering:
Firstly, the width and height of red, green and blue channel images are respectively reduced to 1/4 of the original width and height, gaussian low-pass filtering is used for the images with reduced sizes, and then the images are reset to the original size, so that a blurred image containing average brightness information of each block of the red, green and blue channel images is obtained, and the blurred image is marked as I blurred. The maximum pixel brightness of the red, green and blue channel blurred images is noted as max (I blurred), the original red, green and blue channel images are noted as I 0, and the corrected red, green and blue channel images I 1 are noted as I 1＝I0-(max(Iblurred)-I0);
Substep c.2: sharpening the brightness-corrected red, green and blue channel images using an anti-mask;
Firstly, normalizing the brightness of images of three channels to 0-255, and converting the brightness into 8-bit images;
Multiplying the red, green and blue channel images corrected by the brightness of the step c.1 by using the ROI mask generated in the step B, and performing boundary filling on the retina area of the obtained image, wherein the purpose of the boundary filling is to prevent the following gaussian low pass filtering from erroneously recognizing the image edge as a high frequency area so as to obtain an incorrect result;
The images of the red, green and blue channels were filtered using gaussian low pass filtering and then filtered using means, the resulting image was designated I 2 and again filtered using gaussian low pass filtering, designated I gaussian. The sharpened image I 3 obtained in this step is I 3＝I2-0.8×(I2-Igaussian);
Substep c.3: performing brightness equalization on the sharpened red, green and blue channel images;
First, the pixel numbers of all the brightness of the red, green and blue channel images in the ROI mask area are counted respectively. When the sum of certain pixel numbers with the lowest brightness exceeds 0.05 of the total pixel number in the ROI mask area, if the highest brightness L 1 in the pixels exceeds 30, the low brightness value L low is set to 10, otherwise (L 1 -20). When the number of certain pixels with highest brightness exceeds 0.05 of the total number of pixels in the ROI mask area, if the lowest brightness L 2 in the pixels exceeds 235, setting the value L high of 'high brightness' to 255, otherwise setting the value to (L 2 +20); the luminance-equalized image I 4 obtained in this step is I 4＝(255÷(Lhigh-Llow))×(I3-Llow);
Substep c.4: performing CLAHE histogram equalization on the fundus images of the red, green and blue channels with the equalized brightness;
Substep c.5: gray-scale equalization is performed on the red, green and blue channel images after histogram equalization, the gray-scale average value of the red, green and blue channels is set to omega (in this embodiment, the gray-scale average values of the three channels are set to 223, 119 and 59 respectively), and in the region of the ROI mask, the sum a of the pixel number n and the pixel brightness of the red, green and blue channel images is counted respectively, and the gray-scale equalized image I 5 obtained in this step is
Substep c.6: multiplying the ROI mask of the retina area extracted in the step B with the red, green and blue channel fundus images with balanced gray scale respectively and combining the images into a pseudo-color fundus image;
fig. 2 is a schematic diagram of an countermeasure network structure on which the fundus image domain transformation method of the present invention relies;
In fig. 1, G represents a generator, D represents a discriminator, a solid line represents an update generator parameter, and a broken line represents an update discriminator parameter; the input of the generator is a pseudo-color fundus image, and the output is a generated enhanced image; the input of the discriminator is the generated enhanced image and true color fundus image; patchNCE Loss updating generator parameters by calculating a multi-layer loss between image blocks of the input and output images; GAN Loss updates the generator and the discriminator parameters by calculating the resistive Loss of the input discriminator image.
Fig. 3 is a schematic flow chart of a fundus image domain conversion method and an embodiment of the present invention, specifically including the following steps:
step E: building a training set and a testing set:
Wherein, the images in the training set and the testing set are retina fundus images collected clinically; the training set contains a pseudo-color fundus image synthesized from multispectral fundus pictures and a true-color fundus image shot by a common fundus camera, and the pseudo-color fundus image and the true-color fundus image do not need to be matched; the images in the test set are pseudo-color fundus images synthesized from the multispectral fundus images, and the test set does not need true-color fundus images as references;
in the embodiment, 92 multispectral pseudo-color fundus images and 94 Zhang Zhen color fundus images collected clinically are used as training sets, and 50 multispectral pseudo-color fundus images are used as test sets;
step F: building and training an enhanced network model: as shown in fig. 1, the network model in the figure is composed of a generator and a discriminator, the generator inputs a multispectral pseudo-color retina fundus image, the output is an enhanced image, the Loss function comprises generation of a contrast Loss GAN Loss and a multi-image block contrast Loss PatchNCE Loss between the input image and the output image, each layer of the generator is connected with each layer of the discriminator by adopting a jump connection mode, and the final layer of the discriminator convolutionally outputs and predicts the final layer of the discriminator, which comprises the following specific steps:
step F.1: a generator is constructed. The input of the generator is a multispectral pseudo-color retina fundus image, the pseudo-color fundus image is enhanced by the generator, and the generated image cannot be distinguished from the true or false true color fundus image as far as possible;
In this embodiment, in order to ensure that structural information such as a video disc and a blood vessel of an input fundus image can be retained, the generator connects the input layer with the corresponding output layer in a jump connection manner. And performing K times of downsampling and corresponding K times of upsampling, and converting the image with the channel number of 3 into 2 K+5 dimensions and then reducing to 3 dimensions. The convolution kernel size of the downsampling is S, the step length is d, and the convolution kernel size, the step length and the channel number of the downsampling correspond to the downsampling. Between the downsampling and upsampling operations, T residual blocks are used, each including convolution, regularization and ReLU activation functions, the activation functions being followed by a second convolution and regularization, the input and output dimensions of each convolution layer in the residual blocks being 2 K+5, each layer performing a batch normalization operation. Each layer is activated using a ReLU function except that the outermost activation function is a hyperbolic tangent function.
Step F.2: constructing a discriminator. The function of the discriminator is to judge whether the image is a true color fundus image or not, and the true or false can be successfully judged through training;
In the embodiment, the discriminator adopts PatchGAN structure, and has P layers of convolution, the front P-1 layer carries out downsampling operation on the input image, and the image with the channel number of 3 is converted into 2 P+5 dimensions; the number of channels in the last layer is 1;
the front P-1 layer comprises a convolution and activation function, the activation function is LeakyReLU with a negative slope of k, the layers 2 to P-1 also comprise batch normalization operation, and a 1-dimensional prediction graph is directly output after the convolution of the last layer;
step F.3: constructing a model loss function;
In this embodiment, the data is represented as Wherein the method comprises the steps ofRepresenting a multispectral pseudo-color fundus image and a true-color fundus image, respectively. Our aim is to transform the style of the pseudo-colour fundus image x to produce a/>, as similar as possible to the true colour fundus image yThe generator may be expressed as/>Training to make/>Approaching y. The discriminator may be expressed as D x→d ε [0,1], where D is close to 0 when x is a pseudo-color fundus image and is close to 1 when x is a true-color fundus image. The Loss function of the method includes two parts, namely generation of the counterloss GAN Loss and inter-block Loss PatchNCE Loss of the network input and output images. Wherein, the Loss of antagonism GAN Loss is generated by/>The expression is shown in the formula (1):
Wherein G and D represent a generator and a discriminator, respectively, and X and Y represent source-domain (pseudo-color) and target-domain (true-color) fundus images, respectively;
PatchNCE Loss function of The expression is shown in the formula (2):
where H denotes a two-layer MLP network, z l denotes a feature stack that the output of the generator first layer produces through the H network, A feature representing the stack of features at a location in space; similarly,/>Expressed as output of generator/>The feature obtained as a network input; /(I)Representing all but this position of the feature stack in space,/>The constraint between the enhanced image and the real image is obtained;
The purpose of GAN Loss is to generate a realistic fundus image, and PatchNCE Loss is to retain fundus structure information such as optic disc blood vessels in the enhanced image. The final objective function, i.e., the constructed loss function, is shown in equation (3):
Wherein lambda X and lambda Y are respectively And/>The weight lost;
Step F.4: model training, parameter updating and saving. During training, the multispectral pseudo-color fundus image is input into the generator G to generate a reinforced pseudo-color fundus image, and the reinforced pseudo-color fundus image and the true-color fundus image are respectively input into the discriminator D to calculate and generate the countermeasures loss Subsequently inputting the true color fundus image into a generator G, generating an enhanced true color fundus image, calculating a multi-layer loss/> between blocks of the twice input and output generator imagesBack propagation and parameter optimization is then performed, and after the training of the first n rounds is finished, the learning rate is gradually adjusted to 0 in a linear manner in the next n rounds of training, so as to fine tune the model. Finally, the trained generator G is saved.
Step G: when the test is carried out, a trained generator G is loaded, and the multispectral pseudo-color retina fundus image is input into the generator to obtain an enhanced fundus image.
From this point on, the whole process of multispectral pseudo-color retinal fundus image enhancement is achieved. Experiments prove that the method can effectively realize the conversion from the pseudo-color fundus image to the true color style, well restore details such as retinal fundus image vascularity, and the like, has true and reliable generation results, and overcomes the defect that the generation of an antagonism network is too dependent on a standard fundus data set.
From the foregoing, it will be appreciated that embodiments of the invention can be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. The foregoing is a preferred embodiment of the present invention, and the present invention should not be limited to the embodiment and the disclosure of the drawings. All equivalents and modifications that come within the spirit of the disclosure are desired to be protected.
Claims (9)
1. A fundus image domain conversion method is characterized in that: the method comprises the steps of enhancing and synthesizing a pseudo-color fundus image by using a multispectral image, and converting the pseudo-color fundus image style into a true color style based on deep learning;
the multispectral image is enhanced and synthesized into a pseudo-color fundus image, and the method comprises the following steps:
Step 1: extracting multispectral fundus images and selecting three wave band images to be used as red, green and blue channels for generating pseudo-color fundus images according to detailed information characteristics of fundus images of different wave bands;
Step 2: extracting ROI masks of retina areas in the red, green and blue channel images generated in the step 1;
step 3: enhancing the red, green and blue channel images and synthesizing a pseudo-color fundus image, comprising the sub-steps of:
Step 3.1: brightness correction is carried out on fundus images of red, green and blue channels, and fundus images of the red, green and blue channels after brightness correction are generated;
step 3.2: sharpening the red, green and blue channel images subjected to the brightness correction in the step 3.1 by using an anti-mask to generate sharpened red, green and blue channel fundus images;
Step 3.3: performing brightness equalization on the sharpened red, green and blue channel fundus images to generate red, green and blue channel fundus images with balanced brightness;
step 3.4: performing CLAHE histogram equalization on the red, green and blue channel fundus images with balanced brightness to generate red, green and blue channel fundus images with balanced histograms;
step 3.5: gray balance is carried out on the red, green and blue channel fundus images with balanced histograms, and red, green and blue channel fundus images with balanced gray scale are generated;
Step 3.6: multiplying the ROI mask of the retina area extracted in the step2 with the red, green and blue channel fundus images with balanced gray scale generated in the step 3.5 respectively and combining the images into a pseudo-color fundus image;
The deep learning-based method for converting the pseudo-color fundus image style into the true color style comprises fundus image style conversion model training and fundus image style conversion model implementation;
the fundus image style conversion model training comprises the following steps:
Step 4, shooting a true color fundus image by using a fundus camera and a pseudo color fundus image synthesized from the multispectral fundus image, and forming a training set together; forming a test set using pseudo-color fundus images synthesized from the multispectral fundus images;
step 5, inputting the training set into a generated countermeasure network, inputting pseudo-color fundus images, sequentially passing through an encoder and a decoder of the generator, outputting a judging result by the generated output images, calculating loss based on a loss function, and continuously optimizing parameters of the generator and the judging device to obtain a trained generated countermeasure network;
the generation countermeasure network comprises a generator and a discriminator; the generator comprises an encoder and a decoder;
the encoder includes a plurality of convolutional layers, and the decoder is structured opposite to the encoder;
The input of the generator is a pseudo-color fundus image synthesized in the steps 1 to 3.6, the generator extracts different layers of characteristic data, carries out negative sample extraction and contrast learning on the different layers of characteristic data, adjusts style information of the characteristic data, and splices and outputs the characteristic data to form an enhanced image;
the discriminator converts the output image of the generator into a one-dimensional vector and discriminates whether the output image of the generator is a real fundus image or not based on the one-dimensional vector;
The Loss functions include a contrast Loss function GAN Loss and a multi-image block contrast Loss function PatchNCE Loss;
the fundus image style conversion model implementation comprises the following steps:
Step 6, inputting the test set into a trained generated countermeasure network, enabling pseudo-color fundus images in the test set to pass through an encoder and a decoder of a generator in sequence, enabling the generated output images to pass through a discriminator and outputting discrimination results, and obtaining the trained generated countermeasure network;
And 7, obtaining a trained fundus image which is output by the generation countermeasure network and is subjected to style conversion.
2. A fundus image domain converting method according to claim 1, wherein: in step 1, multispectral fundus images, i.e., fundus images taken at different bands.
3. A fundus image domain converting method according to claim 1, wherein: the generator comprises several layers using a jump connection.
4. A fundus image domain transformation method according to claim 3, wherein: the encoder downsamples an input image and obtains high-dimensional image features; the decoder performs an up-sampling process on the acquired high-dimensional image features to obtain an output image.
5. A fundus image domain transformation method according to claim 4, wherein: and whether the output image based on the one-dimensional vector judgment generator is a real fundus image or not is judged specifically by training the discriminator and optimizing parameters of the generator and the discriminator according to the loss function.
6. The fundus image domain converting method according to claim 5, wherein: the GAN Loss is used for enabling the generated image to be similar to the image of the target domain as much as possible; the image blocks at the corresponding positions of the input and the output are matched, the image blocks at other positions of the same image are used as negative samples, and the loss is recorded as PatchNCE Loss, so that the same structure between the input image and the output image is ensured.
7. A fundus image domain converting method according to claim 1, wherein: the brightness correction in step 3.1 is based on Gao Sidi-pass filtering to remove the gray-scale non-uniformity of the image.
8. A fundus image domain converting method according to claim 1, wherein: step 4, the pseudo-color fundus images in the training set are not required to be matched with the true-color fundus images one by one; the image in the test set is a pseudo-color fundus image and an achromatic fundus image.
9. A fundus image domain transformation system, characterized by: comprising the following steps: a processor, a memory communicatively coupled to the processor, and instructions stored in the memory and executable on the processor; the instructions, when executed, perform the multi-spectral image enhancement and synthesis of pseudo-color fundus images, fundus image style conversion model training, and fundus image domain conversion implementation in a fundus image domain conversion method according to any one of claims 1 to 8, specifically comprising: firstly, synthesizing a pseudo-color fundus image from a multispectral fundus picture; then inputting the synthesized pseudo-color fundus image and the true-color fundus image shot by the common fundus camera into a neural network for training, and obtaining a trained generation countermeasure network; the pseudo-color fundus image is synthesized from the multispectral fundus picture and used as the trained input data for generating the countermeasure network, so that the fundus image with the true color style is obtained, the countermeasure network is generated for learning how to convert the color attribute of the pseudo-color fundus image into the color attribute of the true color fundus image and retain all lines and contours in the fundus, and the fundus image with the true color style retains the lines and contours of the input fundus image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111600197.9A CN114298206B (en) | 2021-12-24 | 2021-12-24 | Fundus image domain conversion method and fundus image domain conversion system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111600197.9A CN114298206B (en) | 2021-12-24 | 2021-12-24 | Fundus image domain conversion method and fundus image domain conversion system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114298206A CN114298206A (en) | 2022-04-08 |
| CN114298206B true CN114298206B (en) | 2024-05-24 |
Family
ID=80969272
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111600197.9A Active CN114298206B (en) | 2021-12-24 | 2021-12-24 | Fundus image domain conversion method and fundus image domain conversion system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114298206B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110264424A (en) * | 2019-06-20 | 2019-09-20 | 北京理工大学 | A kind of fuzzy retinal fundus images Enhancement Method based on generation confrontation network |
| WO2019240257A1 (en) * | 2018-06-15 | 2019-12-19 | キヤノン株式会社 | Medical image processing device, medical image processing method and program |
| CN110610152A (en) * | 2019-09-10 | 2019-12-24 | 西安电子科技大学 | Multispectral cloud detection method based on discriminative feature learning unsupervised network |
| JP2020058800A (en) * | 2018-10-10 | 2020-04-16 | キヤノン株式会社 | Image processing device, image processing method, and image processing program |
| CN111340743A (en) * | 2020-02-18 | 2020-06-26 | 云南大学 | Semi-supervised multispectral and panchromatic remote sensing image fusion method and system |
| CN111402179A (en) * | 2020-03-12 | 2020-07-10 | 南昌航空大学 | Image synthesis method and system combining countermeasure autoencoder and generation countermeasure network |
| KR20210103717A (en) * | 2020-02-14 | 2021-08-24 | 세종대학교산학협력단 | Multispectral image conversion method and apparatus |
-
2021
- 2021-12-24 CN CN202111600197.9A patent/CN114298206B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019240257A1 (en) * | 2018-06-15 | 2019-12-19 | キヤノン株式会社 | Medical image processing device, medical image processing method and program |
| JP2020058800A (en) * | 2018-10-10 | 2020-04-16 | キヤノン株式会社 | Image processing device, image processing method, and image processing program |
| CN110264424A (en) * | 2019-06-20 | 2019-09-20 | 北京理工大学 | A kind of fuzzy retinal fundus images Enhancement Method based on generation confrontation network |
| CN110610152A (en) * | 2019-09-10 | 2019-12-24 | 西安电子科技大学 | Multispectral cloud detection method based on discriminative feature learning unsupervised network |
| KR20210103717A (en) * | 2020-02-14 | 2021-08-24 | 세종대학교산학협력단 | Multispectral image conversion method and apparatus |
| CN111340743A (en) * | 2020-02-18 | 2020-06-26 | 云南大学 | Semi-supervised multispectral and panchromatic remote sensing image fusion method and system |
| CN111402179A (en) * | 2020-03-12 | 2020-07-10 | 南昌航空大学 | Image synthesis method and system combining countermeasure autoencoder and generation countermeasure network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114298206A (en) | 2022-04-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | Low-light image enhancement via progressive-recursive network | |
| CN109919869B (en) | Image enhancement method and device and storage medium | |
| Jiang et al. | A switched view of Retinex: Deep self-regularized low-light image enhancement | |
| CN110276767A (en) | Image processing method and device, electronic equipment, computer readable storage medium | |
| CN109816666B (en) | Symmetrical full convolution neural network model construction method, fundus image blood vessel segmentation device, computer equipment and storage medium | |
| CN111402146A (en) | Image processing method and image processing apparatus | |
| CN112446880B (en) | Image processing method, electronic device and readable storage medium | |
| CN113822830B (en) | Multi-exposure image fusion method based on depth perception enhancement | |
| CN111488912B (en) | Laryngeal disease diagnosis system based on deep learning neural network | |
| CN111415304A (en) | Underwater vision enhancement method and device based on cascade deep network | |
| CN116681636B (en) | Light infrared and visible light image fusion method based on convolutional neural network | |
| CN117994154B (en) | Intelligent image denoising method based on sensor | |
| CN113344804B (en) | Training method of low-light image enhancement model and low-light image enhancement method | |
| CN115578284A (en) | Multi-scene image enhancement method and system | |
| CN111223110A (en) | Microscopic image enhancement method and device and computer equipment | |
| CN113096023A (en) | Neural network training method, image processing method and device, and storage medium | |
| CN115641391A (en) | Infrared image colorizing method based on dense residual error and double-flow attention | |
| CN114066780A (en) | 4k endoscope image defogging method and device, electronic equipment and storage medium | |
| CN116503422A (en) | Eye cup video disc segmentation method based on attention mechanism and multi-scale feature fusion | |
| CN115797205A (en) | Unsupervised single image enhancement method and system based on Retinex fractional order variation network | |
| CN116883429A (en) | Liver and liver tumor segmentation method based on parallel residual attention | |
| CN109872337B (en) | Eye fundus image optic disc segmentation method based on rapid mean shift | |
| CN107133932A (en) | Retina image preprocessing method and device and computing equipment | |
| US20190220956A1 (en) | Image processing method, image processing device and nonvolatile storage medium | |
| CN113554568A (en) | Unsupervised circulating rain removal network method based on self-supervision constraint and unpaired data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |