CN112837805A - Deep learning-based eyelid topological morphology feature extraction method - Google Patents
Deep learning-based eyelid topological morphology feature extraction method Download PDFInfo
- Publication number
- CN112837805A CN112837805A CN202110036779.2A CN202110036779A CN112837805A CN 112837805 A CN112837805 A CN 112837805A CN 202110036779 A CN202110036779 A CN 202110036779A CN 112837805 A CN112837805 A CN 112837805A
- Authority
- CN
- China
- Prior art keywords
- eyelid
- module
- sampling
- output
- cornea
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 210000000744 eyelid Anatomy 0.000 title claims abstract description 138
- 238000013135 deep learning Methods 0.000 title claims abstract description 15
- 238000000605 extraction Methods 0.000 title claims abstract description 7
- 210000004087 cornea Anatomy 0.000 claims abstract description 62
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 25
- 238000012549 training Methods 0.000 claims abstract description 15
- 241000669069 Chrysomphalus aonidum Species 0.000 claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 5
- 210000001747 pupil Anatomy 0.000 claims abstract description 4
- 238000005070 sampling Methods 0.000 claims description 108
- 230000011218 segmentation Effects 0.000 claims description 61
- 238000011176 pooling Methods 0.000 claims description 26
- 210000003786 sclera Anatomy 0.000 claims description 11
- 230000002123 temporal effect Effects 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000012952 Resampling Methods 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 6
- 210000001061 forehead Anatomy 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 238000009792 diffusion process Methods 0.000 claims description 3
- 230000001815 facial effect Effects 0.000 claims description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 claims description 2
- 208000032610 autosomal dominant 2 intellectual disability Diseases 0.000 claims 3
- 201000000188 autosomal dominant non-syndromic intellectual disability 2 Diseases 0.000 claims 3
- 238000005259 measurement Methods 0.000 abstract description 7
- 230000000877 morphologic effect Effects 0.000 abstract description 4
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 3
- 201000010099 disease Diseases 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000011514 reflex Effects 0.000 description 3
- 206010015995 Eyelid ptosis Diseases 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 201000003004 ptosis Diseases 0.000 description 2
- 208000028006 Corneal injury Diseases 0.000 description 1
- 208000004350 Strabismus Diseases 0.000 description 1
- 206010044604 Trichiasis Diseases 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 206010005159 blepharospasm Diseases 0.000 description 1
- 230000000744 blepharospasm Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 208000035475 disorder Diseases 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 208000030533 eye disease Diseases 0.000 description 1
- 210000000720 eyelash Anatomy 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 206010023332 keratitis Diseases 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000007420 reactivation Effects 0.000 description 1
- 238000004171 remote diagnosis Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/193—Preprocessing; Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A10/00—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE at coastal zones; at river basins
- Y02A10/40—Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Human Computer Interaction (AREA)
- Ophthalmology & Optometry (AREA)
- Databases & Information Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Pathology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an eyelid topological morphology feature extraction method based on deep learning. The method specifically comprises the following steps: acquiring an electronic digital photo of a normal person, processing the electronic digital photo to construct an ROI (region of interest) image training set, and inputting the ROI image training set into a convolutional neural network to be trained to obtain a trained convolutional neural network; positioning a region of interest (ROI) of the eye by using a face recognition method for the electronic digital photo to be detected, obtaining an image of the ROI to be detected, inputting the image of the ROI to be detected into a trained convolutional neural network to output an image with an eyelid contour line and a cornea contour line, determining a circular scale and a pupil center of the electronic digital photo to be detected, and extracting eyelid topological morphological characteristics of a single eye. The method uses the convolutional neural network to segment the eyelid and cornea structures, and uses MeanShift clustering to determine the center of the pupil and then automatically calculate the parameters of the eyelid-related structures, thereby obtaining the accuracy equivalent to that of manual measurement.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an eyelid topological morphology feature extraction method based on deep learning.
Background
Normal eyelid position is the basis for achieving normal eye function, and assessment of eyelid morphology and position is important for reshaping the eye (e.g., ptosis, trichiasis), ocular surface disorders (e.g., exposed keratitis), and Graves' eye disease.
Currently, the clinically common scales manually measure the upper eyelid margin reflex distance (MRD1), lower eyelid margin reflex distance (MRD2), and palpebral fissure size (PF) of a patient for assessing the eyelid position. However, accurate measurements require long experience and a high degree of cooperation from the person who is measuring, while the reproducibility and stability of manual measurements are poor. At the same time, these linear indicators do not fully reflect the complete eyelid contour morphology. By analyzing the electronic photograph, the problem of poor reproducibility and stability of manual measurement can be solved, however, when the traditional automatic analysis method such as a Canny boundary detection algorithm is used, the interference of eyelashes can be encountered, so that the eyelid boundary cannot be accurately identified, and meanwhile, the ordinary method of fitting the circle center by three points to determine the pupil center is also subjected to certain defects due to the non-perfect circle of the iris. To realize full-automatic eyelid structure analysis, accurate eyelid boundary identification and pupil center positioning must be based on. The eyelid topological feature extraction method based on deep learning is constructed, the deep convolutional neural network is used for accurately segmenting the cornea and the eyelid boundary, the pupil center is positioned, the method is a key technology for measuring and evaluating the eyelid topological feature, and urgent clinical requirements are met in automatic and remote diagnosis and evaluation of eyelid-related diseases.
Disclosure of Invention
In order to overcome the existing problems in the background art, the invention aims to provide an eyelid topological feature extraction method based on deep learning, which realizes automatic identification of eyelid-related structures and automatic measurement and calculation of eyelid topological features.
The technical scheme adopted by the invention is as follows:
the invention comprises the following steps:
step 1: collecting electronic digital photos of normal people, and forming a facial photo data set by the electronic digital photos;
step 2: processing the electronic digital photos marked with the eyelid contour lines and the cornea contour lines to obtain binary segmentation images, and forming a binary segmentation image data set by the binary segmentation images of all the electronic digital photos;
and step 3: positioning an eye region-of-interest (ROI) position of a binary segmentation image in a binary segmentation image data set by using a face recognition method to obtain an ROI region binary segmentation image of the binary segmentation image, and forming an ROI image training set by the ROI region binary segmentation images of all the binary segmentation images;
and 4, step 4: constructing an AGN (extension-gated Network) -based convolutional neural Network;
and 5: inputting the ROI image training set obtained in the step (3) into the convolutional neural network obtained in the step (4) to obtain a trained convolutional neural network;
step 6: positioning a region of interest (ROI) of an eye by using a face recognition method for an electronic digital photo to be detected, obtaining an image of the ROI to be detected, inputting the image of the ROI to be detected into a trained convolutional neural network, outputting classification probability that each pixel point is a cornea, an eyelid and a background by the trained convolutional neural network, judging the classification probability of each pixel point according to a preset threshold, classifying each pixel point, forming a cornea region, an eyelid region and a background region of the image of the ROI to be detected after classifying various pixel points, and finally outputting an image with an eyelid contour line and a cornea contour line;
and 7: repeating the steps for a plurality of times to randomly select three pixel points with eyelid contour lines and cornea contour lines in the cornea contour line image and fit the circle centers of circles where the three pixel points are located, determining a clustering center by using a clustering method for the circle centers obtained by multiple times of fitting, and taking the clustering center as a pupil center;
and 8: a round mark is pasted at the forehead of the electronic digital photo to be detected in the step 6, the round mark is detected in an HSV color space by using a Hough coding method, and a round scale is obtained through calculation;
and step 9: and (3) calculating the image with the eyelid contour line and the cornea contour line obtained in the step (6) by using the circular scale obtained in the step (8) and the pupil center positioned in the step (7) to obtain the upper eyelid edge reflection distance (MRD1), the lower eyelid edge emission distance (MRD2), the eyelid fissure size (PF), the upper eyelid length, the lower eyelid length, the cornea area, the nose side area and the temporal side area of the single eye, and forming eyelid topological morphology characteristics by the upper eyelid edge reflection distance (MRD1), the lower eyelid edge emission distance (MRD2), the eyelid fissure size (PF), the upper eyelid length, the lower eyelid length, the cornea area, the nose side area and the temporal side area.
The electronic digital photo in the step 1 and the electronic digital photo to be detected in the step 6 are both required to be full face, a round mark is pasted at the forehead, and the shot person is positioned at the first eye position right in front of the eye.
The step 2 specifically comprises the following steps:
respectively converting the eyelid contour line image and the cornea contour line image corresponding to the electronic digital photo in the step 1 into an eyelid binary segmentation image and a cornea binary segmentation image by using a water diffusion filling method, overlapping the eyelid binary segmentation image and the corresponding cornea binary segmentation image to obtain a binary segmentation image, and forming a binary segmentation image data set by all the binary segmentation images.
The ROI areas include the upper eyelid, lower eyelid, cornea, pupil, and sclera visibility region.
The convolutional neural network in the step 4 comprises a down-sampling module and an up-sampling module, wherein the down-sampling module mainly comprises a first convolutional Pooling module, a second convolutional Pooling module, a third convolutional Pooling module and a fourth convolutional Pooling module which are sequentially connected, the convolutional Pooling module mainly comprises a down-sampling convolutional module and a maximum Pooling module which are sequentially connected, the down-sampling convolutional module mainly comprises a first convolutional layer (conv), a first Batch Normalization layer (BN), a first ReLU active layer, a second convolutional layer, a second Batch Normalization layer and a second ReLU active layer which are sequentially connected, and the maximum Pooling module comprises two maximum Pooling layers (Max Pooling); the up-sampling module comprises a convolution module, four up-sampling convolution modules, four gate control units, four up-sampling sub-modules and an up-sampling convolution layer, wherein the up-sampling convolution module and the down-sampling convolution module have the same structure, and the up-sampling sub-modules mainly comprise B-spline interpolation operation; the output of the first gate control unit and the output of the first up-sampling sub-module are subjected to characteristic splicing and then input into the first up-sampling convolution module; the output of the first up-sampling convolution module and the output of the third down-sampling convolution module are input into a second gating unit, the output of the first up-sampling convolution module is also input into a second up-sampling sub-module, and the output of the second gating unit and the output of the second up-sampling sub-module are subjected to characteristic splicing and then input into the second up-sampling convolution module; the output of the second up-sampling convolution module and the output of the second down-sampling convolution module are input into a third gate control unit, the output of the second up-sampling convolution module is also input into a third up-sampling sub-module, and the output of the third gate control unit and the output of the third up-sampling sub-module are subjected to characteristic splicing and then input into a third up-sampling convolution module; the output of the third up-sampling convolution module and the output of the first down-sampling convolution module are input into a fourth gating unit, the output of the third up-sampling convolution module is also input into a fourth up-sampling sub-module, and the output of the fourth gating unit and the output of the fourth up-sampling sub-module are subjected to characteristic splicing and then input into a fourth up-sampling convolution module; and the output of the fourth up-sampling convolution module is input to the up-sampling convolution layer, the output of the up-sampling convolution layer is input to the Softmax classification layer, and finally the semantic segmentation result of the ROI area image to be detected is obtained.
The gate control unit is specifically as follows: the first input and the second input of the gate control unit are subjected to pixel addition and input to a third ReLU active layer after passing through respective gate control convolutional layers, the third ReLU active layer is subjected to resampling after passing through a third convolutional layer and a first Sigmoid active layer in sequence, the output after resampling is subjected to jump connection with the second input and then output, and the output after jump connection is used as the output of the gate control unit;
the first input is the output of a downsampling convolution module; the second input is the output of the convolution module or the output of the up-sampling convolution module; the skip connection is a pixel-by-pixel multiplication of the resampled output with the second input with a weight alpha.
The calculation of the circular scale is specifically as follows:
and detecting the original mark in the HSV color space by using a Hough coding method, taking the distance between two pixels with the longest distance on the edge of the circle mark as the diameter of the circle mark, taking the number of the pixels occupied by the diameter of the circle mark as the pixel value corresponding to the actual diameter of the circle mark, and calculating the actual diameter of the circle mark divided by the number of the pixels corresponding to the actual diameter of the circle mark to obtain the circular scale.
In step 9, MRD1 is the vertical distance from the pupil center to the upper eyelid margin, MRD2 is the vertical distance from the pupil center to the upper eyelid margin, PF is the vertical distance from the upper eyelid margin to the lower eyelid margin and passing through the pupil center, the length of the upper and lower eyelids is the geometric length of the upper and lower eyelid margins with the inner outer canthus as the starting point, the area of the exposed part of the cornea is the area of the cornea when the cornea is at the first eye position, the area of the sclera is the area of the sclera on the nasal side of the cornea when the nasal side is at the first eye position, and the area of the sclera is the area of the sclera on the temporal side of the cornea when the eyelid is at the first eye position.
Compared with the prior art, the invention has the advantages and beneficial effects that:
the eyelid structure of the electronic digital photo is segmented by using the deep convolutional neural network, and compared with a manual measurement method, the eyelid structure segmentation method has better repeatability and stability, short matching time of a required patient and easily obtained photo materials, and provides technical support for the realization of remote medical treatment and automatic diagnosis.
Compared with the traditional segmentation method, the deep neural network-based segmentation method has the advantages that a more accurate segmentation effect can be obtained, the interference of nearby tissue structures is small, and the eyelid topological structure analysis has a more accurate structural basis.
The method uses the MeanShift clustering based on the Gaussian kernel to obtain the clustering center which is set as the pupil center, reduces the positioning deviation between the fitting circle center generated by the non-perfect circle of the iris and the actual pupil center, ensures that the calculation of the eyelid related morphological parameters is more accurate and objective, further improves the accuracy and reliability of the method, assists the objective evaluation of the eyelid morphological parameters of remote and multi-center, and objectively carries out automatic diagnosis on the related diseases.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention;
FIG. 2 is a schematic diagram of a convolutional neural network of the present invention;
FIG. 3 is a schematic of the attention mechanism of the present invention;
fig. 4 is a schematic diagram of eyelid topomorphism measurement parameters of the present invention.
Detailed Description
The invention is further described with reference to the accompanying drawings and the detailed description.
As shown in fig. 1, the present invention comprises the steps of:
step 1: 1581 electronic digital photos of normal people are collected from an ophthalmologic center of a certain hospital, the shooting range is required to be a whole face, a flat round mark with the diameter of 10mm is pasted at the forehead, and a shot person is positioned at a first eye position right in front of the eye. Patients with ptosis, blepharospasm, strabismus or corneal trauma, and poor quality pictures with blurred pictures were excluded from the study. The photos are taken by a Canon EOS 500D single lens reflex with a 100mm macro lens, and the photos are uploaded to a computer to obtain electronic digital photos with the resolution of 4752 x 3618. Constructing a facial photo data set by using the electronic digital photos;
step 2: processing the electronic digital photos marked with the eyelid contour lines and the cornea contour lines to obtain binary segmentation images, and forming a binary segmentation image data set by the binary segmentation images of all the electronic digital photos;
the step 2 specifically comprises the following steps:
respectively converting the eyelid contour line image and the cornea contour line image corresponding to the electronic digital photo in the step 1 into an eyelid binary segmentation image and a cornea binary segmentation image by using a water diffusion filling method, overlapping the eyelid binary segmentation image and the corresponding cornea binary segmentation image to obtain a binary segmentation image, and forming a binary segmentation image data set by all the binary segmentation images.
And step 3: positioning an eye region-of-interest (ROI) position of a binary segmentation image in a binary segmentation image data set by using a face recognition method to obtain an ROI region binary segmentation image of the binary segmentation image, and forming an ROI image training set by the ROI region binary segmentation images of all the binary segmentation images; the ROI area includes the upper eyelid, lower eyelid, cornea, pupil, and sclera visibility region.
And 4, step 4: constructing an AGN (extension-gated Network) -based convolutional neural Network;
as shown in fig. 2, the convolutional neural network in step 4 includes a down-sampling module and an up-sampling module, the down-sampling module mainly includes a first convolution Pooling module, a second convolution Pooling module, a third convolution Pooling module and a fourth convolution Pooling module, the convolution Pooling module mainly includes a down-sampling convolution module and a maximum Pooling module, the down-sampling convolution module mainly includes a first convolution layer (conv), a first Batch Normalization layer (BN), a first re-lu activation layer, a second convolution layer, a second Batch Normalization layer and a second re-activation layer, which are sequentially connected, and the maximum Pooling module includes two maximum Pooling layers (Max Pooling); the up-sampling module comprises a convolution module, four up-sampling convolution modules, four gate control units, four up-sampling sub-modules and an up-sampling convolution layer, wherein the up-sampling convolution module and the down-sampling convolution module have the same structure, and the up-sampling sub-modules mainly comprise B-spline interpolation operation; the output of the first gate control unit and the output of the first up-sampling sub-module are subjected to characteristic splicing and then input into the first up-sampling convolution module; the output of the first up-sampling convolution module and the output of the third down-sampling convolution module are input into a second gating unit, the output of the first up-sampling convolution module is also input into a second up-sampling sub-module, and the output of the second gating unit and the output of the second up-sampling sub-module are subjected to characteristic splicing and then input into the second up-sampling convolution module; the output of the second up-sampling convolution module and the output of the second down-sampling convolution module are input into a third gate control unit, the output of the second up-sampling convolution module is also input into a third up-sampling sub-module, and the output of the third gate control unit and the output of the third up-sampling sub-module are subjected to characteristic splicing and then input into a third up-sampling convolution module; the output of the third up-sampling convolution module and the output of the first down-sampling convolution module are input into a fourth gating unit, the output of the third up-sampling convolution module is also input into a fourth up-sampling sub-module, and the output of the fourth gating unit and the output of the fourth up-sampling sub-module are subjected to characteristic splicing and then input into a fourth up-sampling convolution module; and the output of the fourth up-sampling convolution module is input to the up-sampling convolution layer, the output of the up-sampling convolution layer is input to the Softmax classification layer, and finally the semantic segmentation result of the ROI area image to be detected is obtained.
As shown in fig. 3, the gate control unit specifically includes: the first input and the second input of the gate control unit respectively pass through respective 1 × 1 gate control convolution layers, then are subjected to pixel sum addition and are input into a third ReLU activation layer, the 1 × 1 gate control convolution layers convert the number of characteristic channels of the first input and the second input into the same number, the third ReLU activation layer sequentially passes through a 1 × 1 third convolution layer and a first Sigmoid activation layer and then is subjected to resampling, the output after resampling is subjected to jump connection with the second input and then is output, the number of the characteristic channels is reduced to 1 by the 1 × 1 third convolution layer, and the output after jump connection is used as the output of the gate control unit;
the first input is the output of the downsampling convolution module; the second input is the output of the convolution module or the output of the up-sampling convolution module; the skip connection is a pixel-by-pixel multiplication of the resampled output with the second input with a weight alpha.
And 5: 1378 pieces of ROI area binary segmentation images in the ROI image training set obtained in the step 3 are randomly selected and input into the convolutional neural network in the step 4. Of these, 882 participants (1764 eyes) were used as the training set, 220 participants (440 eyes) were used as the validation set, and 276 participants (552 eyes) were used as the test set.
During training, the set learning rate is 0.001, the training round is 100 rounds, the learning rate is attenuated at 20 rounds, the attenuation rate is 0.1, the learning rate of each training is smaller than or equal to the learning rate of the previous training, and a trained convolutional neural network is obtained for eyelid and cornea segmentation of the electronic digital photos;
step 6: positioning a region of interest (ROI) of an eye by using a face recognition method for an electronic digital photo to be detected, obtaining an image of the ROI to be detected, inputting the image of the ROI to be detected into a trained convolutional neural network, outputting classification probability that each pixel point is a cornea, an eyelid and a background by the trained convolutional neural network, judging the classification probability of each pixel point according to a preset threshold, classifying each pixel point, forming a cornea region, an eyelid region and a background region of the image of the ROI to be detected after classifying various pixel points, and finally outputting an image with an eyelid contour line and a cornea contour line; the background area is an image except a cornea area and an eyelid area in the ROI area image to be measured. The eyelid contour line is the boundary between the eyelid area and the background area, and the cornea contour line is the boundary between the cornea area and the eyelid area.
And 7: repeating the steps for a plurality of times to randomly select three pixel points with eyelid contour lines and cornea contour lines in the cornea contour line image and fit the circle centers of circles where the three pixel points are located, determining a clustering center by using a MeanShift clustering method of Gaussian kernels for the circle centers obtained by multiple times of fitting, and taking the clustering center as a pupil center;
and 8: a round mark is pasted at the forehead of the electronic digital photo to be detected in the step 6, the round mark is detected in an HSV color space by using a Hough coding method, and a round scale is obtained through calculation;
the calculation of the circular scale is specifically as follows:
the original mark is detected in HSV color space by using a Hough coding method, the distance between two pixels with the longest distance on the edge of the circle mark is taken as the diameter of the circle mark, the number of the pixels occupied by the diameter of the circle mark is taken as the pixel value corresponding to the actual diameter of the circle mark of 10mm, and the number of the pixels corresponding to the actual diameter of the circle mark of 10mm divided by the actual diameter of the circle mark of 10mm is calculated as a circular scale R.
And step 9: and (4) selecting 203 pieces of ROI area binary segmentation images which are not used for convolutional neural network training in the ROI image training set in the step (5), repeating the step (6-8), and performing cornea and eyelid structure segmentation on the 203 pieces of ROI area binary segmentation images to obtain 203 pieces of images with eyelid contour lines and cornea contour lines.
Step 10: calculating the image with the eyelid contour line and the cornea contour line obtained in the step 9 by using the circular scale obtained in the step 8 and the pupil center positioned in the step 7 to obtain the upper eyelid edge reflection distance MRD1, the lower eyelid edge emission distance MRD2, the eyelid fissure size PF and the upper eyelid length L of the single eyeulLength L of lower eyelidll、Corneal region AcNasal area AnAnd temporal area AtThe eyelid topological morphology features are composed of an upper eyelid edge reflection distance MRD1, a lower eyelid edge emission distance MRD2, a palpebral fissure size PF, an upper eyelid length, a lower eyelid length, a cornea area, a nose side area and a temporal side area.
As shown in fig. 4, in step 10, the eyelid topological characteristics are all established at the first eye position of the subject directly in front of the eye, MRD1 is the vertical distance from the pupil center to the upper eyelid margin, MRD2 is the vertical distance from the pupil center to the upper eyelid margin, PF is the vertical distance from the upper eyelid margin to the lower eyelid margin and passing through the pupil center, i.e., the sum of MRD1 and MRD2, the lengths of the upper and lower eyelids are the geometric lengths of the upper and lower eyelid margins with the inner and outer canthus as the starting points, the corneal area is the area of the exposed part of the cornea at the first eye position, the nasal area is the area of the scleral area of the eyelid fissure at the nasal side of the cornea at the first eye position, and the temporal area is the area of the scleral area of the eyelid fissure at the temporal side of the cornea at the first eye position.
The specific calculation method satisfies the formulas (1) to (8):
MRD1＝NMRD1×R (1)
MRD2＝NMRD2×R (2)
PF＝MRD1+MRD2 (3)
At＝Nt×R2 (4)
An＝Nn×R2 (5)
Ac＝Nc×R2 (6)
Lul＝Nul×R (7)
Lll＝Nll×R (8)
the number of pixels of the upper eyelid edge reflection distance MRD1 is NMRD1, the number of pixels of the lower eyelid edge reflection distance MRD2 is NMRD2, Nul is the number of pixels of the upper eyelid length, Nll is the number of pixels of the lower eyelid length, Nc is the number of pixels in a cornea area, Nn is the number of pixels in a nose area, Nt is the number of pixels in a temporal area, and R is a circular scale.
The method realizes accurate cornea and upper and lower eyelid segmentation through deep learning. The automatic measurement of the eyelid topological morphological parameters based on the invention has high accuracy and good repeatability, and can be applied to the fields of automatic disease diagnosis, telemedicine, surgical evaluation and the like.
Claims (8)
1. An eyelid topological morphology feature extraction method based on deep learning is characterized by comprising the following steps: the method comprises the following steps:
step 1: collecting electronic digital photos of normal people, and forming a facial photo data set by the electronic digital photos;
step 2: processing the electronic digital photos marked with the eyelid contour lines and the cornea contour lines to obtain binary segmentation images, and forming a binary segmentation image data set by the binary segmentation images of all the electronic digital photos;
and step 3: positioning the position of an eye region of interest of a binary segmentation image in a binary segmentation image data set by using a face recognition method to obtain an ROI (region of interest) binary segmentation image of the binary segmentation image, and forming an ROI image training set by the ROI binary segmentation images of all the binary segmentation images;
and 4, step 4: constructing a convolutional neural network based on AGN;
and 5: inputting the ROI image training set obtained in the step (3) into the convolutional neural network obtained in the step (4) to obtain a trained convolutional neural network;
step 6: positioning the position of an eye region of interest by using a face recognition method for an electronic digital photo to be detected, obtaining an ROI image to be detected, inputting the ROI image to be detected into a trained convolutional neural network, outputting classification probability of each pixel point as a cornea, an eyelid and a background by the trained convolutional neural network, judging the classification probability of each pixel point according to a preset threshold, classifying each pixel point, forming a cornea region, an eyelid region and a background region of the ROI image to be detected after classifying various pixel points, and finally outputting an image with an eyelid contour line and a cornea contour line;
and 7: repeating the steps for a plurality of times to randomly select three pixel points with eyelid contour lines and cornea contour lines in the cornea contour line image and fit the circle centers of circles where the three pixel points are located, determining a clustering center by using a clustering method for the circle centers obtained by multiple times of fitting, and taking the clustering center as a pupil center;
and 8: a round mark is pasted at the forehead of the electronic digital photo to be detected in the step 6, the round mark is detected in an HSV color space by using a Hough coding method, and a round scale is obtained through calculation;
and step 9: and (3) calculating the image with the eyelid contour line and the cornea contour line obtained in the step (6) by using the circular scale obtained in the step (8) and the pupil center positioned in the step (7) to obtain the upper eyelid edge reflection distance (MRD1), the lower eyelid edge emission distance (MRD2), the eyelid fissure size (PF), the upper eyelid length, the lower eyelid length, the cornea area, the nose side area and the temporal side area of the single eye, and forming eyelid topological morphology characteristics by the upper eyelid edge reflection distance (MRD1), the lower eyelid edge emission distance (MRD2), the eyelid fissure size (PF), the upper eyelid length, the lower eyelid length, the cornea area, the nose side area and the temporal side area.
2. The method for extracting eyelid topological feature based on deep learning of claim 1, wherein: the electronic digital photo in the step 1 and the electronic digital photo to be detected in the step 6 are both required to be full face, a round mark is pasted at the forehead, and the shot person is positioned at the first eye position right in front of the eye.
3. The method for extracting eyelid topological feature based on deep learning of claim 1, wherein: the step 2 specifically comprises the following steps:
respectively converting the eyelid contour line image and the cornea contour line image corresponding to the electronic digital photo in the step 1 into an eyelid binary segmentation image and a cornea binary segmentation image by using a water diffusion filling method, overlapping the eyelid binary segmentation image and the corresponding cornea binary segmentation image to obtain a binary segmentation image, and forming a binary segmentation image data set by all the binary segmentation images.
4. The method for extracting eyelid topological feature based on deep learning of claim 1, wherein: the ROI areas include the upper eyelid, lower eyelid, cornea, pupil, and sclera visibility region.
5. The method for extracting eyelid topological feature based on deep learning of claim 1, wherein: the convolution neural network in the step 4 comprises a down-sampling module and an up-sampling module, wherein the down-sampling module mainly comprises a first convolution pooling module, a second convolution pooling module, a third convolution pooling module and a fourth convolution pooling module which are sequentially connected, the convolution pooling module mainly comprises a down-sampling convolution module and a maximum pooling module which are sequentially connected, the down-sampling convolution module mainly comprises a first convolution layer, a first batch normalization layer, a first ReLU activation layer, a second convolution layer, a second batch normalization layer and a second ReLU activation layer which are sequentially connected, and the maximum pooling module comprises two maximum pooling layers; the up-sampling module comprises a convolution module, four up-sampling convolution modules, four gate control units, four up-sampling sub-modules and an up-sampling convolution layer, wherein the up-sampling convolution module and the down-sampling convolution module have the same structure, and the up-sampling sub-modules mainly comprise B-spline interpolation operation; the output of the first gate control unit and the output of the first up-sampling sub-module are subjected to characteristic splicing and then input into the first up-sampling convolution module; the output of the first up-sampling convolution module and the output of the third down-sampling convolution module are input into a second gating unit, the output of the first up-sampling convolution module is also input into a second up-sampling sub-module, and the output of the second gating unit and the output of the second up-sampling sub-module are subjected to characteristic splicing and then input into the second up-sampling convolution module; the output of the second up-sampling convolution module and the output of the second down-sampling convolution module are input into a third gate control unit, the output of the second up-sampling convolution module is also input into a third up-sampling sub-module, and the output of the third gate control unit and the output of the third up-sampling sub-module are subjected to characteristic splicing and then input into a third up-sampling convolution module; the output of the third up-sampling convolution module and the output of the first down-sampling convolution module are input into a fourth gating unit, the output of the third up-sampling convolution module is also input into a fourth up-sampling sub-module, and the output of the fourth gating unit and the output of the fourth up-sampling sub-module are subjected to characteristic splicing and then input into a fourth up-sampling convolution module; and the output of the fourth up-sampling convolution module is input to the up-sampling convolution layer, the output of the up-sampling convolution layer is input to the Softmax classification layer, and finally the semantic segmentation result of the ROI area image to be detected is obtained.
6. The method for extracting eyelid topological feature based on deep learning of claim 5, wherein: the gate control unit is specifically as follows: the first input and the second input of the gate control unit are subjected to pixel addition and input to a third ReLU active layer after passing through respective gate control convolutional layers, the third ReLU active layer is subjected to resampling after passing through a third convolutional layer and a first Sigmoid active layer in sequence, the output after resampling is subjected to jump connection with the second input and then output, and the output after jump connection is used as the output of the gate control unit;
the first input is the output of a downsampling convolution module; the second input is the output of the convolution module or the output of the up-sampling convolution module; the skip connection is a pixel-by-pixel multiplication of the resampled output with the second input with a weight alpha.
7. The method for extracting eyelid topological feature based on deep learning of claim 1, wherein: the calculation of the circular scale is specifically as follows:
and detecting the original mark in the HSV color space by using a Hough coding method, taking the distance between two pixels with the longest distance on the edge of the circle mark as the diameter of the circle mark, taking the number of the pixels occupied by the diameter of the circle mark as the pixel value corresponding to the actual diameter of the circle mark, and calculating the actual diameter of the circle mark divided by the number of the pixels corresponding to the actual diameter of the circle mark to obtain the circular scale.
8. The method for extracting eyelid topological feature based on deep learning of claim 1, wherein: in step 9, MRD1 is the vertical distance from the pupil center to the upper eyelid margin, MRD2 is the vertical distance from the pupil center to the upper eyelid margin, PF is the vertical distance from the upper eyelid margin to the lower eyelid margin and passing through the pupil center, the length of the upper and lower eyelids is the geometric length of the upper and lower eyelid margins with the inner outer canthus as the starting point, the area of the exposed part of the cornea is the area of the cornea when the cornea is at the first eye position, the area of the sclera is the area of the sclera on the nasal side of the cornea when the nasal side is at the first eye position, and the area of the sclera is the area of the sclera on the temporal side of the cornea when the eyelid is at the first eye position.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110036779.2A CN112837805B (en) | 2021-01-12 | 2021-01-12 | Eyelid topological morphology feature extraction method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110036779.2A CN112837805B (en) | 2021-01-12 | 2021-01-12 | Eyelid topological morphology feature extraction method based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112837805A true CN112837805A (en) | 2021-05-25 |
| CN112837805B CN112837805B (en) | 2024-03-29 |
Family
ID=75929655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110036779.2A Active CN112837805B (en) | 2021-01-12 | 2021-01-12 | Eyelid topological morphology feature extraction method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112837805B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113902910A (en) * | 2021-12-10 | 2022-01-07 | 中国科学院自动化研究所 | Vision measurement method and system |
| CN114694236A (en) * | 2022-03-08 | 2022-07-01 | 浙江大学 | Eyeball motion segmentation positioning method based on cyclic residual convolution neural network |
| CN115762787A (en) * | 2022-11-24 | 2023-03-07 | 浙江大学 | Eyelid disease surgery curative effect evaluation method and system based on eyelid topological morphology analysis |
| CN115909470A (en) * | 2022-11-24 | 2023-04-04 | 浙江大学 | Deep learning-based full-automatic eyelid disease postoperative appearance prediction system and method |
| WO2024037579A1 (en) * | 2022-08-18 | 2024-02-22 | 上海市内分泌代谢病研究所 | Palpebral fissure height measurement method and apparatus, and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573219A (en) * | 2018-03-27 | 2018-09-25 | 上海电力学院 | A kind of eyelid key point accurate positioning method based on depth convolutional neural networks |
| CN109994202A (en) * | 2019-03-22 | 2019-07-09 | 华南理工大学 | A method of the face based on deep learning generates prescriptions of traditional Chinese medicine |
| CN111127431A (en) * | 2019-12-24 | 2020-05-08 | 杭州求是创新健康科技有限公司 | Dry eye disease grading evaluation system based on regional self-adaptive multitask neural network |
| AU2020102885A4 (en) * | 2020-10-20 | 2020-12-17 | Xijing University | Disease recognition method of winter jujube based on deep convolutional neural network and disease image |
-
2021
- 2021-01-12 CN CN202110036779.2A patent/CN112837805B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573219A (en) * | 2018-03-27 | 2018-09-25 | 上海电力学院 | A kind of eyelid key point accurate positioning method based on depth convolutional neural networks |
| CN109994202A (en) * | 2019-03-22 | 2019-07-09 | 华南理工大学 | A method of the face based on deep learning generates prescriptions of traditional Chinese medicine |
| CN111127431A (en) * | 2019-12-24 | 2020-05-08 | 杭州求是创新健康科技有限公司 | Dry eye disease grading evaluation system based on regional self-adaptive multitask neural network |
| AU2020102885A4 (en) * | 2020-10-20 | 2020-12-17 | Xijing University | Disease recognition method of winter jujube based on deep convolutional neural network and disease image |
Non-Patent Citations (1)
| Title |
|---|
| JI SHAO等: "deep-learning-based image analysis of eyelid morphology in thyroid-associated ophthalmopathy", 《QIMS》, 1 January 2021 (2021-01-01) * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113902910A (en) * | 2021-12-10 | 2022-01-07 | 中国科学院自动化研究所 | Vision measurement method and system |
| CN114694236A (en) * | 2022-03-08 | 2022-07-01 | 浙江大学 | Eyeball motion segmentation positioning method based on cyclic residual convolution neural network |
| WO2024037579A1 (en) * | 2022-08-18 | 2024-02-22 | 上海市内分泌代谢病研究所 | Palpebral fissure height measurement method and apparatus, and storage medium |
| CN115762787A (en) * | 2022-11-24 | 2023-03-07 | 浙江大学 | Eyelid disease surgery curative effect evaluation method and system based on eyelid topological morphology analysis |
| CN115909470A (en) * | 2022-11-24 | 2023-04-04 | 浙江大学 | Deep learning-based full-automatic eyelid disease postoperative appearance prediction system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112837805B (en) | 2024-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112837805B (en) | Eyelid topological morphology feature extraction method based on deep learning | |
| CN110400289B (en) | Fundus image recognition method, fundus image recognition device, fundus image recognition apparatus, and fundus image recognition storage medium | |
| WO2020199593A1 (en) | Image segmentation model training method and apparatus, image segmentation method and apparatus, and device and medium | |
| Gao et al. | Automatic feature learning to grade nuclear cataracts based on deep learning | |
| Hassan et al. | Joint segmentation and quantification of chorioretinal biomarkers in optical coherence tomography scans: A deep learning approach | |
| CN111986211A (en) | Deep learning-based ophthalmic ultrasonic automatic screening method and system | |
| KR20190087272A (en) | Method for diagnosing glaucoma using fundus image and apparatus therefor | |
| JP2014530730A (en) | System and method for identifying eye symptoms | |
| CN114694236B (en) | Eyeball motion segmentation positioning method based on cyclic residual convolution neural network | |
| Lin et al. | Retinal image quality assessment for diabetic retinopathy screening: A survey | |
| CN109697719A (en) | A kind of image quality measure method, apparatus and computer readable storage medium | |
| WO2022048171A1 (en) | Method and apparatus for measuring blood vessel diameter in fundus image | |
| CN114343563A (en) | Method, device and system for assisting dry eye diagnosis and typing through multi-modal fusion | |
| JPWO2019073962A1 (en) | Image processing apparatus and program | |
| Kaya | A novel method for optic disc detection in retinal images using the cuckoo search algorithm and structural similarity index | |
| Reethika et al. | Diabetic retinopathy detection using statistical features | |
| Giancardo | Automated fundus images analysis techniques to screen retinal diseases in diabetic patients | |
| WO2024037587A1 (en) | Palpebral fissure height measurement method and apparatus, and storage medium | |
| TWI742782B (en) | Establishing method of retinal layer thickness detection model, retinal layer thickness detection system, eye care device, method for detecting retinal layer thickness and method for assessing and predicting dementia | |
| Pavlov et al. | Methods and computer tools for identifying diabetes-induced fundus pathology | |
| KR20210033902A (en) | Method, apparatus and software program for cervical cancer diagnosis using image analysis of artificial intelligence based technology | |
| CN115886717B (en) | Eye crack width measuring method, device and storage medium | |
| US20230284903A1 (en) | System and method for acquisition and quantification of images with ocular staining | |
| George et al. | Oct segmentation using convolutional neural network | |
| TWI821063B (en) | Establishing method of retinal layer autosegmentation model, retinal layer quantitative system, eye care device, method for detecting retinal layer thickness and retinal layer area, and method for assessing and predicting neurodegenerative disease |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |