CN112837805B - Eyelid topological morphology feature extraction method based on deep learning - Google Patents
Eyelid topological morphology feature extraction method based on deep learning Download PDFInfo
- Publication number
- CN112837805B CN112837805B CN202110036779.2A CN202110036779A CN112837805B CN 112837805 B CN112837805 B CN 112837805B CN 202110036779 A CN202110036779 A CN 202110036779A CN 112837805 B CN112837805 B CN 112837805B
- Authority
- CN
- China
- Prior art keywords
- eyelid
- output
- module
- input
- sampling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 210000000744 eyelid Anatomy 0.000 title claims abstract description 132
- 238000013135 deep learning Methods 0.000 title claims abstract description 15
- 238000000605 extraction Methods 0.000 title claims abstract description 13
- 210000004087 cornea Anatomy 0.000 claims abstract description 61
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 24
- 238000012549 training Methods 0.000 claims abstract description 15
- 241000669069 Chrysomphalus aonidum Species 0.000 claims abstract description 14
- 238000004364 calculation method Methods 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000005070 sampling Methods 0.000 claims description 87
- 230000011218 segmentation Effects 0.000 claims description 58
- 238000011176 pooling Methods 0.000 claims description 29
- 210000001508 eye Anatomy 0.000 claims description 28
- 230000004913 activation Effects 0.000 claims description 15
- 230000002123 temporal effect Effects 0.000 claims description 13
- 208000032610 autosomal dominant 2 intellectual disability Diseases 0.000 claims description 10
- 201000000188 autosomal dominant non-syndromic intellectual disability 2 Diseases 0.000 claims description 10
- 210000003786 sclera Anatomy 0.000 claims description 9
- 238000010606 normalization Methods 0.000 claims description 8
- 210000001061 forehead Anatomy 0.000 claims description 6
- 101100077919 Neurospora crassa (strain ATCC 24698 / 74-OR23-1A / CBS 708.71 / DSM 1257 / FGSC 987) mrd-1 gene Proteins 0.000 claims description 5
- 210000001747 pupil Anatomy 0.000 claims description 5
- 230000009191 jumping Effects 0.000 claims description 4
- 238000012952 Resampling Methods 0.000 claims description 3
- 238000009792 diffusion process Methods 0.000 claims description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 3
- 210000000887 face Anatomy 0.000 claims description 2
- 230000001815 facial effect Effects 0.000 claims description 2
- 238000005259 measurement Methods 0.000 abstract description 8
- 238000013528 artificial neural network Methods 0.000 abstract description 2
- 238000011156 evaluation Methods 0.000 description 5
- 230000011514 reflex Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 201000010099 disease Diseases 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 230000000877 morphologic effect Effects 0.000 description 3
- 206010015995 Eyelid ptosis Diseases 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 201000003004 ptosis Diseases 0.000 description 2
- 208000023328 Basedow disease Diseases 0.000 description 1
- 208000028006 Corneal injury Diseases 0.000 description 1
- 208000015023 Graves' disease Diseases 0.000 description 1
- 208000023715 Ocular surface disease Diseases 0.000 description 1
- 208000004350 Strabismus Diseases 0.000 description 1
- 206010044604 Trichiasis Diseases 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 206010005159 blepharospasm Diseases 0.000 description 1
- 230000000744 blepharospasm Effects 0.000 description 1
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 238000003776 cleavage reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 210000000720 eyelash Anatomy 0.000 description 1
- 238000010304 firing Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 206010023332 keratitis Diseases 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 238000004171 remote diagnosis Methods 0.000 description 1
- 230000007017 scission Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/193—Preprocessing; Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A10/00—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE at coastal zones; at river basins
- Y02A10/40—Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Human Computer Interaction (AREA)
- Ophthalmology & Optometry (AREA)
- Databases & Information Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Pathology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an eyelid topological morphology feature extraction method based on deep learning. The method comprises the following steps: collecting an electronic digital photo of a normal person, processing the electronic digital photo, constructing an ROI image training set, and inputting the ROI image training set into a convolutional neural network to be trained to obtain a trained convolutional neural network; and positioning the position (ROI) of an eye region of interest (ROI) of the electronic digital photo to be detected by using a face recognition method, obtaining an ROI region image to be detected, inputting the ROI region image to be detected into a trained convolutional neural network to output an image with eyelid contour lines and cornea contour lines, determining a circular scale and pupil center of the electronic digital photo to be detected, and extracting eyelid topological morphology features of a single eye. The invention uses convolution neural network to segment eyelid and cornea structure, and uses MeanShift cluster to determine pupil center, and then carries out eyelid related structure parameter automatic calculation to obtain accuracy equivalent to manual measurement.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an eyelid topological morphology feature extraction method based on deep learning.
Background
Normal eyelid position is the basis for achieving normal function of the eyeball, and evaluation of the morphology and position of the eyelid is important for ocular reshaping (e.g., ptosis, trichiasis), ocular surface diseases (e.g., exposed keratitis), graves' disease, and the like.
Currently, a scale is commonly used in clinical practice to manually measure the patient's upper lid edge reflex distance (MRD 1), lower lid edge reflex distance (MRD 2), and lid break size (PF) for assessing the eyelid position. However, accurate measurement requires long experience of the measurer and high coordination of the measurer, and reproducibility and stability of manual measurement are poor. At the same time, these linear indicators do not fully reflect the complete eyelid contour morphology features. The problem of poor reproducibility and stability of manual measurement can be solved by analyzing the electronic photograph, however, the traditional automatic analysis method such as Canny boundary detection algorithm can encounter interference of eyelashes, so that eyelid boundary can not be accurately identified, and meanwhile, the common method for determining pupil center by fitting circle centers at three points by using non-perfect circles of the iris is subject to certain defects. To achieve full-automatic eyelid structure analysis, accurate eyelid boundary identification and pupil center positioning must be based. The method for extracting eyelid topological morphology features based on deep learning is constructed, the cornea and eyelid boundary are accurately segmented by using a deep convolutional neural network, and the pupil center is positioned, so that the method is a key technology for realizing eyelid morphological feature measurement and evaluation, and has urgent clinical requirements in automation and remote diagnosis and evaluation of eyelid related diseases.
Disclosure of Invention
In order to overcome the problems in the prior art, the invention aims to provide an eyelid topological morphology feature extraction method based on deep learning, which realizes automatic eyelid related structure identification and automatic eyelid topological morphology feature measurement and calculation.
The technical scheme adopted by the invention is as follows:
the invention comprises the following steps:
step 1: collecting an electronic digital photo of a normal person, and forming a facial photo data set by the electronic digital photo;
step 2: processing the electronic digital photos marked with the eyelid contour lines and the cornea contour lines to obtain binary segmentation images, and forming a binary segmentation image dataset by the binary segmentation images of all the electronic digital photos;
step 3: locating the position (ROI) of an eye region of interest (ROI) of a binary segmentation image data set by using a face recognition method to obtain an ROI region binary segmentation image of the binary segmentation image, and forming an ROI image training set by the ROI region binary segmentation images of all the binary segmentation images;
step 4: constructing an AGN (Attention-oriented Network) -based convolutional neural Network;
step 5: inputting the ROI image training set obtained in the step 3 into the convolutional neural network in the step 4 to obtain a trained convolutional neural network;
step 6: positioning an eye region of interest (ROI) of an electronic digital photo to be detected by using a face recognition method to obtain an ROI region image to be detected, inputting the ROI region image to be detected into a trained convolutional neural network, outputting classification probability of each pixel point as cornea, eyelid and background by the trained convolutional neural network, judging the classification probability of each pixel point according to a preset threshold value, classifying each pixel point, classifying various pixel points to form cornea region, eyelid region and background region of the ROI region image to be detected, and finally outputting an image with eyelid contour lines and cornea contour lines;
step 7: repeating the steps for a plurality of times, randomly selecting three pixel points with eyelid contour lines and cornea contour lines in cornea contour line images, fitting the circle centers of circles where the three pixel points are located, determining a clustering center for the circle centers obtained by multiple fitting by using a clustering method, and taking the clustering center as a pupil center;
step 8: a circle mark is stuck at the forehead of the electronic digital photo to be detected in the step 6, the circle mark is detected in the HSV color space by using a Hough coding method, and a circular scale is obtained through calculation;
step 9: calculating the image with eyelid contour lines and cornea contour lines obtained in the step 6 by using the circular scale obtained in the step 8 and pupil centers positioned in the step 7 to obtain an upper eyelid edge reflection distance (MRD 1), a lower eyelid edge emission distance (MRD 2), a eyelid fissure size (PF), an upper eyelid length, a lower eyelid length, a cornea region, a nasal side region and a temporal side region area of a single eye, and forming eyelid topological morphology features by the upper eyelid edge reflection distance (MRD 1), the lower eyelid edge emission distance (MRD 2), the eyelid fissure size (PF), the upper eyelid length, the lower eyelid length, the cornea region, the nasal side region and the temporal side region area.
The electronic digital photos in the step 1 and the electronic digital photos to be detected in the step 6 are all required to be full faces, round marks are attached to the forehead, and the photographed person is in a first eye position right in front of the eyes.
The step 2 specifically comprises the following steps:
and (2) respectively converting the eyelid contour line image and the cornea contour line image corresponding to the electronic digital photo in the step (1) into an eyelid binary segmentation image and a cornea binary segmentation image by using a water diffusion filling method, carrying out superposition processing on the eyelid binary segmentation image and the corresponding cornea binary segmentation image to obtain a binary segmentation image, and forming a binary segmentation image data set by all the binary segmentation images.
The ROI area includes an upper eyelid, a lower eyelid, a cornea, a pupil, and a sclera visibility region.
The convolutional neural network in the step 4 comprises a downsampling module and an upsampling module, wherein the downsampling module is mainly formed by sequentially connecting a first convolutional Pooling module, a second convolutional Pooling module, a third convolutional Pooling module and a fourth convolutional Pooling module, the convolutional Pooling module is mainly formed by sequentially connecting a downsampling convolutional module and a maximum Pooling module, the downsampling convolutional module is mainly formed by sequentially connecting a first convolutional layer (Convoluational layer, conv), a first batch normalization layer (Batch Normalization, BN), a first ReLU activation layer, a second convolutional layer, a second batch normalization layer and a second ReLU activation layer, and the maximum Pooling module comprises two maximum Pooling layers (Max Pooling); the up-sampling module comprises a convolution module, four up-sampling convolution modules, four gate control units, four up-sampling sub-modules and an up-sampling convolution layer, wherein the up-sampling convolution modules and the down-sampling convolution modules have the same structure, and the up-sampling sub-modules mainly comprise B spline interpolation operations; the output of the fourth maximum pooling module is input to the convolution module, the output of the convolution module and the output of the fourth downsampling convolution module are input to the first gating unit, the output of the convolution module is also input to the first upsampling submodule, and the output of the first gating unit and the output of the first upsampling submodule are input to the first upsampling convolution module after characteristic splicing; the output of the first up-sampling convolution module and the output of the third down-sampling convolution module are input into a second gating unit, the output of the first up-sampling convolution module is also input into a second up-sampling sub-module, and the output of the second gating unit and the output of the second up-sampling sub-module are input into the second up-sampling convolution module after characteristic splicing; the output of the second up-sampling convolution module and the output of the second down-sampling convolution module are input into a third gating unit, the output of the second up-sampling convolution module is also input into a third up-sampling sub-module, and the output of the third gating unit and the output of the third up-sampling sub-module are input into the third up-sampling convolution module after characteristic splicing; the output of the third up-sampling convolution module and the output of the first down-sampling convolution module are input into a fourth gating unit, the output of the third up-sampling convolution module is also input into a fourth up-sampling sub-module, and the output of the fourth gating unit and the output of the fourth up-sampling sub-module are input into the fourth up-sampling convolution module after characteristic splicing; the output of the fourth up-sampling convolution module is input to an up-sampling convolution layer, the output of the up-sampling convolution layer is input to a Softmax classification layer, and finally the semantic segmentation result of the ROI region image to be detected is obtained.
The gate control unit specifically comprises: the first input and the second input of the gating unit respectively pass through the respective gating convolution layers, then carry out pixel addition and input to a third ReLU activation layer, the third ReLU activation layer sequentially passes through the third convolution layer and the first Sigmoid activation layer and then carries out resampling, the resampled output is output after being connected with the second input in a jumping way, and the output after being connected in a jumping way is used as the output of the gating unit;
the first input is the output of a downsampling convolution module; the second input is the output of the convolution module or the output of the up-sampling convolution module; the skip connection is a pixel-by-pixel multiplication of the resampled output with a second input by a weight α.
The calculation of the circular scale is specifically as follows:
detecting an original mark in an HSV color space by using a Hough coding method, taking the distance between two longest pixel points on the edge of a circle mark as the diameter of the circle mark, taking the number of the pixel points occupied by the diameter of the circle mark as the pixel value corresponding to the diameter of an actual circle mark, and calculating the diameter of the actual circle mark divided by the number of the pixel points corresponding to the diameter of the actual circle mark to obtain a circular scale.
In the step 9, MRD1 is the vertical distance from the pupil center to the upper eyelid margin, MRD2 is the vertical distance from the pupil center to the upper eyelid margin, PF is the vertical distance from the upper eyelid margin to the lower eyelid margin and through the pupil center, the upper eyelid length and the lower eyelid length are the geometric lengths of the upper eyelid margin and the lower eyelid margin with the inner canthus as the starting points, respectively, the cornea area is the area of the exposed portion of the cornea at the first eye position, the nasal side area is the sclera area on the nasal side of the cornea during the first eye position, and the temporal side area is the sclera area on the temporal side of the cornea during the first eye position.
Compared with the prior art, the invention has the following advantages:
compared with a manual measurement method, the method has better repeatability and stability, the required patient matching time is short, the photo materials are easy to obtain, and technical support is provided for realizing remote medical treatment and automatic diagnosis.
Compared with the traditional segmentation method, the segmentation based on the deep neural network can obtain a more accurate segmentation effect, is less interfered by nearby tissue structures, and enables eyelid topological morphology analysis to have a more accurate structure basis.
According to the invention, the MeanShift clustering based on the Gaussian kernel is used, the clustering center is set as the pupil center, so that the positioning deviation between the fitting circle center and the actual pupil center, which is generated by the fact that the iris is a non-perfect circle, is reduced, the calculation of the eyelid-related morphological parameters is more accurate and objective, the accuracy and reliability of the method are further improved, objective evaluation of the eyelid-related morphological parameters in a remote and multi-center manner is assisted, and the related diseases are objectively diagnosed automatically.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention;
FIG. 2 is a schematic diagram of a convolutional neural network of the present invention;
FIG. 3 is a schematic diagram of the attention mechanism of the present invention;
fig. 4 is a schematic diagram of eyelid topology morphology measurement parameters of the present invention.
Detailed Description
The invention is further described below with reference to the drawings and the detailed description.
As shown in fig. 1, the present invention includes the steps of:
step 1: an electronic digital photo of 1581 normal person is collected from an ophthalmologic center of a certain hospital, the required shooting range is a full face, a flat circle mark with the diameter of 10mm is attached to the forehead, and the photographed person is at a first eye position right in front of the eyes. Poor quality pictures such as ptosis, blepharospasm, strabismus, or corneal trauma patients and photo blurry were excluded from the study. The photo is shot by a Canon EOS 500D single lens with a 100mm micro lens, and the electronic digital photo with the resolution of 4752 x 3618 is obtained by uploading the photo to a computer. Constructing a face photo data set by using the electronic digital photo;
step 2: processing the electronic digital photos marked with the eyelid contour lines and the cornea contour lines to obtain binary segmentation images, and forming a binary segmentation image dataset by the binary segmentation images of all the electronic digital photos;
the step 2 is specifically as follows:
and (2) respectively converting the eyelid contour line image and the cornea contour line image corresponding to the electronic digital photo in the step (1) into an eyelid binary segmentation image and a cornea binary segmentation image by using a water diffusion filling method, carrying out superposition processing on the eyelid binary segmentation image and the corresponding cornea binary segmentation image to obtain a binary segmentation image, and forming a binary segmentation image data set by all the binary segmentation images.
Step 3: locating the position (ROI) of an eye region of interest (ROI) of a binary segmentation image data set by using a face recognition method to obtain an ROI region binary segmentation image of the binary segmentation image, and forming an ROI image training set by the ROI region binary segmentation images of all the binary segmentation images; the ROI area includes the upper eyelid, lower eyelid, cornea, pupil and sclera visibility region.
Step 4: constructing an AGN (Attention-oriented Network) -based convolutional neural Network;
as shown in fig. 2, the convolutional neural network in step 4 includes a downsampling module and an upsampling module, the downsampling module is mainly composed of a first convolutional Pooling module, a second convolutional Pooling module, a third convolutional Pooling module and a fourth convolutional Pooling module which are sequentially connected, the convolutional Pooling module is mainly composed of a downsampling convolutional module and a maximum Pooling module which are sequentially connected, the downsampling convolutional module is mainly composed of a first convolutional layer (Convoluational layer, conv), a first batch normalization layer (Batch Normalization, BN), a first ReLU activation layer, a second convolutional layer, a second batch normalization layer and a second ReLU activation layer which are sequentially connected, and the maximum Pooling module includes two maximum Pooling layers (Max Pooling); the up-sampling module comprises a convolution module, four up-sampling convolution modules, four gate control units, four up-sampling sub-modules and an up-sampling convolution layer, wherein the up-sampling convolution modules and the down-sampling convolution modules have the same structure, and the up-sampling sub-modules mainly comprise B spline interpolation operations; the output of the fourth maximum pooling module is input to the convolution module, the output of the convolution module and the output of the fourth downsampling convolution module are input to the first gating unit, the output of the convolution module is also input to the first upsampling submodule, and the output of the first gating unit and the output of the first upsampling submodule are input to the first upsampling convolution module after characteristic splicing; the output of the first up-sampling convolution module and the output of the third down-sampling convolution module are input into a second gating unit, the output of the first up-sampling convolution module is also input into a second up-sampling sub-module, and the output of the second gating unit and the output of the second up-sampling sub-module are input into the second up-sampling convolution module after characteristic splicing; the output of the second up-sampling convolution module and the output of the second down-sampling convolution module are input into a third gating unit, the output of the second up-sampling convolution module is also input into a third up-sampling sub-module, and the output of the third gating unit and the output of the third up-sampling sub-module are input into the third up-sampling convolution module after characteristic splicing; the output of the third up-sampling convolution module and the output of the first down-sampling convolution module are input into a fourth gating unit, the output of the third up-sampling convolution module is also input into a fourth up-sampling sub-module, and the output of the fourth gating unit and the output of the fourth up-sampling sub-module are input into the fourth up-sampling convolution module after characteristic splicing; the output of the fourth up-sampling convolution module is input to an up-sampling convolution layer, the output of the up-sampling convolution layer is input to a Softmax classification layer, and finally the semantic segmentation result of the ROI region image to be detected is obtained.
As shown in fig. 3, the gating unit specifically includes: the first input and the second input of the gating unit respectively pass through a respective 1X 1 gating convolution layer and then are subjected to pixel addition and input to a third ReLU activation layer, the 1X 1 gating convolution layer converts the number of characteristic channels of the first input and the second input into the same number, the third ReLU activation layer sequentially passes through the 1X 1 third convolution layer and the first Sigmoid activation layer and then is subjected to resampling, the resampled output is subjected to jump connection with the second input and then is output, the 1X 1 third convolution layer reduces the number of the characteristic channels to 1, and the output after the jump connection is used as the output of the gating unit;
the first input is the output of the downsampling convolution module; the second input is the output of the convolution module or the output of the up-sampling convolution module; the skip connection is a pixel-by-pixel multiplication of the resampled output with a second input by a weight α.
Step 5: and (3) randomly taking 1378 ROI region binary segmentation images in the ROI image training set obtained in the step (3), and inputting the images into the convolutional neural network in the step (4). With 882 participants (1764 eyes) as the training set, 220 participants (440 eyes) as the verification set, and 276 participants (552 eyes) as the test set.
When training, the set learning rate is 0.001, the training round is 100, the learning rate is attenuated at 20 rounds, the attenuation rate is 0.1, the learning rate of each training is smaller than or equal to the learning rate of the previous training, and the trained convolutional neural network is obtained for eyelid and cornea segmentation of the electronic digital photo;
step 6: positioning an eye region of interest (ROI) of an electronic digital photo to be detected by using a face recognition method to obtain an ROI region image to be detected, inputting the ROI region image to be detected into a trained convolutional neural network, outputting classification probability of each pixel point as cornea, eyelid and background by the trained convolutional neural network, judging the classification probability of each pixel point according to a preset threshold value, classifying each pixel point, classifying various pixel points to form cornea region, eyelid region and background region of the ROI region image to be detected, and finally outputting an image with eyelid contour lines and cornea contour lines; the background area is an image except the cornea area and the eyelid area in the ROI area image to be detected. The eyelid contour line is the boundary between the eyelid area and the background area, and the cornea contour line is the boundary between the cornea area and the eyelid area.
Step 7: repeating the steps for a plurality of times, randomly selecting three pixel points with eyelid contour lines and cornea contour lines in cornea contour line images, fitting the circle centers of circles where the three pixel points are located, determining a clustering center for the circle centers obtained by multiple fitting by using a MeanShift clustering method of Gaussian kernels, and taking the clustering center as a pupil center;
step 8: a circle mark is stuck at the forehead of the electronic digital photo to be detected in the step 6, the circle mark is detected in the HSV color space by using a Hough coding method, and a circular scale is obtained through calculation;
the calculation of the circular scale is specifically as follows:
detecting an original mark in an HSV color space by using a Hough coding method, taking the distance between two longest pixel points on the edge of a circle mark as the diameter of the circle mark, taking the number of the pixel points occupied by the diameter of the circle mark as the pixel value corresponding to the diameter of an actual circle mark of 10mm, and calculating the number of the pixel points corresponding to the diameter of the actual circle mark of 10mm divided by the diameter of the actual circle mark as a circular scale R.
Step 9: and (3) selecting 203 ROI region binary segmentation images which are not used for convolutional neural network training in the ROI image training set in the step (5), repeating the steps (6-8), and carrying out cornea and eyelid structure segmentation on the 203 ROI region binary segmentation images to obtain 203 images with eyelid contour lines and cornea contour lines.
Step 10: calculating the image with eyelid contour line and cornea contour line obtained in the step 9 by using the circular scale obtained in the step 8 and the pupil center positioned in the step 7 to obtain the upper monocular imageEyelid edge reflex distance MRD1, lower eyelid edge reflex distance MRD2, eyelid cleavage size PF, upper eyelid length L ul Length of lower eyelid L ll、 Cornea region A c Nasal area A n And temporal area A t Eyelid topology morphology features are composed of upper lid edge reflex distance MRD1, lower lid edge firing distance MRD2, lid break size PF, upper lid length, lower lid length, corneal zone, nasal zone and temporal zone area.
As shown in fig. 4, in step 10, eyelid topology characteristics are established at a first eye position of the subject in front of the eye, MRD1 is a vertical distance from the pupil center to the upper eyelid margin, MRD2 is a vertical distance from the pupil center to the upper eyelid margin, PF is a vertical distance from the upper eyelid margin to the lower eyelid margin and through the pupil center, that is, a sum of MRD1 and MRD2, upper and lower eyelid lengths are respectively a geometric length of the upper and lower eyelid margins with inner and outer canthus as a starting point, a cornea area is an area of an exposed portion of the cornea at the first eye position, a nasal area is an area of a sclera region of the cornea on the nasal side of the eyelid split at the first eye position, and a temporal area is an area of a sclera region of the cornea on the temporal side of the eyelid split at the first eye position.
The specific calculation method satisfies the formulas (1) to (8):
MRD1＝N MRD1 ×R (1)
MRD2＝N MRD2 ×R (2)
PF＝MRD1+MRD2 (3)
A t ＝N t ×R 2 (4)
A n ＝N n ×R 2 (5)
A c ＝N c ×R 2 (6)
L ul ＝N ul ×R (7)
L ll ＝N ll ×R (8)
wherein NMRD1 is the number of pixels of the upper eyelid edge reflection distance MRD1, NMRD2 is the number of pixels of the lower eyelid edge reflection distance MRD2, nul is the number of pixels of the upper eyelid length, nll is the number of pixels of the lower eyelid length, nc is the number of pixels in the cornea region, nn is the number of pixels in the nasal region, nt is the number of pixels in the temporal region, and R is a circular scale.
The method realizes accurate cornea and upper and lower eyelid segmentation through deep learning. The eyelid topology morphology parameter based on the invention is adopted for automatic measurement, has higher accuracy and better repeatability, and can be applied to the fields of disease automatic diagnosis, remote medical treatment, operation evaluation and the like.
Claims (8)
1. The eyelid topological morphology feature extraction method based on deep learning is characterized by comprising the following steps of: the method comprises the following steps:
step 1: collecting an electronic digital photo of a normal person, and forming a facial photo data set by the electronic digital photo;
step 2: processing the electronic digital photos marked with the eyelid contour lines and the cornea contour lines to obtain binary segmentation images, and forming a binary segmentation image dataset by the binary segmentation images of all the electronic digital photos;
step 3: locating the position of an eye region of interest on the binary segmentation image in the binary segmentation image data set by using a face recognition method to obtain an ROI region binary segmentation image of the binary segmentation image, and forming an ROI image training set by the ROI region binary segmentation images of all the binary segmentation images;
step 4: constructing an AGN-based convolutional neural network;
step 5: inputting the ROI image training set obtained in the step 3 into the convolutional neural network in the step 4 to obtain a trained convolutional neural network;
step 6: positioning the position of an eye region of interest by using a face recognition method for an electronic digital photo to be detected to obtain an ROI region image to be detected, inputting the ROI region image to be detected into a trained convolutional neural network, outputting classification probability of each pixel point as cornea, eyelid and background by the trained convolutional neural network, judging the classification probability of each pixel point according to a preset threshold value, classifying each pixel point, classifying various pixel points to form cornea region, eyelid region and background region of the ROI region image to be detected, and finally outputting an image with eyelid contour lines and cornea contour lines;
step 7: repeating the steps for a plurality of times, randomly selecting three pixel points with eyelid contour lines and cornea contour lines in cornea contour line images, fitting the circle centers of circles where the three pixel points are located, determining a clustering center for the circle centers obtained by multiple fitting by using a clustering method, and taking the clustering center as a pupil center;
step 8: a circle mark is stuck at the forehead of the electronic digital photo to be detected in the step 6, the circle mark is detected in the HSV color space by using a Hough coding method, and a circular scale is obtained through calculation;
step 9: calculating the image with eyelid contour lines and cornea contour lines obtained in the step 6 by using the circular scale obtained in the step 8 and pupil centers positioned in the step 7 to obtain an upper eyelid edge reflection distance (MRD 1), a lower eyelid edge emission distance (MRD 2), a eyelid fissure size (PF), an upper eyelid length, a lower eyelid length, a cornea region, a nasal side region and a temporal side region area of a single eye, and forming eyelid topological morphology features by the upper eyelid edge reflection distance (MRD 1), the lower eyelid edge emission distance (MRD 2), the eyelid fissure size (PF), the upper eyelid length, the lower eyelid length, the cornea region, the nasal side region and the temporal side region area.
2. The deep learning-based eyelid topology morphology feature extraction method of claim 1, wherein: the electronic digital photos in the step 1 and the electronic digital photos to be detected in the step 6 are all required to be full faces, round marks are attached to the forehead, and the photographed person is in a first eye position right in front of the eyes.
3. The deep learning-based eyelid topology morphology feature extraction method of claim 1, wherein: the step 2 specifically comprises the following steps:
and (2) respectively converting the eyelid contour line image and the cornea contour line image corresponding to the electronic digital photo in the step (1) into an eyelid binary segmentation image and a cornea binary segmentation image by using a water diffusion filling method, carrying out superposition processing on the eyelid binary segmentation image and the corresponding cornea binary segmentation image to obtain a binary segmentation image, and forming a binary segmentation image data set by all the binary segmentation images.
4. The deep learning-based eyelid topology morphology feature extraction method of claim 1, wherein: the ROI area includes an upper eyelid, a lower eyelid, a cornea, a pupil, and a sclera visibility region.
5. The deep learning-based eyelid topology morphology feature extraction method of claim 1, wherein: the convolutional neural network in the step 4 comprises a downsampling module and an upsampling module, wherein the downsampling module mainly comprises a first convolutional pooling module, a second convolutional pooling module, a third convolutional pooling module and a fourth convolutional pooling module which are sequentially connected, the convolutional pooling module mainly comprises a downsampling convolutional module and a maximum pooling module which are sequentially connected, and the downsampling convolutional module mainly comprises a first convolutional layer, a first batch normalization layer, a first ReLU activation layer, a second convolutional layer, a second batch normalization layer and a second ReLU activation layer which are sequentially connected, and the maximum pooling module comprises two maximum pooling layers; the up-sampling module comprises a convolution module, four up-sampling convolution modules, four gate control units, four up-sampling sub-modules and an up-sampling convolution layer, wherein the up-sampling convolution modules and the down-sampling convolution modules have the same structure, and the up-sampling sub-modules mainly comprise B spline interpolation operations; the output of the fourth maximum pooling module is input to the convolution module, the output of the convolution module and the output of the fourth downsampling convolution module are input to the first gating unit, the output of the convolution module is also input to the first upsampling submodule, and the output of the first gating unit and the output of the first upsampling submodule are input to the first upsampling convolution module after characteristic splicing; the output of the first up-sampling convolution module and the output of the third down-sampling convolution module are input into a second gating unit, the output of the first up-sampling convolution module is also input into a second up-sampling sub-module, and the output of the second gating unit and the output of the second up-sampling sub-module are input into the second up-sampling convolution module after characteristic splicing; the output of the second up-sampling convolution module and the output of the second down-sampling convolution module are input into a third gating unit, the output of the second up-sampling convolution module is also input into a third up-sampling sub-module, and the output of the third gating unit and the output of the third up-sampling sub-module are input into the third up-sampling convolution module after characteristic splicing; the output of the third up-sampling convolution module and the output of the first down-sampling convolution module are input into a fourth gating unit, the output of the third up-sampling convolution module is also input into a fourth up-sampling sub-module, and the output of the fourth gating unit and the output of the fourth up-sampling sub-module are input into the fourth up-sampling convolution module after characteristic splicing; the output of the fourth up-sampling convolution module is input to an up-sampling convolution layer, the output of the up-sampling convolution layer is input to a Softmax classification layer, and finally the semantic segmentation result of the ROI region image to be detected is obtained.
6. The deep learning-based eyelid topology morphology feature extraction method of claim 5, wherein: the gate control unit specifically comprises: the first input and the second input of the gating unit respectively pass through the respective gating convolution layers, then carry out pixel addition and input to a third ReLU activation layer, the third ReLU activation layer sequentially passes through the third convolution layer and the first Sigmoid activation layer and then carries out resampling, the resampled output is output after being connected with the second input in a jumping way, and the output after being connected in a jumping way is used as the output of the gating unit;
the first input is the output of a downsampling convolution module; the second input is the output of the convolution module or the output of the up-sampling convolution module; the skip connection is a pixel-by-pixel multiplication of the resampled output with a second input by a weight α.
7. The deep learning-based eyelid topology morphology feature extraction method of claim 1, wherein: the calculation of the circular scale is specifically as follows:
detecting an original mark in an HSV color space by using a Hough coding method, taking the distance between two longest pixel points on the edge of a circle mark as the diameter of the circle mark, taking the number of the pixel points occupied by the diameter of the circle mark as the pixel value corresponding to the diameter of an actual circle mark, and calculating the diameter of the actual circle mark divided by the number of the pixel points corresponding to the diameter of the actual circle mark to obtain a circular scale.
8. The deep learning-based eyelid topology morphology feature extraction method of claim 1, wherein: in the step 9, MRD1 is the vertical distance from the pupil center to the upper eyelid margin, MRD2 is the vertical distance from the pupil center to the upper eyelid margin, PF is the vertical distance from the upper eyelid margin to the lower eyelid margin and through the pupil center, the upper eyelid length and the lower eyelid length are the geometric lengths of the upper eyelid margin and the lower eyelid margin with the inner canthus as the starting points, respectively, the cornea area is the area of the exposed portion of the cornea at the first eye position, the nasal side area is the sclera area on the nasal side of the cornea during the first eye position, and the temporal side area is the sclera area on the temporal side of the cornea during the first eye position.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110036779.2A CN112837805B (en) | 2021-01-12 | 2021-01-12 | Eyelid topological morphology feature extraction method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110036779.2A CN112837805B (en) | 2021-01-12 | 2021-01-12 | Eyelid topological morphology feature extraction method based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112837805A CN112837805A (en) | 2021-05-25 |
| CN112837805B true CN112837805B (en) | 2024-03-29 |
Family
ID=75929655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110036779.2A Active CN112837805B (en) | 2021-01-12 | 2021-01-12 | Eyelid topological morphology feature extraction method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112837805B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113902910B (en) * | 2021-12-10 | 2022-04-08 | 中国科学院自动化研究所 | Vision measurement method and system |
| CN114694236B (en) * | 2022-03-08 | 2023-03-24 | 浙江大学 | Eyeball motion segmentation positioning method based on cyclic residual convolution neural network |
| CN115281601A (en) * | 2022-08-18 | 2022-11-04 | 上海市内分泌代谢病研究所 | Eye crack width measuring device and using method thereof |
| CN115886717B (en) * | 2022-08-18 | 2023-09-29 | 上海佰翊医疗科技有限公司 | Eye crack width measuring method, device and storage medium |
| CN115909470B (en) * | 2022-11-24 | 2023-07-07 | 浙江大学 | Deep learning-based full-automatic eyelid disease postoperative appearance prediction system and method |
| CN115762787B (en) * | 2022-11-24 | 2023-07-07 | 浙江大学 | Eyelid disease operation curative effect evaluation method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573219A (en) * | 2018-03-27 | 2018-09-25 | 上海电力学院 | A kind of eyelid key point accurate positioning method based on depth convolutional neural networks |
| CN109994202A (en) * | 2019-03-22 | 2019-07-09 | 华南理工大学 | A method of the face based on deep learning generates prescriptions of traditional Chinese medicine |
| CN111127431A (en) * | 2019-12-24 | 2020-05-08 | 杭州求是创新健康科技有限公司 | Dry eye disease grading evaluation system based on regional self-adaptive multitask neural network |
| AU2020102885A4 (en) * | 2020-10-20 | 2020-12-17 | Xijing University | Disease recognition method of winter jujube based on deep convolutional neural network and disease image |
-
2021
- 2021-01-12 CN CN202110036779.2A patent/CN112837805B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573219A (en) * | 2018-03-27 | 2018-09-25 | 上海电力学院 | A kind of eyelid key point accurate positioning method based on depth convolutional neural networks |
| CN109994202A (en) * | 2019-03-22 | 2019-07-09 | 华南理工大学 | A method of the face based on deep learning generates prescriptions of traditional Chinese medicine |
| CN111127431A (en) * | 2019-12-24 | 2020-05-08 | 杭州求是创新健康科技有限公司 | Dry eye disease grading evaluation system based on regional self-adaptive multitask neural network |
| AU2020102885A4 (en) * | 2020-10-20 | 2020-12-17 | Xijing University | Disease recognition method of winter jujube based on deep convolutional neural network and disease image |
Non-Patent Citations (1)
| Title |
|---|
| deep-learning-based image analysis of eyelid morphology in thyroid-associated ophthalmopathy;JI SHAO等;《QIMS》;20210101;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112837805A (en) | 2021-05-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112837805B (en) | Eyelid topological morphology feature extraction method based on deep learning | |
| CN110400289B (en) | Fundus image recognition method, fundus image recognition device, fundus image recognition apparatus, and fundus image recognition storage medium | |
| Gao et al. | Automatic feature learning to grade nuclear cataracts based on deep learning | |
| CN109472781B (en) | Diabetic retinopathy detection system based on serial structure segmentation | |
| EP2888718B1 (en) | Methods and systems for automatic location of optic structures in an image of an eye, and for automatic retina cup-to-disc ratio computation | |
| Hassan et al. | Joint segmentation and quantification of chorioretinal biomarkers in optical coherence tomography scans: A deep learning approach | |
| CN111986211A (en) | Deep learning-based ophthalmic ultrasonic automatic screening method and system | |
| KR102155381B1 (en) | Method, apparatus and software program for cervical cancer decision using image analysis of artificial intelligence based technology | |
| WO2022048171A1 (en) | Method and apparatus for measuring blood vessel diameter in fundus image | |
| JP6734475B2 (en) | Image processing device and program | |
| CN117764957A (en) | Glaucoma image feature extraction training system based on artificial neural network | |
| JP2008073280A (en) | Eye-fundus image processor | |
| Reethika et al. | Diabetic retinopathy detection using statistical features | |
| Giancardo | Automated fundus images analysis techniques to screen retinal diseases in diabetic patients | |
| WO2024037587A1 (en) | Palpebral fissure height measurement method and apparatus, and storage medium | |
| Dutta et al. | Automatic evaluation and predictive analysis of optic nerve head for the detection of glaucoma | |
| KR20210033902A (en) | Method, apparatus and software program for cervical cancer diagnosis using image analysis of artificial intelligence based technology | |
| Singh et al. | Assessment of disc damage likelihood scale (DDLS) for automated glaucoma diagnosis | |
| Azeroual et al. | Convolutional Neural Network for Segmentation and Classification of Glaucoma. | |
| CN111291706B (en) | Retina image optic disc positioning method | |
| KR20220138069A (en) | Method, apparatus and software program for cervical cancer diagnosis using image analysis of artificial intelligence based technology | |
| Bhardwaj et al. | A computational framework for diabetic retinopathy severity grading categorization using ophthalmic image processing | |
| Biswas et al. | Grading Quality of Color Retinal Images to Assist Fundus Camera Operators | |
| Chalakkal | Automatic Retinal Image Analysis to Triage Retinal Pathologies | |
| CN112651921B (en) | Glaucoma visual field data region extraction method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |