CN110705468B - Eye movement range identification method and system based on image analysis - Google Patents
Eye movement range identification method and system based on image analysis Download PDFInfo
- Publication number
- CN110705468B CN110705468B CN201910940365.5A CN201910940365A CN110705468B CN 110705468 B CN110705468 B CN 110705468B CN 201910940365 A CN201910940365 A CN 201910940365A CN 110705468 B CN110705468 B CN 110705468B
- Authority
- CN
- China
- Prior art keywords
- eye
- image
- point
- iris
- contour
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 230000004424 eye movement Effects 0.000 title claims abstract description 55
- 238000010191 image analysis Methods 0.000 title claims abstract description 23
- 210000001508 eye Anatomy 0.000 claims abstract description 363
- 238000005286 illumination Methods 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 22
- 238000012937 correction Methods 0.000 claims abstract description 17
- 210000005252 bulbus oculi Anatomy 0.000 claims abstract description 16
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 230000000877 morphologic effect Effects 0.000 claims description 13
- 230000000694 effects Effects 0.000 claims description 10
- 238000003708 edge detection Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000005260 corrosion Methods 0.000 claims description 7
- 230000007797 corrosion Effects 0.000 claims description 7
- 101150055297 SET1 gene Proteins 0.000 claims description 3
- 230000000295 complement effect Effects 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 claims description 3
- 238000011084 recovery Methods 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 15
- 238000004458 analytical method Methods 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 4
- 208000018737 Parkinson disease Diseases 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003628 erosive effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000000744 eyelid Anatomy 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 230000004770 neurodegeneration Effects 0.000 description 1
- 208000015122 neurodegenerative disease Diseases 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 210000003786 sclera Anatomy 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/19—Sensors therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/193—Preprocessing; Feature extraction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Ophthalmology & Optometry (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an eye movement range identification method and system based on image analysis and processing, and relates to the field of image analysis and processing. The method comprises the following steps: carrying out primary positioning on a human eye image on an acquired video frame, preprocessing the human eye image, removing illumination influence, and determining an eye crack outline and an eye crack central point on the human eye image after illumination correction; further determining the iris outline and the iris center in the human eye image; the eye movement Range is determined based on the determined Iris Center point Center _ Iris and eye crack Center point Center _ eye. The eye movement range identification method and system based on image analysis can automatically analyze and detect based on the collected human eye movement video frames, and extract effective eyeball movement range characteristics by using an image processing method.
Description
Technical Field
The invention relates to the field of image analysis and processing, in particular to an eye movement range identification method and system based on image analysis.
Background
The judgment of certain symptoms or diseases by a clinician needs to be concluded on the basis of some behavior indexes or parameters in combination with other diagnosis theories and clinical experience. For example, in the second most common neurodegenerative disease, Parkinson's Disease (PD), it is clinically found that the eye movement range of PD patients is much smaller than that of normal people, so that a clinician can make a judgment by combining the behavior parameters of patients such as the eye movement range and the clinical experience. The detection of the eye movement range is mainly realized by using an eye tracker to track the sight of human eyes at present. However, existing eye trackers are very expensive and difficult to store and analyze in large quantities; secondly, subject freedom is also limited during the experiment. In addition, due to the fact that the sight line tracking technology is not completely mature and the characteristics of eye movement are caused, data can be interrupted in the experimental process, a plurality of interference signals exist, and the accuracy of sight line tracking is reduced finally. If the eye movement range of the human eye can be analyzed without the help of an eye movement instrument, the cost of the eye movement analysis is greatly reduced, and the accuracy of the analysis is improved.
Disclosure of Invention
The invention provides an eye movement range recognition method and system based on image analysis, aiming at the technical problems in the prior art, the digital signal processing technology is utilized to analyze the collected video frames of human eye movement, extract the iris center and the eye crack center of human eyes, calculate the relative distance between the iris center and the eye crack center, and further realize the analysis and evaluation of the human eye movement range.
The technical scheme adopted by the invention is as follows:
an eye movement range identification method based on image analysis comprises the following steps:
(1) carrying out primary positioning on human eye images on the acquired video frames I _ frames, namely carrying out primary positioning on all acquired images containing human eyes within the range of human eyes to obtain the human eye images I _ eye after the primary positioning;
(2) preprocessing the human eye image I _ eye, and removing illumination influence to obtain a human eye image I _ eye _ color after illumination correction;
(3) determining an eye crack Contour _ eye and an eye crack central point Center _ eye for the eye image I _ eye _ color after the illumination correction;
(4) further determining Iris outline Contour _ Iris and Iris Center _ Iris in the human eye image I _ eye _ color after the illumination correction;
(5) the eye movement Range is determined based on the determined Iris Center point Center _ Iris and eye crack Center point Center _ eye.
Further, the step (1) specifically includes:
(1.1) carrying out primary positioning on the human eye image of the first frame of video frame:
setting a first frame video frame as an I _ frame (1), carrying out primary positioning on the I _ frame (1) comprises calibrating a human face by using a deep learning training model, searching the position of eye cracks of human eyes from calibrated characteristic points, and fitting the calibrated point of the human eyes by using a rectangle so as to determine an eye image I _ eye (1) containing the primary positioning of the human eyes;
(1.2) carrying out initial positioning on the human eye image on the rest video frames:
(1.2.1) determining the human eye activity Range _ eye of the current video frame;
supposing that the current video frame is the nth frame, the image frame is marked as I _ frame (n), and the human eye movement Range Range _ eye of the current frame is determined by using the human eye image I _ eye (n-1) after the initial positioning of the previous frame;
the abscissa of the left and right boundary points of the Range _ eye of the active Range is Range _ left _ x and Range _ right _ x, and the ordinate of the upper and lower boundary points is Range _ top _ y and Range _ bottom _ y, which are respectively:
Range_left_x＝eye_left_x-W
Range_right_x＝eye_right_x+W
Range_top_y＝eye_top_y-H
Range_bottom_y＝eye_bottom_y+H；
wherein, W and H are the width and height of the human eye image I _ eye (n-1) respectively;
(1.2.2) carrying out primary human eye positioning on the current video frame based on the human eye activity Range _ eye of the current video frame, and sequentially finishing the primary human eye positioning on all the video frames;
dividing the human eye moving Range Range _ eye into a plurality of windows by using a sliding Window method, wherein the Window size is W multiplied by H, the Step length in the horizontal direction is set to be Step _ len _ x, the Step length in the vertical direction is set to be Step _ len _ y, and the top left corner vertex of the first Window corresponds to the top left corner vertex of the current human eye moving Range Range _ eye;
calculating the similarity between each Window and the previous human eye image I _ eye (n-1), searching the Window with the highest similarity, and taking the Window with the highest similarity as the human eye image I _ eye (n) of the current frame.
Further, the specific method for preprocessing the human eye image in the step (2) is as follows: and for the initially positioned human eye image I _ eye, eliminating the influence caused by uneven illumination on the extracted human eye image I _ eye by using a multi-scale Retinex algorithm with color recovery to obtain a preprocessed image I _ eye _ color.
Further, the step (3) of determining the eye crack Contour _ eye and the eye crack Center _ eye of the photo-corrected eye image I _ eye _ color specifically includes:
(3.1) acquiring a Scharr gradient component image corresponding to the human eye image I _ eye _ color after illumination correction:
(3.1.1) extracting the Scharr gradient of the image I _ eye _ color in the RGB space;
firstly, calculating the gradient of an image I _ eye _ color by using a Scharr operator, wherein the Scharr operator comprises a Scharr operator Gx in the horizontal direction and a Scharr operator Gy in the vertical direction;
if the gradient value of the (I, j) th pixel point of the eye image I _ eye _ color after the illumination correction in the horizontal direction is Gx (I, j), and the gradient value in the vertical direction is Gy (I, j), the total gradient value G (I, j) at the pixel point (I, j) is:
(3.1.2) converting the preliminarily extracted Scharr gradient image I _ Scharr into an HSV space, and extracting a corresponding V component to obtain a corresponding component image I _ ScharrV;
(3.2) carrying out edge detection on the human eye image I _ eye _ color by using an edge detection algorithm to obtain a corresponding gradient image I _ HED;
(3.3) determining the position of the eye crack Contour Contour _ eye based on the gradient component image I _ ScharrV and the gradient image I _ HED;
further, the step (3.3) of determining the position of the eye crack Contour _ eye specifically includes:
(3.3.1) combining the gradient component image I _ ScharrV and the gradient image I _ HED for extracting a corresponding binary image;
the maximum value of the pixel values of all the pixel points in the gradient component image I _ ScharrV is set as V _ Vmax, and the binary image I _ binary1 obtained when the threshold is set as V _ Vmax/4 is:
the maximum value of the pixel values of all the pixel points in the gradient image I _ HED is set as V _ HEDmax, and the binary image I _ binary2 obtained when the threshold is set as V _ HEDmax/3 is:
and (3) performing phase comparison on the binarized image I _ binary1 and the binarized image I _ binary2, and taking the intersection of the two to obtain a processed binary image I _ binary, namely:
I_binary(i,j)＝I_binary1(i,j)&I_binary2(i,j)；
wherein, I _ binary (I, j), I _ binary1(I, j) and I _ binary2(I, j) respectively represent the pixel values of the (I, j) th pixel in the binary images I _ binary, I _ binary1 and I _ binary 2;
(3.3.2) performing morphological processing on the extracted binary image I _ binary;
firstly, performing expansion operation on the binary image I _ binary to obtain I _ partition, and then performing corrosion operation to obtain an image I _ closing, wherein a calculation formula of the closing operation is as follows:
wherein A is 2 x 2 square structural element, I _ die c Is the complement of I _ dilate;
(3.3.3) further removing redundant small objects in the image and extracting the maximum connected domain in the image;
and removing small noise points in the image I _ closing obtained after the closed operation by using a method for removing small targets in morphology to obtain a corresponding image I _ morphology, and further extracting a maximum connected domain from the image I _ morphology as a corresponding eye fissure Contour Contour _ eye.
Further, the specific step of extracting the maximum connected component in the image in the step (3.3.3) is as follows:
3.3.3.1) first, all contours in the image I _ morphology are extracted, denoted as C _ mor _ set, i.e.:
C_mor_set＝{C_mor 1 ，C_mor 2 ，…，C_mor k1 ，…，C_mor n1 }
wherein, C _ mor k1 (1. ltoreq. k 1. ltoreq. n1) represents the k1 th contour, n1 is the total number of contours in the image I _ morphology;
3.3.3.2) then, the Area of each contour is calculated, resulting in an Area set Area _ set, namely:
Area_set＝{Area 1 ，Area 2 ，…，Area k1 ，…，Area n1 }
wherein, Area k1 Area representing the k1 th contour;
3.3.3.3) removing connected domains with the area less than 300 to obtain corresponding images I _ RemoveSmallObjects;
3.3.3.4) find the Contour C _ mormax with the largest area in the image I _ removesmalllobjects, which is the corresponding eye-crack Contour _ eye.
Further, the step (3.4) of determining the Center point Center _ eye of the eye crack Contour _ eye specifically comprises the following steps:
for the eye crack Contour _ eye, finding the left and right boundary points of the Contour, respectively Point _ left and Point _ right, namely:
Point_left_x＝min(Contour_eye_x)
Point_right_x＝max(Contour_eye_x)
wherein Point _ left _ x and Point _ right _ x represent the abscissa of Point _ left and Point _ right, and content _ eye _ x represents the abscissa of the points constituting the eye crack Contour;
searching the vertical coordinates Point _ left _ y and Point _ right _ y corresponding to the left boundary Point _ left and the right boundary Point _ right by using the horizontal coordinates Point _ left _ x and Point _ right _ x of the left boundary Point and the right boundary Point _ right;
taking the average value of the abscissa points Point _ left _ x and Point _ right _ x of the left and right boundary points Point _ left and Point _ right as the abscissa value Center _ eye _ x of the eye crack Center Point _ eye, taking the average value of the ordinate points Point _ left _ y and Point _ right _ y of Point _ left and Point _ right as the ordinate value Center _ eye _ y of the eye crack Center _ eye, and obtaining the pixel coordinates (Center _ eye _ x, Center _ eye _ y) of the eye crack Center Point _ eye, namely:
Center_eye_x＝(Point_left_x+Point_right_x)/2
Center_eye_y＝(Point_left_y+Point_right_y)/2。
further, the step (4) of determining the Iris outline Contour _ Iris and the Iris Center _ Iris in the image specifically includes:
(4.1) image binarization
Calculating an Otsu threshold value Otsu _ thresh for the gradient image I _ ScharrV by using an Otsu threshold value method, and performing binarization processing on the gradient image I _ Scharr by using the Otsu threshold value Otsu _ thresh to obtain a corresponding binary image I _ binary 3:
(4.2) morphological Corrosion treatment
Performing morphological erosion operation on the image I _ binary3 by using a 3 x 3 square structural element B, removing burrs at the edge of the iris, disconnecting the connection with a noise point, and obtaining a corresponding image I _ exposure:
(4.3) hole filling and finding the largest connected domain
Filling holes caused by lamplight in video recording in the iris outline of the image I _ erosion by using a morphological hole filling method to obtain a corresponding image I _ holeruled; further extracting all connected domains in the image I _ holeruled, and taking the connected domain with the largest area as the position of the iris;
(4.4) determining the corresponding Iris Center point Center _ Iris in the Iris Contour _ Iris:
calculating the centroid of the Iris outline Contour _ Iris, wherein the centroid point is the Center point (Center _ Iris _ x, Center _ Iris _ y) of the Iris, and the centroid calculation formula is as follows:
wherein px (k) and py (k) (1 ≦ k ≦ m) respectively represent the abscissa and ordinate of the kth point on the Iris Contour Contour _ Iris, and Contour _ Iris (i, j) represents the pixel value of the (i, j) th pixel point.
Further, the specific step of finding the largest connected domain in the step (4.3) includes:
(4.3.1) first, all contours in the image I _ holeruled are extracted, forming a contour set C _ hole _ set, namely:
C_hole_set＝{C_hole 1 ，C_hole 2 ，…，C_hole k2 ，…，C_hole n2 }
wherein, C _ hole k2 (1. ltoreq. k 2. ltoreq. n2) represents the k2 th contour, n2 is the total number of contours in the image I _ holeruled;
(4.3.2) Next, the Area of each contour is calculated, resulting in the Area set Area _ set1, namely:
Area_set1＝{Area 1 ，Area 2 ，…，Area k2 ，…，Area n2 }
wherein, Area k2 (1. ltoreq. k 2. ltoreq. n2) represents the area of the k2 th contour;
(4.3.3) searching the Contour C _ holomax with the largest area, wherein the Contour is the corresponding Iris Contour Contour _ Iris.
Further, the specific step of determining the eye movement Range based on the determined Iris Center point _ Iris and eye fissure Center point _ eye in the step (5) includes:
after the eye crack Center point _ eye and the Iris Center point _ Iris in each frame of image are located and determined, calculating the motion amplitude Mag of the Iris, wherein the position values of the motion amplitude Mag in the x direction and the y direction are Mag _ x and Mag _ y respectively:
Mag_x＝Center_Iris_x-Center_eye_x
Mag_y＝Center_Iris_y-Center_eye_y；
calculating the maximum value and the minimum value of the iris motion amplitude Mag in the x direction and the y direction, namely the maximum value and the minimum value of Mag _ x and Mag _ y, respectively, taking the maximum value of Mag _ x as the maximum amplitude Mag _ right of the eyeball moving to the right in the horizontal direction, taking the minimum value of Mag _ x as the maximum amplitude Mag _ left of the eyeball moving to the left in the horizontal direction, taking the maximum value of Mag _ y as the maximum amplitude Mag _ bottom of the eyeball moving downwards in the vertical direction, and taking the minimum value of Mag _ y as the maximum amplitude Mag _ top of the eyeball moving upwards in the vertical direction, namely:
Mag_right＝max(Mag_x)
Mag_left＝min(Mag_x)
Mag_bottom＝max(Mag_y)
Mag_top＝min(Mag_y)
the eye movement Range corresponding to the image frame is: [ Mag _ left: Mag _ right, Mag _ top: Mag _ bottom ].
On the other hand, the invention also provides an eye movement range recognition system based on image analysis, which is characterized in that the system is a recognition system formed on the basis of module units corresponding to any one of the eye movement range recognition method steps, and is used for recognizing the eye movement range in the acquired video frame.
In summary, due to the adoption of the technical scheme, the invention has the beneficial effects that:
the eye movement range identification method and system based on image analysis can automatically analyze and detect based on the collected human eye movement video frames, and extract effective eyeball movement range characteristics by using an image processing method.
Drawings
Fig. 1 is a schematic view of a process of primary positioning of an eye image according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a first frame right-eye image according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a corresponding position of the right-eye movement range of the second frame in the image according to the embodiment of the present invention.
Fig. 4 is a schematic view of a part of a window of a sliding window method according to an embodiment of the present invention.
Fig. 5 is a flowchart illustrating a process of eliminating an illumination effect on a human eye image according to an embodiment of the present invention.
Fig. 6 is a schematic flow chart of determining an eye crack contour and an eye crack center point for an eye image according to an embodiment of the present invention.
Fig. 7 is a schematic diagram of a gradient image provided by an embodiment of the present invention.
Fig. 8 is a schematic diagram of gradient images and components of an HSV space according to an embodiment of the present disclosure.
Fig. 9 is a schematic diagram of a gradient image after edge detection according to an embodiment of the present invention.
Fig. 10 is a schematic diagram of a binarized image according to an embodiment of the present invention.
Fig. 11 is a schematic diagram of a binarized image according to an embodiment of the present invention.
Fig. 12 is a schematic diagram of a binarized image according to an embodiment of the present invention.
Fig. 13 is a schematic diagram of an image after the closing operation processing according to the embodiment of the present invention.
Fig. 14 is a schematic diagram of a connected domain provided by an embodiment of the present invention.
Fig. 15 is a schematic view of a crack profile provided by an embodiment of the present invention.
Fig. 16 is a schematic diagram of an eye crack contour and an eye crack center point according to an embodiment of the present invention.
Fig. 17 is a schematic flow chart of determining an iris outline and an iris center of an image according to an embodiment of the present invention.
Fig. 18 is a schematic diagram of a binary image according to an embodiment of the present invention.
Fig. 19 is a schematic diagram of a morphological processing provided by an embodiment of the invention.
FIG. 20 is a schematic view of a filled cavity according to an embodiment of the present invention.
Fig. 21 is a schematic diagram of an iris profile provided by an embodiment of the invention.
Fig. 22 is a schematic diagram of iris centroid points provided by an embodiment of the present invention.
Fig. 23 is a schematic flow chart of determining an eye movement range according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the following description of the technical solutions of the present invention with reference to the accompanying drawings of the present invention is made clearly and completely, and other similar embodiments obtained by a person of ordinary skill in the art without any creative effort based on the embodiments in the present application shall fall within the protection scope of the present application.
It should be noted that the image feature processing method, the feature extraction method, the signal identification and classification method provided by the present invention and the corresponding embodiments are all only to research and improve the processing and identification method itself of the image signal, and although the range identification result achieved for the image signal collected by the human eye movement range can be used as an evaluation reference, the evaluation result is only an auxiliary evaluation in the clinical or medical field, and the specific treatment method still needs and mainly depends on the clinical experience of the doctor and the treatment method provided by the doctor.
Example 1
The embodiment is an eye movement range identification method based on image analysis, which comprises the following steps:
(1) carrying out primary positioning on human eye images of the acquired video frames, namely carrying out primary positioning on all acquired images containing human eyes within the range of human eyes, as shown in figure 1;
(1.1) carrying out primary positioning on the human eye image of the first frame of video frame:
the first frame of video is set as I _ frame (1), the acquired image is an image acquired by an image acquisition device for an object, the acquisition range of the image acquisition device can be large, and therefore, the acquired image can be initially positioned for determining the position of the eye cleft of the human eye. The I _ frame (1) is preliminarily positioned, namely a training model of deep learning is utilized to calibrate a human face, the position of eye cleft of human eyes is searched from calibrated characteristic points, and a rectangle is utilized to fit the calibrated points of the human eyes, so that the human eye image I _ eye (1) containing the initial positioning of the human eyes is determined;
in one embodiment, the face calibration can be realized by using 68 trained Dlib face marker models, which are marked as Point [1:68], wherein the serial numbers of the calibration points of the left and right eyes of a person are [43:48] and [37:42], respectively. The primary positioning of human eyes is realized by using the index points of the left eye and the right eye, and the obtained human eye images I _ eye (1) respectively comprise human eye images I _ eye _ left (1) and I _ eye _ right (1) of the left eye and the right eye of a human, wherein the human eye images I _ eye _ left (1) and the human eye images I _ eye _ right (1) are W multiplied by H three-dimensional color images, and W and H respectively represent the width and the height of the human eye images.
Taking the right eye as an example, if the sequence number of the leftmost point of the right eye fissure is 37, the sequence number of the rightmost point is 40, the sequence number of the topmost point is 38 or 39, and the sequence number of the bottommost point is 41 or 42, then the abscissa eye _ left _ x and eye _ right _ x of the left and right boundary points of the human right eye image I _ eye _ right (1), and the ordinate of the upper and lower boundary points are eye _ top _ y and eye _ bottom _ y, respectively:
eye_left_x＝Point[37]_x-l_x
eye_right_x＝Point[40]_x+l_x
eye_top_y＝min(Point[38]_y,Point[39]_y)-l_y
eye_bottom_y＝max(Point[41]_y,Point[42]_y)+l_y
wherein Point [ alpha ] x and Point [ alpha ] y respectively represent the abscissa and ordinate of the Point corresponding to the sequence number alpha (37 ≦ alpha ≦ 42), and l _ x and l _ y respectively represent the increase of the width and height of the human right eye image I _ eye _ right (1). In the present embodiment, l _ x-l _ y-40 is selected for the parameter.
The width W and height H of the human right eye image I _ eye _ right (1) are respectively:
W＝eye_right_x-eye_left_x
H＝eye_bottom_y-eye_top_y
the corresponding first frame right eye image I _ eye _ right (1) is shown in fig. 2.
It should be noted that, in the actual image processing process, the processing may be performed according to the unit of I _ eye (1), or may also be performed respectively according to the unit of I _ eye _ left (1) and I _ eye _ right (1), and the processing substantially includes processing the left eye and the right eye in the captured image, and for the purpose of describing the image processing process more simply and clearly, the subsequent steps in this embodiment will be described by using I _ eye (1).
(1.2) carrying out initial positioning on the human eye image on the rest video frames;
(1.2.1) determining the human eye activity Range _ eye of the current video frame;
assuming that the current video frame is the nth frame, the image frame is denoted as I _ frame (n), the eye image I _ eye (n-1) after the initial positioning of the previous frame is used to determine the eye movement Range _ eye of the current frame, the abscissa of the left and right boundary points of the movement Range _ eye is Range _ left _ x and Range _ right _ x, and the ordinate of the upper and lower boundary points is Range _ top _ y and Range _ bottom _ y, which are respectively:
Range_left_x＝eye_left_x-W
Range_right_x＝eye_right_x+W
Range_top_y＝eye_top_y-H
Range_bottom_y＝eye_bottom_y+H
the size of the human eye moving Range of the nth frame (n ≧ 2) is 3W × 3H, and the position corresponding to the right eye moving Range in the second frame image is shown in fig. 3, for example.
(1.2.2) carrying out human eye initial positioning on the current video frame based on the human eye activity Range _ eye of the current video frame;
dividing the Range _ eye of the human eye movement Range into a plurality of windows by using a sliding Window method (the Window size is W multiplied by H), wherein the Window windows are obtained by the following method:
(1.2.2.1) setting the Step size in the horizontal direction as Step _ len _ x, the Step size in the vertical direction as Step _ len _ y, and setting the vertex at the upper left corner of the first window to correspond to the vertex at the upper left corner of the Range _ eye of the human eye activity Range of the current frame, then:
Window(1)_left_x＝Range_left_x
Window(1)_right_x＝Range_left_x+W
Window(1)_top_y＝Range_top_y
Window(1)_bottom_y＝Range_top_y+H
wherein, Window (1) _ left _ x, Window (1) _ right _ x, Window (1) _ top _ y, and Window (1) _ bottom _ y respectively represent the left, right, upper, and lower boundaries of the first Window.
(1.2.2.2) the corresponding boundary point coordinates of the kth Window of the kth _ row and the kth _ col column are as follows:
Window(k)_left_x＝Range_left_x+(k_col-1)*Step_len_x
Window(k)_right_x＝Range_left_x+(k_col-1)*Step_len_x+W
Window(k)_top_y＝Range_top_y+(k_row-1)*Step_len_y
Window(k)_bottom_y＝Range_top_y+(k_row-1)*Step_len_y+H
wherein 1 ≦ k _ row ≦ int (2H/(Setp _ len _ y)) +1, 1 ≦ k _ col ≦ int (2W/(Setp _ len _ x)) +1, k ═ k _ row-1) ((int (2W/(Setp _ len _ x)) +1) + k _ col, Window (k) _ left _ x, Window (k) _ right _ x, Window (k) _ top _ y, and Window (k) _ bottom _ y respectively represent the left boundary, right boundary, upper boundary, and lower boundary of the kth Window.
In one embodiment, Step _ len _ x is selected to be Step _ len _ y is selected to be 25, and then a partial Window obtained by the sliding Window method is shown in fig. 4.
(1.2.2.3) calculating the similarity between each Window and the previous human eye image I _ eye (n-1), finding the Window with the highest similarity, and taking the Window with the highest similarity as the human eye image I _ eye (n) of the current frame.
In one embodiment, a template matching method-average difference matching method is adopted for the similarity algorithm, and the specific calculation steps are as follows:
traversing each pixel point in the k Window and the human eye image I _ eye (n-1) of the previous frame, and calculating the square difference Diff (k) between the corresponding pixel points, wherein the calculation formula of the square difference Diff is as follows:
wherein, Window (k) (I, j) and I _ eye (n-1) (I, j) respectively represent the pixel values of the (I, j) th pixel point of the k-th Window and the previous frame human eye image I _ eye (n-1).
All windows have a square difference Diff, and the Window corresponding to the Diff with the smallest distance value is the human eye image I _ eye (n) of the current frame I _ frame (n).
(2) And (4) preprocessing the human eye image and removing the illumination influence.
For the initially positioned human eye image I _ eye, because the gray value of the image is unevenly distributed in the process of image acquisition due to uneven illumination, the human eye image I _ eye needs to be preprocessed so as to realize illumination correction processing. In one embodiment, a multiscale retinex (msrcr) algorithm with color recovery is used to eliminate the influence of uneven illumination on the extracted human eye image I _ eye to obtain a preprocessed image I _ eye _ color, and the steps of the algorithm are shown in fig. 5.
(2.1) calculating the logarithm values I _ eye _ R _ log, I _ eye _ G _ log and I _ eye _ B _ log at R, G, B for the pixel values I _ eye _ R, I _ eye _ G and I _ eye _ B of three channels for each pixel point of the eye image I _ eye, and calculating the Mean value _ R, Mean _ G, Mean _ B and the Mean square error Var _ R, Var _ G, Var _ B in each channel, namely:
I_eye_R_log(i，j)＝log(I_eye_R(i，j))
I_eye_G_iog(i，j)＝log(I_eye_G(i，j))
I_eye_B_log(i，j)＝log(I_eye_B(i，j))
(2.2) finding the maximum values Max _ R, Max _ G and Max _ B and the minimum values Min _ R, Min _ G and Min _ B of the pixel means of the respective channels, namely:
Max_R＝max(Mean_R)
Max_G＝max(Mean_G)
Max_B＝max(Mean_B)
Min_R＝min(Mean_R)
Min_G＝min(Mean_G)
Min_B＝min(Mean_B)
(2.3) linearly mapping the pixel logarithm values I _ eye _ R _ log, I _ eye _ G _ log and I _ eye _ B _ log of each channel to [0, 255] for normalization, wherein R _ R, R _ G and R _ B are obtained after conversion, namely:
(2.4) an image formed by the values of the three channels R _ R, R _ G and R _ B is the human eye image I _ eye _ color after the illumination correction.
(3) For the eye image I _ eye _ color after the illumination correction, determining the eye crack Contour _ eye and the eye crack Center point _ eye, the steps of the algorithm are shown in fig. 6:
(3.1) acquiring a Scharr gradient image corresponding to the human eye image I _ eye _ color after illumination correction, wherein the method comprises the following steps:
(3.1.1) extracting the Scharr gradient of the image I _ eye _ color in the RGB space; because the pixel values of the boundaries of the iris, the sclera and the upper and lower eyelids contained in the human eye image I _ eye _ color after illumination correction are greatly different, namely the gradient value is large, the gradient information in the image can be well extracted by using the gradient detection operator, and the extraction of the eye crack outline is realized.
Firstly, the gradient of the image I _ eye _ color is calculated by using the Scharr operator, and the Scharr operator Gx in the horizontal direction and the Scharr operator Gy in the vertical direction are respectively as follows:
if the gradient value of the (I, j) th pixel point of the human eye image I _ eye _ color after the illumination correction in the horizontal direction is Gx (I, j), and the gradient value in the vertical direction is Gy (I, j), the total gradient value G (I, j) at the pixel point (I, j) is:
after the gradient G (I, j) is calculated for the eye image I _ eye _ color, a corresponding gradient image I _ Scharr is obtained, as shown in fig. 7. As can be seen from the image shown in fig. 7, the contour of the eye fissure is already obvious, however, many complex noise textures still exist in the preliminarily extracted gradient image I _ Scharr, and further processing and optimization are needed.
(3.1.2) converting the preliminarily extracted gradient image I _ Scharr into HSV space, and extracting a corresponding V component to obtain a corresponding component image I _ ScharrV.
In order to highlight the eye crack contour to be extracted, the preliminarily extracted gradient image I _ Scharr is converted from RGB space to HSV space, and the gradient image I _ ScharrHSV of HSV space is obtained, and the corresponding image I _ ScharrHSV and H, S, V components thereof are respectively shown in fig. 8(a) - (d).
From the graphs in fig. 8, it can be seen that the eye crack contour in the gradient component image I _ ScharrV corresponding to the V component of I _ ScharrHSV in graph (d) is relatively complete and the noise in the image is small, so the gradient component image I _ ScharrV is further selected for further analysis.
(3.2) carrying out edge detection on the human eye image I _ eye _ color by using an edge detection algorithm to obtain a corresponding gradient image I _ HED;
in one embodiment, edge detection is performed on the human eye image I _ eye _ color by using an HED edge detection algorithm established in a convolutional neural network and a deep surveillance network, so that multi-scale and multi-level feature learning can be realized. The corresponding gradient image I _ HED is obtained as shown in fig. 9.
(3.3) determining the position of the eye crack Contour Contour _ eye based on the gradient component image I _ ScharrV and the gradient image I _ HED;
(3.3.1) combining the gradient component image I _ ScharrV and the gradient image I _ HED for extracting a corresponding binary image;
assuming that the maximum value of the pixel values of all the pixel points in the gradient component image I _ ScharrV is V _ Vmax, it was found through research in one embodiment that when the threshold is set to V _ Vmax/4, the obtainable binarized image I _ binary1 is:
the binarized image I _ binary1 is shown in fig. 10;
assuming that the maximum value of the pixel values of all the pixel points in the gradient image I _ HED is V _ HEDmax, it is found through research in one embodiment that when the threshold is set to V _ HEDmax/3, the obtainable binary image I _ binary2 is:
the binarized image I _ binary2 is shown in fig. 11.
And (3) performing phase comparison on the binarized image I _ binary1 and the binarized image I _ binary2, and taking the intersection of the two to obtain a processed binary image I _ binary, namely:
I_binary(i,j)＝I_binary1(i,j)&I_binary2(i,j)
wherein, I _ bank (I, j), I _ bank 1(I, j) and I _ bank 2(I, j) respectively represent pixel values of the (I, j) th pixel point in the binary images I _ bank, I _ bank 1 and I _ bank 2. The corresponding binary image I _ binary is shown in fig. 12.
(3.3.2) morphological processing is carried out on the extracted binary image I _ binary
The binary image I _ binary obtained by the foregoing steps still has burrs connected with the eye-crack contour, as shown in fig. 12, and therefore, it is necessary to further perform a morphological closing operation to achieve image edge smoothing. The closed operation is to perform expansion operation on the binary image I _ binary to obtain I _ dilate, and then perform corrosion operation to obtain an image I _ closing, wherein the calculation formula of the closed operation is as follows:
wherein A is 2 x 2 square structural element, I _ die c Is the complement of I _ dilate.
Fig. 13 shows the corresponding image I _ closing after the closing operation processing.
(3.3.3) further removing redundant small objects in the image and extracting the maximum connected domain in the image;
the image I _ closing obtained after the closed operation is shown as fig. 13, wherein a certain small isolated noise point still exists, the small noise point in the image I _ closing is removed by using a method for removing a small target in morphology to obtain a corresponding image I _ morphology, and a maximum connected domain is further extracted from the image I _ morphology, wherein the specific extraction steps are as follows:
3.3.3.1) first, all contours in the image I _ morphology are extracted, denoted as C _ mor _ set, i.e.:
C_mor_set＝{C_mor 1 ，C_mor 2 ，…，C_mor k1 ，…，C_mor n1 }
wherein, C _ mor k1 (1. ltoreq. k 1. ltoreq. n1) represents the k1 th contour, n1 is the total number of contours in the image I _ morphology;
3.3.3.2) then, the Area of each contour is calculated, resulting in an Area set Area _ set, namely:
Area_set＝{Area 1 ，Area 2 ，…，Area k1 ，…，Area n1 }
wherein, Area k1 Indicating the area of the k1 th contour.
3.3.3.3) remove connected components with an area less than 300, resulting in the corresponding image I _ RemoveSmallObjects, as shown in FIG. 14.
3.3.3.4) find the Contour C _ mormax with the largest area in the image I _ removesmallcobjects, which is the corresponding eye crack Contour _ eye, as shown in fig. 15.
(3.4) determining the Center point Center _ eye of the crack Contour Contour _ eye
For the eye crack Contour _ eye, finding the left and right boundary points of the Contour, respectively Point _ left and Point _ right, namely:
Point_left_x＝min(Contour_eye_x)
Point_right_x＝max(Contour_eye_x)
here, Point _ left _ x and Point _ right _ x represent the abscissa of Point _ left and Point _ right, and content _ eye _ x represents the abscissa of the Point constituting the eye-crack Contour.
And searching the vertical coordinates Point _ left _ y and Point _ right _ y corresponding to the left and right boundary points Point _ left and Point _ right in the eye crack Contour _ eye by using the horizontal coordinates Point _ left _ x and Point _ right _ x of the left and right boundary points Point _ left and Point _ right.
Taking the average of the abscissa Point _ left _ x and Point _ right _ x of the left and right boundary points Point _ left and Point _ right as the abscissa value Center _ eye _ x of the eye crack Center Point _ eye, and taking the average of the ordinate Point _ left _ y and Point _ right _ y of the Point _ left and Point _ right as the ordinate value Center _ eye _ y of the eye crack Center Point _ eye, the pixel coordinates (Center _ eye _ x, Center _ eye _ y) of the eye crack Center Point Center _ eye are obtained, that is:
Center_eye_x＝(Point_left_x+Point_right_x)/2
Center_eye_y＝(Point_left_y+Point_right_y)/2
the eye crack Contour _ eye and the eye crack Center _ eye are shown in fig. 16.
(4) The Iris Contour, Contour _ Iris, and Iris Center _ Iris in the image are determined, and the algorithm steps are shown in fig. 17.
(4.1) image binarization
An Otsu threshold value Otsu _ thresh is calculated for the gradient image I _ ScharrV by an Otsu threshold value method, and binarization processing is performed on the gradient image I _ Scharr by the threshold value Otsu _ thresh to obtain a corresponding binary image I _ binary3, that is:
the binary image I _ binary3 is shown in fig. 18.
(4.2) morphological Corrosion treatment
As shown in fig. 18, there are many noises connected to the iris in the binarized image I _ binary3, and the image I _ binary3 is morphologically corroded by using 3 × 3 square structural elements B, thereby removing burrs at the edge of the iris and breaking the connection with noise points, and the corresponding image I _ cross is obtained as shown in fig. 19.
(4.3) filling holes and finding the maximum connected domain
The image I _ exposure is shown in fig. 19, wherein holes caused by light in video recording exist in the iris outline, and the holes of the image I _ exposure are filled by using a morphological hole filling method to obtain a corresponding image I _ holedistorted, as shown in fig. 20.
Further extracting all connected domains in the image I _ holeruled, and taking the connected domain with the largest area as the position of the iris, wherein the specific steps comprise:
(4.3.1) first, all contours in the image I _ holeruled are extracted, forming a contour set C _ hole _ set, namely:
C_hole_set＝{C_hole 1 ，C_hole 2 ，…，C_hole k2 ，…，C_hole n2 }
wherein, C _ hole k2 (1. ltoreq. k 2. ltoreq. n2) represents the k2 th contour, and n2 is the total number of contours in the image I _ holeruled.
(4.3.2) Next, the Area of each contour is calculated, resulting in the Area set Area _ set1, namely:
Area_set1＝{Area 1 ，Area 2 ，…，Area k2 ，…，Area n2 }
wherein, Area k2 (1. ltoreq. k 2. ltoreq. n2) represents the area of the k2 th contour.
(4.3.3) finding the Contour C _ holemax with the largest area, which is the corresponding Iris Contour Contour _ Iris, and the Iris Contour is shown in FIG. 21.
(4.4) determining the corresponding Iris Center _ Iris in the Iris Contour Contour _ Iris
Calculating the centroid of the Iris outline Contour _ Iris, wherein the centroid point is the Center point (Center _ Iris _ x, Center _ Iris _ y) of the Iris, and the centroid calculation formula is as follows:
wherein px (k) and py (k) (k is more than or equal to 1 and less than or equal to m) respectively represent the abscissa and the ordinate of the kth point on the Iris outline Contour _ Iris, and Contour _ Iris (i, j) represents the pixel value of the (i, j) th pixel point.
The Iris center point content _ Iris is shown in fig. 22.
(5) The eye movement Range is determined based on the determined Iris Center point Center _ Iris and the eye fissure Center point Center _ eye, as shown in fig. 23.
After the eye crack Center point _ eye and the Iris Center point _ Iris in each frame of image are located and determined, calculating the motion amplitude Mag of the Iris, wherein the position values of the motion amplitude Mag in the x direction and the y direction are Mag _ x and Mag _ y respectively:
Mag_x＝Center_Iris_x-Center_eye_x
Mag_y＝Center_Iris_y-Center_eye_y
calculating the maximum value and the minimum value of the iris motion amplitude Mag in the x direction and the y direction, namely the maximum value and the minimum value of Mag _ x and Mag _ y, respectively, taking the maximum value of Mag _ x as the maximum amplitude Mag _ right of the eyeball moving to the right in the horizontal direction, taking the minimum value of Mag _ x as the maximum amplitude Mag _ left of the eyeball moving to the left in the horizontal direction, taking the maximum value of Mag _ y as the maximum amplitude Mag _ bottom of the eyeball moving downwards in the vertical direction, and taking the minimum value of Mag _ y as the maximum amplitude Mag _ top of the eyeball moving upwards in the vertical direction, namely:
Mag_right＝max(Mag_x)
Mag_left＝min(Mag_x)
Mag_bottom＝max(Mag_y)
Mag_top＝min(Mag_y)
the eye movement Range corresponding to the image frame is: [ Mag _ left: Mag _ right, Mag _ top: Mag _ bottom ].
The eye movement range identification method of the embodiment can be completed through the processing steps, the corresponding identification result can be obtained, and the motion amplitude curves of the eyeballs corresponding to the eye movement range in the horizontal direction (x _ axis) and the vertical direction (y _ axis) can be respectively drawn in the collected video image frame.
Example 2
The present embodiment is an eye movement range recognition system based on image analysis, and the system is a recognition system composed of module units corresponding to the recognition method in any of the foregoing embodiments, and is used for recognizing an eye movement range in a captured video image frame.
Through the embodiments provided by the invention, automatic analysis and detection can be carried out based on the collected human eye movement video image frames, and the effective eyeball movement range characteristics are extracted by using an image processing method.
All of the features disclosed in this specification, or all of the steps in any method or process so disclosed, may be combined in any combination, except combinations of features and/or steps that are mutually exclusive.
The invention is not limited to the foregoing embodiments. The invention extends to any novel feature or any novel combination of features disclosed in this specification and any novel method or process steps or any novel combination of features disclosed.
Claims (10)
1. An eye movement range identification method based on image analysis is characterized by comprising the following steps:
(1) carrying out primary positioning on human eye images on the acquired video frames I _ frames, namely carrying out primary positioning on all acquired images containing human eyes within the range of human eyes to obtain the human eye images I _ eye after the primary positioning;
(2) preprocessing the human eye image I _ eye, and removing illumination influence to obtain a human eye image I _ eye _ color after illumination correction;
(3) determining an eye crack Contour ContoureEye for the eye image I _ eye _ color after the illumination correction, and determining an eye crack central point Center _ eye from the eye crack Contour ContoureEye;
(4) further determining an Iris outline Contour _ Iris in the human eye image I _ eye _ color after the illumination correction, and determining an Iris Center _ Iris from the Iris outline Contour _ Iris;
(5) determining an eye movement Range based on the determined Iris Center point Center _ Iris and eye fissure Center point Center _ eye;
the step (1) specifically comprises:
(1.1) initially positioning the human eye image of the first frame video frame:
setting a first frame video frame as an I _ frame (1), carrying out primary positioning on the I _ frame (1) comprises calibrating a human face by using a deep learning training model, searching the position of eye cracks of human eyes from calibrated characteristic points, and fitting the calibrated point of the human eyes by using a rectangle so as to determine an eye image I _ eye (1) containing the primary positioning of the human eyes;
(1.2) carrying out initial positioning on the human eye image on the rest video frames:
(1.2.1) determining the human eye activity Range _ eye of the current video frame;
assuming that the current video frame is the nth frame, the image frame is marked as I _ frame (n), and the human eye movement Range _ eye of the current frame is determined by using the human eye image I _ eye (n-1) after the initial positioning of the previous frame;
the abscissa of the left and right boundary points of the Range _ eye of the active Range is Range _ left _ x and Range _ right _ x, and the ordinate of the upper and lower boundary points is Range _ top _ y and Range _ bottom _ y, which are respectively:
Range_left_x＝eye_left_x-W
Range_right_x＝eye_right_x+W
Range_top_y＝eye_top_y-H
Range_bottom_y＝eye_bottom_y+H；
wherein, W and H are the width and height of the human eye image I _ eye (n-1) respectively;
(1.2.2) carrying out primary human eye positioning on the current video frame based on the human eye activity Range _ eye of the current video frame, and sequentially finishing the primary human eye positioning on all the video frames;
dividing the human eye activity Range _ eye into a plurality of windows by using a sliding Window method, wherein the Window size is WxH, the Step length in the horizontal direction is set to be Step _ len _ x, the Step length in the vertical direction is set to be Step _ len _ y, and then the top left corner vertex of the first Window corresponds to the top left corner vertex of the current human eye activity Range _ eye;
calculating the similarity between each Window and the previous human eye image I _ eye (n-1), searching the Window with the highest similarity, and taking the Window with the highest similarity as the human eye image I _ eye (n) of the current frame.
2. The eye movement range recognition method based on image analysis as claimed in claim 1, wherein the specific method for preprocessing the human eye image in the step (2) is as follows: and for the initially positioned human eye image I _ eye, eliminating the influence caused by uneven illumination on the extracted human eye image I _ eye by using a multi-scale Retinex algorithm with color recovery to obtain a preprocessed image I _ eye _ color.
3. The eye movement range recognition method based on image analysis as claimed in claim 2, wherein the step (3) of determining the eye crack Contour _ eye and the eye crack Center point _ eye for the photo-corrected eye image I _ eye _ color specifically comprises:
(3.1) acquiring a Scharr gradient component image corresponding to the human eye image I _ eye _ color after illumination correction:
(3.1.1) extracting the Scharr gradient of the image I _ eye _ color in the RGB space;
firstly, calculating the gradient of an image I _ eye _ color by using a Scharr operator, wherein the Scharr operator comprises a Scharr operator Gx in the horizontal direction and a Scharr operator Gy in the vertical direction;
if the gradient value of the (I, j) th pixel point of the eye image I _ eye _ color after the illumination correction in the horizontal direction is Gx (I, j), and the gradient value in the vertical direction is Gy (I, j), the total gradient value G (I, j) at the pixel point (I, j) is:
(3.1.2) converting the preliminarily extracted Scharr gradient image I _ Scharr into an HSV space, and extracting a corresponding V component to obtain a corresponding component image I _ ScharrV;
(3.2) carrying out edge detection on the human eye image I _ eye _ color by using an edge detection algorithm to obtain a corresponding gradient image I _ HED;
(3.3) determining the position of the eye crack Contour Contour _ eye based on the gradient component image I _ ScharrV and the gradient image I _ HED;
(3.4) determining the Center _ eye of the eye crack Contour Contour _ eye.
4. The eye movement range recognition method based on image analysis as claimed in claim 3, wherein the step (3.3) of determining the position of the eye crack Contour Contour _ eye specifically comprises:
(3.3.1) combining the gradient component image I _ ScharrV and the gradient image I _ HED for extracting a corresponding binary image;
the maximum value of the pixel values of all the pixel points in the gradient component image I _ ScharrV is set as V _ Vmax, and the binary image I _ binary1 obtained when the threshold is set as V _ Vmax/4 is:
the maximum value of the pixel values of all the pixel points in the gradient image I _ HED is set as V _ HEDmax, and the binary image I _ binary2 obtained when the threshold is set as V _ HEDmax/3 is:
and (3) performing phase comparison on the binarized image I _ binary1 and the binarized image I _ binary2, and taking the intersection of the two to obtain a processed binary image I _ binary, namely:
I_binary(i,j)＝I_binary1(i,j)&I_binary2(i,j)；
wherein, I _ binary (I, j), I _ binary1(I, j) and I _ binary2(I, j) respectively represent the pixel values of the (I, j) th pixel points in the binary images I _ binary, I _ binary1 and I _ binary 2;
(3.3.2) performing morphological processing on the extracted binary image I _ binary;
firstly, performing expansion operation on the binary image I _ binary to obtain I _ dilate, and then performing corrosion operation to obtain an image I _ closing, wherein a calculation formula of the closing operation is as follows:
wherein A is 2 x 2 square structural element, I _ die c Is the complement of I _ dilate;
(3.3.3) further removing redundant small objects in the image and extracting the maximum connected domain in the image;
and removing small noise points in the image I _ closing obtained after the closed operation by using a method for removing small targets in morphology to obtain a corresponding image I _ morphology, and further extracting a maximum connected domain from the image I _ morphology as a corresponding eye fissure Contour Contour _ eye.
5. The eye movement range recognition method based on image analysis as claimed in claim 4, wherein the specific steps of extracting the maximum connected component in the image in the step (3.3.3) are as follows:
3.3.3.1) first, all contours in the image I _ morphology are extracted, denoted as C _ mor _ set, i.e.:
C_mor_set＝{C_mor 1 ，C_mor 2 ，…，C_mor k1 ，…，C_mor n1 }
wherein, C _ mor k1 (1 ≦ k1 ≦ n1) representing the k1 th contour, n1 being the total number of contours in the image I _ morphology;
3.3.3.2) then, the Area of each contour is calculated, resulting in an Area set Area _ set, namely:
Area_set＝{Area 1 ，Area 2 ，…，Area k1 ，…，Area n1 }
wherein, Area k1 Area representing the k1 th contour;
3.3.3.3) removing connected domains with the area less than 300 to obtain corresponding images I _ RemoveSmallObjects;
3.3.3.4) find the Contour C _ mormax with the largest area in the image I _ removesmalllobjects, which is the corresponding eye-crack Contour _ eye.
6. The eye movement range recognition method based on image analysis as claimed in claim 5, wherein the step (3.4) of determining the Center point Center _ eye of the eye crack Contour Contour _ eye comprises the following steps:
for the eye crack Contour _ eye, finding the left and right boundary points of the Contour, respectively Point _ left and Point _ right, namely:
Point_left_x＝min(Contour_eye_x)
Point_right_x＝max(Contour_eye_x)
wherein Point _ left _ x and Point _ right _ x represent the abscissa of Point _ left and Point _ right, and content _ eye _ x represents the abscissa of the points constituting the eye crack Contour;
searching the vertical coordinates Point _ left _ y and Point _ right _ y corresponding to the left boundary Point _ left and the right boundary Point _ right by using the horizontal coordinates Point _ left _ x and Point _ right _ x of the left boundary Point and the right boundary Point _ right;
taking the average value of the abscissa points Point _ left _ x and Point _ right _ x of the left and right boundary points Point _ left and Point _ right as the abscissa value Center _ eye _ x of the eye crack Center Point _ eye, taking the average value of the ordinate points Point _ left _ y and Point _ right _ y of Point _ left and Point _ right as the ordinate value Center _ eye _ y of the eye crack Center _ eye, and obtaining the pixel coordinates (Center _ eye _ x, Center _ eye _ y) of the eye crack Center Point _ eye, namely:
Center_eye_x＝(Point_left_x+Point_right_x)/2
Center_eye_y＝(Point_left_y+Point_right_y)/2。
7. the eye movement range recognition method based on image analysis as claimed in claim 2, wherein the step (4) of determining the Iris outline Contour _ Iris and the Iris Center _ Iris in the image comprises the specific steps of:
(4.1) image binarization
Calculating an Otsu threshold value Otsu _ thresh for the gradient image I _ ScharrV by using an Otsu threshold value method, and performing binarization processing on the gradient image I _ Scharr by using the threshold value Otsu _ thresh to obtain a corresponding binary image I _ binary 3:
(4.2) morphological Corrosion treatment
Performing morphological corrosion operation on the image I _ binary3 by using the 3 × 3 square structural element B, removing burrs at the edge of the iris, disconnecting the connection with a noise point, and obtaining a corresponding image I _ evolution:
(4.3) filling holes and finding the maximum connected domain
Filling holes caused by lamplight in video recording in the iris outline of the image I _ exposure by using a morphological hole filling method to obtain a corresponding image I _ holeruled; further extracting all connected domains in the image I _ holeruled, and taking the connected domain with the largest area as the position of the iris;
(4.4) determining the corresponding Iris Center point Center _ Iris in the Iris Contour _ Iris:
calculating the centroid of the Iris outline Contour _ Iris, wherein the centroid point is the central point (Center _ Iris _ x, Center _ Iris _ y) of the Iris, and the centroid calculation formula is as follows:
wherein px (k) and py (k) (k is more than or equal to 1 and less than or equal to m) respectively represent the abscissa and the ordinate of the kth point on the Iris outline Contour _ Iris, and Contour _ Iris (i, j) represents the pixel value of the (i, j) th pixel point.
8. The eye movement range recognition method based on image analysis as claimed in claim 7, wherein the specific step of finding the maximum connected component in step (4.3) comprises:
(4.3.1) first, all contours in the image I _ holeruled are extracted, forming a contour set C _ hole _ set, namely:
C_hole_set＝[C_hole 1 ，C_hole 2 ，…，C_hole k2 ，…，C_hole n2 }
wherein, C _ hole k2 (1 ≦ k2 ≦ n2) representing the k2 th contour, n2 being the total number of contours in the image I _ holeruled;
(4.3.2) Next, the Area of each contour is calculated, resulting in the Area set Area _ set1, namely:
Area_set1＝{Area 1 ，Area 2 ，…，Area k2 ，…，Area n2 }
wherein, Area k2 (1. ltoreq. k 2. ltoreq. n2) represents the area of the k2 th contour;
(4.3.3) searching the Contour C _ holomax with the largest area, wherein the Contour is the corresponding Iris Contour Contour _ Iris.
9. The eye movement Range recognition method based on image analysis as claimed in claim 2, wherein the step (5) of determining the eye movement Range based on the determined Iris Center point Center _ Iris and eye fissure Center point Center _ eye comprises the specific steps of:
after the eye fissure Center point _ eye and the Iris Center point _ Iris in each frame of image are located and determined, calculating the motion amplitude Mag of the Iris, wherein the position values of the motion amplitude Mag in the x direction and the y direction are Mag _ x and Mag _ y respectively:
Mag_x＝Center_Iris_x-Center_eye_x
Mag_y＝Center_Iris_y-Center_eye_y；
calculating the maximum value and the minimum value of the iris motion amplitude Mag in the x direction and the y direction, namely the maximum value and the minimum value of Mag _ x and Mag _ y, respectively, taking the maximum value of Mag _ x as the maximum amplitude Mag _ right of the eyeball moving to the right in the horizontal direction, taking the minimum value of Mag _ x as the maximum amplitude Mag _ left of the eyeball moving to the left in the horizontal direction, taking the maximum value of Mag _ y as the maximum amplitude Mag _ bottom of the eyeball moving downwards in the vertical direction, and taking the minimum value of Mag _ y as the maximum amplitude Mag _ top of the eyeball moving upwards in the vertical direction, namely:
Mag_right＝max(Mag_x)
Mag_left＝min(Mag_x)
Mag_bottom＝max(Mag_y)
Mag_top＝min(Mag_y)
the eye movement Range corresponding to the image frame is: [ Mag _ left: Mag _ right, Mag _ top: Mag _ bottom ].
10. An eye movement range recognition system based on image analysis, which is characterized in that the system is a recognition system composed of module units corresponding to the steps of the eye movement range recognition method according to any one of claims 1-9, and is used for recognizing the eye movement range in the collected video image frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910940365.5A CN110705468B (en) | 2019-09-30 | 2019-09-30 | Eye movement range identification method and system based on image analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910940365.5A CN110705468B (en) | 2019-09-30 | 2019-09-30 | Eye movement range identification method and system based on image analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110705468A CN110705468A (en) | 2020-01-17 |
| CN110705468B true CN110705468B (en) | 2022-08-30 |
Family
ID=69197842
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910940365.5A Expired - Fee Related CN110705468B (en) | 2019-09-30 | 2019-09-30 | Eye movement range identification method and system based on image analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110705468B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111861977A (en) * | 2020-05-27 | 2020-10-30 | 温州医科大学附属眼视光医院 | Feature extraction method of anterior segment tomogram based on machine vision |
| CN112347860B (en) * | 2020-10-16 | 2023-04-28 | 福建天泉教育科技有限公司 | Gradient-based eye state detection method and computer-readable storage medium |
| CN115908237B (en) * | 2022-08-18 | 2023-09-08 | 上海佰翊医疗科技有限公司 | Eye crack width measuring method, device and storage medium |
| CN115376167B (en) * | 2022-10-26 | 2023-02-24 | 山东圣点世纪科技有限公司 | Palm detection method and system under complex background |
| CN115761871B (en) * | 2022-12-01 | 2023-08-11 | 北京中科睿医信息科技有限公司 | Detection image generation method, device, equipment and medium based on eye movement detection |
| CN116958885B (en) * | 2023-09-19 | 2023-12-05 | 四川大学 | Correcting glasses wearing comfort evaluation method and system based on reading vision |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102426644A (en) * | 2011-08-24 | 2012-04-25 | 夏东 | Human eye cross localization method based on GentleBoost machine learning |
| CN103885589A (en) * | 2014-03-06 | 2014-06-25 | 华为技术有限公司 | Eye movement tracking method and device |
| CN106778499A (en) * | 2016-11-24 | 2017-05-31 | 江苏大学 | A kind of method of quick positioning people's eye iris during iris capturing |
| US10089525B1 (en) * | 2014-12-31 | 2018-10-02 | Morphotrust Usa, Llc | Differentiating left and right eye images |
| WO2019011803A1 (en) * | 2017-07-10 | 2019-01-17 | Trinamix Gmbh | Detector for optically detecting at least one object |
| CN110221699A (en) * | 2019-06-13 | 2019-09-10 | 北京师范大学珠海分校 | A kind of eye movement Activity recognition method of front camera video source |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7085401B2 (en) * | 2001-10-31 | 2006-08-01 | Infowrap Systems Ltd. | Automatic object extraction |
| US7508979B2 (en) * | 2003-11-21 | 2009-03-24 | Siemens Corporate Research, Inc. | System and method for detecting an occupant and head pose using stereo detectors |
-
2019
- 2019-09-30 CN CN201910940365.5A patent/CN110705468B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102426644A (en) * | 2011-08-24 | 2012-04-25 | 夏东 | Human eye cross localization method based on GentleBoost machine learning |
| CN103885589A (en) * | 2014-03-06 | 2014-06-25 | 华为技术有限公司 | Eye movement tracking method and device |
| US10089525B1 (en) * | 2014-12-31 | 2018-10-02 | Morphotrust Usa, Llc | Differentiating left and right eye images |
| CN106778499A (en) * | 2016-11-24 | 2017-05-31 | 江苏大学 | A kind of method of quick positioning people's eye iris during iris capturing |
| WO2019011803A1 (en) * | 2017-07-10 | 2019-01-17 | Trinamix Gmbh | Detector for optically detecting at least one object |
| CN110221699A (en) * | 2019-06-13 | 2019-09-10 | 北京师范大学珠海分校 | A kind of eye movement Activity recognition method of front camera video source |
Non-Patent Citations (2)
| Title |
|---|
| 基于MATLAB的视觉方向识别算法;吴海翔;《中国科技信息》;20171012;全文 * |
| 虹膜识别关键技术研究及实现;周啸;《南京航空航天大学硕士论文》;20120815;第3.4.1节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110705468A (en) | 2020-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110705468B (en) | Eye movement range identification method and system based on image analysis | |
| Adeyanju et al. | Machine learning methods for sign language recognition: A critical review and analysis | |
| CN103870808B (en) | Finger vein identification method | |
| CN109829877A (en) | A kind of retinal fundus images cup disc ratio automatic evaluation method | |
| CN106599804B (en) | Fovea centralis detection method based on multiple features model | |
| CN111582086A (en) | Fatigue driving identification method and system based on multiple characteristics | |
| CN106530283A (en) | SVM (support vector machine)-based medical image blood vessel recognition method | |
| CN101359365A (en) | Iris positioning method based on Maximum between-Cluster Variance and gray scale information | |
| EP2285266A1 (en) | Automatic cup-to-disc ratio measurement system | |
| Harangi et al. | Automatic detection of the optic disc using majority voting in a collection of optic disc detectors | |
| CN107563996A (en) | A kind of new discus nervi optici dividing method and system | |
| CN106651888A (en) | Color fundus image optic cup segmentation method based on multi-feature fusion | |
| CN106778499B (en) | Method for rapidly positioning human iris in iris acquisition process | |
| CN106960199B (en) | Complete extraction method of white eye region of true color eye picture | |
| CN107516083B (en) | Recognition-oriented remote face image enhancement method | |
| Jayanthi et al. | Automatic diagnosis of retinal diseases from color retinal images | |
| CN106570447A (en) | Face photo sunglass automatic removing method based on gray histogram matching | |
| CN115456967A (en) | Arteriovenous internal fistula thrombus detection method and device | |
| CN112069986A (en) | Machine vision tracking method and device for eye movements of old people | |
| CN110929570B (en) | Iris rapid positioning device and positioning method thereof | |
| Pal et al. | Detection of red lesions in retinal fundus images using YOLO V3 | |
| CN114202795A (en) | Method for quickly positioning pupils of old people | |
| Mahdi et al. | Glaucoma Diagnosis Based on Retinal Fundus Image: A Review | |
| CN110751064B (en) | Blink frequency analysis method and system based on image processing | |
| Karmawat et al. | Glaucoma detection using fuzzy C-means optic cup segmentation and feature classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220830 |