CN117058747A - Gaze vector detection method and device and electronic equipment - Google Patents
Gaze vector detection method and device and electronic equipment Download PDFInfo
- Publication number
- CN117058747A CN117058747A CN202311055986.8A CN202311055986A CN117058747A CN 117058747 A CN117058747 A CN 117058747A CN 202311055986 A CN202311055986 A CN 202311055986A CN 117058747 A CN117058747 A CN 117058747A
- Authority
- CN
- China
- Prior art keywords
- pupil
- human eye
- eye image
- pupil center
- current frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000013598 vector Substances 0.000 title claims abstract description 113
- 238000001514 detection method Methods 0.000 title claims description 59
- 210000001747 pupil Anatomy 0.000 claims abstract description 143
- 238000009826 distribution Methods 0.000 claims abstract description 78
- 238000000034 method Methods 0.000 claims abstract description 72
- 238000003384 imaging method Methods 0.000 claims description 19
- 230000015654 memory Effects 0.000 claims description 17
- 230000007704 transition Effects 0.000 claims description 17
- 239000011159 matrix material Substances 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 10
- 238000007781 pre-processing Methods 0.000 claims description 9
- 238000012360 testing method Methods 0.000 claims description 9
- 238000003860 storage Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000007619 statistical method Methods 0.000 claims description 5
- 230000008878 coupling Effects 0.000 abstract description 11
- 238000010168 coupling process Methods 0.000 abstract description 11
- 238000005859 coupling reaction Methods 0.000 abstract description 11
- 210000001508 eye Anatomy 0.000 description 196
- 210000004087 cornea Anatomy 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000004424 eye movement Effects 0.000 description 6
- 210000005252 bulbus oculi Anatomy 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000005286 illumination Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000001179 pupillary effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010924 continuous production Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000000611 regression analysis Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Multimedia (AREA)
- Eye Examination Apparatus (AREA)
Abstract
The invention provides a method, a device and electronic equipment for detecting a gaze vector, wherein the method takes a pupil rotation center as a point with fixed position in space, so that the problems of ambiguous gaze vector and gaze direction and coupling caused by fixed point drift are avoided, further, the gaze vector determined according to pupil center coordinates and pupil rotation center coordinates can accurately reflect the gaze direction, in addition, when the pupil center coordinates are determined, the pupil center is detected on the whole of a preprocessed current frame human eye image, and meanwhile, the pupil center distribution probability of a target subarea to which the detected pupil center coordinates belong is checked, the accuracy of the pupil center coordinates is ensured, and finally, the gaze vector of the determined pupil center coordinates and pupil rotation center coordinates has good accuracy, and the gaze direction can be accurately reflected.
Description
Technical Field
The present invention relates to the field of gaze tracking technologies, and in particular, to a method and an apparatus for detecting a gaze vector, and an electronic device.
Background
Eye tracking technology, which is generally a technology for recording and measuring eye movements (i.e., changes in the direction of the line of sight), has been widely used. For example, in the medical field, eye tracking may be used for disease analysis, as well as to identify and analyze the visual attention patterns of individuals when performing certain tasks (e.g., reading, searching, browsing an image, driving, etc.), thereby quantifying visual attention, exploring human behavior; in the VR and game fields, the problem of fatigue in use can be alleviated by operating VR scenes or games through eyes, in addition, the gaze point area of a user is subjected to image enhancement by capturing the gaze point of the user, high-quality images are presented, and the images of non-gaze point areas are weakened, namely, the gaze point rendering technology is adopted, so that the data processing speed can be improved, the requirements of VR equipment or application on hardware equipment are reduced, the characteristics of human eyes are met, and better experience is brought to the user; in the nerve marketing field, through eye movement tracking, merchants can improve package design, store layout, point-of-sale display and the like for content focused or ignored by users; in the field of human-computer interaction, eye tracking may identify a user's intent by detecting eye movements, thereby helping the user interact with the environment. Thus, eye tracking plays an extremely important role in various fields of daily life.
Currently, the most commonly used eye tracking technique is video analytics (VOG), which captures images of the patient's eyes and face with a head-mounted or stationary camera, and matches eye movement to gaze points in the scene in a non-contact manner, typically involving two steps of gaze vector detection and calibration.
Depending on the configuration of the human eye, the angle λ between the pupillary axis and the line of sight of a particular individual (the particular individual for each person) will not change significantly over a longer period of time, and therefore the direction of the pupillary axis may reflect the direction of the line of sight. The pupil axis is a straight line passing through the center of the pupil and perpendicular to the cornea, and because the pupil and the cornea both rotate along with the eyeball, the relative positions of the pupil and the cornea are unchanged, based on the fact, the included angle between the pupil axis and the normal line of the pupil is kept fixed, and a certain mapping relation exists between the normal line direction of the pupil and the sight line direction. Since the rotation center of the pupil is not in the pupil plane, the pupil center coordinates and the pupil normal line are in one-to-one correspondence (each time there is one pupil center coordinate, there is one pupil normal line corresponding to the pupil center coordinates), and the pupil center coordinates and the sight line direction are also in one-to-one correspondence, so that only one point with fixed position in the space needs to be found, and then the connection line of the point and the pupil center coordinates has one-to-one and determined relationship with the sight line direction. Since the line of sight has a defined mapping relation to the coordinates of the gaze point in the scene, the vector (the line connecting the fixed point to the pupil center coordinates, which may be referred to as a gaze vector reflecting the direction of the line of sight) also has a defined mapping relation to the coordinates of the gaze point.
Among the methods commonly used in the current VOG technology are the PCCR method and the pupil center-canthus method. In the PCCR method: the human eye is irradiated with visible light or infrared light, and the direction of the line of sight is reflected by a pupil center-cornea reflection point (P-CR) vector. In this method, the eyeball is regarded as a standard sphere and the cornea is regarded as a standard sphere, and therefore, the position of the corneal reflection point is theoretically unchanged with the rotation of the eyeball, i.e., the corneal reflection point is a point at which the above position is fixed (i.e., a fixed end point of the above gaze vector). However, since the eyeball is not a standard sphere and the curvature of the cornea is different at different positions, the position of the cornea reflection point varies with the rotation of the eyeball (i.e., the cornea reflection point is not actually a point whose position is fixed in space), and the variation range is large, therefore, when the cornea reflection point is used as a fixed end point of the gaze vector to detect the gaze vector, the gaze vector cannot accurately reflect the line of sight direction, that is, when the gaze vector represented by the pupil center-cornea reflection point reflects the line of sight direction, the error is large. In pupil center-canthus: by recognizing the corner position, the gaze direction is reflected with the pupil center-corner vector as a gaze vector, wherein the corner is considered as a point where the above-described position is fixed. However, the positions of the corners of the eyes of a person can change in the process of making different expressions, which is easy to cause larger errors. In addition, because the shape of the canthus is irregular and the individual difference is larger, the recognition difficulty is increased, and more errors are easily introduced.
In summary, in the above two schemes, the point considered to be fixed in position in the gaze vector for reflecting the direction of the line of sight is not actually fixed in the physical space, and therefore, the relationship between the gaze vector for reflecting the direction of the line of sight and the direction of the line of sight, which is finally detected, is not clear (i.e., the detected gaze vector cannot accurately reflect the direction of the line of sight). Although some errors can be eliminated by calibration and fitting, the fitted mapping is inaccurate; in addition, due to drift of fixed-position points, there may be linear correlation between gaze vectors corresponding to different gaze directions, causing mapping problems of a plurality of gaze vectors corresponding to the same gaze vector.
In addition, pupil detection is easily affected by ambient light and camera angle. For example, a dim environment or uneven illumination condition may cause a dark area of the corner of the eye in the image to be mistakenly identified as the pupil in the binarization process (a human eye image, typically an RGB image or a gray image, is required to be converted into a binarized image when the human eye image is processed, and at this time, if the dark area should be white, but is darker, as the pupil is, eventually, black), and thus the pupil center is incorrectly positioned, which eventually results in tracking failure. Currently, in order to reduce interference caused by other regions and thus reduce misrecognition of the pupil, an existing scheme redefines a region of interest (ROI) for each captured human eye image, i.e., a specific region defined by a formula around the pupil center is regarded as the ROI for the next pupil center detection. For example, T.Santini et al define an ROI having a width equal to twice the long axis of the pupil. S.lee et al define a circular ROI whose radius and center are adjusted by the coordinates of the pupil center. The method of defining the dynamic ROI reduces interference caused by other regions, but the shape and size of the ROI are defined based on experience, which is different from the actual situation of the user. Thus, the ROI may contain a disturbing region or exclude a true pupil region, and then the darkest region in the ROI is identified as the pupil center, resulting in a continuous error.
In summary, the existing gaze vector cannot accurately reflect the direction of the line of sight, and in the existing gaze vector detection method, due to the influence of ambient light and camera angles, the pupil center is often misidentified, so that the accuracy of the detected gaze vector is poor.
Disclosure of Invention
Accordingly, the present invention aims to provide a method, an apparatus and an electronic device for detecting a gaze vector, so as to solve the technical problems that the existing gaze vector cannot accurately reflect the direction of the line of sight, and the accuracy of the detected gaze vector is poor due to the erroneous recognition of the pupil center.
In a first aspect, an embodiment of the present invention provides a method for detecting a gaze vector, including:
acquiring a preprocessed current frame human eye image, and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions;
performing pupil center detection on the preprocessed current frame human eye image to obtain pupil center coordinates in the preprocessed current frame human eye image, and determining a target sub-region to which the pupil center coordinates belong;
Judging whether the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image reaches a preset probability threshold;
if so, determining that the pupil center coordinate is correctly detected;
calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane;
and determining the fixation vector according to the pupil center coordinates and the pupil rotation center coordinates.
Further, acquiring a preprocessed current frame human eye image, and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein the method comprises the following steps:
acquiring a current frame human eye image, and preprocessing the current frame human eye image to obtain a preprocessed current frame human eye image;
determining the original pupil center distribution probability of each sub-region in the preprocessed current frame of human eye image according to the pupil center distribution probability of each sub-region in the previous frame of human eye image and a preset state transition matrix;
When the previous frame of human eye image is the first frame of human eye image, the pupil center distribution probability of each sub-region in the previous frame of human eye image is determined according to the sub-region to which the pupil center obtained after the previous frame of human eye image is subjected to pupil center marking manually, and the element of the ith row and the jth column in the preset state transition matrix represents the probability that the pupil center is in the ith sub-region in the previous frame of human eye image and the pupil center is in the jth sub-region in the current frame of human eye image;
and normalizing the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image to obtain the pupil center distribution probability of each sub-region in the preprocessed current frame human eye image.
Further, determining the original pupil center distribution probability of each sub-region in the preprocessed current frame of human eye image according to the pupil center distribution probability of each sub-region in the previous frame of human eye image and a preset state transition matrix, including:
according to the first expressionCalculating the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein +_ >Representing the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image,/or->Representing the preset state transition matrix,/a>Representing pupil center distribution probability of each sub-region in the last frame of human eye image, wherein the preset state transition matrix is obtained by calculating human eye test data collected in advance through a statistical method, and the human eye test data is a sub-region to which the pupil center obtained by manually marking the pupil center of the collected human eye image belongs.
Further, the method further comprises:
if the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image does not reach the preset probability threshold, determining that the pupil center coordinate is detected incorrectly, and taking the subarea, of which the pupil center distribution probability of each subarea in the preprocessed current frame human eye image reaches the preset probability threshold, as a pupil center detection area;
and detecting the pupil center of the pupil center detection area to obtain correct pupil center coordinates.
Further, calculating the pupil rotation center coordinate according to the line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and the line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, including:
Calculating the intersection point of the straight line of the short axis of the pupil projection ellipse of the previous frame of human eye image and the straight line of the short axis of the pupil projection ellipse of the current frame of human eye image, and determining the coordinate of the intersection point;
and taking the coordinate of the intersection point as the pupil rotation center coordinate.
Further, determining the gaze vector from the pupil center coordinates and the pupil rotation center coordinates includes:
according to the second equationCalculating the gaze vector, wherein (u) t+1 ，v t+1 ) Representing the gaze vector, (x) t+1 ，y t+1 ) Representing the pupil center coordinates, (x s ，y s ) Representing the pupil rotation center coordinates。
Further, the method further comprises:
determining a corresponding gaze direction based on the gaze vector;
and determining the coordinates of the gaze point in the scene image coordinate system according to the mapping relation among the gaze vector, the eye camera coordinate system and the scene camera coordinate system.
In a second aspect, an embodiment of the present invention further provides a device for detecting a gaze vector, including:
the acquisition and determination unit is used for acquiring a preprocessed current frame human eye image and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions;
The pupil center detection unit is used for detecting the pupil center of the preprocessed current frame human eye image, obtaining pupil center coordinates in the preprocessed current frame human eye image, and determining a target sub-region to which the pupil center coordinates belong;
the judging unit is used for judging whether the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image reaches a preset probability threshold value;
the first determining unit is used for determining that the pupil center coordinate is correctly detected if the pupil center coordinate is reached;
the calculation unit is used for calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane;
and the second determining unit is used for determining the gazing vector according to the pupil center coordinates and the pupil rotation center coordinates.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps of the method according to any one of the first aspects when the processor executes the computer program.
In a fourth aspect, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions which, when invoked and executed by a processor, cause the processor to perform the method of any one of the first aspects.
In an embodiment of the present invention, a method for detecting a gaze vector is provided, including: acquiring a preprocessed current frame human eye image, and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions; performing pupil center detection on the preprocessed current frame human eye image to obtain pupil center coordinates in the preprocessed current frame human eye image, and determining a target sub-region to which the pupil center coordinates belong; judging whether the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image reaches a preset probability threshold value or not; if the pupil center coordinate is detected, determining that the pupil center coordinate is detected correctly; calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane; and determining the fixation vector according to the pupil center coordinates and the pupil rotation center coordinates. According to the detection method of the gaze vector, disclosed by the invention, the pupil rotation center is used as a point with fixed position in space, the problems of ambiguous gaze vector and gaze direction and coupling caused by fixed point drift are avoided, the gaze vector determined according to the pupil center coordinates and the pupil rotation center coordinates can accurately reflect the gaze direction, in addition, when the pupil center coordinates are determined, the pupil center detection is carried out on the whole of the eye image of the current frame after preprocessing, meanwhile, the pupil center distribution probability of a target subarea to which the detected pupil center coordinates belong is checked, the accuracy of the pupil center coordinates is ensured, the accuracy of the finally determined gaze vector of the pupil center coordinates and the pupil rotation center coordinates is good, the gaze direction can be accurately reflected, and the technical problems that the gaze direction cannot be accurately reflected by the existing gaze vector and the accuracy of the detected gaze vector is poor caused by the pupil center misrecognition are solved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a method for detecting a gaze vector according to an embodiment of the present invention;
FIG. 2 is a diagram of a sphere, pupil and camera position rotated by the pupil center according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a projection theorem of a circle provided by an embodiment of the present invention;
FIG. 4 shows a sphere O and pupil plane O of an imaging view angle according to an embodiment of the present invention 1 Schematic of (2);
FIG. 5 shows a plane S according to an embodiment of the present invention 2 Schematic representation of the cross section after cutting the ball O;
fig. 6 is a schematic diagram of a gaze vector detection apparatus according to an embodiment of the present invention;
fig. 7 is a schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions of the present invention will be clearly and completely described in connection with the embodiments, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The traditional gazing vector cannot accurately reflect the direction of the sight, and in the traditional gazing vector detection method, due to the influence of ambient light and camera angles, the pupil center is often misidentified, so that the detected gazing vector is poor in accuracy.
Based on the above, in the detection method of the gaze vector, the pupil rotation center is used as a point with fixed position in space, the problems of ambiguous gaze vector and gaze direction and coupling caused by fixed point drift are avoided, the gaze vector determined according to the pupil center coordinate and the pupil rotation center coordinate can accurately reflect the gaze direction, in addition, when the pupil center coordinate is determined, the pupil center detection is carried out on the whole of the eye image of the current frame after preprocessing, meanwhile, the pupil center distribution probability of a target subarea to which the detected pupil center coordinate belongs is checked, the accuracy of the pupil center coordinate is ensured, and the gaze vector of the finally determined pupil center coordinate and pupil rotation center coordinate has good accuracy and can accurately reflect the gaze direction.
For the convenience of understanding the present embodiment, a method for detecting a gaze vector disclosed in the present embodiment will be described in detail.
Embodiment one:
according to an embodiment of the present invention, there is provided an embodiment of a method of gaze vector detection, it being noted that the steps shown in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.
Fig. 1 is a flowchart of a method of detecting a gaze vector according to an embodiment of the present invention, as shown in fig. 1, the method comprising the steps of:
step S102, acquiring a preprocessed current frame human eye image, and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions;
step S104, pupil center detection is carried out on the preprocessed current frame human eye image, pupil center coordinates in the preprocessed current frame human eye image are obtained, and a target sub-region to which the pupil center coordinates belong is determined;
the pupil center detection process is a traditional technology, and specifically, pupil ellipse fitting can be performed by adopting an image processing technology means, so that pupil center coordinates are obtained.
Step S106, judging whether the pupil center distribution probability of the target sub-region in the preprocessed current frame human eye image reaches a preset probability threshold;
step S108, if the detection result is reached, determining that the pupil center coordinate is detected correctly;
in order to overcome the problem that the dark corner area is mistakenly identified as the pupil under the dim environment or uneven illumination condition, the time sequence pupil center position prediction model building method based on the Markov chain in the steps S102 to S108 is explored. Eye movement is a continuous process in which two successive captured pupil center positions approach, and if other areas are erroneously detected, a distance jump occurs. Based on the above, a time sequence prediction method based on a Markov chain is designed, and the error rate is reduced by comparing the pupil center distribution probability of the target subarea to which the detected pupil center position belongs with a preset probability threshold.
Step S110, calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane;
The following theoretical demonstration is made on the position fixing of the pupil rotation center in space and the calculation method thereof:
the pupil center rotates around the pupil rotation center to form a hemispherical surface, and the connecting line of the pupil rotation center and the pupil center is perpendicular to the pupil surface, so that the pupil is a tangent plane circle of the hemispherical surface, and the tangent point is the pupil center. As shown in FIG. 2, the sphere is defined as O, the center of the sphere is the rotation center O point of the pupil, and the pupil is the tangent circle O of the sphere O 1 The circle center is pupil center point O 1 。
The human eye camera imaging corresponds to the projection of the pupil onto the camera imaging plane. According to the projection theorem of a circle, the projection of the pupil on the imaging plane of the camera must be a circle or an ellipse, wherein the short axis of the ellipse is perpendicular to the intersection line of the plane of the pupil and the projection plane, and the projection must pass through the center of the pupil, as shown in fig. 3.
Seen from the camera direction, as shown in FIG. 4, the sphere O and the pupil circle O 1 Is the same as its projection in a plane parallel to the imaging plane of the camera.
Assuming pupil circle O 1 The plane is S 1 For parallel to the camera imaging plane and passing through point O 1 Plane S of (2) 2 Cutting ball O with round surface O 2 The center of circle is point O 2 As shown in fig. 5.
Ball O and round surface O 1 To plane S 2 Projection, which is identical to camera imaging. Projection of sphere center O and point O 2 Overlap due to S 2 Point of passing O 1 Thus the center of circle O 1 Is itself the projection of (c). If find the round surface O 1 In plane S 2 Projection onto the surface, the circular surface O is required 1 Plane S in which 1 And plane S 2 Is a line of intersection l.
Due to the centre of a circle O 1 In plane S 1 In the plane S 2 In, thus let go of O 1 . Also know plane S 1 Is a tangential plane to the sphere O, thus, plane S 1 Has only one intersection point O with the spherical surface O 1 And then plane S 1 With circumference O 2 Also only one intersection point O 1 . Due to l being in plane S 1 In, therefore, l and circumference O 2 With only one point of intersection O 1 . From the foregoing, it can be seen that l and circumference O 2 Are all in plane S 2 In, therefore, l is circumference O 2 Tangent to point O 1 . From the tangential line properties, the straight line O 1 O 2 T l. Due to circle O 1 In plane S 2 The minor axis of the ellipse obtained by internal projection must pass through O 1 And perpendicular to l, therefore, the minor axis must be in line O 1 O 2 And (3) upper part.
It follows that the minor axes of ellipses derived from the projection of different pupil circles onto the camera imaging plane (i.e. pupil projection ellipses) intersect at a point (i.e. point O 2 ) This point is the projection of the pupil rotation center onto the camera imaging plane. The pupil rotation center is fixed in space, so that the pupil rotation center can be used as a fixed end of a fixation vector, the problems of uncertainty and coupling of the fixation vector and the sight line direction caused by fixed point drift are solved, and the pupil rotation center coordinates can be solved through intersection points of short axes of ellipses obtained by projection of different pupil circles on a camera imaging plane.
Step S112, a fixation vector is determined according to the pupil center coordinates and the pupil rotation center coordinates.
In an embodiment of the present invention, a method for detecting a gaze vector is provided, including: acquiring a preprocessed current frame human eye image, and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions; performing pupil center detection on the preprocessed current frame human eye image to obtain pupil center coordinates in the preprocessed current frame human eye image, and determining a target sub-region to which the pupil center coordinates belong; judging whether the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image reaches a preset probability threshold value or not; if the pupil center coordinate is detected, determining that the pupil center coordinate is detected correctly; calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane; and determining the fixation vector according to the pupil center coordinates and the pupil rotation center coordinates. According to the detection method of the gaze vector, disclosed by the invention, the pupil rotation center is used as a point with fixed position in space, the problems of ambiguous gaze vector and gaze direction and coupling caused by fixed point drift are avoided, the gaze vector determined according to the pupil center coordinates and the pupil rotation center coordinates can accurately reflect the gaze direction, in addition, when the pupil center coordinates are determined, the pupil center detection is carried out on the whole of the eye image of the current frame after preprocessing, meanwhile, the pupil center distribution probability of a target subarea to which the detected pupil center coordinates belong is checked, the accuracy of the pupil center coordinates is ensured, the accuracy of the finally determined gaze vector of the pupil center coordinates and the pupil rotation center coordinates is good, the gaze direction can be accurately reflected, and the technical problems that the gaze direction cannot be accurately reflected by the existing gaze vector and the accuracy of the detected gaze vector is poor caused by the pupil center misrecognition are solved.
The above description briefly describes the method for detecting a gaze vector according to the present invention, and the detailed description thereof will be given below.
In an optional embodiment of the present invention, a preprocessed current frame human eye image is obtained, and a pupil center distribution probability of each sub-region in the preprocessed current frame human eye image is determined, which specifically includes the following steps:
(1) Acquiring a current frame human eye image, and preprocessing the current frame human eye image to obtain a preprocessed current frame human eye image;
specifically, the eye image can be captured through a head-mounted or fixed camera, so that the eye image of the current frame is obtained, the eye image of the current frame is preprocessed, and the preprocessed eye image of the current frame is obtained.
The pretreatment comprises the following steps: 1) Defining a region of interest (i.e., pupil center detection area) to reduce computation and eliminate interference; 2) Converting the RGB image into a gray scale map; 3) Smoothing high frequency noise by gaussian filtering; 4) Morphological methods are used to highlight image features.
(2) Determining the original pupil center distribution probability of each sub-region in the human eye image of the current frame after preprocessing according to the pupil center distribution probability of each sub-region in the human eye image of the previous frame and a preset state transition matrix;
When the previous frame of human eye image is the first frame of human eye image, the pupil center distribution probability of each sub-region in the previous frame of human eye image is the probability that the pupil center is in the ith sub-region in the previous frame of human eye image and in the jth sub-region in the current frame of human eye image according to the fact that the obtained sub-region to which the pupil center belongs is determined and obtained after the previous frame of human eye image is subjected to pupil center marking manually;
specifically, according to the first expressionCalculating the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein +.>Representing the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image,/and->Representing a preset state transition matrix->Representing the pupil center distribution probability of each sub-region in the previous frame of human eye image, wherein the preset state transition matrix is obtained by calculating the human eye test data collected in advance through a statistical method, and the human eye test data is the sub-region to which the pupil center is belonged after the collected human eye image is marked by the pupil center manually.
The process is described in detail below:
the current frame human eye imageThe last frame of human eye image is uniformly divided into 5×9 sub-regions (the number of the sub-regions is not particularly limited in the invention), and is denoted as a 1 ，A 2 ，A 3 ，…，A 45 The pupil center coordinate in the previous frame of human eye image (namely the human eye image at the moment t) is (x) t ，y t ) I.e. the pupil center coordinate at time t is (x t ，y t ) The pupil center coordinate in the current frame human eye image (i.e. the human eye image at time t+1) is (x) t+1 ，y t+1 ) Pupil center coordinate at instant t+1 is (x) t+1 ，y t+1 ) Can pass through Q t+1 ＝P·Q t Calculating the original pupil center distribution probability (the original pupil center distribution probability of each subarea at the moment t+1) of each subarea in the current frame human eye image, wherein Q t+1 And Q t Representing the pupil center distribution probability of each subarea in the current frame of human eye image and the pupil center distribution probability of each subarea in the previous frame of human eye image, namely the pupil center distribution probability matrix of each subarea at the moment t and the moment t+1, Q t+1 ＝[q 1，t+1 ，q 2，t+1 ，...，q 45，t+1 ] T ，Q t ＝[q 1，t ，q 2，t ，...，q 45，t ] T 。
q i，t+1 And q i，t (i=1, 2,.,. 45) is that the pupil center is located in sub-region a i Due to (x) t ，y t ) Sub-region A to which it belongs j It is known (that is, the sub-region to which the pupil center coordinate in the previous frame of human eye image belongs is known, and when the previous frame of human eye image is the first frame of human eye image, the sub-region to which the pupil center coordinate in the previous frame of human eye image belongs is the sub-region to which the pupil center obtained after the previous frame of human eye image is manually marked), so, Representing that the pupil center in the previous frame of human eye image is positioned in the subarea A j The probability of (1) is that the pupil center in the previous frame of human eye image is not in the subarea A j The probability of (2) is 0.
P represents a preset state transition matrix,wherein p is i，j Representing the probability that the pupil center is in the ith sub-area in the previous frame of human eye image and the pupil center is in the jth sub-area in the current frame of human eye image, namely that the pupil center is in A at the moment t i And at time t+1 is at A j Probability of p ij ＝P{(x t+1 ，y t+1 )∈A j |(x t ，y t )∈A i }. P was calculated statistically using collected human eye test data over 5 minutes.
The above formula is arranged to obtain:
and further, calculating the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image according to the first arithmetic formula.
(3) Normalizing the original pupil center distribution probability of each subarea in the preprocessed current frame human eye image to obtain the pupil center distribution probability of each subarea in the preprocessed current frame human eye image.
Specifically, in order to unify the metrics, normalization processing is required, and the process is as follows:and obtaining the pupil center distribution probability of each sub-region in the preprocessed current frame human eye image.
In an alternative embodiment of the invention, the method further comprises:
(1) If the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image does not reach the preset probability threshold, determining that the pupil center coordinate is detected incorrectly, and taking the subarea of which the pupil center distribution probability of each subarea in the preprocessed current frame human eye image reaches the preset probability threshold as a pupil center detection area;
specifically, assuming pupil center coordinates (x t+1 ，y t+1 ) Belonging to the target subarea A m (m=1, 2,.,. 45), if(q represents a preset probability threshold), the pupil center coordinates (x) t+1 ，y t+1 ) An error is detected. Will->Is defined as the pupil center detection area.
(2) And (3) performing pupil center detection on the pupil center detection area to obtain correct pupil center coordinates.
In an alternative embodiment of the present invention, the pupil rotation center coordinate is calculated according to a line where a short axis of a pupil projection ellipse of a previous frame of human eye image is located and a line where a short axis of a pupil projection ellipse of a current frame of human eye image is located, and specifically includes the following steps:
(1) Calculating the intersection point of the straight line of the short axis of the pupil projection ellipse of the previous frame of human eye image and the straight line of the short axis of the pupil projection ellipse of the current frame of human eye image, and determining the coordinate of the intersection point;
Specifically, assume that the straight line where the short axis of the pupil projection ellipse (of the current frame human eye image) at time t+1 is located is m t+1 The straight line of the short axis of the pupil projection ellipse (of the previous frame of human eye image) at the moment t is m t An intersection point (x) of two straight lines is obtained s ，y s )。
(2) The coordinates of the intersection point are taken as pupil rotation center coordinates.
In an alternative embodiment of the present invention, the method for determining the gaze vector according to the pupil center coordinates and the pupil rotation center coordinates specifically includes the steps of:
according to the second equationCalculating a gaze vector, wherein (u) t+1 ，v t+1 ) Representing gaze vector, (x) t+1 ，y t+1 ) Representing pupil center coordinates, (x) s ，y s ) Representing pupil rotation center coordinates.
In an alternative embodiment of the invention, the method further comprises:
(1) Determining a corresponding gaze direction based on the gaze vector;
specifically, the corresponding line-of-sight direction is determined from the conversion relationship between the gaze vector and the line-of-sight direction.
(2) And determining the coordinates of the gaze point in the scene image coordinate system according to the mapping relation among the gaze vector, the eye camera coordinate system and the scene camera coordinate system, and further determining the gaze point of the user in the scene image.
In particular, the method comprises the steps of,(u, v) represents the gaze vector (i.e. pupil center-pupil rotation center vector in the human eye image coordinate system), (s, t) represents the coordinates of the gaze point in the scene image coordinate system, a i (i=0, 1,2, 3) and b i (i=0, 1,2,3, 4) represents constants in the map, and these constants are determined to values using a 9-point calibration method.
During the calibration process, the user gazes at nine points at a time while keeping the head stationary. Corresponding pupil center-pupil rotation center vectors (i.e., gaze vectors) are recorded, and constants in the mapping relationship are calculated by regression analysis.
To demonstrate the feasibility of the method of the invention, subjects were allowed to move the eyes for 3 minutes while pupil centers were detected by applying the method and by the PCCR algorithm without applying a markov chain. The processing results were stored as 2 groups each containing 5400 images per frame. In the group where no markov chain was applied, the pupil center in 4826 images was correctly detected, while in the other group, the pupil center in 5219 images was correctly detected. The method can improve the precision by 7.3%. Compared with PCCR which uses cornea reflection points as fixed points, the fixed points selected by the method are fixed in position in a physical space, so that the problems of undefined gaze vector and sight direction and coupling caused by fixed point drifting can be avoided; compared with a dynamic ROI method, the algorithm calculates the distribution probability by a statistical method, has higher reliability (the invention not only extracts a small region of interest (namely, a pupil center detection region, the region of interest is defined in the region near the pupil center), namely, the region of interest is not found near the pupil center, or is found in the whole domain), but also can retain the pupil characteristics and eliminate the interference region by combining global detection (namely, pupil center detection is carried out on the preprocessed current frame human eye image) and local detection (namely, pupil center detection is carried out on the pupil center detection region). Thus, the method has more accurate and reliable performance.
The method is a novel eye tracking method based on two-dimensional images, builds pupil rotation center-pupil center vectors to reflect the sight direction, calculates pupil rotation center coordinates as fixed points, and calibrates in real time, so that the problems of uncertainty and coupling between the gaze vectors and the sight direction caused by fixed point drift are avoided, and the eye tracking precision is improved; and the sources and the characteristics of interference under the non-ideal illumination condition are researched, the bottleneck that the pupil center detection precision is poor under the non-ideal illumination condition in the eye movement tracking field is broken through, a time sequence pupil position prediction model based on a Markov chain is established, the influence of non-uniform light is overcome by narrowing the detection range, and the accurate detection of the pupil center is realized.
Embodiment two:
the embodiment of the invention also provides a device for detecting the gazing vector, which is mainly used for executing the method for detecting the gazing vector provided in the first embodiment of the invention, and the device for detecting the gazing vector provided in the first embodiment of the invention is specifically described below.
Fig. 6 is a schematic diagram of a gaze vector detection apparatus according to an embodiment of the present invention, as shown in fig. 6, the apparatus mainly includes: an acquisition and determination unit 10, a pupil center detection unit 20, a judgment unit 30, a first determination unit 40, a calculation unit 50, and a second determination unit 60, wherein:
The acquisition and determination unit is used for acquiring the preprocessed current frame human eye image and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions;
the pupil center detection unit is used for carrying out pupil center detection on the preprocessed current frame human eye image to obtain pupil center coordinates in the preprocessed current frame human eye image, and determining a target sub-region to which the pupil center coordinates belong;
the judging unit is used for judging whether the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image reaches a preset probability threshold value;
the first determining unit is used for determining that the pupil center coordinate is correctly detected if the pupil center coordinate is reached;
the calculation unit is used for calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane;
And the second determining unit is used for determining a fixation vector according to the pupil center coordinates and the pupil rotation center coordinates.
In an embodiment of the present invention, there is provided a device for detecting a gaze vector, including: acquiring a preprocessed current frame human eye image, and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions; performing pupil center detection on the preprocessed current frame human eye image to obtain pupil center coordinates in the preprocessed current frame human eye image, and determining a target sub-region to which the pupil center coordinates belong; judging whether the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image reaches a preset probability threshold value or not; if the pupil center coordinate is detected, determining that the pupil center coordinate is detected correctly; calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane; and determining the fixation vector according to the pupil center coordinates and the pupil rotation center coordinates. As can be seen from the above description, in the gaze vector detection device of the present invention, the pupil rotation center is used as a point with a fixed position in space, so that the problems of ambiguous gaze vector and gaze direction and coupling caused by fixed point drift are avoided, and further, the gaze vector determined according to the pupil center coordinate and pupil rotation center coordinate can accurately reflect the gaze direction.
Optionally, the acquiring and determining unit is further configured to: acquiring a current frame human eye image, and preprocessing the current frame human eye image to obtain a preprocessed current frame human eye image; determining the original pupil center distribution probability of each sub-region in the human eye image of the current frame after preprocessing according to the pupil center distribution probability of each sub-region in the human eye image of the previous frame and a preset state transition matrix; when the previous frame of human eye image is the first frame of human eye image, the pupil center distribution probability of each sub-region in the previous frame of human eye image is the probability that the pupil center is in the ith sub-region in the previous frame of human eye image and in the jth sub-region in the current frame of human eye image according to the fact that the obtained sub-region to which the pupil center belongs is determined and obtained after the previous frame of human eye image is subjected to pupil center marking manually; normalizing the original pupil center distribution probability of each subarea in the preprocessed current frame human eye image to obtain the pupil center distribution probability of each subarea in the preprocessed current frame human eye image.
Optionally, the acquiring and determining unit is further configured to: according to the first expression Calculating the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein +.>Representing the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image,/and->Representing a preset state transition matrix->Representing the pupil center distribution probability of each sub-region in the previous frame of human eye image, wherein the preset state transition matrix is obtained by calculating the human eye test data collected in advance through a statistical method, and the human eye test data is the sub-region to which the pupil center is belonged after the collected human eye image is marked by the pupil center manually.
Optionally, the device is further configured to: if the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image does not reach the preset probability threshold, determining that the pupil center coordinate is detected incorrectly, and taking the subarea of which the pupil center distribution probability of each subarea in the preprocessed current frame human eye image reaches the preset probability threshold as a pupil center detection area; and (3) performing pupil center detection on the pupil center detection area to obtain correct pupil center coordinates.
Optionally, the computing unit is further configured to: calculating the intersection point of the straight line of the short axis of the pupil projection ellipse of the previous frame of human eye image and the straight line of the short axis of the pupil projection ellipse of the current frame of human eye image, and determining the coordinate of the intersection point; the coordinates of the intersection point are taken as pupil rotation center coordinates.
Optionally, the second determining unit is further configured to: according to the second equationCalculating a gaze vector, wherein (u) t+1 ，v t+1 ) Representing gaze vector, (x) t+1 ，y t+1 ) Representing pupil center coordinates, (x) s ，y s ) Representing pupil rotation center coordinates.
Optionally, the device is further configured to: determining a corresponding gaze direction based on the gaze vector; and determining the coordinates of the gaze point in the scene image coordinate system according to the mapping relation among the gaze vector, the eye camera coordinate system and the scene camera coordinate system.
The device provided by the embodiment of the present application has the same implementation principle and technical effects as those of the foregoing method embodiment, and for the sake of brevity, reference may be made to the corresponding content in the foregoing method embodiment where the device embodiment is not mentioned.
As shown in fig. 7, an electronic device 600 provided in an embodiment of the present application includes: the gaze vector detection system comprises a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine readable instructions executable by the processor 601, the processor 601 and the memory 602 communicate through the bus when the electronic device is running, and the processor 601 executes the machine readable instructions to perform the steps of the gaze vector detection method as described above.
Specifically, the memory 602 and the processor 601 can be general-purpose memories and processors, which are not particularly limited herein, and the gaze vector detection method can be performed when the processor 601 runs a computer program stored in the memory 602.
The processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in the processor 601 or instructions in the form of software. The processor 601 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but may also be a digital signal processor (Digital Signal Processing, DSP for short), application specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA for short), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory 602, and the processor 601 reads information in the memory 602 and performs the steps of the above method in combination with its hardware.
Corresponding to the above gaze vector detection method, the embodiments of the present application further provide a computer readable storage medium storing machine executable instructions, which when invoked and executed by a processor, cause the processor to execute the steps of the gaze vector detection method described above.
The gaze vector detection apparatus provided by the embodiment of the present application may be specific hardware on a device or software or firmware installed on a device. The device provided by the embodiment of the present application has the same implementation principle and technical effects as those of the foregoing method embodiment, and for the sake of brevity, reference may be made to the corresponding content in the foregoing method embodiment where the device embodiment is not mentioned. It will be clear to those skilled in the art that, for convenience and brevity, the specific operation of the system, apparatus and unit described above may refer to the corresponding process in the above method embodiment, which is not described in detail herein.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
As another example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments provided in the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing an electronic device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the vehicle marking method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
It should be noted that: like reference numerals and letters in the following figures denote like items, and thus once an item is defined in one figure, no further definition or explanation of it is required in the following figures, and furthermore, the terms "first," "second," "third," etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above examples are only specific embodiments of the present application, and are not intended to limit the scope of the present application, but it should be understood by those skilled in the art that the present application is not limited thereto, and that the present application is described in detail with reference to the foregoing examples: any person skilled in the art may modify or easily conceive of the technical solution described in the foregoing embodiments, or perform equivalent substitution of some of the technical features, while remaining within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit of the corresponding technical solutions. Are intended to be encompassed within the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (10)
1. A method of gaze vector detection, comprising:
acquiring a preprocessed current frame human eye image, and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions;
Performing pupil center detection on the preprocessed current frame human eye image to obtain pupil center coordinates in the preprocessed current frame human eye image, and determining a target sub-region to which the pupil center coordinates belong;
judging whether the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image reaches a preset probability threshold;
if so, determining that the pupil center coordinate is correctly detected;
calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane;
and determining the fixation vector according to the pupil center coordinates and the pupil rotation center coordinates.
2. The method of claim 1, wherein obtaining a preprocessed current frame human eye image and determining pupil center distribution probabilities of respective sub-regions in the preprocessed current frame human eye image comprises:
acquiring a current frame human eye image, and preprocessing the current frame human eye image to obtain a preprocessed current frame human eye image;
Determining the original pupil center distribution probability of each sub-region in the preprocessed current frame of human eye image according to the pupil center distribution probability of each sub-region in the previous frame of human eye image and a preset state transition matrix;
when the previous frame of human eye image is the first frame of human eye image, the pupil center distribution probability of each sub-region in the previous frame of human eye image is determined according to the sub-region to which the pupil center obtained after the previous frame of human eye image is subjected to pupil center marking manually, and the element of the ith row and the jth column in the preset state transition matrix represents the probability that the pupil center is in the ith sub-region in the previous frame of human eye image and the pupil center is in the jth sub-region in the current frame of human eye image;
and normalizing the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image to obtain the pupil center distribution probability of each sub-region in the preprocessed current frame human eye image.
3. The method according to claim 2, wherein determining the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image according to the pupil center distribution probability of each sub-region in the previous frame human eye image and a preset state transition matrix comprises:
According to the first expressionCalculating the original pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein +_>Representing each sub-in the preprocessed current frame human eye imageOriginal pupil center distribution probability of region, +.>Representing the preset state transition matrix,/a>Representing pupil center distribution probability of each sub-region in the last frame of human eye image, wherein the preset state transition matrix is obtained by calculating human eye test data collected in advance through a statistical method, and the human eye test data is a sub-region to which the pupil center obtained by manually marking the pupil center of the collected human eye image belongs.
4. The method according to claim 1, wherein the method further comprises:
if the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image does not reach the preset probability threshold, determining that the pupil center coordinate is detected incorrectly, and taking the subarea, of which the pupil center distribution probability of each subarea in the preprocessed current frame human eye image reaches the preset probability threshold, as a pupil center detection area;
And detecting the pupil center of the pupil center detection area to obtain correct pupil center coordinates.
5. The method according to claim 1, wherein calculating the pupil rotation center coordinates from a line in which the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a line in which the short axis of the pupil projection ellipse of the current frame of human eye image is located, comprises:
calculating the intersection point of the straight line of the short axis of the pupil projection ellipse of the previous frame of human eye image and the straight line of the short axis of the pupil projection ellipse of the current frame of human eye image, and determining the coordinate of the intersection point;
and taking the coordinate of the intersection point as the pupil rotation center coordinate.
6. The method of claim 1, wherein determining the gaze vector from the pupil center coordinates and the pupil center of rotation coordinates comprises:
according to the second equationCalculating the gaze vector, wherein (u) t+1 ，v t+1 ) Representing the gaze vector, (x) t+1 ，y t+1 ) Representing the pupil center coordinates, (x s ，y s ) Representing the pupil rotation center coordinates.
7. The method according to claim 1, wherein the method further comprises:
determining a corresponding gaze direction based on the gaze vector;
And determining the coordinates of the gaze point in the scene image coordinate system according to the mapping relation among the gaze vector, the eye camera coordinate system and the scene camera coordinate system.
8. A gaze vector detection apparatus, comprising:
the acquisition and determination unit is used for acquiring a preprocessed current frame human eye image and determining pupil center distribution probability of each sub-region in the preprocessed current frame human eye image, wherein each sub-region in the preprocessed current frame human eye image is obtained by uniformly dividing the preprocessed current frame human eye image into a plurality of regions;
the pupil center detection unit is used for detecting the pupil center of the preprocessed current frame human eye image, obtaining pupil center coordinates in the preprocessed current frame human eye image, and determining a target sub-region to which the pupil center coordinates belong;
the judging unit is used for judging whether the pupil center distribution probability of the target subarea in the preprocessed current frame human eye image reaches a preset probability threshold value;
the first determining unit is used for determining that the pupil center coordinate is correctly detected if the pupil center coordinate is reached;
The calculation unit is used for calculating pupil rotation center coordinates according to a straight line where the short axis of the pupil projection ellipse of the previous frame of human eye image is located and a straight line where the short axis of the pupil projection ellipse of the current frame of human eye image is located, wherein the pupil rotation center coordinates are projection coordinates of the pupil rotation center in a camera imaging plane;
and the second determining unit is used for determining the gazing vector according to the pupil center coordinates and the pupil rotation center coordinates.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any of the preceding claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium storing machine executable instructions which, when invoked and executed by a processor, cause the processor to perform the method of any one of the preceding claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311055986.8A CN117058747A (en) | 2023-08-21 | 2023-08-21 | Gaze vector detection method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311055986.8A CN117058747A (en) | 2023-08-21 | 2023-08-21 | Gaze vector detection method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117058747A true CN117058747A (en) | 2023-11-14 |
Family
ID=88662329
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311055986.8A Pending CN117058747A (en) | 2023-08-21 | 2023-08-21 | Gaze vector detection method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117058747A (en) |
-
2023
- 2023-08-21 CN CN202311055986.8A patent/CN117058747A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9202280B2 (en) | Position estimation based rotation of switched off light source | |
| US10692210B2 (en) | Recording medium storing computer program for pupil detection, information processing apparatus, and pupil detecting method | |
| Vicente et al. | Driver gaze tracking and eyes off the road detection system | |
| Guo et al. | Contact-free hand geometry-based identification system | |
| US9936195B2 (en) | Calibration for eye tracking systems | |
| US10614289B2 (en) | Facial tracking with classifiers | |
| EP3225154A2 (en) | Gaze detection apparatus and gaze detection method | |
| EP2665406B1 (en) | Automated determination of arteriovenous ratio in images of blood vessels | |
| Torricelli et al. | A neural-based remote eye gaze tracker under natural head motion | |
| US20160162673A1 (en) | Technologies for learning body part geometry for use in biometric authentication | |
| CN110807427B (en) | Sight tracking method and device, computer equipment and storage medium | |
| US20130243251A1 (en) | Image processing device and image processing method | |
| EP4095744A1 (en) | Automatic iris capturing method and apparatus, computer-readable storage medium, and computer device | |
| JPWO2013122009A1 (en) | Reliability acquisition device, reliability acquisition method, and reliability acquisition program | |
| Haker et al. | Geometric invariants for facial feature tracking with 3D TOF cameras | |
| Bergamasco et al. | Image-space marker detection and recognition using projective invariants | |
| Prasetya et al. | Triangle similarity approach for detecting eyeball movement | |
| CN111274852B (en) | Target object key point detection method and device | |
| Yin et al. | An automatic visual monitoring system for expansion displacement of switch rail | |
| CN117058747A (en) | Gaze vector detection method and device and electronic equipment | |
| George et al. | Escaf: Pupil centre localization algorithm with candidate filtering | |
| Melesse et al. | Appearance-based gaze tracking through supervised machine learning | |
| Cao et al. | Gaze tracking on any surface with your phone | |
| Ponz et al. | Topography-based detection of the iris centre using multiple-resolution images | |
| CN114297720A (en) | Image desensitization method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |