CN114652267A - Eyeball tracking method, system, readable storage medium and electronic equipment - Google Patents
Eyeball tracking method, system, readable storage medium and electronic equipment Download PDFInfo
- Publication number
- CN114652267A CN114652267A CN202210228088.7A CN202210228088A CN114652267A CN 114652267 A CN114652267 A CN 114652267A CN 202210228088 A CN202210228088 A CN 202210228088A CN 114652267 A CN114652267 A CN 114652267A
- Authority
- CN
- China
- Prior art keywords
- light
- corneal
- determining
- liquid crystal
- straight line
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
Abstract
An eye tracking method, system, readable storage medium and electronic device, the method comprising: providing corneal reflections to the user's eye with the light emitters in each device group, respectively; when the illuminator in the current device group is used for providing corneal reflection for the eyes of a user, the refractive index of the zooming liquid crystal lens is adjusted, so that light rays emitted by the illuminator in the current device are refracted by the zooming liquid crystal lens and then deflected; when the light is deflected to meet the preset condition, acquiring incident light information of incident light which is refracted by the zooming liquid crystal lens and then is incident to human eyes from the zooming liquid crystal lens, and acquiring reflected light information of the incident light after the incident light is reflected by a cornea; determining the coordinates of the corneal curvature center according to the incident light information and the reflected light information corresponding to each device group; and determining the fixation point of the human eyes on the screen according to the coordinates of the corneal curvature center. The invention realizes eyeball tracking based on the light emitter and the light sensor, and has low cost, higher accuracy and robustness.
Description
Technical Field
The present invention relates to the field of computer vision, and in particular, to an eye tracking method, system, readable storage medium, and electronic device.
Background
The virtual reality system can utilize eyeball tracking to realize functions of point of regard interaction, focusing, rendering and the like. With the thinning of virtual reality devices, the layout and power consumption limitation of eye tracking using cameras are gradually emerging.
In the prior art, a sight line direction tracking method mainly acquires a human eye image and analyzes features in the human eye image to estimate a sight line direction and a viewpoint. The analysis method of human eye images mainly depends on image analysis technology, and involves complex equipment and algorithms and an accurate calibration process. The hardware manufacturing cost is high, and the eyeball tracking precision is generally low.
Disclosure of Invention
In view of the above, it is desirable to provide an eye tracking method, system, readable storage medium and electronic device for solving the problems of high cost and low accuracy of eye tracking in the prior art.
An eyeball tracking method is applied to an eyeball tracking system, the eyeball tracking system comprises a zoom liquid crystal lens and at least two device groups, each device group is arranged at different positions of the zoom liquid crystal lens, each device group comprises a light emitter and a light receiver, and the light emitter and the light receiver are both arranged at one side of the zoom liquid crystal lens, which is far away from eyes of people, the eyeball tracking method comprises the following steps:
providing corneal reflections to a user's eye with the light emitters in each device group, respectively;
when corneal reflection is provided for eyes of a user by the light emitter in the current device group, the refractive index of the zooming liquid crystal lens is adjusted, so that light rays emitted by the light emitter in the current device are deflected after being refracted by the zooming liquid crystal lens;
when the light is deflected to meet the preset condition, the position and the direction of the starting point of the incident light, which is refracted by the zoom liquid crystal lens and then enters the human eyes by the zoom liquid crystal lens, are obtained to obtain the incident light information corresponding to the light-emitting device,
acquiring the end point position and the direction of the reflected light which reaches the zoom liquid crystal lens after the incident light is reflected by the cornea so as to obtain reflected light information corresponding to the current device, wherein the preset condition is that the intensity of light reflected by the cornea received by the photoreceptor in the current device is the maximum after the light incident to the cornea is reflected by the cornea;
acquiring incident light information and reflected light information corresponding to each device group, and determining the coordinates of the corneal curvature center according to the incident light information and the reflected light information corresponding to each device group;
and determining the fixation point of the human eyes on the screen according to the coordinates of the corneal curvature center.
Further, in the eyeball tracking method, the positions of the light emitter and the light receiver in the same device group are the same, and the step of determining the coordinates of the corneal curvature center based on the incident light information and the reflected light information corresponding to each device group includes:
determining a first straight line according to incident light information or reflected light information corresponding to one of the device groups;
determining a second straight line according to incident light information or reflected light information corresponding to the other device group;
and determining the coordinates of the cornea curvature center according to the intersection point of the first straight line and the second straight line.
Further, in the eyeball tracking method, positions of the light emitter and the light receiver in the same device group are different, and the step of determining the coordinates of the corneal curvature center based on incident light information and reflected light information corresponding to each device group includes:
determining a normal of a first corneal reflection point on the cornea according to incident light information and reflected light information corresponding to one of the device groups;
determining a normal of a second corneal reflection point on the cornea according to incident light information and reflected light information corresponding to another device group;
and determining the coordinates of the corneal curvature center according to the normal line passing through the first corneal reflection point and the normal line passing through the second corneal reflection point.
Further, the method for tracking an eyeball, wherein the step of determining a normal line of a first corneal reflection point on the cornea according to the incident light information and the reflected light information corresponding to one of the device groups, comprises:
respectively determining a third straight line and a fourth straight line according to incident light information and reflected light information corresponding to one of the device groups;
solving a straight line intersection point according to the third straight line and the fourth straight line to obtain a coordinate of a first corneal reflection point on the cornea;
calculating a bisector of the third straight line and the fourth straight line according to the third straight line, the fourth straight line and the coordinates of the first corneal reflection point to obtain a normal of the first corneal reflection point;
the step of determining the normal of a second corneal reflection point on the cornea according to the incident light information and the reflected light information corresponding to another device group comprises:
determining a fifth straight line and a sixth straight line according to incident light information and reflected light information corresponding to another device group;
solving a straight line intersection point according to the fifth straight line and the sixth straight line to obtain a coordinate of a second corneal reflection point on the cornea;
and calculating a bisector of the fifth straight line and the sixth straight line according to the fifth straight line, the sixth straight line and the coordinates of the second corneal reflection point to obtain a normal of the second corneal reflection point.
Further, the above eye tracking method, wherein the step of determining the gaze point of the human eye on the screen according to the coordinates of the corneal curvature center comprises:
determining the current eye optical axis of the user according to the coordinates of the corneal curvature center and the spherical center coordinates of the corneal rotation spherical surface;
determining a fixation point of the human eyes on the screen according to an intersection point of the current eye optical axis and a virtual image plane of the screen;
and correcting the position of the fixation point according to the deviation angle between the optical axis of the eyes and the visual axis of the eyes to obtain the accurate position of the fixation point.
Further, the eyeball tracking method may further include, before the step of correcting the position of the fixation point according to a deviation angle between an optical axis of the eye and a visual axis of the eye:
acquiring coordinates of historical corneal curvature centers of at least four positions in the rotation process of the user's eyes in a fixation point calibration stage;
determining the spherical center coordinates of a corneal rotation spherical surface according to the coordinates of each historical corneal curvature center, wherein the corneal rotation spherical surface is a spherical surface formed by the motion trail of the corneal curvature center;
and calculating the deviation angle of the optical axis of the eye and the visual axis of the eye according to the spherical center coordinate, the coordinate of the historical corneal curvature center and the coordinate of a preset fixation point.
The invention also discloses an eyeball tracking system, which comprises a zoom liquid crystal lens and at least two device groups, wherein each device group is arranged at different positions of the zoom liquid crystal lens, each device group comprises a light emitter and a light receiver, the light emitter and the light receiver are both arranged at one side of the zoom liquid crystal lens far away from eyes, the eyeball tracking system also comprises an eyeball tracking control device, and the eyeball tracking control device comprises:
a corneal reflection module for providing corneal reflection to the user's eye using the light emitters in each device group, respectively;
the adjusting module is used for adjusting the refractive index of the zooming liquid crystal lens when the light emitter in the current device group is used for providing corneal reflection for the eyes of a user, so that light rays emitted by the light emitter in the current device group are deflected after being refracted by the zooming liquid crystal lens;
a first obtaining module, configured to obtain, when the light beam is deflected to meet a preset condition, a starting position and a direction of an incident light beam that is refracted by the zoom liquid crystal lens and then enters the human eye through the zoom liquid crystal lens, so as to obtain incident light information corresponding to the light emitting device,
acquiring the end position and the direction of the reflected light which reaches the zoom liquid crystal lens after the incident light is reflected by the cornea so as to obtain reflected light information corresponding to the current device, wherein the preset condition is that the intensity of light reflected by the cornea received by the photoreceptor in the current device is the maximum after the light incident to the cornea is reflected by the cornea;
the corneal curvature center determining module is used for acquiring incident light information and reflected light information corresponding to each device group and determining the coordinates of a corneal curvature center according to the incident light information and the reflected light information corresponding to each device group;
and the fixation point determining module is used for determining the fixation point of the human eyes on the screen according to the coordinates of the corneal curvature center.
Further, in the eyeball tracking system, the gaze point determination module is specifically configured to:
determining the current eye optical axis of the user according to the coordinates of the corneal curvature center and the spherical center coordinates of the corneal rotating spherical surface;
determining a fixation point of the human eyes on the screen according to an intersection point of the current eye optical axis and a virtual image plane of the screen;
and correcting the position of the fixation point according to the deviation angle between the optical axis of the eyes and the visual axis of the eyes to obtain the accurate position of the fixation point.
Further, the eyeball tracking system further includes:
the second acquisition module is used for acquiring the coordinates of the historical corneal curvature centers of at least four positions in the rotation process of the user eyes in the fixation point calibration stage;
the spherical center coordinate determination module is used for determining the spherical center coordinates of the corneal rotating spherical surface according to the coordinates of each historical corneal curvature center, and the corneal rotating spherical surface is a spherical surface formed by the motion trail of the corneal curvature center;
and the deviation angle calculation module is used for calculating the deviation angle between the optical axis of the eye and the visual axis of the eye according to the spherical center coordinate, the coordinate of the historical corneal curvature center and the coordinate of the preset fixation point.
The invention also discloses a readable storage medium on which a computer program is stored, which program, when executed by a processor, performs the method of any of the above.
The invention also discloses an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of the above when executing the computer program.
The invention controls the refractive index of the zoom liquid crystal lens to enable the light emitted by the light emitter at one side of the zoom liquid crystal lens to scan the eyes at the other side of the lens unit, and the light reflected by the eyes is received by the photoreceptor. The positions of the light emitter and the light receiver, the refractive index of the zoom liquid crystal lens, the output signal of the light receiver and other information are comprehensively utilized to track eyeballs, and the requirements of fixation point interaction, focusing and rendering are met. The invention realizes eyeball tracking based on the sparsely arranged light emitter and light sensor, and has the advantages of low cost, low power consumption, and higher accuracy and robustness.
Drawings
FIG. 1 is a flowchart illustrating an eye tracking method according to a first embodiment of the present invention;
FIG. 2 is a flowchart of step S15 in FIG. 1;
FIG. 3 is a flowchart of the steps for determining the coordinates of the center of curvature of the cornea in a second embodiment of the present invention;
FIG. 4 is a flowchart of the steps for determining the coordinates of the center of curvature of the cornea in a third embodiment of the present invention;
FIG. 5 is a block diagram illustrating an eye tracking system according to a fourth embodiment of the present invention;
FIG. 6 is a block diagram illustrating an eye tracking control apparatus according to a fourth embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device in an embodiment of the invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
These and other aspects of embodiments of the invention will be apparent with reference to the following description and attached drawings. In the description and drawings, particular embodiments of the invention have been disclosed in detail as being indicative of some of the ways in which the principles of the embodiments of the invention may be practiced, but it is understood that the scope of the embodiments of the invention is not limited correspondingly. On the contrary, the embodiments of the invention include all changes, modifications and equivalents coming within the spirit and terms of the claims appended hereto.
The eyeball tracking method in the embodiment of the invention is mainly applied to an eyeball tracking system, the eyeball tracking system comprises a zoom liquid crystal lens and at least two device groups arranged at different positions of the zoom liquid crystal lens, each device group comprises a light emitter and a light receiver, and the light emitter and the light receiver are both arranged at one side of the zoom liquid crystal lens, which is far away from eyes.
The variable focal length liquid crystal lens includes a plurality of liquid crystal cells, and the plurality of liquid crystal cells can control refractive indexes of the liquid crystal cells of the variable focal length liquid crystal lens by using array electrodes, so that the deflection direction of incident light is controlled. A plurality of liquid crystal cells are selected on the zoom liquid crystal lens, and a device group is placed behind the liquid crystal cells, so that the deflection range of infrared light emitted from the light emitter after refraction of the lens can cover a part of the moving area of the cornea. In specific implementation, the liquid crystal unit with small influence on the screen image and the corresponding light emitter and light sensor can be selected to perform eyeball tracking based on refractive index control.
The light emitter and the photoreceptor are used in pairs, and one light emitter and one photoreceptor form one device group. The device groups on the zoom liquid crystal lens are at least two, namely, only two of the device groups are needed to be utilized when eye tracking is carried out, and the technical scheme of the invention can be realized.
Referring to fig. 1, the eyeball tracking method in the first embodiment of the invention includes steps S11-S15.
Step S11, providing corneal reflection to the user' S eye with the light emitters in each device group, respectively.
The light emitter may for example be selected as an infrared light emitter and the corresponding light receptor as an infrared light receptor. The light emitter and the light sensor are arranged behind the variable focus liquid crystal lens, i.e. on the side facing away from the human eye, into which light emitted by the light emitter can be coupled. Corneal reflections are provided separately to the user's eye using the light emitters in each device group. In particular embodiments, only two sets of devices may be used, the light emitters of which emit light into the eye of the user to produce a reflection on the cornea.
Step S12, when corneal reflection is provided to the user' S eye by the light emitter in the current device group, adjusting the refractive index of the zoom liquid crystal lens, so that the light emitted by the light emitter in the current device group is refracted by the zoom liquid crystal lens and then deflected.
Step S13, when the light beam is deflected to meet the preset condition, obtaining the starting position and direction of the incident light beam incident to the human eye from the zoom liquid crystal lens after the light beam is refracted by the zoom liquid crystal lens, so as to obtain the incident light information corresponding to the light emitting device,
and acquiring the end position and the direction of the reflected light which reaches the zoom liquid crystal lens after the incident light is reflected by the cornea so as to obtain the reflected light information corresponding to the current device, wherein the preset condition is that the intensity of the light reflected by the cornea received by the photoreceptor in the current device is the maximum after the light incident to the cornea is reflected by the cornea.
When corneal reflection is provided for human eyes of a user by using the light emitter in the current device group, light emitted by the light emitter is deflected after being refracted by the zoom liquid crystal lens by adjusting the refractive index of the zoom liquid crystal lens. It should be noted that, when adjusting the refractive index of the variable focus liquid crystal lens, the refractive index of a part of the lens units may be controlled, i.e. only the refractive index of the region where the light emitter is located may be adjusted. Moreover, each device group can provide corneal reflection for human eyes of users at the same time, namely, adjust the deflection direction at the same time, and also can provide corneal reflection in sequence, namely, adjust in sequence.
Specifically, the refractive index of the zoom liquid crystal lens can be gradually adjusted according to a certain rule, so that the light refracted by the zoom liquid crystal lens is gradually deflected from the left to the right of the eye, or gradually deflected from the right to the left. During the process of adjusting the light deflection, the cornea of human eyes reflects the light in real time, so that the receivers in the current device group receive the reflected light with different intensities.
And when the light is deflected to meet the preset condition, stopping the adjustment of the refractive index of the zooming liquid crystal lens. The preset condition is that the light emitted by the light emitter is refracted by the cornea and then emitted onto the cornea, and the light intensity reflected to the photoreceptor by the cornea is the maximum. At the moment, the position and the direction of the starting point of the incident light, which is incident to the human eyes from the zoom liquid crystal lens after the light emitted by the current illuminator passes through the zoom liquid crystal lens, are recorded, and then the incident light information corresponding to the current device is obtained. And recording the end position and the direction of the reflected light which is reflected to the corresponding photoreceptor by the incident light through the cornea, namely obtaining the reflected light information corresponding to the current device.
It is understood that the starting point of the incident light may be the position of the liquid crystal cell where the light emitter is located. The direction of the incident light is the direction of the light emitted by the light emitter after being refracted by the zoom liquid crystal lens. The direction of the light emitted by the light emitter is known and, depending on the refractive index of the variable focus liquid crystal lens, the direction of the refracted light can be calculated. The end position of the reflected light is the position of the liquid crystal unit where the photoreceptor is located, and the direction of the reflected light is the direction of the reflected light received by the photoreceptor.
Step S14, acquiring incident light information and reflected light information corresponding to each device group, and determining coordinates of a corneal curvature center according to the incident light information and the reflected light information corresponding to each device group.
Incident light information and reflected light information are obtained for each device group in the above-described manner. And determining the coordinates of the corneal curvature center according to the incident light information and the reflected light information corresponding to each device group.
In specific implementation, a straight line corresponding to an incident light ray can be determined for incident light information corresponding to each device group, and a straight line corresponding to a reflected light can be determined for reflected light information corresponding to each device group. And determining the coordinates of the corneal curvature center according to the straight line corresponding to the incident light and the straight line corresponding to the reflected light.
And step S15, determining the fixation point of the human eye on the screen according to the coordinates of the corneal curvature center.
After the coordinates of the corneal curvature center are determined, the fixation point of the human eyes on the screen can be determined according to the coordinates of the corneal curvature center.
Specifically, as shown in fig. 2, in one embodiment of the present invention, step S15 specifically includes:
step S151, determining the current eye optical axis of the user according to the coordinates of the cornea curvature center and the spherical center coordinates of the cornea rotating spherical surface;
step S152, determining a fixation point of the human eyes on the screen according to the intersection point of the current eye optical axis and the virtual image plane of the screen;
and step S153, correcting the position of the fixation point according to the deviation angle between the eye optical axis and the eye visual axis to obtain the accurate position of the fixation point.
It is understood that, in the above steps, the determination of the gaze point of the human eye on the screen is performed in the gaze point detection stage. In the stage of detecting the fixation point, the starting point and the direction of the optical axis of the eye are calculated according to the position of the rotating spherical center of the cornea and the position of the curvature center of the cornea. The optical axis of the eye is defined as the rotation of the spherical center p from the corneasphereTo the center of curvature p of the corneacorneaRay d ofoptical：
doptical＝(pcornea-psphere)/||pcornea-psphere||2；
The corresponding linear equation is: p ═ psphere+tdoptical。
According to the z-z between the optical axis of the eye and the virtual image plane of the screenscreenThe intersection point of (a) determines the position p of the point of regardgaze；
And then, performing fixation point correction according to the deviation angle kappa between the eye optical axis and the eye visual axis:
wherein alpha is a deflection angle of the intersection point coordinate of the eyeball optical axis and the screen virtual image plane in the x direction relative to the eyeball optical axis when the virtual image plane is vertical to the eyeball optical axis,
is the x-axis component of the gaze point coordinates,
is the z-axis component of the gaze point coordinates.
Further, the spherical center coordinates and the supplementary angle of the cornea rotating spherical surface can be determined in the stage of the fixation point calibration. The specific implementation mode is as follows:
acquiring coordinates of historical corneal curvature centers of at least four positions in the rotation process of the user's eyes in a fixation point calibration stage;
determining the spherical center coordinates of a corneal rotation spherical surface according to the coordinates of each historical corneal curvature center, wherein the corneal rotation spherical surface is a spherical surface formed by the motion trail of the corneal curvature center;
and calculating the deviation angle of the optical axis of the eye and the visual axis of the eye according to the spherical center coordinate, the historical corneal curvature center coordinate and the preset fixation point coordinate.
In the stage of calibrating the fixation point, the method for determining the coordinates of the historical corneal curvature center may refer to the method for determining the coordinates of the corneal curvature center in the first embodiment, which is not described herein again.
Specifically, the equation of the spherical surface S determined by the motion track of the corneal curvature center is (x-x)0)2+(y-y0)2+(z-z0)2＝R2Wherein (x)0,y0,z0) Is the sphere center coordinate and R is the sphere radius. According to at least four corneal curvature center coordinates
Establishing an equation set:
according to the equation set, the spherical center coordinate and the spherical radius of the S can be solved, and the spherical center coordinate of the S is the spherical center coordinate of the cornea rotating spherical surface.
In the stage of calibrating the fixation point, the deviation angle between the optical axis of the eye and the visual axis of the eye can be calculated according to the spherical center coordinates of the corneal rotating spherical surface, the coordinates of the historical corneal curvature center and the coordinates of the preset fixation point.
The eye optic axis is defined as:
doptical＝(pcornea-psphere)/||pcornea-psphere||2。
the visual axis of the eye is defined as the center of curvature p from the corneacorneaTo a preset point of fixation pgazeThe ray (c) of (c):
dvisual＝(pgaze-pcornea)/‖pgaze-pcornea‖2。
the deviation angle k between the eye optical axis and the eye visual axis is:
κ＝acos(<doptical·dvisual>)。
it is understood that after the coordinates of the corneal curvature center are calculated, the fixation point of the human eye on the screen can be determined according to other methods. For example, in another embodiment of the present invention, during the fixation point calibration phase, the position p of the corneal center of curvature may be establishedcorneaAnd point of fixation pgazeThe mapping relationship of (1): p is a radical of formulagaze＝f(pcornea)。
Wherein the mapping relationship f can be composed of multiple groups (p)cornea,pgaze) And fitting or learning the data to obtain the data. In the fixation point detection stage, the position of the fixation point can be calculated from the position of the corneal curvature center based on the mapping relationship f.
Specifically, in the present invention, in step S14, the coordinates of the corneal curvature center can be determined based on the incident light information and the reflected light information corresponding to each device group in two ways. As shown in fig. 3, the method steps for determining the coordinates of the corneal curvature center according to the incident light information and the reflected light information corresponding to each device group in the second embodiment of the present invention are that the light emitter and the light receiver are located at the same position in the same device group. The method comprises the following steps:
step S21, determining a first straight line according to the incident light information or the reflected light information corresponding to one of the device groups.
Step S22, determining a second straight line according to the incident light information or the reflected light information corresponding to another device group.
Step S23, determining coordinates of the corneal center of curvature from the intersection of the first straight line and the second straight line.
This example utilizes two sets of devices, each employing an infrared emitter and an infrared sensor, to determine the coordinates of the corneal center of curvature. Because in each group of devices, the infrared light emitter and the infrared photoreceptor are located at the same position. When the light emitted and deflected from the infrared emitter is directed toward the center of curvature of the cornea, the intensity of the infrared light reflected by the corneal surface is maximized, maximizing the response of the infrared photoreceptor at the same location as the infrared emitter. In this case, the incident light incident from the zoom liquid crystal lens to the cornea and the reflected light reflected from the cornea to the infrared photoreceptor overlap.
The position of one of the infrared emitters in the liquid crystal cell and the direction of the incident light define a line pointing from the liquid crystal cell to the center of curvature of the cornea. Another line pointing from the liquid crystal cell to the center of curvature of the cornea can be defined by the infrared emitter of the other liquid crystal cell. The intersection of two such lines determines the location of the center of curvature of the cornea. Since the incident light and the reflected light overlap, the reflected light received with the infrared photoreceptors of both devices can also locate the position of the center of curvature of the cornea.
Specifically, the two sets of devices include a first light emitter and a first photoreceptor arranged in pair, and a second light emitter and a second photoreceptor arranged in pair, the first light emitter and the first photoreceptor being arranged behind the first lens unit, and the second light emitter and the second photoreceptor being arranged behind the second lens unit. A simplified process for operating an eye tracking system is as follows:
generating first emission light using a first light emitter;
coupling a portion of the first emitted light into a first lens unit;
adjusting the refractive index of the first lens unit to project the first emission light onto the cornea;
detecting first reflected light reflected from the cornea using a first photoreceptor;
adjusting the refractive index of the first lens unit to maximize the intensity of the signal received by the first photoreceptor;
determining a first line through the center of curvature of the cornea;
generating a second emission using a second light emitter;
coupling a portion of the second emitted light into a second lens unit;
adjusting the refractive index of the second lens unit such that the second emitted light is projected onto the cornea;
detecting second reflected light reflected from the cornea using a second photoreceptor;
adjusting the refractive index of the second lens unit to maximize the intensity of the signal received by the second photoreceptor;
a second line is determined that passes through the center of curvature of the cornea.
As shown in fig. 4, a method step of determining coordinates of a corneal curvature center according to incident light information and reflected light information corresponding to each device group in a third embodiment of the present invention is shown, where positions of a light emitter and a light receiver in the same device group are different in this embodiment. The method comprises the following steps:
step S31, determining a third straight line and a fourth straight line according to incident light information and reflected light information corresponding to one of the device groups;
step S32, solving a straight line intersection point according to the third straight line and the fourth straight line to obtain the coordinate of the first corneal reflection point on the cornea;
step S33, calculating a bisector of the third straight line and the fourth straight line according to the third straight line, the fourth straight line and the coordinates of the first corneal reflection point to obtain a normal of the first corneal reflection point;
step S34, determining a fifth straight line and a sixth straight line according to incident light information and reflected light information corresponding to another device group;
step S35, solving a straight line intersection point according to the fifth straight line and the sixth straight line to obtain the coordinate of a second corneal reflection point on the cornea;
step S36, calculating a bisector of the fifth straight line and the sixth straight line according to the fifth straight line, the sixth straight line and the coordinates of the second corneal reflection point to obtain a normal of the second corneal reflection point;
step S37, determining coordinates of the corneal curvature center according to the normal line passing through the first corneal reflection point and the normal line passing through the second corneal reflection point.
This example utilizes two sets of devices, each employing an infrared emitter and an infrared sensor, to determine the coordinates of the corneal center of curvature. From the origin and direction of the infrared emitted light and the origin and direction of the infrared reflected light, the location of the corneal reflection point can be determined. Another set of origin, direction of the infrared emitted light and origin, direction of the infrared reflected light may determine the location of another corneal reflection point.
The origin and direction of the light define a linear parametric equation p ═ p0+ td, where p0As a starting point, d is the normalized direction and t is a parameter.
An equation set is established according to the emitted light and the reflected light, and the straight line intersection point, namely the cornea reflection point p can be solvedreflection：
Wherein p is11Parametric equation of straight line for incident light, p12For the equation of the parameter of the straight line corresponding to the reflected light, P10 is the starting position of the emitted light, and P20 is the ending position of the reflected light.
From the emitted and incident rays, an angular bisector can be calculated, i.e. the normal direction n of the corneal reflection point:
n＝d1+d2；
where d1 and d2 are the direction of the emitted light and the direction of the reflected light, respectively.
According to the normal p at the two corneal reflection points21And p22The position p of the center of curvature of the cornea can be calculatedcorneaNormal line p21And p22The intersection point of (a) is the coordinate of the corneal curvature center.
Wherein the content of the first and second substances,
and
respectively, the coordinates of the first corneal reflection point and the second corneal reflection point.
Specifically, the two groups of devices comprise a third light emitter and a third photoreceptor which are arranged in pair, and a fourth light emitter and a fourth photoreceptor which are arranged in pair, wherein the positions of the third light emitter and the third photoreceptor are different, the third light emitter is arranged behind a third lens unit, and the third photoreceptor is arranged behind a fifth lens unit; meanwhile, a fourth light emitter disposed behind the fourth lens unit and a fourth photoreceptor disposed behind the sixth lens unit are located at different positions. A simplified process for operating an eye tracking system is as follows:
generating a third emission using a third light emitter;
coupling a portion of the third emitted light into a third lens unit;
adjusting the refractive index of the third lens unit such that the third emitted light is projected onto the cornea;
detecting first reflected light reflected from the cornea using a third photoreceptor;
adjusting the refractive index of the third lens unit to maximize the intensity of the signal received by the third photoreceptor;
determining a position of a first corneal reflection point;
generating a fourth emission using a fourth light emitter;
coupling a portion of the fourth emitted light into a fourth lens unit;
adjusting a refractive index of the fourth lens unit such that the fourth emitted light is projected onto the cornea;
detecting fourth reflected light reflected from the cornea using a fourth photoreceptor;
adjusting the refractive index of the fourth lens unit to maximize the intensity of the signal received by the fourth photoreceptor;
the location of the second corneal reflection point is determined.
According to the embodiment of the invention, the refractive index of the zoom liquid crystal lens is controlled, so that light emitted by the light emitter on one side of the zoom liquid crystal lens can scan eyes on the other side of the lens unit, and the light reflected by the eyes is received by the light receiver. The positions of the light emitter and the light receiver, the refractive index of the zoom liquid crystal lens, the output signal of the light receiver and other information are comprehensively utilized to track eyeballs, and the requirements of fixation point interaction, focusing and rendering are met. The embodiment of the invention realizes eyeball tracking based on the sparsely arranged light emitter and light receiver, and has the advantages of low cost, low power consumption, higher accuracy and robustness.
Referring to fig. 5, an eyeball tracking system in a fourth embodiment of the present invention comprises a variable-focus liquid crystal lens 41 and at least two device groups 42, each of the device groups 42 is disposed at a different position of the variable-focus liquid crystal lens 41, each of the device groups 42 comprises a light emitter 421 and a light receiver 422, the light emitter 421 and the light receiver 422 are both disposed at a side of the variable-focus liquid crystal lens 41 far away from the human eye, the eyeball tracking system further comprises an eyeball tracking control device 43, as shown in fig. 6, the eyeball tracking control device 43 comprises:
a corneal reflection module 431 for providing corneal reflection to the user's eye using the light emitters in the respective device groups, respectively;
an adjusting module 432, configured to adjust a refractive index of the zoom liquid crystal lens when corneal reflection is provided to an eye of a user by using the light emitter in the current device group, so that light emitted by the light emitter in the current device is refracted by the zoom liquid crystal lens and then deflected;
a first obtaining module 433, configured to, when the light beam is deflected to meet a preset condition, obtain a starting position and a direction of an incident light beam that is refracted by the zoom liquid crystal lens and then enters the human eye through the zoom liquid crystal lens, so as to obtain incident light information corresponding to the light emitting device,
acquiring the end point position and the direction of the reflected light which reaches the zoom liquid crystal lens after the incident light is reflected by the cornea so as to obtain reflected light information corresponding to the current device, wherein the preset condition is that the intensity of light reflected by the cornea received by the photoreceptor in the current device is the maximum after the light incident to the cornea is reflected by the cornea;
a corneal curvature center determining module 434, configured to obtain incident light information and reflected light information corresponding to each device group, and determine coordinates of a corneal curvature center according to the incident light information and the reflected light information corresponding to each device group;
and a fixation point determining module 435 for determining the fixation point of the human eye on the screen according to the coordinates of the corneal curvature center.
Further, in the eyeball tracking system, the gaze point determination module 435 is specifically configured to:
determining the current eye optical axis of the user according to the coordinates of the corneal curvature center and the spherical center coordinates of the corneal rotating spherical surface;
determining a fixation point of the human eyes on the screen according to an intersection point of the current eye optical axis and a virtual image plane of the screen;
and correcting the position of the fixation point according to the deviation angle between the optical axis of the eyes and the visual axis of the eyes to obtain the accurate position of the fixation point.
Further, the eyeball tracking system further includes:
the second acquisition module is used for acquiring the coordinates of historical corneal curvature centers of at least four positions in the rotation process of the user's eyes in the fixation point calibration stage;
the spherical center coordinate determination module is used for determining the spherical center coordinates of the corneal rotating spherical surface according to the coordinates of each historical corneal curvature center, and the corneal rotating spherical surface is a spherical surface formed by the motion trail of the corneal curvature center;
and the deviation angle calculation module is used for calculating the deviation angle between the optical axis of the eye and the visual axis of the eye according to the spherical center coordinate, the coordinate of the historical corneal curvature center and the coordinate of the preset fixation point.
The eyeball tracking system provided by the embodiment of the invention has the same implementation principle and the same technical effect as the method embodiment, and for the sake of brief description, the corresponding contents in the method embodiment can be referred to for the parts which are not mentioned in the device embodiment.
Referring to fig. 7, an electronic device according to a fourth embodiment of the present invention is shown, which includes a processor 10, a memory 20, and a computer program 30 stored in the memory and executable on the processor, wherein the processor 10 executes the computer program 30 to implement the eye tracking method as described above.
The electronic device may be, but is not limited to, a virtual reality headset, a computer, a server, and the like. Processor 10 may be, in some embodiments, a Central Processing Unit (CPU), controller, microcontroller, microprocessor or other data Processing chip that executes program code stored in memory 20 or processes data.
The memory 20 includes at least one type of readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, and the like. The memory 20 may in some embodiments be an internal storage unit of the electronic device, for example a hard disk of the electronic device. The memory 20 may also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the electronic device. Further, the memory 20 may also include both an internal storage unit and an external storage device of the electronic apparatus. The memory 20 may be used not only to store application software installed in the electronic device and various types of data, but also to temporarily store data that has been output or will be output.
Optionally, the electronic device may further comprise a user interface, a network interface, a communication bus, etc., the user interface may comprise a Display (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface may further comprise a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface. The network interface may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), typically used to establish a communication link between the device and other electronic devices. The communication bus is used to enable connection communication between these components.
It should be noted that the configuration shown in fig. 7 does not constitute a limitation of the electronic device, and in other embodiments the electronic device may include fewer or more components than shown, or some components may be combined, or a different arrangement of components.
The present invention also proposes a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the eye tracking method as described above.
Those of skill in the art will appreciate that the logic and/or steps illustrated in the flowcharts or otherwise described herein, such as an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus (e.g., a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions). For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Further, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is specific and detailed, but not to be understood as limiting the scope of the present invention. It should be noted that various changes and modifications can be made by those skilled in the art without departing from the spirit of the invention, and these changes and modifications are all within the scope of the invention. Therefore, the protection scope of the present patent should be subject to the appended claims.
Claims (10)
1. An eye tracking method, applied to an eye tracking system, the eye tracking system including a variable-focus liquid crystal lens, and at least two device groups, each of the device groups being disposed at a different position of the variable-focus liquid crystal lens, each of the device groups including a light emitter and a light receiver, both of the light emitter and the light receiver being disposed on a side of the variable-focus liquid crystal lens away from a human eye, the eye tracking method comprising:
providing corneal reflections to the user's eye with the light emitters in each device group, respectively;
when corneal reflection is provided for eyes of a user by the light emitter in the current device group, the refractive index of the zooming liquid crystal lens is adjusted, so that light rays emitted by the light emitter in the current device are deflected after being refracted by the zooming liquid crystal lens;
when the light is deflected to meet the preset condition, the position and the direction of the starting point of the incident light, which is refracted by the zoom liquid crystal lens and then enters the human eyes by the zoom liquid crystal lens, are obtained to obtain the incident light information corresponding to the light-emitting device,
acquiring the end point position and the direction of the reflected light which reaches the zoom liquid crystal lens after the incident light is reflected by the cornea so as to obtain reflected light information corresponding to the current device, wherein the preset condition is that the intensity of light reflected by the cornea received by the photoreceptor in the current device is the maximum after the light incident to the cornea is reflected by the cornea;
acquiring incident light information and reflected light information corresponding to each device group, and determining the coordinates of the corneal curvature center according to the incident light information and the reflected light information corresponding to each device group;
and determining the fixation point of the human eyes on the screen according to the coordinates of the corneal curvature center.
2. The eyeball tracking method as set forth in claim 1, wherein the light emitter and the light receiver in the same device group are located at the same position, and the step of determining the coordinates of the center of curvature of the cornea from the incident light information and the reflected light information corresponding to each of the device groups comprises:
determining a first straight line according to incident light information or reflected light information corresponding to one of the device groups;
determining a second straight line according to incident light information or reflected light information corresponding to the other device group;
and determining the coordinates of the corneal curvature center according to the intersection point of the first straight line and the second straight line.
3. The eye tracking method of claim 1, wherein the light emitter and the light sensor in the same device group are located at different positions, and the step of determining the coordinates of the corneal curvature center based on the incident light information and the reflected light information corresponding to each device group comprises:
determining a normal of a first corneal reflection point on the cornea according to incident light information and reflected light information corresponding to one of the device groups;
determining a normal of a second corneal reflection point on the cornea according to incident light information and reflected light information corresponding to another device group;
and determining the coordinates of the corneal curvature center according to the normal line passing through the first corneal reflection point and the normal line passing through the second corneal reflection point.
4. The eye tracking method of claim 3, wherein said step of determining the normal of the first corneal reflection point on the cornea based on the incident light information and the reflected light information corresponding to one of said device groups comprises:
respectively determining a third straight line and a fourth straight line according to incident light information and reflected light information corresponding to one of the device groups;
solving a straight line intersection point according to the third straight line and the fourth straight line to obtain a coordinate of a first corneal reflection point on the cornea;
calculating a bisector of the third straight line and the fourth straight line according to the third straight line, the fourth straight line and the coordinates of the first corneal reflection point to obtain a normal of the first corneal reflection point;
the step of determining the normal of a second corneal reflection point on the cornea according to the incident light information and the reflected light information corresponding to another device group comprises:
determining a fifth straight line and a sixth straight line according to incident light information and reflected light information corresponding to another device group;
solving a straight line intersection point according to the fifth straight line and the sixth straight line to obtain a coordinate of a second corneal reflection point on the cornea;
and calculating a bisector of the fifth straight line and the sixth straight line according to the fifth straight line, the sixth straight line and the coordinates of the second corneal reflection point to obtain a normal of the second corneal reflection point.
5. The eye tracking method of claim 1, wherein said step of determining the gaze point of said human eye on the screen based on the coordinates of said corneal center of curvature comprises:
determining the current eye optical axis of the user according to the coordinates of the corneal curvature center and the spherical center coordinates of the corneal rotating spherical surface;
determining a fixation point of the human eyes on the screen according to an intersection point of the current eye optical axis and a virtual image plane of the screen;
and correcting the position of the fixation point according to the deviation angle of the optical axis of the eyes and the visual axis of the eyes to obtain the accurate position of the fixation point.
6. The eye tracking method according to claim 5, wherein the step of correcting the position of the gaze point according to the deviation angle of the optical axis of the eye from the visual axis of the eye further comprises:
acquiring coordinates of historical corneal curvature centers of at least four positions in the rotation process of the user's eyes in a fixation point calibration stage;
determining the spherical center coordinates of a corneal rotation spherical surface according to the coordinates of the historical corneal curvature centers, wherein the corneal rotation spherical surface is a spherical surface formed by the motion trail of the corneal curvature center;
and calculating the deviation angle of the optical axis of the eye and the visual axis of the eye according to the spherical center coordinate, the coordinate of the historical corneal curvature center and the coordinate of a preset fixation point.
7. An eye tracking system, comprising a zoom liquid crystal lens, and at least two device groups, each of the device groups being disposed at a different position of the zoom liquid crystal lens, each of the device groups including a light emitter and a light sensor, the light emitter and the light sensor being disposed on a side of the zoom liquid crystal lens away from human eyes, the eye tracking system further comprising an eye tracking control device, the eye tracking control device comprising:
a corneal reflection module for providing corneal reflection to the user's eye using the light emitters in each device group, respectively;
the adjusting module is used for adjusting the refractive index of the zooming liquid crystal lens when the light emitter in the current device group is used for providing corneal reflection for the eyes of a user, so that light rays emitted by the light emitter in the current device group are deflected after being refracted by the zooming liquid crystal lens;
a first obtaining module, configured to obtain a starting position and a direction of an incident light ray that is refracted by the variable focus liquid crystal lens and is incident to a human eye through the variable focus liquid crystal lens when the light ray is deflected to meet a preset condition, so as to obtain incident light information corresponding to the light emitting device,
acquiring the end position and the direction of the reflected light which reaches the zoom liquid crystal lens after the incident light is reflected by the cornea so as to obtain reflected light information corresponding to the current device, wherein the preset condition is that the intensity of light reflected by the cornea received by the photoreceptor in the current device is the maximum after the light incident to the cornea is reflected by the cornea;
the corneal curvature center determining module is used for acquiring incident light information and reflected light information corresponding to each device group and determining the coordinates of a corneal curvature center according to the incident light information and the reflected light information corresponding to each device group;
and the fixation point determining module is used for determining the fixation point of the human eyes on the screen according to the coordinates of the corneal curvature center.
8. The eye tracking system of claim 7, wherein the point of regard determination module is specifically configured to:
determining the current eye optical axis of the user according to the coordinates of the corneal curvature center and the spherical center coordinates of the corneal rotating spherical surface;
determining a fixation point of the human eyes on the screen according to an intersection point of the current eye optical axis and a virtual image plane of the screen;
and correcting the position of the fixation point according to the deviation angle between the optical axis of the eyes and the visual axis of the eyes to obtain the accurate position of the fixation point.
9. A readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 6.
10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 6 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210228088.7A CN114652267A (en) | 2022-03-08 | 2022-03-08 | Eyeball tracking method, system, readable storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210228088.7A CN114652267A (en) | 2022-03-08 | 2022-03-08 | Eyeball tracking method, system, readable storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114652267A true CN114652267A (en) | 2022-06-24 |
Family
ID=82028745
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210228088.7A Pending CN114652267A (en) | 2022-03-08 | 2022-03-08 | Eyeball tracking method, system, readable storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114652267A (en) |
-
2022
- 2022-03-08 CN CN202210228088.7A patent/CN114652267A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11604509B1 (en) | Event camera for eye tracking | |
| US10257507B1 (en) | Time-of-flight depth sensing for eye tracking | |
| US11042034B2 (en) | Head mounted display calibration using portable docking station with calibration target | |
| US10416766B1 (en) | Varifocal head-mounted display including modular air spaced optical assembly | |
| US10025384B1 (en) | Eye tracking architecture for common structured light and time-of-flight framework | |
| CN110967166B (en) | Detection method, detection device and detection system of near-eye display optical system | |
| US9959678B2 (en) | Face and eye tracking using facial sensors within a head-mounted display | |
| US8933912B2 (en) | Touch sensitive user interface with three dimensional input sensor | |
| TWI432172B (en) | Pupil location method, pupil positioning system and storage media | |
| CN108205374B (en) | Eyeball tracking module and method of video glasses and video glasses | |
| JP2010259605A (en) | Visual line measuring device and visual line measuring program | |
| JP6631951B2 (en) | Eye gaze detection device and eye gaze detection method | |
| US10430988B2 (en) | Facial animation using facial sensors within a head-mounted display | |
| JP6596678B2 (en) | Gaze measurement apparatus and gaze measurement method | |
| JP2018099174A (en) | Pupil detector and pupil detection method | |
| KR20170031185A (en) | Wide field-of-view depth imaging | |
| CN114360043B (en) | Model parameter calibration method, sight tracking method, device, medium and equipment | |
| CN114652267A (en) | Eyeball tracking method, system, readable storage medium and electronic equipment | |
| US10359845B1 (en) | Display assembly using dynamic liquid crystal array | |
| US11698676B2 (en) | Method and electronic device for eye-tracking | |
| US20230068608A1 (en) | Gaze tracking for a retinal projection display system | |
| US11924536B2 (en) | Augmented reality device including variable focus lenses and operating method thereof | |
| US20230077073A1 (en) | Augmented reality device and method for obtaining depth map by using depth sensor | |
| KR102511078B1 (en) | Apparatus and method for estimating gaze depth information | |
| KR20220170336A (en) | An augmented reality device comprising variable focus lenses and a method for operating the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |