CN109922710A - The method and system of eyes tracking is executed using speckle pattern - Google Patents
The method and system of eyes tracking is executed using speckle pattern Download PDFInfo
- Publication number
- CN109922710A CN109922710A CN201780069498.9A CN201780069498A CN109922710A CN 109922710 A CN109922710 A CN 109922710A CN 201780069498 A CN201780069498 A CN 201780069498A CN 109922710 A CN109922710 A CN 109922710A
- Authority
- CN
- China
- Prior art keywords
- eyes
- speckle pattern
- light beam
- speckle
- tracking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/47—Scattering, i.e. diffuse reflection
- G01N21/4788—Diffraction
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/262—Analysis of motion using transform domain methods, e.g. Fourier domain methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/32—Determination of transform parameters for the alignment of images, i.e. image registration using correlation-based methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/47—Scattering, i.e. diffuse reflection
- G01N21/4788—Diffraction
- G01N2021/479—Speckle
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30041—Eye; Retina; Ophthalmic
Abstract
A kind of method of movement tracking eyes of user includes leading light beam at eyes.The eyes can reflect the part in the light beam.This method further comprises detecting to multiple speckle patterns, and multiple speckle pattern is formed by the part in the light beam of eye reflections in detector.Multiple speckle patterns can be detected with scheduled frame rate.This method further comprises tracking the movement of eyes by tracking multiple speckle patterns frame by frame.
Description
Cross reference to related applications
This application claims the U.S. Provisional Patent Applications for the Serial No. 62/420,292 submitted on November 10th, 2016
The complete disclosure of senior interest, this application is incorporated by reference into.
Background technique
The tracking technique of watching attentively of eyes can be applied to many fields.For example, the tracking of watching attentively of eyes can be used for constructing void
Quasi- reality (VR) or augmented reality (AR) head-wearing device are to provide a user accurately three-dimensional present.Other application may include being used for
The cognition in man-machine interface, activity recognition, image and the video compress, computer vision, medical research of physical disabilities is helped to grind
Study carefully, laser refractive surgery, the vehicle simulator of vehicle-mounted research, the training simulators including movement training simulation device, fatigue detecting
Deng.Although achieving progress in terms of the development for watching tracking technique attentively of eyes, there is still a need for the notes of improved eyes for this field
Depending on tracking.
Summary of the invention
This invention relates generally to the method and systems for watching tracking attentively for eyes.It is more particularly related to
For using the method and system for watching tracking attentively of the eyes of speckle pattern.
According to an embodiment of the invention, a kind of method for the movement for tracking eyes of user includes leading light beam at eyes.
Part in the eye reflections light beam.This method further comprises detecting to multiple speckle patterns, multiple speckle pattern
It is formed by the part in the light beam of eye reflections in detector.Multiple speckle patterns can be examined with scheduled frame rate
It surveys.This method further comprises tracking the movement of eyes by tracking multiple speckle patterns frame by frame.
According to another embodiment of the present invention, a kind of method watched attentively tracking eyes of user includes: using camera with the
First multiple images of one frame rate record eyes；And the start bit of eyes is determined based on the first multiple images of eyes
It sets.This method further comprises leading light beam at eyes.Part in the eye reflections light beam.This method further comprises pair
A speckle pattern is detected more than first, and more than first a speckle pattern is by the part in the light beam of eye reflections in detector
It is formed.More than first a speckle pattern can be detected with the second frame rate for being greater than the first frame rate.This method is into one
Step includes the movement for tracking eyes relative to initial position by tracking a speckle pattern more than first frame by frame.
According to still another embodiment of the invention, a kind of method watched attentively tracking eyes of user includes: acquisition eyes of user
Pre-calibration speckle pattern；The initial position of eyes is determined using the speckle pattern of the pre-calibration；And tracking eyes relative to
The movement of initial position.In one embodiment, the speckle pattern of the pre-calibration of eyes of user is obtained by following steps: by first
Light beam is led at eyes；Multiple first speckle patterns are detected, multiple first speckle pattern is by the first of eye reflections
Part in light beam is formed at detector, the corresponding first position of each of multiple first speckle pattern and eyes
It is corresponding；Using the multiple images of cameras record eyes, when detecting corresponding first speckle pattern, the multiple of eyes is recorded
Each of image；Eyes corresponding with each corresponding first speckle pattern are determined based on the corresponding image of eyes
Corresponding first position；And the corresponding first position of storage multiple first speckle patterns and eyes.In one embodiment,
The initial position for determining eyes using the speckle pattern of pre-calibration is executed by following steps: the second light beam is led at eyes,
Part in the eye reflections second light beam；Multiple second speckle patterns are detected, multiple second speckle pattern by
Part in second light beam of eye reflections is formed at detector；And it based on multiple second speckle patterns and is stored more
The corresponding first position of comparison between a first speckle pattern and the eyes stored determines the initial positions of eyes.?
In one embodiment, movement of the tracking eyes relative to initial position is executed by following steps: third light beam is led into eyes
Locate, the part in the eye reflections third light beam；Multiple third speckle patterns are detected, multiple third speckle pattern
It is formed by the part in the third light beam of eye reflections in detector, multiple third speckle pattern is with scheduled frame rate
Detected；And the movement for by tracking the multiple speckle patterns of third frame by frame tracking eyes relative to initial position.
In further embodiment of the invention, a kind of method identifying user includes the First view of the first user of acquisition
The speckle pattern of the pre-calibration of eyeball.The speckle pattern of the pre-calibration of the first eye of the first user is obtained by following steps: by first
Light beam is led at first eye, which reflects the part in first light beam；Multiple first speckle patterns are examined
It surveys, the part in the first light beam that multiple first speckle pattern is reflected by first eye formed in detector, and multiple the
Each of one speckle pattern and the corresponding first position of first eye are corresponding；The more of first eye are recorded using camera
A image records each of multiple image of the first eye when detecting corresponding first speckle pattern；It is based on
The corresponding image of first eye determines corresponding the first of first eye corresponding with each corresponding first speckle pattern
Position；And the corresponding first position of storage multiple first speckle patterns and first eye.This method further comprises: by
Two light beams are led at the eyes of user, the part in the eye reflections second light beam；Multiple second speckle patterns are examined
It surveys, multiple second speckle pattern is formed by the part in the second light beam of eye reflections in detector, and multiple second dissipates
Each of spot pattern and the corresponding second position of eyes are corresponding, and by comparing multiple second speckle patterns with it is multiple
First speckle pattern determines that user is the first user.In some embodiments, by the part in the first light beam of eye reflections
It is reflected diffusely or specularly with each of the part in the second light beam by eye reflections by eyes.
Many advantages better than routine techniques are realized by means of the present invention.For example, the embodiment of the present invention can be with
Provide a mean for the method and be that the movement of tracking speckle pattern is detected come the fixation movement accurately to small size eyes
System, such as in the micron-scale or smaller level.May be very steady using the tracking of watching attentively of the eyes of speckle, and tracking quality
Position relative insensitivity for sensor relative to eyes.The eyes of embodiment according to the present invention watch attentively tracking with
Conventional method is compared to that can have lower power consumption, since it is desired that less calculation amount and quantity of light source are (for example, with based on camera
Eyes, which track four commonly required LED, which to be compared, only needs a light source).These methods also can have lower cost, because
They only need a light source and a detector, and do not need camera lens.
By combine following text and attached drawing these and other embodiment that the present invention will be described in more detail and they
Many advantages and feature.
Detailed description of the invention
Fig. 1 is shown when four images (a) of the eyes of eyes captured when watching various positions attentively to image (d).
Fig. 2 schematically shows the movement of the eyes along three independent axes.
Fig. 3 shows the illustrative speckle pattern generated from laser designator.
Fig. 4 schematically shows how speckle pattern is formed at imaging sensor.
Fig. 5 A and Fig. 5 B show two snapshots of the speckle pattern shot relative to each other with small time interval.
Fig. 6 schematically shows the exemplary setting for watching tracking attentively for eyes of embodiment according to the present invention.
Fig. 7 A shows the front view of the eyes by camera capture of embodiment according to the present invention.
Fig. 7 B shows the speckle pattern of embodiment according to the present invention captured by detector.
Fig. 8 shows how being mounted on light source, detector and optional eyes camera for embodiment according to the present invention
It can be by the schematic diagram on a pair of glasses of user's wearing.
When Fig. 9 A to Fig. 9 F is shown when eyes are moved from one side to the other by detector with 10ms (that is, 0.01 second)
Between be spaced capture eyes speckle pattern six sequence frames.
Figure 10 A to 10C is shown with three successive frames of the detector of the interval 10ms capture, which shows
How speckle pattern is formed on the detector.
Figure 11 A shows the snapshot of the front view for the eyes of embodiment according to the present invention captured by eyes camera.
Figure 11 B show embodiment according to the present invention as detector with capture Figure 10 A shown in eyes forward sight
The snapshot of the speckle pattern for the eyes generated that the snapshot of figure captures simultaneously.
Figure 11 C shows the motion profile of the eyes from initial position to final position of embodiment according to the present invention.
Figure 12 A shows snapshot (two of the speckle pattern at two different detectors of embodiment according to the present invention
Left panel), and by two different camerals capture eyes front view and side view snapshot (two right panels).
Figure 12 B and Figure 12 C respectively illustrate the eyes of embodiment according to the present invention detected by two detectors
Track.
The snapshot of the front view of eyes when Figure 13 A shows the blink of embodiment according to the present invention.
The image at detector when Figure 13 B shows the blink of embodiment according to the present invention.
Figure 14 A shows embodiment according to the present invention: in (i) two right panels by two different eyes cameras
The snapshot of the front view of the eyes of capture and side view simultaneously；And in (ii) two left panels by two different detectors
The snapshot of the speckle pattern of capture.
Figure 14 B shows a part of the track of the movement of the eyes of embodiment according to the present invention.
Figure 14 C shows the tracking quality score according to the time of embodiment according to the present invention.
Figure 15 A to Figure 15 C is similar with Figure 14 A to Figure 14 C, but be embodiment according to the present invention different time it is fast
According to.
Figure 16 shows the simplified flowchart for watching tracking attentively of the display eyes of embodiment according to the present invention.
Figure 17 shows the simplified flowcharts for watching tracking attentively of display eyes according to another embodiment of the present invention.
Figure 18 is the simple flow for showing the method for the movement of display tracking eyes of user of embodiment according to the present invention
Figure.
Figure 19 is the simple flow for showing the method for display tracking eyes of user watched attentively of embodiment according to the present invention
Figure.
Figure 20 is the simplified stream for showing the method for display tracking eyes of user according to another embodiment of the present invention watched attentively
Cheng Tu.
Figure 21 is the method for showing the speckle pattern of the pre-calibration of display acquisition eyes of user of embodiment according to the present invention
Simplified flowchart.
Figure 22 is to show the display of embodiment according to the present invention to determine the start bits of eyes using the speckle pattern of pre-calibration
The simplified flowchart for the method set.
Figure 23 is method of the display tracking eyes relative to the movement of initial position for showing embodiment according to the present invention
Simplified flowchart.
Figure 24 is the simplified flowchart for showing the method for display identification user of embodiment according to the present invention.
Specific embodiment
This invention relates generally to the method and systems for watching tracking attentively for eyes.It is more particularly related to
For using speckle pattern to carry out the method and system for watching tracking attentively of eyes.Most popular design is based on video at present
Eye tracker, also referred to as passive tracking.When viewer sees certain stimulation, camera can be focused on one or two eyes simultaneously
Record their movement.Fig. 1 shows four images (a) to (d) when the eyes of eyes captured when watching attentively at various positions.
By tracking (x, y) coordinate of pupil center, watching attentively for eyes can be monitored.
Most of modern times eye trackers generate cornea using pupil center and infrared (such as, near-infrared) non-collimated light
It reflects (CR).The focus that vector between pupil center and corneal reflection can be used on gauging surface or direction of gaze.It uses
Infrared or near-infrared (also referred to as active light) eye tracking technology of two kinds of general types: bright pupil and dark pupil.Their difference exists
In position of the irradiation source relative to optical device.If irradiation and light path coaxial, eyes serve as retroeflector because light from
Retinal reflex generates the bright pupil effect for being similar to blood-shot eye illness.If irradiation source deviates optical path, pupil seems very dark, because
Reflection from retina is directed away from camera.
Fig. 2 schematically shows the movements of the possible eyes along three independent axes: horizontal displacement (the first dimension
(dimension)), vertical displacement (the second dimension), horizontal rotation (third dimension), vertical rotation (fourth dimension degree), spin
(cyclotorsion) (the 5th dimension) and axial displacement (6 DOF degree).
The embodiment provides the method and systems watched attentively that eyes are tracked for using speckle pattern.Speckle
Pattern is by the intensity pattern for interfering generation of one group of wavefront.Speckle pattern typically occurs in the list of such as laser etc
In the diffusing reflection of coloured light.Speckle effect be with identical frequency, it is that a large amount of waves of out of phase and amplitude are interfered as a result, this
A little waves are added together to form composite wave, and the amplitude of the composite wave changes at random, therefore intensity also changes at random.Fig. 3 show from
The illustrative speckle pattern that green laser indicator generates.
Fig. 4 schematically shows how on imaging sensor 430 to form speckle pattern.When surface 420 is such as swashed
When the coherent source 410 of light device etc irradiates, according to diffraction theory, each point on illuminated surface serves as secondary spherical wave
Source.Light in scattered optical field at arbitrary point from the wave of each point scattering on illuminated surface 420 by constituting.If surface
420 is sufficiently coarse to generate the path length difference more than a wavelength, then causes the phase change greater than 2 π, synthesize the width of light
The random variation of degree, therefore its intensity also changes at random.For example, as shown in figure 4, scattering light wave may reverse phase (out relative to each other
Of phase) interfere and generates blackening 442 at (for example, at point A) certain positions on imaging sensor 430.Equally,
Scatter light wave may relative to each other at certain positions on the interference of same phase (in phase) and imaging sensor 430 (for example,
At point B) generate speck 444.Therefore, speckle pattern can be considered as the hologram generated by rough surface.
When the rough surface irradiated by coherent light (for example, laser beam) is imaged, observed in the plane of delineation
Speckle pattern；This is referred to as " subjective speckle pattern ".Referred to as " subjectivity " is because the detailed construction of speckle pattern depends on seeing
See system parameter.For example, speckle size changes if the size of lens aperture changes.When the phase scattered from rough surface
When dry light is fallen on the other surface, which forms " objective speckle pattern ".If photographic plate or another 2D optical sensor
In the scattered optical field of not lens, then obtain feature depending on system geometry and coherent source wavelength it is objective
Speckle pattern.
The embodiment of the present invention includes for by leading light beam at eyes and tracking the diffusing reflection by eyes to light beam
The speckle pattern formed with mirror-reflection moves to track the method and system of the movement of eyes.Eye is tracked using speckle pattern
Watching attentively for eyeball can provide several advantages.Although the texture of eyeball surface may be too smooth and can not be detected by other means
It arrives, but observable speckle pattern can be generated in it.When eye movement, speckle pattern will be moved relative to sensor, because
It is generated for speckle pattern by eyeball and is consequently fixed to eyeball.It, can be accurately to eyeball by tracking the movement of speckle pattern
Small movements detected, such as the movement of micron order or smaller rank.
Fig. 5 A and Fig. 5 B are shown relative to each other with two of the speckle pattern on the detector of small time interval capture
A snapshot, wherein target moves a small distance during the time interval.As can be seen, speckle pattern phase
For detector motion.For example, the speck court in Fig. 5 A at the A of position moves left the position near the detector edge in Fig. 5 B
A'.Similarly, the speck in Fig. 5 A at position B and C is also respectively towards the position B' and C' moved left in Fig. 5 B.It is scattered in order to track
The movement of spot pattern, it may be necessary to which the beam spot movement in interframe target is much smaller than beam spot size, and may need Change detection
Beam spot movement on device is much smaller than detector size.For example, in some embodiments, it is from about 5mm to about that range, which can be used,
The beam spot size of 10mm.When eyes are in interframe movement, the beam spot position on eyes is correspondingly moved.For example, for 1000
The frame rate of frame/second (fps), giving beam spot size is 5mm, then the movement of the eyes carried out with about 5,000mm/ seconds rate will
In beam spot size.Similarly, when eyes are in interframe movement, the speckle pattern on detector is also correspondingly moved.Therefore, may be used
High speed detector is advantageously used, for example, with the detector that 1000fps or faster rate operate, to meet above-mentioned want
It asks.
Fig. 6 schematically shows the exemplary setting for watching tracking attentively for eyes of embodiment according to the present invention.
Light source 610 can be thrown into light beam 620 at eyes with tilt angle.Eyes can be with the part in the reflected beams 620.Detector
630 can be positioned at ocular vicinity, to detect the speckle pattern formed by the part in the light beam of eye reflections.Fig. 7 A is shown
By the front view of the eyes of camera capture.Speck 702 is the beam spot on eyes.Fig. 7 B is shown to be dissipated by what detector 620 captured
Spot pattern.As can see in figure 7b, the interference fringe generated by speckle reflection is high-visible in speckle pattern.One
In a little embodiments, light source 610 can be configured as the light beam in transmitting infrared wavelength range.In some embodiments, light source 610
It can be configured as the light beam in transmitting near infrared wavelength region (for example, 0.75 μm to 2 μm).In some embodiments, light source
610 may include the coherent source of such as laser etc, or the partially coherent light source of such as light emitting diode (LED) etc,
So as to detect speckle pattern.
What Fig. 8 showed embodiment according to the present invention can be how by light source 810, detector 830 and optional eyes
Camera 840 is mounted on can be by the schematic diagram on a pair of glasses 850 of user's wearing.Light source 810 can throw into illumination beam 820
At the eyes of user.Eyes can form speckle pattern to the diffusing reflection of light beam 820 and mirror-reflection at detector 830.?
In some embodiments, light source 810 may include the coherent source of such as laser etc.Eyes camera 840 can capture eyes
Image, can therefrom determine the position of watching attentively of initial eyes, discuss in greater detail below.
Fig. 9 A to Fig. 9 F is shown when eyes are moved from one side to the other by detector with 10ms's (i.e. 0.01 second)
Six sequence frames of the speckle pattern of time interval capture generated by eye reflections light beam.As can be seen, speckle pattern
Fig. 9 F is moved to from Fig. 9 A to the left across detector region.For example, the clear zone A in Fig. 9 A is in Fig. 9 B and Fig. 9 C progressively towards left side
It is mobile, then detector region is removed in Fig. 9 D to Fig. 9 F.Similarly, the dark space B in Fig. 9 B just starts in figures 9 b and 9
It is existing, it is mobile progressively towards left side in Fig. 9 V to Fig. 9 E, then detector region is removed in Fig. 9 F.
Figure 10 A to Figure 10 C is shown with three successive frames of the detector of the interval 10ms capture, which shows
How speckle pattern can be formed on the image sensor.Here, although without visible interference fringe, it can be seen that
The speckle pattern generated by the diffusing reflection of eyes.
By tracking the movement of speckle pattern, the movement of eyes can be tracked.It according to some embodiments of the present invention, can be with
The movement of speckle pattern is tracked using optical flow algorithm.Optical flow or light stream are by between viewer's (eyes or camera) and scene
Relative motion caused by object in visual scene, surface and edge the pattern obviously moved.The orderly image of sequence can
To allow for estimation to be displaced as instantaneous picture speed or discrete picture.The method for determining light stream may include phase related side
Method, block-based method, difference method, Huo En-mountain gram (Horn-Schunck) method, Buxton-Buxton
(Buxton-Buxton) method, Bu Laike-Jie Pusen (Black-Jepson) method and general variational method, discrete optimization side
Method etc..According to an embodiment of the invention, passing through the sequence frames for the speckle pattern that the phase related algorithm of light stream is applied to eyes
To track the movement of eyes.
Figure 11 A to Figure 11 C shows some example results for watching tracking attentively of the eyes of embodiment according to the present invention.
Figure 11 A shows the snapshot of the video of the front view of the eyes captured by eyes camera.Figure 11 B is shown before capturing eyes
By the snapshot of the speckle pattern of the eyes of detector capture while the snapshot of view.Figure 11 C, which is shown, obtains Figure 11 A and figure
The track of the movement of the eyes slave starting point to terminating point of the snapshot of 11B.Trunnion axis and vertical axes in Figure 11 C be respectively with
The horizontal position of arbitrary unit (for example, according to detector pixel) and vertical position.After the calibration, the position of watching attentively of eyes can
To be converted into gaze angle.Frame rate for detecting speckle pattern is about 100 frames/second.The duration of tracking is about 20
Second.As can see in Figure 11 C, when eyes move back and forth up and down, the movement of eyes is continuously tracked.
In order to show the robustness for watching tracking attentively of the eyes using speckle pattern, two detectors can be located in eye
The movement of eyes is tracked near eyeball.Figure 12 A shows snapshot (two of the speckle pattern by two different detector captures
Left panel), and by two different cameras captures eyes front view and side view snapshot (two right panels).Figure
12B and Figure 12 C respectively illustrates the track of the eyes detected by two detectors.Trunnion axis in Figure 12 B and Figure 12 C and perpendicular
D-axis is horizontal position and the vertical position of the eyes using arbitrary unit respectively.For each of two detectors, use
In detection speckle pattern frame rate be about 100 frames/second.The duration of tracking is about 20 seconds.As can be seen, two rails
Mark is consistent with each other, it was demonstrated that this eyes to watch tracking attentively quite steady.
According to an embodiment of the invention, as long as eyes are not batted an eyelid, can be moved by tracking the speckle pattern of the eyes come
The movement of eyes is continuously tracked.When blink, speckle pattern may disappear, and therefore, the correlation between a frame and next frame can
It can lose.Figure 13 A shows the snapshot of the front view of the eyes when eyes blink.Figure 13 B is shown in eyes blink
Image at the detector of capture simultaneously.As shown in Figure 13 B, when eyes are blinked, speckle pattern disappears.If blink continues
Time long enough, such as a few tens of milliseconds, the then speckle pattern captured before and after blink may no longer have phase each other
Correlation.Therefore, eyes are watched tracking attentively and may be interrupted because it is such as in accordance with some embodiments for eyes watch attentively with
Some algorithms of the optical flow algorithm of track etc may need the phase correlation between sequence frames.
According to some embodiments, the instruction of watching tracking quality of the tracking quality score as eyes can be used.For example, with
Track mass fraction can be the measurement of the degree of correlation between the sequence frames of speckle pattern.It in some embodiments, can basis
Correlation peak calculates tracking quality score.If the correlation between sequence frames is perfectly, the value of correlation peak can be connect
It is bordering on 1.If the correlation between sequence frames is poor (may be due to blink), correlation peak may be significantly less than 1.
Figure 14 A to Figure 14 C and Figure 15 A to Figure 15 C shows the tracking of watching attentively of the eyes of embodiment according to the present invention
Some example results.Two right panels in Figure 14 A respectively illustrate the side of the eyes by two different eyes camera captures
The snapshot of view and front view.Two left panels of Figure 14 A show the snapshot of side view and front view in capture eyes
Simultaneously by the snapshot of the speckle pattern of two different detectors capture.Figure 14 B shows the partial traces of the movement of eyes.Figure
Trunnion axis and vertical axes in 14B are with the horizontal position of the eyes of arbitrary unit and vertical position respectively.Figure 14 C shows root
According to the tracking quality score of time.Vertical axes are the tracking quality scores with arbitrary unit.Trunnion axis is frame number (with 100
Frame/second corresponding 10 milliseconds of interval).Tracking quality score in period shown in as shown, is relatively high.Figure 15 A
It is similar with Figure 14 A to Figure 14 C to Figure 15 C, but it is snapshot in different time periods.As shown in fig. 15, it since eyes are blinked, dissipates
Spot pattern loss.Figure 15 C shows tracking quality score at about 1060ms, i.e., at the time of eyes start blink, declines suddenly.
As it appears from the above, can realize the accurate tracking of the relative motion to eyes by tracking speckle pattern.But speckle
What pattern can not provide the absolute eyes of eyes watches position attentively.According to an embodiment of the invention, the initial absolute eyes of eyes
Position of watching attentively can be by such as being determined based on the another method of the method for camera etc.In watching attentively for the initial eyes of determination
After position, the movement of watching position of the eyes relative to initial eyes is more accurately tracked using speckle tracking.Similarly,
When losing correlation when eyes are blinked, the fixation position of new initial absolute eyes can be determined by the method based on camera
It sets.Establish new initial eyes watch position attentively after, speckle tracking can be restarted.
Figure 16 shows the simplification of the method 1600 for watching tracking attentively of display eyes according to some embodiments of the present invention
Flow chart.Method 1600 includes watching tracking (1602) attentively using camera execution eyes, and determine whether to have obtained starting eye
Eyeball watches position (1604) attentively.The note of the eyes using camera can be executed by using the multiple images of cameras record eyes
Depending on tracking.It can choose one of multiple images to determine that starting eyes watch position attentively.For example, some images may be because
Eyes are blinking and are having relatively poor quality.Based on selected image, the coordinate system of pre-calibration can be used to have determined
Beginning eyes watch position attentively.For example, as shown in Figure 1, can determine that (x, y) of pupil center is sat relative to the origin of pre-calibration
Mark.
If not yet obtain starting eyes watches position attentively, method 1600 can continue to use watching attentively for the eyes of camera
It tracks (1602), the fixation position until obtaining starting eyes is set to only.Once obtain starting eyes watches position, method attentively
1600 can start to execute and watch tracking (1606) attentively using the eyes of speckle pattern.As set forth above, it is possible to by the way that light beam is led
It is detected at eyes and to speckle pattern, the part in light beam which is gone out by eye reflections is in detector
It is formed, the eyes to execute using speckle pattern watch tracking attentively.Light beam can be produced by the coherent source of such as laser etc
It is raw, or generated by the partially coherent light source of such as light emitting diode (LED) etc, so as to detect speckle pattern.It can be with
The algorithm of phase related algorithm by using such as optical flow approach etc tracks speckle pattern frame by frame, to track the fortune of eyes
It is dynamic.
If user blinks, since the correlation between successive frame is lost, it may lose and track eye using speckle pattern
The ability of the movement of eyeball.Therefore, method 1600 further include determine whether to have occurred and that blink and whether due to blink and
It loses correlation (1608).In some embodiments, determining whether correlation has lost may include calculating correlation peak
It is worth (as described above), and the value of correlation peak is compared with predetermined threshold.If the value of correlation peak is equal to or higher than pre-
Determine threshold value, then can determine that correlation is not yet lost.On the contrary, can be determined if the value of correlation peak is lower than predetermined threshold
Correlation has been lost.
If it is determined that correlation is not yet lost, then method 1600 can continue through tracking speckle pattern come eyes watch attentively with
Track (1606).If it is determined that correlation has been lost, then method 1600, which may return to, watches tracking attentively using the eyes of camera
(1602).When obtaining new when watching position (1604) attentively of starting eyes by using camera tracking, method 1600 can pass through
Tracking speckle pattern watches tracking (1606) attentively restart eyes.
As described above, the tracking of watching attentively using the eyes of speckle pattern can be than using the watching attentively for eyes of camera to track more
Accurately.In some embodiments, method 1600 can choose the best camera frame for watching position attentively for determining starting eyes, and
And speckle tracking is executed between camera frame to obtain better accuracy.In some embodiments, method 1600 can be with phase
When high frame rate (for example, 10,000fps) execution speckle tracking, and phase is executed with much lower frame rate (for example, 3fps)
Machine tracking.
According to another embodiment of the present invention, initial absolute eyes can be obtained by using the speckle pattern of pre-calibration
Watch position attentively.It is exclusive that eyes speckle pattern can be everyone.Before the watching attentively of eyes of tracking user, it can be directed to
User executes calibration process.In a calibration process, can obtain with the various eyes of user to watch position attentively corresponding multiple
Speckle pattern.Various eyes watch attentively position can by based on camera method or other suitable eyes watch tracking attentively
To determine.Thus, it is possible to obtain the exclusive speckle pattern of user and storing in systems.Speckle pattern may include multiple pre-calibrations
Speckle pattern and its corresponding eyes watch position attentively.
Figure 17 shows the simple flows of the method 1700 for watching tracking attentively of the display eyes of embodiment according to the present invention
Figure.Method 1700 includes executing eyes using the speckle pattern of the pre-calibration of user to watch tracking (1702) attentively, and determine whether
Obtained starting eyes watches position (1704) attentively.As described above, speckle pattern may include multiple pre-calibrations speckle pattern and
Its corresponding eye watches position attentively.It can be by the way that light beam be led at eyes and at the detector of ocular vicinity
Multiple initial speckle patterns are detected, to obtain multiple initial speckle patterns.Light beam can be by the phase of such as laser etc
Dry light source generates, or is generated by the partially coherent light source of such as light emitting diode (LED) etc, so as to detect speckle pattern
Case.Multiple initial speckle patterns can be compared with the speckle pattern of the pre-calibration in speckle pattern, with the initial eyes of determination
Watch position attentively.For example, it may be determined that one of multiple initial speckle patterns and the speckle in the speckle pattern for being stored in user
Pattern match.It is then possible to use the correspondence eyes of matched speckle pattern watch attentively position as starting eyes fixation position
It sets.
If not yet obtaining initial position, what method 1700 can continue to use the eyes of speckle pattern watches tracking attentively
(1702).Once obtaining initial position, then system can start to track relative to starting eyes by tracking speckle pattern
That watches the eyes of position attentively watches (1706) attentively.Can by using the phase related algorithm of such as optical flow approach etc algorithm by
Frame tracks speckle pattern, to track the movement of eyes.
Method 1700 further includes determining whether to have occurred and that blink (1708), and whether lost phase due to blink
Closing property (1708), it is similar with the method 1600 discussed above for Figure 16.If it is determined that correlation is not yet lost, then method
1700 can continue speckle tracking (1706).If it is determined that correlation has been lost, then method 1700 may return to step
1702 watch tracking (1702) attentively using the eyes of speckle pattern to execute.When obtaining new initial position (1704), method 1700
It can restart speckle tracking (1706).In some embodiments, method 1700 can with quite high frame rate (for example,
10,000fps) speckle tracking is executed, and executes the tracking for using speckle pattern with much lower frame rate (for example, 3fps).
Figure 18 is the simplification for showing the method 1800 of the movement of display tracking eyes of user of embodiment according to the present invention
Flow chart.Method 1800 includes that light beam is led at eyes to (1802).Light beam can be by the coherent source of such as laser etc
It generates, or is generated by the partially coherent light source of such as light emitting diode (LED) etc, so as to detect speckle pattern.Eye
Eyeball can be with the part in the reflected beams.It can be reflected diffusely or specularly by eyes by the part in the light beam of eye reflections.Side
Method 1800 further includes detecting to multiple speckle patterns, and multiple speckle pattern is being examined by the part in the light beam of eye reflections
It surveys at device and forms (1804).Multiple speckle patterns are detected with scheduled frame rate.Method 1800 further includes by tracking frame by frame
Multiple speckle patterns track the movements (1806) of eyes.In some embodiments, pre-determined frame rate is greater than about 5,000 frame/second
And it is less than about 15,000 frames/second.In one embodiment, pre-determined frame rate is about 10,000 frame/second.In some other embodiments
In, pre-determined frame rate is greater than about 50 frames/second and is less than about 15,000 frame/second.In some embodiments, it is executed using optical flow algorithm
Track multiple speckle patterns.In one embodiment, optical flow algorithm uses phase correlation method.
Figure 19 is the simplification for showing the method 1900 of display tracking eyes of user watched attentively of embodiment according to the present invention
Flow chart.Method 1900 includes using camera with the first multiple images (1902) of the first frame rate record eyes；And it is based on
First multiple images of eyes determine the initial positions (1904) of eyes.In some embodiments, it can choose more than first
One of image determines the initial positions of eyes.Based on selected image, the coordinate system of pre-calibration can be used to determine eye
The initial position of eyeball.For example, as shown in Figure 1, can determine (x, y) coordinate of pupil center relative to the origin of pre-calibration.
Method 1900 can also include that light beam is led at eyes to (1906).Light beam can be by the phase of such as laser etc
Dry light source generates, or is generated by the partially coherent light source of such as light emitting diode (LED) etc.Eyes can be in the reflected beams
Part.It can be reflected diffusely or specularly by eyes by the part in the light beam of eye reflections.Method 1900 can be further
Including detecting to more than first a speckle patterns, which is being examined by the part in the light beam of eye reflections
It surveys at device and forms (1908), and the movement for tracking eyes relative to initial position by tracking a speckle more than first frame by frame
(1910).In some embodiments, multiple speckle patterns are detected with the second frame rate for being greater than the first frame rate.One
In a little embodiments, the first frame rate is less than about 10fps, and the second frame rate is greater than about 50fps and is less than about 15,000fps.One
In a embodiment, the first frame rate is about 3 frames/second, and the second frame rate is about 10,000 frame/second.
In some embodiments, method 1900 can also include determining that eyes have blinked (1920).For example, can be based on
Determine whether eyes have blinked such as the tracking quality score that is discussed above for Figure 16 and Figure 17.In response to determining eyes
It has been blinked that, method 1900 may return to step 1902 to determine new initial position.For example, method 1900 may include passing through
Following manner determines new initial position: using the second multiple images of cameras record eyes, and more than second based on eyes figure
As determining new initial position；And more than second a speckle patterns by being formed at detector with the detection of the second frame rate are simultaneously
More than second a speckle patterns are tracked, frame by frame to track movement of the eyes relative to new initial position.If it is determined that eyes not yet blink
Eye, then method 1900 can continue step 1908 and step 1910
Figure 20 is the simplification for showing the method 2000 of tracking eyes of user according to some embodiments of the present invention watched attentively
Flow chart.Method 2000 may include the speckle pattern (2010) for obtaining the pre-calibration of eyes of user, use the speckle of the pre-calibration
Figure determines the initial positions (2020) of eyes, and movement (2030) of the tracking eyes relative to the initial position, such as following
It is further described about Figure 21 to Figure 23.
Figure 21 is the method 2010 for showing the speckle pattern of the pre-calibration of acquisition eyes of user of embodiment according to the present invention
Simplified flowchart.Method 2010 includes that the first light beam is led at eyes (2011).First light beam can be by such as laser
Etc coherent source generate, or by such as light emitting diode (LED) etc partially coherent light source generate.Eyes can be anti-
Penetrate the part in the first light beam.Method 2010 further includes detecting to multiple first speckle patterns, multiple first speckle pattern
Case forms (2012) in detector by the part in the first light beam of eye reflections.Each of multiple first speckle patterns
It is corresponding with corresponding eyes first position.Method 2010 further includes the multiple images (2013) using cameras record eyes.Work as inspection
When measuring corresponding first speckle pattern, each of multiple images of eyes are recorded.Method 2010 further includes based on eyes
The determining corresponding eyes first position (2014) corresponding with each corresponding first speckle pattern of respective image, and storage is more
A first speckle pattern and corresponding eyes first position (2015).
Figure 22 is the initial position that eyes are determined using the speckle pattern of pre-calibration for showing embodiment according to the present invention
Method 2020 simplified flowchart.Method 2020 includes that the second light beam is led at eyes (2021).Second light beam can be by
The coherent source of such as laser etc generates, or is generated by the partially coherent light source of such as light emitting diode (LED) etc.
Eyes can reflect the part in the second light beam.Method 2020 further includes detecting to multiple second speckle patterns, multiple
Second speckle pattern is formed (2022) by the part in the second light beam of eye reflections in detector, and is based on multiple second
Speckle pattern is compared between multiple first speckle patterns stored and the corresponding eyes first position that is stored is come
Determine the initial position (2023) of eyes.
Figure 23 is the method 2030 for showing the tracking eyes of embodiment according to the present invention relative to the movement of initial position
Simplified flowchart.Method 2030 includes that third light beam is led at eyes (2031).Third light beam can be by such as laser
Etc coherent source generate, or by such as light emitting diode (LED) etc partially coherent light source generate.Eyes can be anti-
Penetrate the part in third light beam.Method 2030 further includes detecting to multiple third speckle patterns, multiple third speckle pattern
Case forms (2032) in detector by the part in the third light beam of eye reflections.Multiple thirds are detected with pre-determined frame rate to dissipate
Spot pattern.Method 2030 further includes the fortune for tracking eyes relative to initial position by tracking the multiple speckle patterns of third frame by frame
It is dynamic.
As described above, eyes speckle pattern is that everyone is exclusive.Therefore, eyes speckle pattern can be used for user's knowledge
Not.Figure 24 is the simplified flowchart for showing the method 2400 of identification user of embodiment according to the present invention.Method 2400 includes
First light beam is led at first eye (2402).First light beam can be generated by the coherent source of such as laser etc, or
Person is generated by the partially coherent light source of such as light emitting diode (LED) etc.First eye can reflect the portion in the first light beam
Point.Method 2400 further includes detecting to multiple first speckle patterns, and multiple first speckle pattern is reflected by first eye
The first light beam in part (2404) are formed at detector.Each of multiple first speckle patterns and first eye
Corresponding first position is corresponding.Method 2400 further includes the multiple images (2406) using cameras record first eye.Work as detection
When to corresponding first speckle pattern, each of multiple images of first eye are recorded.Method 2400 further includes based on the
The corresponding first position of the determining first eye corresponding with each corresponding first speckle pattern of the corresponding images of one eyes
(2408), and the corresponding first position (2410) of multiple first speckle patterns and first eye is stored.Method 2400 further includes
Second light beam is led at the eyes of user (2412).Second light beam can be generated by the coherent source of such as laser etc,
Or it is generated by the partially coherent light source of such as light emitting diode (LED) etc.Eyes can reflect the part in the second light beam.
Method 2400 further includes detecting to multiple second speckle patterns, multiple second speckle pattern by eye reflections the second light
Part in beam forms (2414) at detector.The corresponding second of each of multiple second speckle patterns and eyes
Set correspondence.Method 2400 further includes by being compared to multiple second speckle patterns and multiple first speckle patterns to determine use
Family is the first user (2416).
It should be understood that specific steps shown in each of Figure 16 to Figure 24 provide and implement according to the present invention
The ad hoc approach of example.According to alternate embodiment, other sequences of step can also be performed.For example, alternate embodiment of the invention
Above step can be executed with different sequences.In addition, each step shown in Figure 16 to Figure 24 may include multiple sub-steps
Suddenly, these sub-steps can be executed with the various sequences for being suitable for each step.In addition, depending on specific application, can add
Or remove additional step.Many changes, modifications and substitution will be recognized in one of those of ordinary skill in the art.
It should also be understood that example described herein and embodiment are for illustration purposes only, and suggest those skilled in the art
Various slight modifications or change are carried out to it, these modifications and changes include in spirit and scope and appended power
In the range of benefit requires.
Claims (20)
1. a kind of method for the movement for tracking eyes of user, which comprises
Light beam is led at the eyes, the part in light beam described in the eye reflections；
Multiple speckle patterns are detected, the multiple speckle pattern is existed by the part in the light beam of the eye reflections
It is formed at detector, the multiple speckle pattern is detected with scheduled frame rate；And
The movement of the eyes is tracked by tracking the multiple speckle pattern frame by frame.
2. according to the method described in claim 1, wherein, by the part in the light beam of the eye reflections by the eyes
It is reflected diffusely or specularly.
3. according to the method described in claim 1, wherein, the pre-determined frame rate is greater than about 5,000 frame/second and is less than about 15,
000 frame/second.
4. according to the method described in claim 3, wherein, the pre-determined frame rate is about 10,000 frame/second.
5. according to the method described in claim 1, wherein, the pre-determined frame rate is greater than about 50 frames/second and is less than about 15,000
Frame/second.
6. according to the method described in claim 1, wherein, the multiple speckle pattern of tracking is executed using optical flow algorithm.
7. according to the method described in claim 6, wherein, the optical flow algorithm includes phase related algorithm.
8. according to the method described in claim 1, wherein, the light beam includes infra-red radiation.
9. a kind of method watched attentively for tracking eyes of user, which comprises
The first multiple images of the eyes are recorded with the first frame rate using camera；
The initial position of the eyes is determined based on first multiple images of the eyes；
Light beam is led at the eyes, the part in light beam described in the eye reflections；
More than first the first speckle patterns are detected, a speckle pattern more than described first by the eye reflections the light
Part in beam is formed at detector, and a speckle pattern more than described first is the second frame speed to be greater than first frame rate
What rate was detected；And
Movement of the eyes relative to the initial position is tracked by tracking a speckle pattern more than described first frame by frame.
10. according to the method described in claim 9, wherein, by the part in the light beam of the eye reflections by the eye
Eyeball is reflected diffusely or specularly.
11. according to the method described in claim 9, wherein, the multiple speckle pattern of tracking is executed using optical flow algorithm.
12. according to the method for claim 11, wherein the optical flow algorithm includes phase related algorithm.
13. according to the method described in claim 9, wherein, first frame rate is less than about 10 frames/second, and described second
Frame rate is greater than about 50 frames/second and is less than about 15,000 frames/second.
14. according to the method described in claim 9, further comprising:
Determine that the eyes have been blinked；
New initial position is determined by following steps:
The second multiple images of the eyes are recorded using the camera；And
The new initial position is determined based on second multiple images of the eyes；And it is tracked by following steps
Movement of the eyes relative to the new initial position:
More than the second a speckle patterns formed at the detector are detected with the second frame rate；And
A speckle pattern more than described second is tracked frame by frame.
15. a kind of method watched attentively for tracking eyes of user, which comprises
The speckle pattern of the pre-calibration of the eyes of user is obtained by following steps:
First light beam is led at the eyes, the part in the first light beam described in the eye reflections；
Multiple first speckle patterns are detected, the multiple first speckle pattern by the eye reflections first light
Part in beam is formed at detector, each of the multiple first speckle pattern corresponding with the eyes first
Position is corresponding；
The multiple images that the eyes are recorded using camera record the eye when detecting corresponding first speckle pattern
Each of described multiple images of eyeball；
The institute of the eyes corresponding with each corresponding first speckle pattern is determined based on the corresponding image of the eyes
State corresponding first position；And
Store the corresponding first position of the multiple first speckle pattern and the eyes；
The initial position of the eyes is determined using the speckle pattern of the pre-calibration by following steps:
Second light beam is led at the eyes, the part in the second light beam described in the eye reflections；
Multiple second speckle patterns are detected, the multiple second speckle pattern by the eye reflections second light
Part in beam is formed at the detector；And
Based on the multiple second speckle pattern compared between multiple first speckle patterns stored and the institute that is stored
The corresponding first position of eyes is stated to determine the initial position of the eyes；And
Movement of the eyes relative to the initial position is tracked by following steps:
Third light beam is led at the eyes, the part in third light beam described in the eye reflections；
Multiple third speckle patterns are detected, the multiple third speckle pattern by the eye reflections the third light
Part in beam is formed at the detector, and the multiple third speckle pattern is detected with scheduled frame rate；With
And
Movement of the eyes relative to the initial position is tracked by tracking the multiple speckle patterns of the third frame by frame.
16. according to the method for claim 15, wherein by first light beam of the eye reflections part, by
Part in second light beam of the eye reflections and by the part in the third light beam of the eye reflections
Each be reflected diffusely or specularly by the eyes.
17. according to the method for claim 15, wherein execute the tracking multiple speckle patterns of third using optical flow algorithm
Case.
18. according to the method for claim 17, wherein the optical flow algorithm includes phase related algorithm.
19. according to the method for claim 15, wherein the pre-determined frame rate is greater than about 50 frames/second and less than about 15,
000 frame/second.
20. according to the method for claim 15, further comprising:
Determine that the eyes have been blinked；
The new initial position of the eyes is determined by following steps:
4th light beam is led at the eyes, the part in the 4th light beam described in the eye reflections；
Multiple 4th speckle patterns are detected, the multiple 4th speckle pattern by the eye reflections the 4th light
Part in beam is formed at the detector；And
Based on the multiple 4th speckle pattern compared between multiple first speckle patterns stored and the institute that is stored
The corresponding first position of eyes is stated to determine the new initial position of the eyes；And
Movement of the eyes relative to the new initial position is tracked by following steps:
5th light beam is led at the eyes, the part in the 5th light beam described in the eye reflections；
Multiple 5th speckle patterns are detected, the multiple 5th speckle pattern by the eye reflections the 5th light
Part in beam is formed at the detector, and the multiple 5th speckle pattern is detected with the scheduled frame rate
's；And
Movement of the eyes relative to the new initial position is tracked by tracking more than the described 5th a speckle pattern frame by frame.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201662420292P | 2016-11-10 | 2016-11-10 | |
| US62/420,292 | 2016-11-10 | ||
| PCT/US2017/060303 WO2018089329A1 (en) | 2016-11-10 | 2017-11-07 | Method and system for eye tracking using speckle patterns |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109922710A true CN109922710A (en) | 2019-06-21 |
| CN109922710B CN109922710B (en) | 2022-05-13 |
Family
ID=62064441
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201780069498.9A Active CN109922710B (en) | 2016-11-10 | 2017-11-07 | Method and system for performing eye tracking using speckle patterns |
Country Status (9)
| Country | Link |
|---|---|
| US (2) | US10948981B2 (en) |
| EP (1) | EP3537950A4 (en) |
| JP (2) | JP7008702B2 (en) |
| KR (1) | KR102555925B1 (en) |
| CN (1) | CN109922710B (en) |
| AU (1) | AU2017356886A1 (en) |
| CA (1) | CA3042263A1 (en) |
| IL (1) | IL266390B (en) |
| WO (1) | WO2018089329A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113759545A (en) * | 2020-06-02 | 2021-12-07 | 罗伯特·博世有限公司 | Method for operating a pair of smart glasses |
| US11237631B2 (en) | 2016-11-10 | 2022-02-01 | Magic Leap, Inc. | Method and system for eye tracking using speckle patterns |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20240046291A (en) | 2017-03-22 | 2024-04-08 | 매직 립, 인코포레이티드 | Depth based foveated rendering for display systems |
| CA3059373A1 (en) * | 2017-04-24 | 2018-11-01 | Magic Leap, Inc. | Tracking optical flow of backscattered laser speckle patterns |
| CN112136094A (en) | 2018-03-16 | 2020-12-25 | 奇跃公司 | Depth-based foveated rendering for display systems |
| US10884492B2 (en) * | 2018-07-20 | 2021-01-05 | Avegant Corp. | Relative position based eye-tracking system |
| CN110022321A (en) * | 2019-04-09 | 2019-07-16 | 李丰 | A kind of VR building detection method and VR equipment based on virtual reality |
| CN110471532B (en) * | 2019-08-19 | 2023-04-18 | 业成科技（成都）有限公司 | Vehicle-mounted device integrating eyeball tracking and gesture control |
| KR20210136623A (en) * | 2020-05-08 | 2021-11-17 | 한국전자통신연구원 | A device tracking gaze and method threrefor |
| DE102022202208A1 (en) | 2022-03-04 | 2023-09-07 | Robert Bosch Gesellschaft mit beschränkter Haftung | Method for pupil detection and/or pupil tracking, optical system and data glasses |
| CN115760750B (en) * | 2022-11-15 | 2023-11-21 | 北京的卢铭视科技有限公司 | DOE quality detection method, DOE quality detection system, DOE quality detection electronic device and storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1269103A (en) * | 1997-06-28 | 2000-10-04 | 英国国防部 | Autostereoscopic display |
| JP2001229368A (en) * | 2000-02-15 | 2001-08-24 | Niles Parts Co Ltd | Eye state detector |
| CN101336089A (en) * | 2006-01-26 | 2008-12-31 | 诺基亚公司 | Eye tracker equipment |
| US20100053555A1 (en) * | 2008-08-27 | 2010-03-04 | Locarna Systems, Inc. | Method and apparatus for tracking eye movement |
| CN102802502A (en) * | 2010-03-22 | 2012-11-28 | 皇家飞利浦电子股份有限公司 | System and method for tracking the point of gaze of an observer |
| WO2013167864A1 (en) * | 2012-05-11 | 2013-11-14 | Milan Momcilo Popovich | Apparatus for eye tracking |
| US20130315491A1 (en) * | 2008-08-01 | 2013-11-28 | The Trustees Of Columbia University In The City Of New York | Systems and methods for matching and imaging tissue characteristics |
| CN103576857A (en) * | 2012-08-09 | 2014-02-12 | 托比技术股份公司 | Fast wake-up in gaze tracking system |
| WO2014188149A1 (en) * | 2013-05-20 | 2014-11-27 | Milan Momcilo Popovich | Holographic waveguide eye tracker |
| US20150301599A1 (en) * | 2014-04-18 | 2015-10-22 | Magic Leap, Inc. | Eye tracking systems and method for augmented or virtual reality |
| WO2016046514A1 (en) * | 2014-09-26 | 2016-03-31 | LOKOVIC, Kimberly, Sun | Holographic waveguide opticaltracker |
| WO2016124668A1 (en) * | 2015-02-04 | 2016-08-11 | IT-Universitetet i København | A gaze tracker and a gaze tracking method |
| CN105934902A (en) * | 2013-11-27 | 2016-09-07 | 奇跃公司 | Virtual and augmented reality systems and methods |
| CN106028911A (en) * | 2014-03-04 | 2016-10-12 | 南加利福尼亚大学 | Extended duration optical coherence tomography (OCT) system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| IL184868A0 (en) * | 2007-07-26 | 2008-03-20 | Univ Bar Ilan | Motion detection system and method |
| JP2015011579A (en) | 2013-06-28 | 2015-01-19 | 株式会社東芝 | Line-of-sight detector and line-of-sight detection method |
| US9911046B1 (en) * | 2014-11-13 | 2018-03-06 | The United States Of America, As Represented By The Secretary Of The Navy | Method and apparatus for computer vision analysis of spin rate of marked projectiles |
| EP3371973B1 (en) * | 2015-11-06 | 2023-08-09 | Facebook Technologies, LLC | Eye tracking using a diffraction pattern of coherent light on the surface of the eye |
| CN109922710B (en) | 2016-11-10 | 2022-05-13 | 奇跃公司 | Method and system for performing eye tracking using speckle patterns |
-
2017
- 2017-11-07 CN CN201780069498.9A patent/CN109922710B/en active Active
- 2017-11-07 EP EP17869673.8A patent/EP3537950A4/en active Pending
- 2017-11-07 KR KR1020197016452A patent/KR102555925B1/en active IP Right Grant
- 2017-11-07 US US15/805,635 patent/US10948981B2/en active Active
- 2017-11-07 AU AU2017356886A patent/AU2017356886A1/en not_active Abandoned
- 2017-11-07 WO PCT/US2017/060303 patent/WO2018089329A1/en unknown
- 2017-11-07 CA CA3042263A patent/CA3042263A1/en active Pending
- 2017-11-07 JP JP2019524195A patent/JP7008702B2/en active Active
-
2019
- 2019-05-01 IL IL266390A patent/IL266390B/en unknown
-
2021
- 2021-02-04 US US17/168,006 patent/US11237631B2/en active Active
-
2022
- 2022-01-11 JP JP2022002426A patent/JP7186906B2/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1269103A (en) * | 1997-06-28 | 2000-10-04 | 英国国防部 | Autostereoscopic display |
| JP2001229368A (en) * | 2000-02-15 | 2001-08-24 | Niles Parts Co Ltd | Eye state detector |
| CN101336089A (en) * | 2006-01-26 | 2008-12-31 | 诺基亚公司 | Eye tracker equipment |
| US20130315491A1 (en) * | 2008-08-01 | 2013-11-28 | The Trustees Of Columbia University In The City Of New York | Systems and methods for matching and imaging tissue characteristics |
| US20100053555A1 (en) * | 2008-08-27 | 2010-03-04 | Locarna Systems, Inc. | Method and apparatus for tracking eye movement |
| CN102802502A (en) * | 2010-03-22 | 2012-11-28 | 皇家飞利浦电子股份有限公司 | System and method for tracking the point of gaze of an observer |
| WO2013167864A1 (en) * | 2012-05-11 | 2013-11-14 | Milan Momcilo Popovich | Apparatus for eye tracking |
| US20150289762A1 (en) * | 2012-05-11 | 2015-10-15 | Milan Momcilo Popovich | Apparatus for eye tracking |
| CN103576857A (en) * | 2012-08-09 | 2014-02-12 | 托比技术股份公司 | Fast wake-up in gaze tracking system |
| WO2014188149A1 (en) * | 2013-05-20 | 2014-11-27 | Milan Momcilo Popovich | Holographic waveguide eye tracker |
| CN105934902A (en) * | 2013-11-27 | 2016-09-07 | 奇跃公司 | Virtual and augmented reality systems and methods |
| CN106028911A (en) * | 2014-03-04 | 2016-10-12 | 南加利福尼亚大学 | Extended duration optical coherence tomography (OCT) system |
| US20150301599A1 (en) * | 2014-04-18 | 2015-10-22 | Magic Leap, Inc. | Eye tracking systems and method for augmented or virtual reality |
| WO2016046514A1 (en) * | 2014-09-26 | 2016-03-31 | LOKOVIC, Kimberly, Sun | Holographic waveguide opticaltracker |
| WO2016124668A1 (en) * | 2015-02-04 | 2016-08-11 | IT-Universitetet i København | A gaze tracker and a gaze tracking method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11237631B2 (en) | 2016-11-10 | 2022-02-01 | Magic Leap, Inc. | Method and system for eye tracking using speckle patterns |
| CN113759545A (en) * | 2020-06-02 | 2021-12-07 | 罗伯特·博世有限公司 | Method for operating a pair of smart glasses |
Also Published As
| Publication number | Publication date |
|---|---|
| IL266390A (en) | 2019-06-30 |
| WO2018089329A1 (en) | 2018-05-17 |
| US20180129282A1 (en) | 2018-05-10 |
| JP2020516326A (en) | 2020-06-11 |
| KR102555925B1 (en) | 2023-07-13 |
| US10948981B2 (en) | 2021-03-16 |
| CA3042263A1 (en) | 2018-05-17 |
| EP3537950A4 (en) | 2019-10-30 |
| US20210232221A1 (en) | 2021-07-29 |
| IL266390B (en) | 2022-04-01 |
| JP7008702B2 (en) | 2022-02-10 |
| US11237631B2 (en) | 2022-02-01 |
| JP7186906B2 (en) | 2022-12-09 |
| KR20190079667A (en) | 2019-07-05 |
| JP2022058561A (en) | 2022-04-12 |
| AU2017356886A1 (en) | 2019-05-02 |
| CN109922710B (en) | 2022-05-13 |
| EP3537950A1 (en) | 2019-09-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7186906B2 (en) | Method and system for eye tracking using speckle patterns | |
| JP6902075B2 (en) | Line-of-sight tracking using structured light | |
| CN107533362B (en) | Eye tracking device and method for operating an eye tracking device | |
| US8457352B2 (en) | Methods and apparatus for estimating point-of-gaze in three dimensions | |
| US6659611B2 (en) | System and method for eye gaze tracking using corneal image mapping | |
| EP2731490A1 (en) | System and method for remote measurement of optical focus | |
| US20170014026A1 (en) | Eye alignment monitor and method | |
| JP7165994B2 (en) | Methods and devices for collecting eye measurements | |
| CN108537103B (en) | Living body face detection method and device based on pupil axis measurement | |
| Kowalik | Do-it-yourself eye tracker: impact of the viewing angle on the eye tracking accuracy | |
| NZ793781A (en) | Method and system for eye tracking using speckle patterns | |
| TW202207865A (en) | Eye tracking systems and methods |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |