US11307659B2 - Low-power eye tracking system - Google Patents

Low-power eye tracking system Download PDF

Info

Publication number
US11307659B2
US11307659B2 US17/021,901 US202017021901A US11307659B2 US 11307659 B2 US11307659 B2 US 11307659B2 US 202017021901 A US202017021901 A US 202017021901A US 11307659 B2 US11307659 B2 US 11307659B2
Authority
US
United States
Prior art keywords
user
hmd
eyes
camera
respect
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US17/021,901
Other languages
English (en)
Other versions
US20210081038A1 (en
Inventor
Tom Sengelaub
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Apple Inc
Original Assignee
Apple Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Apple Inc filed Critical Apple Inc
Priority to US17/021,901 priority Critical patent/US11307659B2/en
Assigned to APPLE INC. reassignment APPLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Sengelaub, Tom
Priority to CN202010986395.2A priority patent/CN112527099A/zh
Publication of US20210081038A1 publication Critical patent/US20210081038A1/en
Application granted granted Critical
Publication of US11307659B2 publication Critical patent/US11307659B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/14—Arrangements specially adapted for eye photography
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/20—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from infrared radiation only
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/65—Control of camera operation in relation to power supply
    • H04N23/651—Control of camera operation in relation to power supply for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • H04N5/232411—
    • H04N5/23245—
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
    • G01S5/16—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using electromagnetic waves other than radio waves
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • Virtual reality allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment.
  • virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene.
  • MR mixed reality
  • MR combines computer generated information (referred to as virtual content) with real world images or a real world view to augment, or add content to, a user's view of the world.
  • the simulated environments of VR and/or the mixed environments of MR may thus be utilized to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of the viewer's environment, interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the Internet, or the like.
  • An eye tracker is a device for estimating eye positions and eye movement. Eye tracking systems have been used in research on the visual system, in psychology, psycholinguistics, marketing, and as input devices for human-computer interaction. In the latter application, typically the intersection of a person's point of gaze with a desktop monitor is considered.
  • Embodiments of methods and apparatus for tracking relative movement of a device with respect to a user's head are described in which sensors (referred to herein as head motion sensors or head odometers) are placed at one or more positions in or on the device.
  • sensors referred to herein as head motion sensors or head odometers
  • the controller may execute an algorithm that performs a three-dimensional (3D) reconstruction using images captured by the eye tracking cameras to generate 3D models of the user's eyes.
  • Signals from the head odometers may be used to detect movement of the device with respect to the user's eyes.
  • 3D reconstruction may be performed only when movement of the device with respect to the user's eyes has been detected, thus significantly reducing power consumption by the eye tracking system.
  • magnitude and direction of the detected motion may be determined, and 3D models of the user's eyes previously generated by the 3D reconstruction method may be adjusted according to the magnitude and direction of the detected motion of the HMD.
  • Embodiments of methods and apparatus for tracking relative movement of the user's eyes with respect to the HMD using eye odometers are also described.
  • sensors referred to herein as eye motion sensors or eye odometers
  • the eye odometers may be used as a low-power component to track relative movement of the user's eyes with respect to the device in intervals between the processing of frames captured by the eye tracking cameras at a frame rate.
  • Embodiments of methods and apparatus for tracking relative movement of the user's eyes with respect to the HMD using low-resolution images are also described.
  • the eye tracking cameras themselves may capture low-resolution frames, for example by binning pixels on the camera sensor or by capturing horizontal and vertical stripes or lines of pixels on the camera sensor rather than entire frames. This low-resolution information may be used to track relative movement of the user's eyes with respect to the device in intervals between the processing of full, high-resolution frames captured by the eye tracking cameras.
  • These eye tracking methods and apparatus may allow the frame rate of the eye tracking cameras to be reduced, for example from 120 frames per second to 10 frames per second or less, and may also allow 3D reconstruction to be performed much less often, thus significantly reducing power consumption by the eye tracking system.
  • the eye tracking methods may be used alone or in combination in various embodiments.
  • Embodiments of HMDs that include both head odometers and eye odometers, or alternatively that include both head odometers and eye tracking cameras that capture low-resolution images to track relative movement of the eyes are described.
  • Embodiments of an eye tracking system for an HMD that include both head odometers and eye odometers, or alternatively both head odometers and eye tracking cameras that capture low-resolution images to track relative movement of the eyes may, for example, further reduce the frequency at which 3D reconstruction is performed, and may also reduce the frequency at which two-dimensional (2D) image processing of frames captured by the eye tracking cameras is performed to further reduce power consumption of the eye tracking system.
  • FIG. 1 illustrates an example VR/AR HMD that implements an eye tracking system, according to some embodiments.
  • FIG. 2 illustrates an example VR/AR HMD that implements an eye tracking system that includes sensors to detect movement of the HMD with respect to the user's eyes, according to some embodiments.
  • FIG. 3 is a flowchart of an eye tracking method in which sensors are used to detect movement of the HMD with respect to the user's eyes and in which 3D reconstruction is performed only when movement of the HMD is detected, according to some embodiments.
  • FIG. 4 is a flowchart of an eye tracking method in which sensors are used to detect movement of the HMD with respect to the user's eyes and in which a 3D model of the eye is adjusted when movement of the HMD is detected, according to some embodiments.
  • FIG. 5 illustrates an example VR/AR HMD that implements an eye tracking system that includes sensors that are used to track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • FIG. 6 is a flowchart of an eye tracking method in which sensors are used to track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • FIG. 7 illustrates an example VR/AR HMD that implements an eye tracking system in which low-resolution frames are captured by the eye tracking cameras and used to track movement of the eyes in intervals between the processing of high-resolution frames captured by the eye tracking cameras, according to some embodiments.
  • FIGS. 8A and 8B illustrate example low-resolution frames captured by the eye tracking cameras, according to some embodiments.
  • FIG. 9 is a flowchart of an eye tracking method in which low-resolution frames are captured by the eye tracking cameras and used to track movement of the eyes in intervals between the processing of high-resolution frames captured by the eye tracking cameras, according to some embodiments.
  • FIG. 10 illustrates an example VR/AR HMD that implements an eye tracking system that includes head odometers that detect movement of the HMD with respect to the user's eyes and eye odometers that track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • FIG. 11 is a flowchart of an eye tracking method in which head odometers are used detect movement of the HMD with respect to the user's eyes and eye odometers are used to track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • FIG. 12 is a block diagram illustrating an example VR/AR system that includes components of an eye tracking system as illustrated in FIGS. 2 through 11 , according to some embodiments.
  • FIG. 13 illustrates an alternative VR/AR device that includes an eye tracking system with head odometers that detect movement of the device with respect to the user's eyes and eye odometers that track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • Configured To Various units, circuits, or other components may be described or claimed as “configured to” perform a task or tasks.
  • “configured to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs those task or tasks during operation. As such, the unit/circuit/component can be said to be configured to perform the task even when the specified unit/circuit/component is not currently operational (e.g., is not on).
  • the units/circuits/components used with the “configured to” language include hardware—for example, circuits, memory storing program instructions executable to implement the operation, etc.
  • a unit/circuit/component is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. ⁇ 112, paragraph (f), for that unit/circuit/component.
  • “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue.
  • “Configure to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.
  • Second “First,” “Second,” etc. As used herein, these terms are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.).
  • a buffer circuit may be described herein as performing write operations for “first” and “second” values.
  • the terms “first” and “second” do not necessarily imply that the first value must be written before the second value.
  • a VR/AR system may include a device such as a headset, helmet, goggles, or glasses (referred to herein as a head-mounted device (HMD)) that includes a display (e.g., left and right displays) for displaying frames including left and right images in front of a user's eyes to thus provide three-dimensional (3D) virtual views to the user.
  • HMD head-mounted device
  • a VR/AR system may also include a controller.
  • the controller may be implemented in the HMD, or alternatively may be implemented at least in part by an external device (e.g., a computing system) that is communicatively coupled to the HMD via a wired or wireless interface.
  • the controller may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), and/or other components for processing and rendering video and/or images.
  • ISPs image signal processors
  • GPUs graphics processing units
  • codecs coder/decoders
  • the controller may render frames (each frame including a left and right image) that include virtual content based at least in part on the inputs obtained from cameras and other sensors on the HMD, and may provide the frames to a projection system of the HMD for display.
  • the VR/AR system may include an eye tracking system (which may also be referred to as a gaze tracking system).
  • an eye tracking system for VR/AR systems are described that include at least one eye tracking camera (e.g., infrared (IR) cameras) positioned at each side of the user's face and configured to capture images of the user's eyes.
  • the eye tracking system may also include a light source (e.g., an IR light source) that emits light (e.g., IR light) towards the user's eyes. A portion of the IR light is reflected off the user's eyes to the eye tracking cameras.
  • IR infrared
  • the eye tracking cameras capture images of the user's eyes from the IR light reflected off the eyes. Images captured by the eye tracking system may be analyzed by the controller to detect features (e.g., pupil), position, and movement of the user's eyes, and/or to detect other information about the eyes such as pupil dilation. For example, the point of gaze on the display estimated from the eye tracking images may enable gaze-based interaction with content shown on the near-eye display of the HMD.
  • Other applications may include, but are not limited to, creation of eye image animations used for avatars in a VR/AR environment.
  • An HMD may be a mobile device with an internal source of power. Thus, reducing power consumption to increase the life of the power source is a concern. Eye tracking systems (both the camera and processing components) consume power. Embodiments of methods and apparatus for providing low-power eye tracking systems are described that may reduce the amount of power consumed by the eye tracking hardware and software components of an HMD.
  • a key to providing accurate eye tracking is knowing the location of the user's eyes with respect to the eye tracking cameras.
  • the controller may execute an algorithm that performs a three-dimensional (3D) reconstruction using images captured by the eye tracking cameras to generate 3D models of the user's eyes.
  • the 3D models of the eyes indicate the 3D position of the eyes with respect to the eye tracking cameras, which allows the eye tracking algorithms executed by the controller to accurately track eye movement.
  • a key element in accurate eye tracking is robustness in regards to device movement on the user's head.
  • the HMD may move on the user's head during use.
  • the user may remove the device and put it back on.
  • an initial calibration of the eye tracking system performed using 3D reconstruction may be invalidated.
  • a solution to this ambiguity is to perform the 3D reconstruction for every frame captured by the eye tracking cameras.
  • 3D reconstruction is expensive computationally, and consumes a significant amount of power.
  • Embodiments of methods and apparatus for tracking relative movement of a device with respect to a user's head are described in which sensors (referred to herein as head motion sensors or head odometers) are placed at one or more positions in or on the device, for example at or near the user's ears to primarily track pitch and at or near the bridge of the nose to primarily track y movement. Signals from the head odometers may be used to detect movement of the device with respect to the user's eyes. This may allow 3D reconstruction to be performed only when movement of the device with respect to the user's eyes has been detected, thus significantly reducing power consumption by the eye tracking system.
  • sensors referred to herein as head motion sensors or head odometers
  • Signals from the head odometers may be used to detect movement of the device with respect to the user's eyes. This may allow 3D reconstruction to be performed only when movement of the device with respect to the user's eyes has been detected, thus significantly reducing power consumption by the eye tracking system.
  • 2D image processing of frames captured by the eye tracking cameras may be performed to track the user's eyes.
  • a 3D reconstruction may be performed periodically (e.g., once a second, or once every N frames) to prevent error/drift accumulation even if movement of the device has not been detected.
  • magnitude and direction of the detected motion may be determined, and 3D models of the user's eyes previously generated by the 3D reconstruction method may be adjusted according to the magnitude and direction of the detected motion of the HMD.
  • a 3D reconstruction may be performed periodically (e.g., once a second, or once every N frames) to prevent error/drift accumulation.
  • sensors e.g., photosensors or photodiodes, referred to herein as eye motion sensors or eye odometers
  • eye motion sensors or eye odometers may be used as a low-power component to track relative movement of the user's eyes with respect to the device in intervals between the processing of frames captured by the eye tracking cameras.
  • the 3D models generated from the images captured by the eye tracking cameras provide absolute gaze information, while the data captured by the eye odometers in intervals between the processing of the camera frames is processed to provide a relative update to the previously known and trusted 3D models.
  • embodiments of methods and apparatus for reducing power consumption of an eye tracking system are described in which the eye tracking cameras may capture low-resolution frames, for example by binning pixels on the camera sensor or by capturing horizontal and vertical stripes or lines of pixels on the camera sensor rather than entire frames (high-resolution frames).
  • This low-resolution information may be used to track relative movement of the user's eyes with respect to the device in intervals between the processing of full, high-resolution frames captured by the eye tracking cameras. This may allow 3D reconstruction to be performed much less often, thus significantly reducing power consumption by the eye tracking system.
  • the eye tracking cameras themselves may be viewed as “eye odometers” when capturing and processing the low-resolution frames.
  • a high-resolution frame captured by a camera sensor contains more pixels than a low-resolution frame captured by the same camera sensor.
  • a high-resolution frame may include most or all pixels that the camera sensor is capable of capturing, while a low-resolution frame may include significantly fewer pixels than the same camera sensor is capable of capturing.
  • a camera sensor may be switched, for example by a controller comprising one or more processors, from a high-resolution mode in which high-resolution frames are captured, to a low-resolution mode in which low-resolution frames are captured, for example by binning pixels on the camera sensor or by capturing horizontal and vertical stripes or lines of pixels on the camera sensor.
  • head odometer methods and eye odometer methods described above may be used alone or in combination.
  • Embodiments of HMDs that include head odometers to reduce power consumption of the eye tracking system are described.
  • Embodiments of HMDs that include eye odometers (either implemented by sensors or by the eye tracking cameras) to reduce power consumption and bandwidth usage of the eye tracking system are also described.
  • embodiments of HMDs that include both head odometers and eye odometers are described.
  • Embodiments of an eye tracking system for an HMD may, for example, further reduce the frequency at which 3D reconstruction is performed, and may also reduce the frequency at which 2D image processing of frames captured by the eye tracking cameras is performed to further reduce power consumption of the eye tracking system.
  • an eye tracking system While embodiments of an eye tracking system are generally described herein as including at least one eye tracking camera positioned at each side of the user's face to track the gaze of both of the user's eyes, an eye tracking system may also be implemented that includes at least one eye tracking camera positioned at only one side of the user's face to track the gaze of only one of the user's eyes.
  • FIG. 1 shows a side view of an example VR/AR HMD 100 that implements an eye tracking system, according to some embodiments.
  • HMD 100 as illustrated in FIG. 1 is given by way of example, and is not intended to be limiting.
  • the shape, size, and other features of an HMD 100 may differ, and the locations, numbers, types, and other features of the components of an HMD 100 may vary.
  • VR/AR HMD 100 may include, but is not limited to, a display 110 and two optical lenses (eyepieces) 120 , mounted in a wearable housing or frame.
  • HMD 100 may be positioned on the user 190 's head such that the display 110 and eyepieces 120 are disposed in front of the user's eyes 192 . The user looks through the eyepieces 120 onto the display 110 .
  • a controller 160 for the VR/AR system may be implemented in the HMD 100 , or alternatively may be implemented at least in part by an external device (e.g., a computing system) that is communicatively coupled to HMD 100 via a wired or wireless interface.
  • Controller 160 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), and/or other components for processing and rendering video and/or images.
  • Controller 160 may render frames (each frame including a left and right image) that include virtual content based at least in part on the inputs obtained from the sensors, and may provide the frames to a projection system of the HMD 100 for display to display 110 .
  • FIG. 12 further illustrates components of an HMD and VR/AR system, according to some embodiments.
  • the eye tracking system may include, but is not limited to, one or more eye tracking cameras 140 and an IR light source 130 .
  • IR light source 130 e.g., IR LEDs
  • At least one eye tracking camera 140 e.g., an IR camera, for example a 400 ⁇ 400 pixel count camera or a 600 ⁇ 600 pixel count camera, that operates at 850 nm or 940 nm, or at some other IR wavelength, and that captures frames at a rate of 60-120 frames per second (FPS)
  • FPS frames per second
  • the eye tracking cameras 140 may be positioned in the HMD 100 on each side of the user 190 's face to provide a direct view of the eyes 192 , a view of the eyes 192 through the eyepieces 120 , or a view of the eyes 192 via reflection off hot mirrors or other reflective components.
  • the location and angle of eye tracking camera 140 is given by way of example, and is not intended to be limiting. While FIG. 1 shows a single eye tracking camera 140 located on each side of the user 190 's face, in some embodiments there may be two or more eye tracking cameras 140 on each side of the user 190 's face.
  • a portion of IR light emitted by light source(s) 130 reflects off the user 190 's eyes and is captured by the eye tracking cameras 140 to image the user's eyes 192 .
  • Images captured by the eye tracking cameras 140 may be analyzed by controller 160 to detect features (e.g., pupil), position, and movement of the user's eyes 192 , and/or to detect other information about the eyes 192 such as pupil dilation.
  • the point of gaze on the display 110 may be estimated from the eye tracking images to enable gaze-based interaction with content shown on the display 110 .
  • the information collected by the eye tracking system may be used to adjust the rendering of images to be projected, and/or to adjust the projection of the images by the projection system of the HMD 100 , based on the direction and angle at which the user 190 's eyes are looking.
  • Embodiments of an HMD 100 with an eye tracking system as illustrated in FIG. 1 may, for example, be used in augmented or mixed (AR) applications to provide augmented or mixed reality views to the user 190 .
  • HMD 100 may include one or more sensors, for example located on external surfaces of the HMD 100 , that collect information about the user 190 's external environment (video, depth information, lighting information, etc.); the sensors may provide the collected information to controller 160 of the VR/AR system.
  • the sensors may include one or more visible light cameras (e.g., RGB video cameras) that capture video of the user's environment that may be used to provide the user 190 with a virtual view of their real environment.
  • visible light cameras e.g., RGB video cameras
  • video streams of the real environment captured by the visible light cameras may be processed by the controller of the HMD 100 to render augmented or mixed reality frames that include virtual content overlaid on the view of the real environment, and the rendered frames may be provided to the projection system of the HMD 100 for display on display 110 .
  • the display 110 emits light in the visible light range and does not emit light in the IR range, and thus does not introduce noise in the eye tracking system.
  • Embodiments of the HMD 100 with an eye tracking system as illustrated in FIG. 1 may also be used in virtual reality (VR) applications to provide VR views to the user 190 .
  • the controller of the HMD 100 may render or obtain virtual reality (VR) frames that include virtual content, and the rendered frames may be provided to the projection system of the HMD 100 for display on display 110 .
  • VR virtual reality
  • a key to providing accurate eye tracking is knowing the location of the user's eyes 192 with respect to the eye tracking cameras 140 .
  • the controller 160 may perform a 3D reconstruction using images captured by the eye tracking cameras 140 to generate 3D models of the user's eyes 192 .
  • the 3D models of the eyes 192 indicate the 3D position of the eyes 192 with respect to the eye tracking cameras 140 which allows the eye tracking algorithms executed by the controller 160 to accurately track eye movement.
  • a key element in accurate eye tracking is robustness in regards to device movement on the user's head.
  • An initial calibration performed using 3D reconstruction may be invalidated by movement or removal of the HMD 190 .
  • FIG. 2 illustrates an example VR/AR HMD 200 that implements an eye tracking system that includes sensors (referred to as head motion sensors or head odometers) to detect movement of the HMD 200 with respect to the user's eyes 292 , according to some embodiments.
  • HMD 200 as illustrated in FIG. 2 is given by way of example, and is not intended to be limiting.
  • the shape, size, and other features of an HMD 200 may differ, and the locations, numbers, types, and other features of the components of an HMD 200 may vary.
  • VR/AR HMD 200 may include, but is not limited to, a display 210 and two eyepieces 220 , mounted in a wearable housing or frame.
  • a controller 260 for the VR/AR system may be implemented in the HMD 200 , or alternatively may be implemented at least in part by an external device that is communicatively coupled to HMD 200 via a wired or wireless interface.
  • Controller 260 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), and/or other components for processing and rendering video and/or images.
  • FIG. 12 further illustrates components of an HMD and VR/AR system, according to some embodiments.
  • the HMD 200 may include an eye tracking system that includes, but is not limited to, one or more eye tracking cameras 240 and an IR light source 230 .
  • IR light source 230 e.g., IR LEDs
  • At least one eye tracking camera 240 (e.g., an IR camera, for example a 400 ⁇ 400 pixel count camera or a 600 ⁇ 600 pixel count camera, that operates at 850 nm or 940 nm, or at some other IR wavelength, and that captures frames at a rate of 60-120 frames per second (FPS)) is located at each side of the user 290 's face.
  • the eye tracking cameras 240 may be positioned in the HMD 200 on each side of the user 290 's face to provide a direct view of the eyes 292 , a view of the eyes 292 through the eyepieces 220 , or a view of the eyes 292 via reflection off hot mirrors or other reflective components.
  • FIG. 2 shows a single eye tracking camera 240 located on each side of the user 290 's face, in some embodiments there may be two or more eye tracking cameras 240 on each side of the user 290 's face.
  • the eye tracking system may also include sensors 242 (referred to herein as head motion sensors or head odometers) placed at one or more positions in or on the device, for example at or near the user's ears ( 242 A) to primarily track pitch and at or near the bridge of the nose ( 242 B) to primarily track y movement.
  • Sensors from the head odometers 242 A and 242 B may be used to detect movement of the HMD 200 with respect to the user's eyes. This may allow 3D reconstruction to be performed only when movement of the HMD 200 with respect to the user's eyes has been detected, thus significantly reducing power consumption by the eye tracking system.
  • 2D image processing of frames captured by the eye tracking cameras 240 may be performed to track the user's eyes 292 .
  • a 3D reconstruction may be performed periodically (e.g., once a second, or once every N frames) to prevent error/drift accumulation even if movement of the HMD 200 has not been detected.
  • magnitude and direction of the detected motion may be determined, and 3D models of the user's eyes 292 previously generated by the 3D reconstruction method may be adjusted according to the magnitude and direction of the detected motion of the HMD 200 .
  • a 3D reconstruction may be performed periodically (e.g., once a second, or once every N frames) to prevent error/drift accumulation.
  • Example sensors Examples of different technologies that may be used to implement head odometers 242 in an eye tracking system are given below in the section titled Example sensors.
  • FIG. 3 is a flowchart of an eye tracking method in which sensors are used to detect movement of the HMD with respect to the user's eyes and in which 3D reconstruction is performed only when movement of the HMD is detected, according to some embodiments.
  • the method of FIG. 3 may, for example, be performed in a VR/AR system as illustrated in FIG. 2 .
  • a 3D reconstruction method may be performed to generate 3D models of the user's eyes using frame(s) captured by the eye tracking cameras.
  • the 3D reconstruction may be performed at initialization/calibration of the HMD.
  • the 3D models of the eyes indicate the 3D position of the eyes with respect to the eye tracking cameras, which allows the eye tracking algorithms executed by the controller to accurately track eye movement with respect to the HMD.
  • 2D image processing of frames captured by the eye tracking cameras may be performed to track movement of the user's eyes with respect to the HMD.
  • the eye movement tracked by the 2D image processing may, for example, be used to determine the point of gaze on the display of the HMD.
  • relative movement of the HMD with respect to the user's head may be tracked using sensors on the HMD.
  • a key element in accurate eye tracking is robustness in regards to device movement on the user's head. If the HMD moves on the user's head during use, the initial calibration of the eye tracking system performed at element 310 using 3D reconstruction may be invalidated. However, there is an inherent ambiguity in an eye tracking system between whether the user's eyes move with respect to the cameras/HMD or whether the cameras/HMD move with respect to the user's eyes.
  • sensors referred to as head motion sensors or head odometers
  • Signals from the head odometers may be processed by the controller to track movement of the HMD with respect to the user's eyes.
  • the eye tracking method may continue to perform 2D image processing to track movement of the user's eyes with respect to the HMD as indicated at 320 .
  • the method returns to element 310 to again perform the 3D reconstruction method to generate new 3D models of the user's eyes using frame(s) captured by the eye tracking cameras, which allows the eye tracking algorithms executed by the controller to continue to accurately track eye movement with respect to the HMD.
  • the method of FIG. 3 may allow 3D reconstruction to be performed only when movement of the HMD with respect to the user's eyes has been detected, thus significantly reducing power consumption by the eye tracking system.
  • 2D image processing of frames captured by the eye tracking cameras (which is much less expensive computationally than 3D image processing) may be performed to track the user's eyes.
  • a 3D reconstruction may be performed periodically (e.g., once a second, or once every N frames) to prevent error/drift accumulation even if movement of the HMD with respect to the user's eyes has not been detected.
  • FIG. 4 is a flowchart of an eye tracking method in which sensors are used to detect movement of the HMD with respect to the user's eyes and in which a 3D model of the eye is adjusted when movement of the HMD is detected, according to some embodiments.
  • the method of FIG. 4 may, for example, be performed in a VR/AR system as illustrated in FIG. 2 .
  • 2D image processing of frames captured by the eye tracking cameras may be performed to track movement of the user's eyes with respect to the HMD.
  • the eye movement tracked by the 2D image processing may, for example, be used to determine the point of gaze on the display of the HMD.
  • relative movement of the HMD with respect to the user's head may be tracked using sensors on the HMD.
  • sensors referred to as head motion sensors or head odometers
  • signals from the head odometers may be processed by the controller to track movement of the HMD with respect to the user's eyes.
  • the eye tracking method may continue to perform 2D image processing to track movement of the user's eyes with respect to the HMD as indicated at 420 .
  • the controller detects movement of the HMD with respect to the user's eyes, then as indicated at 450 the 3D models generated at 410 may be adjusted by the controller, and the method may continue to perform 2D image processing to track movement of the user's eyes with respect to the HMD as indicated at 420 .
  • Magnitude and direction of the movement may be determined from the head odometer signals.
  • a 3D reconstruction may be performed periodically (e.g., once a second, or once every N frames) to recalibrate the eye tracking system.
  • the method of FIG. 4 may allow 3D reconstruction to be performed less frequently than the method of FIG. 3 , thus further reducing power consumption by the eye tracking system.
  • 2D image processing of frames captured by the eye tracking cameras (which is much less expensive computationally than 3D image processing) may be performed to track the user's eyes.
  • the 3D models can be adjusted without requiring an expensive 3D reconstruction.
  • a 3D reconstruction may need to be performed only occasionally to prevent error/drift accumulation.
  • FIG. 5 illustrates an example VR/AR HMD that implements an eye tracking system that includes sensors that are used to track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • HMD 500 as illustrated in FIG. 5 is given by way of example, and is not intended to be limiting. In various embodiments, the shape, size, and other features of an HMD 500 may differ, and the locations, numbers, types, and other features of the components of an HMD 500 may vary.
  • VR/AR HMD 500 may include, but is not limited to, a display 510 and two eyepieces 520 , mounted in a wearable housing or frame.
  • a controller 560 for the VR/AR system may be implemented in the HMD 500 , or alternatively may be implemented at least in part by an external device that is communicatively coupled to HMD 500 via a wired or wireless interface.
  • Controller 560 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), and/or other components for processing and rendering video and/or images.
  • FIG. 12 further illustrates components of an HMD and VR/AR system, according to some embodiments.
  • the HMD 500 may include an eye tracking system that includes, but is not limited to, one or more eye tracking cameras 540 and an IR light source 530 .
  • IR light source 530 e.g., IR LEDs
  • At least one eye tracking camera 540 (e.g., an IR camera, for example a 400 ⁇ 400 pixel count camera or a 600 ⁇ 600 pixel count camera, that operates at 850 nm or 940 nm, or at some other IR wavelength, and that may be capable of capturing frames at a rate of 60-120 frames per second (FPS)) is located at each side of the user 590 's face.
  • the eye tracking cameras 540 may be positioned in the HMD 500 on each side of the user 590 's face to provide a direct view of the eyes 592 , a view of the eyes 592 through the eyepieces 520 , or a view of the eyes 592 via reflection off hot mirrors or other reflective components.
  • FIG. 5 shows a single eye tracking camera 540 located on each side of the user 590 's face, in some embodiments there may be two or more eye tracking cameras 540 on each side of the user 590 's face.
  • the eye tracking system may also include sensors 544 (e.g., photosensors or photodiodes, referred to herein as eye motion sensors or eye odometers) placed at one or more positions in the HMD 500 to augment the eye tracking cameras 540 .
  • the eye odometers 544 may be used as a low-power component to track relative movement of the user's eyes 592 with respect to the HMD 500 in intervals between the processing of frames captured by the cameras 540 to generate 3D models of the user's eyes.
  • the 3D models generated from the images captured by the eye tracking cameras 540 provide absolute gaze information at the frame rate of the cameras, while the data captured by the eye odometers 544 is processed in intervals between the processing of the camera frames to provide relative updates to the 3D models generated from the captured images.
  • This may allow the frame rate of the eye tracking cameras 540 to be reduced, for example from 120 frames per second to 10 frames per second or less, and may also allow 3D reconstruction based on the captured frames to be performed much less often (e.g., 10 times a second or less), thus significantly reducing power consumption by the eye tracking system.
  • Reducing the frame rate of the eye tracking cameras 540 by augmenting eye tracking with the eye odometers 544 may also significantly reduce bandwidth usage and latency of the eye tracking system. Examples of different technologies that may be used to implement eye odometers 544 in an eye tracking system are given below in the section titled Example sensors.
  • FIG. 6 is a flowchart of an eye tracking method in which sensors are used to track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • the method of FIG. 6 may, for example, be performed in a VR/AR system as illustrated in FIG. 5 .
  • one or more frames may be captured by the eye tracking cameras.
  • a 3D reconstruction method may be performed to generate 3D models of the user's eyes using the frame(s) captured by the eye tracking cameras.
  • the 3D reconstruction may be performed at initialization/calibration of the HMD.
  • the 3D models of the eyes indicate the 3D position of the eyes with respect to the eye tracking cameras, which allows the eye tracking algorithms executed by the controller to accurately track eye movement with respect to the HMD.
  • relative movement of the user's eyes with respect to the HMD may be tracked using data captured by and collected from the eye odometers (e.g., photosensors or photodiodes).
  • the eye tracking system may continue to track the user's eyes using the data from the eye odometers until movement of the HMD with respect to the user's head is detected, a time limit elapses or a confidence threshold in the eye odometer data is exceeded.
  • the method returns to element 610 to capture one or more frames using the eye tracking cameras and perform the 3D reconstruction method 620 to generate new 3D models of the user's eyes using frame(s) captured by the eye tracking cameras. The method then resumes tracking the eyes using the eye odometer information at 630 .
  • FIG. 7 illustrates an example VR/AR HMD that implements an eye tracking system in which low-resolution frames are captured by the eye tracking cameras and used to track movement of the eyes in intervals between the processing of high-resolution frames captured by the eye tracking cameras, according to some embodiments.
  • HMD 700 as illustrated in FIG. 7 is given by way of example, and is not intended to be limiting. In various embodiments, the shape, size, and other features of an HMD 700 may differ, and the locations, numbers, types, and other features of the components of an HMD 700 may vary.
  • VR/AR HMD 700 may include, but is not limited to, a display 710 and two eyepieces 720 , mounted in a wearable housing or frame.
  • a controller 760 for the VR/AR system may be implemented in the HMD 700 , or alternatively may be implemented at least in part by an external device that is communicatively coupled to HMD 700 via a wired or wireless interface.
  • Controller 760 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), and/or other components for processing and rendering video and/or images.
  • FIG. 12 further illustrates components of an HMD and VR/AR system, according to some embodiments.
  • the HMD 700 may include an eye tracking system that includes, but is not limited to, one or more eye tracking cameras 740 and an IR light source 730 .
  • IR light source 730 e.g., IR LEDs
  • At least one eye tracking camera 740 (e.g., an IR camera, for example a 400 ⁇ 400 pixel count camera or a 600 ⁇ 600 pixel count camera, that operates at 850 nm or 940 nm, or at some other IR wavelength, and that captures frames at a rate of 60-120 frames per second (FPS)) is located at each side of the user 790 's face.
  • the eye tracking cameras 740 may be positioned in the HMD 700 on each side of the user 790 's face to provide a direct view of the eyes 792 , a view of the eyes 792 through the eyepieces 720 , or a view of the eyes 792 via reflection off hot mirrors or other reflective components.
  • FIG. 7 shows a single eye tracking camera 740 located on each side of the user 790 's face, in some embodiments there may be two or more eye tracking cameras 740 on each side of the user 790 's face.
  • the eye tracking cameras 740 may capture low-resolution frames, for example by binning pixels on the camera sensor as shown in FIG. 8A or by capturing horizontal and vertical stripes of pixels on the camera sensor rather than entire frames as shown in FIG. 8B . In these embodiments, the cameras 740 may operate in high-resolution mode or in low-resolution mode.
  • the controller 760 may signal the cameras 740 to switch to low-resolution mode during or after processing one or more high-resolution frames to generate 3D models of the eyes 792 , and signal the cameras 740 to switch to high-resolution mode, for example upon detecting movement of the HMD 700 with respect to the user's head 790 .
  • the low-resolution images may be used to track relative movement of the user's eyes 792 with respect to the HMD 700 in intervals between the processing of full, high-resolution frames captured by the eye tracking cameras by the controller 760 . This may allow high-resolution frames to be captured much less often, and also 3D reconstruction to be performed much less often, thus significantly reducing power consumption by the cameras 740 and controller 760 in the eye tracking system.
  • the eye tracking cameras 740 themselves may thus be viewed as “eye odometers” when capturing and processing the low-resolution frames in intervals between the capturing and processing of high-resolution frames.
  • FIGS. 8A and 8B illustrate non-limiting example low-resolution frames that may be captured by an eye tracking camera 740 of FIG. 7 , according to some embodiments.
  • FIG. 8A shows an example of a frame at 640 ⁇ 640 resolution (400k pixels) in high-resolution mode. In low-resolution mode, the camera 740 bins 32 ⁇ 32 blocks of pixels into one pixel, thus resulting in 20 ⁇ 20 resolution (400 pixels) that may be processed in 2D to track movement of the user's eyes in intervals between capturing and processing full, high-resolution frames in 3D.
  • FIG. 8B shows an example in which horizontal and vertical lines of pixels are captured rather than an entire frame in low-resolution mode. The captured lines may be processed in 2D to track movement of the user's eyes in intervals between capturing and processing full, high-resolution frames in 3D.
  • FIG. 9 is a flowchart of an eye tracking method in which low-resolution frames are captured by the eye tracking cameras and used to track movement of the eyes in intervals between the processing of high-resolution frames captured by the eye tracking cameras, according to some embodiments.
  • the method of FIG. 9 may, for example, be performed in a VR/AR system as illustrated in FIG. 7 .
  • one or more high-resolution frames may be captured by the eye tracking cameras.
  • a 3D reconstruction method may be performed to generate 3D models of the user's eyes using the high-resolution frame(s) captured by the eye tracking cameras.
  • the 3D reconstruction may be performed at initialization/calibration of the HMD.
  • the 3D models of the eyes indicate the 3D position of the eyes with respect to the eye tracking cameras, which allows the eye tracking algorithms executed by the controller to accurately track eye movement with respect to the HMD.
  • the eye tracking cameras may switch to low-resolution mode to capture low-resolution frames, for example by binning pixels on the camera sensor as illustrated in FIG. 8A or by capturing horizontal and vertical lines of pixels on the camera sensor as illustrated in FIG. 8B .
  • the controller may signal the eye tracking cameras to switch to low-resolution mode during or after processing captured high-resolution frames(s) to generate 3D models of the eyes.
  • movement of the user's eyes with respect to the HMD may be tracked using the low-resolution images captured in low-resolution mode.
  • the eye tracking system may continue to capture low-resolution images and track the user's eyes with respect to the HMD using the low-resolution information until movement of the HMD with respect to the user's head is detected, a time limit elapses, or a confidence threshold is exceeded.
  • the method returns to element 910 to capture one or more high-resolution frames using the eye tracking cameras and perform the 3D reconstruction method 920 to generate new 3D models of the user's eyes using frame(s) captured by the eye tracking cameras.
  • the controller may signal the eye tracking cameras to switch to high-resolution mode upon detecting movement of the HMD with respect to the user's head, determining that the time limit has elapsed, or determining that the confidence threshold has been exceeded.
  • the eye tracking system then resumes capturing low-resolution images at 930 and tracking the eyes using the low-resolution images at 940 .
  • head odometer methods and eye odometer methods described above in reference to FIGS. 2 through 9 may be used in combination.
  • Embodiments of HMDs that include both head odometers and eye odometers are described in reference to FIGS. 10 and 11 .
  • FIG. 10 illustrates an example VR/AR HMD that implements an eye tracking system that includes head odometers that detect movement of the HMD with respect to the user's eyes and eye odometers that track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • HMD 1000 as illustrated in FIG. 10 is given by way of example, and is not intended to be limiting. In various embodiments, the shape, size, and other features of an HMD 1000 may differ, and the locations, numbers, types, and other features of the components of an HMD 1000 may vary.
  • VR/AR HMD 1000 may include, but is not limited to, a display 1010 and two eyepieces 1020 , mounted in a wearable housing or frame.
  • a controller 1060 for the VR/AR system may be implemented in the HMD 1000 , or alternatively may be implemented at least in part by an external device that is communicatively coupled to HMD 1000 via a wired or wireless interface.
  • Controller 1060 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), and/or other components for processing and rendering video and/or images.
  • FIG. 12 further illustrates components of an HMD and VR/AR system, according to some embodiments.
  • the HMD 1000 may include an eye tracking system that includes, but is not limited to, one or more eye tracking cameras 1040 and an IR light source 1030 .
  • IR light source 1030 e.g., IR LEDs
  • At least one eye tracking camera 1040 (e.g., an IR camera, for example a 400 ⁇ 400 pixel count camera or a 600 ⁇ 600 pixel count camera, that operates at 850 nm or 940 nm, or at some other IR wavelength, and that captures frames at a rate of 60-120 frames per second (FPS)) is located at each side of the user 1090 's face.
  • the eye tracking cameras 1040 may be positioned in the HMD 1000 on each side of the user 1090 's face to provide a direct view of the eyes 1092 , a view of the eyes 1092 through the eyepieces 1020 , or a view of the eyes 1092 via reflection off hot mirrors or other reflective components.
  • FIG. 10 shows a single eye tracking camera 1040 located on each side of the user 1090 's face, in some embodiments there may be two or more eye tracking cameras 1040 on each side of the user 1090 's face.
  • the eye tracking system may also include head odometers 1042 placed at one or more positions in or on the device, for example at or near the user's ears ( 1042 A) to primarily track pitch and at or near the bridge of the nose ( 1042 B) to primarily track y movement.
  • Signals from the head odometers 1042 A and 1042 B may be processed by controller 1060 to detect movement of the HMD 1000 with respect to the user's eyes. This may allow 3D reconstruction to be performed by controller 1060 only when movement of the HMD 1000 with respect to the user's eyes has been detected, thus significantly reducing power consumption by the eye tracking system.
  • magnitude and direction of the detected motion may be determined by controller 1060 , and 3D models of the user's eyes 1092 previously generated by the 3D reconstruction method may be adjusted by controller 1060 according to the magnitude and direction of the detected motion of the HMD 1000 .
  • the eye tracking system may also include eye odometers 1044 placed at one or more positions in the HMD 1000 to augment the eye tracking cameras 1040 .
  • the eye odometers 1044 may be used as a low-power component to track relative movement of the user's eyes 1092 with respect to the HMD 500 in intervals between the processing of frames captured by the cameras 1040 .
  • the 3D models generated by controller 1060 from the images captured by the eye tracking cameras 1040 provide absolute gaze information, while the data captured by the eye odometers 1044 in intervals between the processing of camera frames is processed by controller 1060 to provide a relative update to the previously known and trusted 3D models.
  • the eye tracking cameras 1040 may capture low-resolution frames in intervals between capturing high-resolution frames, for example by binning pixels on the camera sensor or by capturing horizontal and vertical stripes or lines of pixels on the camera sensor rather than entire frames.
  • This low-resolution information may be processed by controller 1060 to track relative movement of the user's eyes 1092 with respect to the HMD 1000 in intervals between full, high-resolution frames captured by the eye tracking cameras 1040 and processed by the controller 1060 .
  • FIG. 11 is a flowchart of an eye tracking method in which head odometers are used detect movement of the HMD with respect to the user's eyes and eye odometers are used to track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras, according to some embodiments.
  • the method of FIG. 11 may, for example, be performed in a VR/AR system as illustrated in FIG. 10 .
  • one or more frames may be captured by the eye tracking cameras.
  • a 3D reconstruction method may be performed to generate 3D models of the user's eyes using the frame(s) captured by the eye tracking cameras.
  • the 3D reconstruction may be performed at initialization/calibration of the HMD.
  • the 3D models of the eyes indicate the 3D position of the eyes with respect to the eye tracking cameras, which allows the eye tracking algorithms executed by the controller to accurately track eye movement with respect to the HMD.
  • movement of the user's eyes with respect to the HMD may be tracked in 2D space using data captured by and collected from the eye odometers on the HMD.
  • relative movement of the HMD with respect to the user's head may be tracked using head odometers on the HMD.
  • the method returns to element 1110 to again perform the 3D reconstruction method to generate new 3D models of the user's eyes using frame(s) captured by the eye tracking cameras.
  • magnitude and direction of the movement may be determined from the head odometer signals. This information may then be used by the controller to adjust the 3D models, which allows the eye tracking algorithms executed by the controller to continue to accurately track eye movement with respect to the HMD in 2D space without requiring 3D reconstruction to be performed.
  • the eye tracking method may continue at 1130 to track movement of the user's eyes with respect to the HMD using data from the eye odometers and at 1140 to track movement of the HMD with respect to the HMD using data from the head odometers.
  • the eye tracking method may continue at 1130 to track movement of the user's eyes with respect to the HMD using data from the eye odometers and at 1140 to track movement of the HMD with respect to the HMD using data from the head odometers.
  • the method returns to element 1110 to again perform the 3D reconstruction method to generate new 3D models of the user's eyes using frame(s) captured by the eye tracking cameras.
  • Embodiments of an eye tracking system for an HMD may, for example, further reduce the frequency at which 3D reconstruction is performed, and may also reduce the frequency at which 2D image processing of frames captured by the eye tracking cameras is performed to further reduce power consumption of the eye tracking system.
  • the following provides non-limiting examples of different technologies that may be used to implement head motion and eye motion sensors in an eye tracking system as illustrated in FIGS. 2 through 13 .
  • These technologies can be broken into three broad categories: drift compensation (1D) technologies, odometry (2D) technologies, and absolute tracking (5D) technologies.
  • Drift compensation technologies may work by contact, and may include odometry sensors in the HMD, for example in a nose pad of the HMD that make contact on the bridge of the user's nose and at the temple area of the HMD that make contact near the user's ears.
  • the odometry sensors may measure force.
  • the sensors may be capacitive sensors.
  • accelerometers with a smart baseline may be used.
  • Odometry technologies may be broken into five categories: electric, optical, acoustic, radar, and pressure technologies.
  • Electric odometry technologies may include, but are not limited to, EOG/EMG, ECG, and capacity sensor (waveguide or triangulation) technologies.
  • Optical odometry technologies may include, but are not limited to, photosensor, photodiode array, proximity sensor, and scanner (TOF, intensity, or phase) technologies.
  • Acoustic odometry technologies may include, but are not limited to, sonar (continuous wave, TOF, or resonance) technologies.
  • Radar odometry technologies may include, but are not limited to, continuous wave or TOF radar technologies.
  • Pressure odometry technologies may include, but are not limited to, an IMU (inertial measurement unit) or other sensor that senses pressure change from eye movement.
  • IMU inertial measurement unit
  • Absolute tracking technologies may be broken into two categories: optical and sonar technologies.
  • Optical absolute tracking technologies may include, but are not limited to, single pixel TOF sensor technology or various camera technologies.
  • the camera technologies may include, but are not limited to, DVS, CIS (RGB+IR), compressed sensing, skipping/binning (e.g., slit camera), and thermal imaging (short, medium, long, pyro-electric, or thermo-electric) technologies.
  • Sonar absolute tracking technologies may include, but are not limited to, transceiver array, line of sight, and non-line of sight sonar technologies.
  • FIG. 12 is a block diagram illustrating an example VR/AR system that includes components of an eye tracking system as illustrated in FIGS. 2 through 11 , according to some embodiments.
  • a VR/AR system may include an HMD 2000 such as a headset, helmet, goggles, or glasses.
  • HMD 2000 may implement any of various types of virtual reality projector technologies.
  • the HMD 2000 may include a VR projection system that includes a projector 2020 that displays frames including left and right images on screens or displays 2022 A and 2022 B that are viewed by a user through eyepieces 2220 A and 2220 B.
  • the VR projection system may, for example, be a DLP (digital light processing), LCD (liquid crystal display), or LCoS (liquid crystal on silicon) technology projection system.
  • DLP digital light processing
  • LCD liquid crystal display
  • LCoS liquid crystal on silicon
  • objects at different depths or distances in the two images may be shifted left or right as a function of the triangulation of distance, with nearer objects shifted more than more distant objects.
  • 3D three-dimensional
  • HMD 2000 may include a controller 2030 configured to implement functionality of the VR/AR system and to generate frames (each frame including a left and right image) that are displayed by the projector 2020 .
  • HMD 2000 may also include a memory 2032 configured to store software (code 2034 ) of the VR/AR system that is executable by the controller 2030 , as well as data 2038 that may be used by the VR/AR system when executing on the controller 2030 .
  • HMD 2000 may also include one or more interfaces (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.
  • External device 2100 may be or may include any type of computing system or computing device, such as a desktop computer, notebook or laptop computer, pad or tablet device, smartphone, hand-held computing device, game controller, game system, and so on.
  • controller 2030 may be a uniprocessor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or another suitable number). Controller 2030 may include central processing units (CPUs) configured to implement any suitable instruction set architecture, and may be configured to execute instructions defined in that instruction set architecture. For example, in various embodiments controller 2030 may include general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of the processors may commonly, but not necessarily, implement the same ISA.
  • ISAs instruction set architectures
  • each of the processors may commonly, but not necessarily, implement the same ISA.
  • Controller 2030 may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, out of order, in order, speculative, non-speculative, etc., or combinations thereof. Controller 2030 may include circuitry to implement microcoding techniques. Controller 2030 may include one or more processing cores each configured to execute instructions. Controller 2030 may include one or more levels of caches, which may employ any size and any configuration (set associative, direct mapped, etc.). In some embodiments, controller 2030 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuitry. Generally, a GPU may be configured to render objects to be displayed into a frame buffer (e.g., one that includes pixel data for an entire frame).
  • GPU graphics processing unit
  • a GPU may include one or more graphics processors that may execute graphics software to perform a part or all of the graphics operation, or hardware acceleration of certain graphics operations.
  • controller 2030 may include one or more other components for processing and rendering video and/or images, for example image signal processors (ISPs), coder/decoders (codecs), etc.
  • ISPs image signal processors
  • codecs coder/decoders
  • Memory 2032 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such as mDDR3, etc., or low power versions of the SDRAMs such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc.
  • DRAM dynamic random access memory
  • SDRAM synchronous DRAM
  • DDR double data rate
  • DDR double data rate
  • RDRAM RAMBUS DRAM
  • SRAM static RAM
  • one or more memory devices may be coupled onto a circuit board to form memory modules such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc.
  • the devices may be mounted with an integrated circuit implementing system in a chip-on-chip configuration, a package-on-package configuration, or a multi-chip module configuration.
  • the HMD 2000 may include one or more sensors 2050 that collect information about the user's environment (video, depth information, lighting information, etc.).
  • the sensors 2050 may provide the information to the controller 2030 of the VR/AR system.
  • sensors 2050 may include, but are not limited to, visible light cameras (e.g., video cameras).
  • HMD 2000 may be positioned on the user's head such that the displays 2022 A and 2022 B and eyepieces 2220 A and 2220 B are disposed in front of the user's eyes 2292 A and 2292 B.
  • IR light sources 2230 A and 2230 B e.g., IR LEDs
  • HMD 2000 may be positioned in the HMD 2000 (e.g., around the eyepieces 2220 A and 2220 B, or elsewhere in the HMD 2000 ) to illuminate the user's eyes 2292 A and 2292 B with IR light.
  • Eye tracking cameras 2240 A and 2240 B are located at each side of the user's face.
  • the eye tracking cameras 2240 may be positioned in the HMD 2000 to provide a direct view of the eyes 2292 , a view of the eyes 2292 through the eyepieces 2220 , or a view of the eyes 2292 via reflection off hot mirrors or other reflective components.
  • eye tracking cameras 2240 A and 2240 B are given by way of example, and is not intended to be limiting. In some embodiments, there may be a single eye tracking camera 2240 located on each side of the user's face. In some embodiments there may be two or more eye tracking cameras 2240 on each side of the user's face. For example, in some embodiments, a wide-angle camera 2240 and a narrower-angle camera 2240 may be used on each side of the user's face.
  • Eye tracking information captured by the cameras 2240 A and 2240 B may be provided to the controller 2030 .
  • the controller 2030 may analyze the eye tracking information (e.g., images of the user's eyes 2292 A and 2292 B) to determine eye position and movement and/or other features of the eyes 2292 A and 2292 B.
  • the controller 2030 may perform a 3D reconstruction using images captured by the eye tracking cameras 2240 A and 2240 B to generate 3D models of the user's eyes 2292 A and 2292 B.
  • the 3D models of the eyes 2292 A and 2292 B indicate the 3D position of the eyes 2292 A and 2292 B with respect to the eye tracking cameras 2240 A and 2240 , which allows the eye tracking algorithms executed by the controller to accurately track eye movement.
  • HMD 2000 may include sensors 2242 (referred to herein as head motion sensors or head odometers) placed at one or more positions in or on the device, for example at or near the user's ears ( 2242 A) to primarily track pitch and at or near the bridge of the nose ( 2242 B) to primarily track y movement. Signals from the head odometers 2242 A and 2242 B may be used to detect movement of the HMD 2000 with respect to the user's eyes 2292 A and 2292 B. This may allow 3D reconstruction to be performed only when movement of the HMD 2000 with respect to the user's eyes 2292 A and 2292 B has been detected, thus significantly reducing power consumption by the eye tracking system.
  • sensors 2242 referred to herein as head motion sensors or head odometers
  • 2D image processing of frames captured by the eye tracking cameras 2240 A and 2240 B may be performed to track the user's eyes 2292 A and 2292 B.
  • a 3D reconstruction may be performed periodically (e.g., once a second, or once every N frames) to prevent error/drift accumulation even if movement of the HMD 2000 has not been detected.
  • magnitude and direction of the detected motion may be determined, and 3D models of the user's eyes 2292 A and 2292 B that were previously generated by the 3D reconstruction method may be adjusted according to the magnitude and direction of the detected motion of the HMD 2000 .
  • a 3D reconstruction may be performed periodically (e.g., once a second, or once every N frames) to prevent error/drift accumulation.
  • HMD 2000 may include sensors 2244 (e.g., photosensors or photodiodes, referred to herein as eye motion sensors or eye odometers) placed at one or more positions in the device to augment the eye tracking cameras 2240 A and 2240 B.
  • the eye odometers 2244 A and 2244 B may be used as a low-power component to track relative movement of the user's eyes 2292 A and 2292 B with respect to the cameras 2240 in intervals between the processing of frames captured by the cameras 2240 A and 2240 B.
  • the eye tracking cameras 2240 A and 2240 B may capture lower-resolution frames, for example by binning pixels or by capturing horizontal and vertical stripes of pixels, rather than entire high-resolution frames.
  • the low-resolution frames captured by cameras 2240 A and 2240 B may be used to track relative movement of the user's eyes 2292 A and 2292 B with respect to the cameras 2240 A and 2240 B in intervals between the processing of full, high-resolution frames captured by the eye tracking cameras 2240 A and 2240 B. This may allow 3D reconstruction to be performed much less often, thus significantly reducing power consumption by the eye tracking system.
  • the eye tracking cameras 2240 A and 2240 B themselves may be viewed as “eye odometers” when capturing and processing the lower-resolution frames.
  • Embodiments of an HMD 2000 may include head odometers 2242 to reduce power consumption of the eye tracking system.
  • Embodiments of an HMD 2000 may include eye odometers 2244 (either implemented by sensors or by the eye tracking cameras) to reduce power consumption and bandwidth usage of the eye tracking system.
  • embodiments of an HMD 2000 may include both head odometers 2242 and eye odometers 2244 .
  • Embodiments that include both head odometers 2242 and eye odometers 2244 may, for example, further reduce the frequency at which 3D reconstruction is performed, and may also reduce the frequency at which 2D image processing of frames captured by the eye tracking cameras 2240 is performed to further reduce power consumption of the eye tracking system.
  • the eye tracking information obtained and analyzed by the controller 2030 may be used by the controller in performing various VR or AR system functions.
  • the point of gaze on the displays 2022 A and 2022 B may be estimated from images captured by the eye tracking cameras 2240 A and 2240 B; the estimated point of gaze may, for example, enable gaze-based interaction with content shown on the displays 2022 A and 2022 B.
  • Other applications of the eye tracking information may include, but are not limited to, creation of eye image animations used for avatars in a VR or AR environment.
  • the information obtained from the eye tracking cameras 2240 A and 2240 B may be used to adjust the rendering of images to be projected, and/or to adjust the projection of the images by the projector 2020 of the HMD 2000 , based on the direction and angle at which the user's eyes are looking.
  • brightness of the projected images may be modulated based on the user's pupil dilation as determined by the eye tracking system.
  • the HMD 2000 may be configured to render and display frames to provide an augmented or mixed reality (AR) view for the user at least in part according to sensor 2050 inputs.
  • the AR view may include renderings of the user's environment, including renderings of real objects in the user's environment, based on video captured by one or more video cameras that capture high-quality, high-resolution video of the user's environment for display.
  • the AR view may also include virtual content (e.g., virtual objects, virtual tags for real objects, avatars of the user, etc.) generated by VR/AR system and composited with the projected view of the user's real environment.
  • Embodiments of the HMD 2000 as illustrated in FIG. 12 may also be used in virtual reality (VR) applications to provide VR views to the user.
  • the controller 2030 of the HMD 2000 may render or obtain virtual reality (VR) frames that include virtual content, and the rendered frames may be provided to the projector 2020 of the HMD 2000 for display to displays 2022 A and 2022 B.
  • VR virtual reality
  • FIG. 13 illustrates an example alternative VR/AR device 3000 that includes an eye tracking system with head odometers 3042 that detect movement of the device 3000 with respect to the user's eyes 3092 and eye odometers 3044 that track movement of the eyes 3092 in intervals between the processing of frames captured by the eye tracking cameras 3040 , according to some embodiments. Note that device 3000 as illustrated in FIG.
  • the device 3000 is given by way of example, and is not intended to be limiting. In various embodiments, the shape, size, and other features of the device 3000 may differ, and the locations, numbers, types, and other features of the components of the device 3000 may vary. In this device 3000 , rather than having display panels that the user views through eyepieces, the device 3000 includes lenses 3010 that allow light from the outside environment to pass, and that are also configured to reflect light emitted by a projector/controller 3080 representing virtual content towards the user's eyes 3092 , for example using a holographic film.
  • Device 3000 may include an eye tracking system that, in addition to eye tracking cameras 3040 , includes head odometers 3042 that detect movement of the device 3000 with respect to the user's eyes 3092 as described herein, eye odometers 3044 that track movement of the eyes 3092 in intervals between the processing of frames captured by the eye tracking cameras 3040 as described herein, or both head odometers 3042 and eye odometers 3044 as described herein.
  • eye tracking system that, in addition to eye tracking cameras 3040 , includes head odometers 3042 that detect movement of the device 3000 with respect to the user's eyes 3092 as described herein, eye odometers 3044 that track movement of the eyes 3092 in intervals between the processing of frames captured by the eye tracking cameras 3040 as described herein, or both head odometers 3042 and eye odometers 3044 as described herein.
  • the methods described herein may be implemented in software, hardware, or a combination thereof, in different embodiments.
  • the order of the blocks of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.
  • Various modifications and changes may be made as would be obvious to a person skilled in the art having the benefit of this disclosure.
  • the various embodiments described herein are meant to be illustrative and not limiting. Many variations, modifications, additions, and improvements are possible. Accordingly, plural instances may be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Human Computer Interaction (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Ophthalmology & Optometry (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)
US17/021,901 2019-09-18 2020-09-15 Low-power eye tracking system Active US11307659B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/021,901 US11307659B2 (en) 2019-09-18 2020-09-15 Low-power eye tracking system
CN202010986395.2A CN112527099A (zh) 2019-09-18 2020-09-18 低功率眼睛跟踪系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201962902329P 2019-09-18 2019-09-18
US17/021,901 US11307659B2 (en) 2019-09-18 2020-09-15 Low-power eye tracking system

Publications (2)

Publication Number Publication Date
US20210081038A1 US20210081038A1 (en) 2021-03-18
US11307659B2 true US11307659B2 (en) 2022-04-19

Family

ID=74868585

Family Applications (4)

Application Number Title Priority Date Filing Date
US17/021,901 Active US11307659B2 (en) 2019-09-18 2020-09-15 Low-power eye tracking system
US17/021,920 Active US11442540B2 (en) 2019-09-18 2020-09-15 Eye tracking using low resolution images
US17/021,912 Active 2040-10-30 US11726560B2 (en) 2019-09-18 2020-09-15 Eye tracking using eye odometers
US18/347,485 Pending US20230341935A1 (en) 2019-09-18 2023-07-05 Eye Tracking Using Eye Odometers

Family Applications After (3)

Application Number Title Priority Date Filing Date
US17/021,920 Active US11442540B2 (en) 2019-09-18 2020-09-15 Eye tracking using low resolution images
US17/021,912 Active 2040-10-30 US11726560B2 (en) 2019-09-18 2020-09-15 Eye tracking using eye odometers
US18/347,485 Pending US20230341935A1 (en) 2019-09-18 2023-07-05 Eye Tracking Using Eye Odometers

Country Status (2)

Country Link
US (4) US11307659B2 (zh)
CN (3) CN112527099A (zh)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11307659B2 (en) 2019-09-18 2022-04-19 Apple Inc. Low-power eye tracking system
US12000931B2 (en) * 2020-05-15 2024-06-04 Analog Devices International Unlimited Company Continuous wave time of flight system
US11789266B1 (en) * 2020-06-29 2023-10-17 Snap Inc. Dynamic sensor selection for visual inertial odometry systems
US20220383512A1 (en) * 2021-05-27 2022-12-01 Varjo Technologies Oy Tracking method for image generation, a computer program product and a computer system
DE102021114026A1 (de) 2021-05-31 2022-12-01 Bayerische Motoren Werke Aktiengesellschaft Posenbestimmung von Datenbrillen in Fahrzeugen
SE545387C2 (en) * 2021-06-30 2023-07-25 Tobii Ab Method, computer program product, control unit and head-mounted display for conserving energy in an eye tracking system
US11874958B2 (en) * 2021-09-24 2024-01-16 Apple Inc. Eye detection methods and devices
EP4333449A4 (en) 2021-09-27 2024-10-16 Samsung Electronics Co Ltd PORTABLE DEVICE WITH CAMERA AND CONTROL METHOD THEREFOR
US20230094572A1 (en) * 2021-09-27 2023-03-30 Eyetech Digital Systems, Inc. Systems and Methods for Passive Calibration in Eye-Tracking System
US11994675B2 (en) * 2022-05-16 2024-05-28 Google Llc Head-mounted device for tracking screen time
US11995226B2 (en) 2022-06-10 2024-05-28 Sony Interactive Entertainment Inc. Dynamic vision sensor tracking based on light source occlusion
US12064682B2 (en) 2022-06-10 2024-08-20 Sony Interactive Entertainment Inc. Deployment of dynamic vision sensor hybrid element in method for tracking a controller and simultaneous body tracking, slam or safety shutter
US12059609B2 (en) 2022-06-10 2024-08-13 Sony Interactive Entertainment Inc. Asynchronous dynamic vision sensor LED AI tracking system and method
US12029971B2 (en) 2022-06-10 2024-07-09 Sony Interactive Entertainment LLC Hybrid pixel dynamic vision sensor tracking using IR and ambient light (or depth sensor)
US20230400915A1 (en) * 2022-06-10 2023-12-14 Sony Interactive Entertainment Inc. Dynamic vision sensor based eye and/or facial tracking

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150109192A1 (en) 2013-10-18 2015-04-23 Pixart Imaging Inc. Image sensing system, image sensing method, eye tracking system, eye tracking method
US20160364881A1 (en) 2015-06-14 2016-12-15 Sony Computer Entertainment Inc. Apparatus and method for hybrid eye tracking
US20170068315A1 (en) * 2015-09-07 2017-03-09 Samsung Electronics Co., Ltd. Method and apparatus for eye tracking
US20170192236A1 (en) 2015-12-31 2017-07-06 Beijing Pico Technology Co., Ltd. Method of Adapting a Virtual Reality Helmet
US20180184002A1 (en) 2016-12-23 2018-06-28 Microsoft Technology Licensing, Llc Eye Tracking Using Video Information and Electrooculography Information
US20180271364A1 (en) * 2017-03-24 2018-09-27 Osaka University Method and apparatus for detecting ocular movement disorders
US20190197765A1 (en) * 2017-12-22 2019-06-27 Magic Leap, Inc. Method of occlusion rendering using raycast and live depth
US20200348750A1 (en) 2019-04-30 2020-11-05 Valve Corporation Display system with dynamic light output adjustment for maintaining constant brightness
US20210081040A1 (en) 2019-09-18 2021-03-18 Apple Inc. Eye Tracking Using Low Resolution Images

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6873714B2 (en) * 2002-02-19 2005-03-29 Delphi Technologies, Inc. Auto calibration and personalization of eye tracking system using larger field of view imager with higher resolution
US7206022B2 (en) * 2002-11-25 2007-04-17 Eastman Kodak Company Camera system with eye monitoring
US9256987B2 (en) * 2013-06-24 2016-02-09 Microsoft Technology Licensing, Llc Tracking head movement when wearing mobile device
US9411417B2 (en) * 2014-07-07 2016-08-09 Logitech Europe S.A. Eye gaze tracking system and method
US9983709B2 (en) * 2015-11-02 2018-05-29 Oculus Vr, Llc Eye tracking using structured light
EP3329316B1 (en) * 2016-03-11 2023-09-20 Facebook Technologies, LLC Corneal sphere tracking for generating an eye model
US10268266B2 (en) * 2016-06-29 2019-04-23 Microsoft Technology Licensing, Llc Selection of objects in three-dimensional space
US10877556B2 (en) * 2016-10-21 2020-12-29 Apple Inc. Eye tracking system
US20180316901A1 (en) * 2017-04-26 2018-11-01 Ford Global Technologies, Llc Event reconstruct through image reporting
US10739850B2 (en) * 2017-09-29 2020-08-11 Sony Interactive Entertainment Inc. Prescription glasses with eye gaze tracking and electro optical signaling to a HMD
EP3514606A1 (en) * 2018-01-23 2019-07-24 BAE SYSTEMS plc Eye tracking for head-worn display

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150109192A1 (en) 2013-10-18 2015-04-23 Pixart Imaging Inc. Image sensing system, image sensing method, eye tracking system, eye tracking method
US20160364881A1 (en) 2015-06-14 2016-12-15 Sony Computer Entertainment Inc. Apparatus and method for hybrid eye tracking
US20170068315A1 (en) * 2015-09-07 2017-03-09 Samsung Electronics Co., Ltd. Method and apparatus for eye tracking
US20170192236A1 (en) 2015-12-31 2017-07-06 Beijing Pico Technology Co., Ltd. Method of Adapting a Virtual Reality Helmet
US20180184002A1 (en) 2016-12-23 2018-06-28 Microsoft Technology Licensing, Llc Eye Tracking Using Video Information and Electrooculography Information
US20180271364A1 (en) * 2017-03-24 2018-09-27 Osaka University Method and apparatus for detecting ocular movement disorders
US20190197765A1 (en) * 2017-12-22 2019-06-27 Magic Leap, Inc. Method of occlusion rendering using raycast and live depth
US20200348750A1 (en) 2019-04-30 2020-11-05 Valve Corporation Display system with dynamic light output adjustment for maintaining constant brightness
US20210081040A1 (en) 2019-09-18 2021-03-18 Apple Inc. Eye Tracking Using Low Resolution Images
US20210081039A1 (en) 2019-09-18 2021-03-18 Apple Inc. Eye Tracking Using Eye Odometers

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
U.S. Appl. No. 17/021,912, filed Sep. 15, 2020, Tom Sengelaub.
U.S. Appl. No. 17/021,920, filed Sep. 15, 2020, Tom Sengelaub.

Also Published As

Publication number Publication date
US20210081040A1 (en) 2021-03-18
US20210081039A1 (en) 2021-03-18
CN112527099A (zh) 2021-03-19
CN112515625A (zh) 2021-03-19
CN112515624A (zh) 2021-03-19
US20230341935A1 (en) 2023-10-26
US11726560B2 (en) 2023-08-15
US11442540B2 (en) 2022-09-13
CN112515624B (zh) 2024-04-30
US20210081038A1 (en) 2021-03-18

Similar Documents

Publication Publication Date Title
US11307659B2 (en) Low-power eye tracking system
US11217021B2 (en) Display system having sensors
US10877556B2 (en) Eye tracking system
US11330241B2 (en) Focusing for virtual and augmented reality systems
US11360557B2 (en) Eye tracking system
US9710973B2 (en) Low-latency fusing of virtual and real content
US9230368B2 (en) Hologram anchoring and dynamic positioning
US11792531B2 (en) Gaze-based exposure
US20130326364A1 (en) Position relative hologram interactions
US20140176591A1 (en) Low-latency fusing of color image data
US20240053823A1 (en) Eye Tracking System
US12039632B2 (en) Synthesized camera arrays for rendering novel viewpoints
US20230421914A1 (en) Gaze-Based Exposure
US11327561B1 (en) Display system
US20230315201A1 (en) Stray light mitigation in optical systems
US20240211039A1 (en) Visual Axis Enrollment
US20240103618A1 (en) Corrected Gaze Direction and Origin
US20240105046A1 (en) Lens Distance Test for Head-Mounted Display Devices
US20240104967A1 (en) Synthetic Gaze Enrollment
US20240104958A1 (en) User Eye Model Match Detection
WO2024138044A1 (en) Visual axis enrollment

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: APPLE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SENGELAUB, TOM;REEL/FRAME:053785/0587

Effective date: 20200910

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE