WO2024138044A1 - Inscription d'axe visuel - Google Patents

Inscription d'axe visuel Download PDF

Info

Publication number
WO2024138044A1
WO2024138044A1 PCT/US2023/085526 US2023085526W WO2024138044A1 WO 2024138044 A1 WO2024138044 A1 WO 2024138044A1 US 2023085526 W US2023085526 W US 2023085526W WO 2024138044 A1 WO2024138044 A1 WO 2024138044A1
Authority
WO
WIPO (PCT)
Prior art keywords
eye
content
line
kappa angle
images
Prior art date
Application number
PCT/US2023/085526
Other languages
English (en)
Inventor
Hao Qin
Hua Gao
Tom Sengelaub
Original Assignee
Apple Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US18/534,398 external-priority patent/US20240211039A1/en
Application filed by Apple Inc. filed Critical Apple Inc.
Publication of WO2024138044A1 publication Critical patent/WO2024138044A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted

Definitions

  • Extended reality (XR) systems such as mixed reality (MR) or augmented reality (AR) systems combine computer generated information (referred to as virtual content) with real world images or a real -world view to augment, or add content to, a user’ s view of the world.
  • XR systems may thus be utilized to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of the viewer’s environment, interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the Internet, or the like.
  • HMDs head-mounted devices
  • XR extended reality
  • HMDs may include wearable devices such as headsets, helmets, goggles, or glasses.
  • An XR system may include an HMD which may include one or more cameras that may be used to capture still images or video frames of the user’s environment.
  • the HMD may include lenses positioned in front of the eyes through which the wearer can view the environment.
  • virtual content may be displayed on or projected onto these lenses to make the virtual content visible to the wearer while still being able to view the real environment through the lenses.
  • the HMD may include gaze tracking technology.
  • the gaze tracking may be performed for one or for both eyes.
  • a multidimensional personalized model of the user’s eye(s) may be generated from one or more images of the eye(s) captured by eye tracking camera(s). This can be done unobtrusively, without prompting, as the user moves their eyes.
  • the personalized eye model(s) generated from the images captured as the eyes move may include information such as a cornea surface model, iris and pupil model, eye center, entrance pupil, and a pupillary or optical axis (a vector which passes through the geometric eye center and the entrance pupil).
  • an eye’s actual gaze direction corresponds to a visual axis, which is offset from the calculated optical axis of the eye model.
  • another part of the calibration or enrollment process is to estimate the visual axis, or kappa angle between the optical axis and the visual axis.
  • Embodiments of methods and apparatus to inconspicuously and unobtrusively estimate and enroll the visual axis for the eye(s) are described.
  • a line of text (or other content) is displayed at a known vertical location and virtual depth that the user can then read.
  • This line of text may be content that the user needs to read as part of the enrollment process, such as “Please read and agree to the following terms of use.”
  • the line of text may be the only content displayed (or displayed at the known virtual depth) at the time. In other words, in some embodiments, only a single horizontal line of content such as text needs to be displayed.
  • the eye tracking cameras may capture images of the eye. This data may then be processed to estimate a stimulus plane.
  • the error between the estimated stimulus plane and the ground truth stimulus plane (the actual location of the line of text in virtual space) may then be determined and used to estimate the kappa angle, and thus the true visual axis of the eye.
  • This method of estimating the visual axis avoids an extra step in the eye enrollment process, and is thus more efficient and less intrusive than previous methods.
  • the user reads a line of text, and the eye model and visual axis are enrolled. No explicit eye enrollment process with prompts to the user is necessary.
  • the personalized eye model and the estimated kappa angle/visual axis may then be used in various algorithms, for example in the gaze estimation process for a gaze-based interface, during use of the device.
  • FIG. 1 graphically illustrates an N-dimensional model of an eye, according to some embodiments.
  • FIG. 2 broadly illustrates an unobtrusive visual axis enrollment process, according to some embodiments.
  • FIG. 3 graphically illustrates an eye enrollment process, according to some embodiments.
  • FIG. 4 illustrates an unobtrusive visual axis enrollment process in more detail, according to some embodiments.
  • FIG. 5A is a high-level flowchart of an eye enrollment process, according to some embodiments.
  • FIG. 5B is a high-level flowchart of an unobtrusive visual axis enrollment process, according to some embodiments.
  • FIGS. 6A through 6C illustrate example devices in which the methods of FIGS. 1 through 5B may be implemented, according to some embodiments.
  • FIG. 7 is a block diagram illustrating an example device that may include components and implement methods as illustrated in FIGS. 1 through 5B, according to some embodiments.
  • a unit/circuit/component is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. ⁇ 112, paragraph (f), for that unit/circuit/component.
  • “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue.
  • “Configure to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.
  • a determination may be solely based on those factors or based, at least in part, on those factors.
  • a determination may be solely based on those factors or based, at least in part, on those factors.
  • HMDs head-mounted devices
  • XR extended reality
  • HMDs may include wearable devices such as headsets, helmets, goggles, or glasses.
  • An XR system may include an HMD which may include one or more cameras that may be used to capture still images or video frames of the user’s environment.
  • the HMD may include lenses positioned in front of the eyes through which the wearer can view the environment.
  • virtual content may be displayed on or projected onto these lenses to make the virtual content visible to the wearer while still being able to view the real environment through the lenses.
  • the HMD may include gaze tracking technology.
  • one or more infrared (IR) light sources emit IR light towards a user’ s eye. A portion of the IR light is reflected off the eye and captured by an eye tracking camera. Images captured by the eye tracking camera may be input to a glint and pupil detection process, for example implemented by one or more processors of a controller of the HMD. Results of the process are passed to a gaze estimation process, for example implemented by one or more processors of the controller, to estimate the user’s current point of gaze.
  • This method of gaze tracking may be referred to as PCCR (Pupil Center Corneal Reflection) tracking. Note that the gaze tracking may be performed for one or for both eyes.
  • PCCR Personal Center Corneal Reflection
  • a multidimensional personalized model of the user’ s eye(s) may be generated from one or more images of the eye(s) captured by the eye tracking camera as described above.
  • FIG. 1 graphically illustrates an N- dimensional model 100 of an eye, according to some embodiments. Physical components of an eye may include a sclera 102, cornea 104, iris 106, and pupil 108.
  • an N-dimensional model of the user’s eye 100 may be generated from one or more images of the eye 100.
  • one or more infrared (IR) light sources emit IR light towards a user’s eye.
  • a portion of the IR light is reflected off the eye and captured by an eye tracking camera.
  • Two or more images captured by the eye tracking camera may be input to an eye model generating process, for example implemented by one or more processors of a controller of the HMD.
  • the process may determine the shapes and relationships of the eye’s components based at least in part on positions of the glints (reflections of the point light sources) in the two or more captured images. This information may then be used to generate the personalized eye model.
  • the personalized eye model may include information such as a cornea surface model, iris and pupil model, eye center 112, entrance pupil 110, and pupillary or optical axis 120 (a vector which passes through the eye center 112 and the entrance pupil 110). This personalized eye model may then be used in various algorithms, for example in the gaze estimation process, during use of the device.
  • an eye’s actual gaze direction corresponds to a visual axis 122, which is offset from the calculated optical axis 120 of the eye model.
  • another part of the calibration or enrollment process is to estimate the visual axis 122, or kappa angle 124 between the optical axis 120 and the visual axis 122.
  • Embodiments of methods and apparatus to inconspicuously and unobtrusively estimate and enroll the visual axis 122 for the eye(s) are described.
  • a relatively long line 230 of text (or other content) is displayed as a stimulus at a known vertical location and virtual depth that the user can then read.
  • This line of text 230 may include content that the user needs to read as part of the enrollment process, such as “Please read and agree to the following terms of use.”
  • the line of text 230 may be the only content (or the only content at that close of a distance) displayed at the time, and the alphanumeric characters in the line may be of a relatively small vertical size.
  • the eye tracking cameras may capture images of the eye(s) 290. This image data may then be processed to estimate a stimulus plane. The error between the estimated stimulus plane and the ground truth stimulus plane (the actual location of the line of text in virtual space) may then be used to estimate the kappa angle and thus the visual axis of the eye(s) 290.
  • the gaze data with the strongest vergence can be located using eye pose only.
  • the displayed text 230 (the stimulus) is a thin line, which allows the pitch component of the kappa angle to be determined.
  • left/right yaw angles can be estimated under the constraint that the vergence point is always at the same depth for different gaze angles.
  • the personalized eye model and the estimated kappa angle/visual axis may then be used in various algorithms, for example in the gaze estimation process for a gaze-based interface, during use of the device.
  • This method of estimating the visual axis using an unobtrusive stimulus avoids an extra step in the eye enrollment process, and is thus more efficient and less intrusive than previous methods.
  • the user reads the line of text, and the eye model and visual axis are enrolled. No explicit eye enrollment process with prompts to the user is necessary.
  • FIG. 3 graphically illustrates an eye enrollment process, according to some embodiments.
  • an N-dimensional personalized eye model 304 may be generated from two or more images of the eye(s) captured by the eye tracking camera of the eye(s) (gaze input 302). This can be done in the background during a user enrollment process after the user puts on and/or turns on the device, and can be done without specifically prompting the user to do anything specific for eye enrollment.
  • the personalized eye model 304 may include information such as a cornea surface model, iris and pupil model, eye center, entrance pupil, and pupillary or optical axis 308 (a vector which passes through the eye center and the entrance pupil).
  • an eye’s actual gaze direction corresponds to a visual axis, which is offset from the calculated optical axis 306 of the eye model 304.
  • a visual axis 308, or kappa angle between the optical axis 306 and the visual axis 308, is estimated.
  • a relatively long line of text (or other content) is displayed as a stimulus at a known vertical location and virtual depth that the user can then read.
  • the eye tracking cameras capture images of the eye(s) (gaze input 312).
  • This gaze input 312 may then be processed to estimate a stimulus plane.
  • the error between the estimated stimulus plane and the ground truth stimulus plane (the actual location of the line of text in virtual space) may then be used to estimate the kappa angle and thus the visual axis of the eye(s).
  • the gaze data with the strongest vergence can be located using eye pose only.
  • the displayed text (the stimulus) is a thin line, which allows the pitch component of the kappa angle to be determined.
  • left/right yaw angles can be estimated under the constraint that the vergence point is always at the same depth for different gaze angles.
  • the personalized eye model 304 and the visual axis 308 may then be used in various algorithms, for example in gaze-based interaction functions of the device.
  • the eye tracking cameras capture images of the user’s eyes during these functions, and gaze tracking algorithms of the device process these images using the eye model 304 and visual axis 308 to determine gaze direction of the eyes, with gaze direction corrected from the optical axis 306 to the visual axis 308 according to the kappa angle.
  • FIG. 4 illustrates an unobtrusive visual axis enrollment process in more detail, according to some embodiments.
  • the process may begin with an initial default kappa angle (e.g., 0).
  • Stimulus plane (ground truth) 430 corresponds to the line of text (or other content) displayed at a known vertical location and virtual depth, and the circles represent the vergence point.
  • a stimulus plane 432 is estimated from gaze tracking data (using the eye model and default kappa angle).
  • the error 434 between the estimated stimulus plane 432 and the ground truth stimulus plane 430 can then be estimated. This error 434 can be used to estimate the true kappa angle.
  • FIG. 5A is a high-level flowchart of an eye enrollment process, according to some embodiments.
  • an N-dimensional personalized eye model is generated.
  • a visual axis is estimated.
  • FIG. 5B provides more detail of the visual axis estimation process.
  • the eye model and visual axis may be stored.
  • the eye model and kappa angle may be used in performing gaze tracking during use of the device.
  • FIG. 5B is a high-level flowchart of an unobtrusive visual axis enrollment process, according to some embodiments.
  • the process may begin with an initial default kappa angle (e.g., 0).
  • a line of virtual text may be displayed at a known virtual depth and height as a stimulus.
  • the line of text may be the only content displayed (or displayed at the known virtual depth) at the time. In other words, in some embodiments, only a single horizontal line of content such as text needs to be displayed.
  • eye tracking cameras capture images of the eyes; eye pose data is generated from the captured image data.
  • a stimulus plane is estimated from the gaze tracking data (using the eye model and default kappa angle). As indicated at 626, a kappa angle is estimated based at least in part on the data generated from the pose of the eye as the user reads the text. The error 434 between the estimated stimulus plane and the ground truth stimulus plane (the line of text can then be estimated; this error can then be used to estimate the true kappa angle. The kappa angle indicates the true visual axis of the eye.
  • FIGS. 6A through 6C illustrate example devices in which the methods of FIGS. 1 through 5B may be implemented, according to some embodiments.
  • the HMDs 1000 as illustrated in FIGS. 6 A through 6C are given by way of example, and are not intended to be limiting.
  • the shape, size, and other features of an HMD 1000 may differ, and the locations, numbers, types, and other features of the components of an HMD 1000 and of the eye imaging system.
  • FIG. 6A shows a side view of an example HMD 1000
  • FIGS. 6B and 6C show alternative front views of example HMDs 1000, with FIG. 6A showing device that has one lens 1030 that covers both eyes and FIG. 6B showing a device that has right 1030 A and left 1030B lenses.
  • HMD 1000 may include lens(es) 1030, mounted in a wearable housing or frame 1010. HMD 1000 may be worn on a user’s head (the “wearer”) so that the lens(es) is disposed in front of the wearer’s eyes.
  • an HMD 1000 may implement any of various types of display technologies or display systems.
  • HMD 1000 may include a display system that directs light that forms images (virtual content) through one or more layers of waveguides in the lens(es) 1020; output couplers of the waveguides (e.g., relief gratings or volume holography) may output the light towards the wearer to form images at or near the wearer’s eyes.
  • HMD 1000 may include a direct retinal projector system that directs light towards reflective components of the lens(es); the reflective lens(es) is configured to redirect the light to form images at the wearer’s eyes.
  • HMD 1000 may also include one or more sensors that collect information about the wearer’s environment (video, depth information, lighting information, etc.) and about the wearer (e.g., eye or gaze tracking sensors).
  • the sensors may include one or more of, but are not limited to one or more eye tracking cameras 1020 (e.g., infrared (IR) cameras) that capture views of the user’s eyes, one or more world-facing or PoV cameras 1050 (e.g., RGB video cameras) that can capture images or video of the real-world environment in a field of view in front of the user, and one or more ambient light sensors that capture lighting information for the environment.
  • Cameras 1020 and 1050 may be integrated in or attached to the frame 1010.
  • HMD 1000 may also include one or more light sources 1080 such as LED or infrared point light sources that emit light (e.g., light in the IR portion of the spectrum) towards the user’s eye or eyes.
  • a controller 1060 for the XR system may be implemented in the HMD 1000, or alternatively may be implemented at least in part by an external device (e.g., a computing system or handheld device) that is communicatively coupled to HMD 1000 via a wired or wireless interface.
  • Controller 1060 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), system on a chip (SOC), CPUs, and/or other components for processing and rendering video and/or images.
  • controller 1060 may render frames (each frame including a left and right image) that include virtual content based at least in part on inputs obtained from the sensors and from an eye tracking system, and may provide the frames to the display system.
  • Memory 1070 for the XR system may be implemented in the HMD 1000, or alternatively may be implemented at least in part by an external device (e.g., a computing system) that is communicatively coupled to HMD 1000 via a wired or wireless interface.
  • the memory 1070 may, for example, be used to record video or images captured by the one or more cameras 1050 integrated in or attached to frame 1010.
  • Memory 1070 may include any type of memory, such as dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such as mDDR3, etc., or low power versions of the SDRAMs such as LPDDR2, etc ), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc.
  • DRAM dynamic random-access memory
  • SDRAM synchronous DRAM
  • DDR double data rate
  • DDR double data rate
  • RDRAM RAMBUS DRAM
  • SRAM static RAM
  • one or more memory devices may be coupled onto a circuit board to form memory modules such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc.
  • the devices may be mounted with an integrated circuit implementing system in a chip-on-chip configuration, a package-on- package configuration, or a multi-chip module configuration.
  • DRAM may be used as temporary storage of images or video for processing, but other
  • FIGS. 6 A through 6C only show light sources 1080 and cameras 1020 and 1050 for one eye
  • embodiments may include light sources 1080 and cameras 1020 and 1050 for each eye, and gaze tracking may be performed for both eyes.
  • the light sources, 1080, eye tracking camera 1020 and PoV camera 1050 may be located elsewhere than shown.
  • Embodiments of an HMD 1000 as illustrated in FIGS. 6A-6C may, for example, be used in augmented or mixed (AR) applications to provide augmented or mixed reality views to the wearer.
  • HMD 1000 may include one or more sensors, for example located on external surfaces of the HMD 1000, that collect information about the wearer’s external environment (video, depth information, lighting information, etc.); the sensors may provide the collected information to controller 1060 of the XR system.
  • the sensors may include one or more visible light cameras 1050 (e.g., RGB video cameras) that capture video of the wearer’s environment that, in some embodiments, may be used to provide the wearer with a virtual view of their real environment.
  • visible light cameras 1050 e.g., RGB video cameras
  • video streams of the real environment captured by the visible light cameras 1050 may be processed by the controller 1060 of the HMD 1000 to render augmented or mixed reality frames that include virtual content overlaid on the view of the real environment, and the rendered frames may be provided to the display system.
  • input from the eye tracking camera 1020 may be used in a PCCR gaze tracking process executed by the controller 1060 to track the gaze/pose of the user’s eyes for use in rendering the augmented or mixed reality content for display.
  • one or more of the methods as illustrated in FIGS. 1 through 5B may be implemented in the HMD to provide unobtrusive visual axis enrollment for the HMD 1000.
  • FIG. 7 is a block diagram illustrating an example device that may include components and implement methods as illustrated in FIGS. 1 through 5B, according to some embodiments.
  • an XR system may include a device 2000 such as a headset, helmet, goggles, or glasses.
  • Device 2000 may implement any of various types of display technologies.
  • device 2000 may include a transparent or translucent display 2060 (e.g., eyeglass lenses) through which the user may view the real environment and a medium integrated with display 2060 through which light representative of virtual images is directed to the wearer’s eyes to provide an augmented view of reality to the wearer.
  • a transparent or translucent display 2060 e.g., eyeglass lenses
  • device 2000 may include a controller 2060 configured to implement functionality of the XR system and to generate frames (each frame including a left and right image) that are provided to display 2030.
  • device 2000 may also include memory 2070 configured to store software (code 2074) of the XR system that is executable by the controller 2060, as well as data 2078 that may be used by the XR system when executing on the controller 2060.
  • memory 2070 may also be used to store video captured by camera 2050.
  • device 2000 may also include one or more interfaces (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device (not shown) via a wired or wireless connection.
  • interfaces e.g., a Bluetooth technology interface, USB interface, etc.
  • controller 2060 may be implemented by the external device.
  • the external device may be or may include any type of computing system or computing device, such as a desktop computer, notebook or laptop computer, pad or tablet device, smartphone, hand-held computing device, game controller, game system, and so on.
  • controller 2060 may be a uniprocessor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or another suitable number).
  • Controller 2060 may include central processing units (CPUs) configured to implement any suitable instruction set architecture, and may be configured to execute instructions defined in that instruction set architecture.
  • CPUs central processing units
  • controller 2060 may include general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of the processors may commonly, but not necessarily, implement the same ISA. Controller 2060 may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, out of order, in order, speculative, non- speculative, etc., or combinations thereof. Controller 2060 may include circuitry to implement microcoding techniques. Controller 2060 may include one or more processing cores each configured to execute instructions.
  • ISAs instruction set architectures
  • Controller 2060 may include one or more levels of caches, which may employ any size and any configuration (set associative, direct mapped, etc.).
  • controller 2060 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuitry.
  • GPU graphics processing unit
  • a GPU may be configured to render objects to be displayed into a frame buffer (e.g., one that includes pixel data for an entire frame).
  • a GPU may include one or more graphics processors that may execute graphics software to perform a part or all of the graphics operation, or hardware acceleration of certain graphics operations.
  • controller 2060 may include one or more other components for processing and rendering video and/or images, for example image signal processors (ISPs), coder/decoders (codecs), etc.
  • ISPs image signal processors
  • codecs codecs
  • Memory 2070 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such as mDDR3, etc., or low power versions of the SDRAMs such as LPDDR2, etc ), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc.
  • DRAM dynamic random access memory
  • SDRAM synchronous DRAM
  • DDR double data rate SDRAM
  • RDRAM RAMBUS DRAM
  • SRAM static RAM
  • one or more memory devices may be coupled onto a circuit board to form memory modules such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc.
  • the devices may be mounted with an integrated circuit implementing system in a chip-on-chip configuration, a package-on-package configuration, or a multi-chip module configuration.
  • DRAM may be used as temporary storage of images or video for processing, but other storage options may be used to store processed data, such as Flash or other “hard drive” technologies.
  • device 2000 may include one or more sensors that collect information about the user’s environment (video, depth information, lighting information, etc.). The sensors may provide the information to the controller 2060 of the XR system.
  • the sensors may include, but are not limited to, at least one visible light camera (e.g., an RGB video camera) 2050, ambient light sensors, and at least on eye tracking camera 2020.
  • device 2000 may also include one or more IR light sources; light from the light sources reflected off the eye may be captured by the eye tracking camera 2020. Gaze tracking algorithms implemented by controller 2060 may process images or video of the eye captured by the camera 2020 to determine eye pose and gaze direction. In addition, one or more of the methods as illustrated in FIGS. 1 through 5 may be implemented in device 2000 to provide unobtrusive visual axis enrollment for the device 2000.
  • device 2000 may be configured to render and display frames to provide an augmented or mixed reality (MR) view for the user based at least in part according to sensor inputs, including input from the eye tracking camera 2020.
  • the MR view may include renderings of the user’s environment, including renderings of real objects in the user’s environment, based on video captured by one or more video cameras that capture high-quality, high-resolution video of the user’ s environment for display.
  • the MR view may also include virtual content (e.g., virtual objects, virtual tags for real objects, avatars of the user, etc.) generated by the XR system and composited with the displayed view of the user’s real environment.
  • a real environment refers to an environment that a person can perceive (e.g., see, hear, feel) without use of a device.
  • an office environment may include furniture such as desks, chairs, and filing cabinets; structural items such as doors, windows, and walls; and objects such as electronic devices, books, and writing instruments.
  • a person in a real environment can perceive the various aspects of the environment, and may be able to interact with objects in the environment.
  • An extended reality (XR) environment is partially or entirely simulated using an electronic device.
  • XR extended reality
  • a user may see or hear computer generated content that partially or wholly replaces the user’s perception of the real environment.
  • a user can interact with an XR environment.
  • the user’s movements can be tracked and virtual objects in the XR environment can change in response to the user’s movements.
  • a device presenting an XR environment to a user may determine that a user is moving their hand toward the virtual position of a virtual object, and may move the virtual object in response.
  • a user’s head position and/or eye gaze can be tracked and virtual objects can move to stay in the user’s line of sight.
  • Examples of XR include augmented reality (AR), virtual reality (VR) and mixed reality (MR).
  • AR augmented reality
  • VR virtual reality
  • MR mixed reality
  • XR can be considered along a spectrum of realities, where VR, on one end, completely immerses the user, replacing the real environment with virtual content, and on the other end, the user experiences the real environment unaided by a device. In between are AR and MR, which mix virtual content with the real environment.
  • VR generally refers to a type of XR that completely immerses a user and replaces the user’ s real environment.
  • VR can be presented to a user using a head mounted device (HMD), which can include a near-eye display to present a virtual visual environment to the user and headphones to present a virtual audible environment.
  • HMD head mounted device
  • the movement of the user can be tracked and cause the user’s view of the environment to change.
  • a user wearing a HMD can walk in the real environment and the user will appear to be walking through the virtual environment they are experiencing.
  • the user may be represented by an avatar in the virtual environment, and the user’s movements can be tracked by the HMD using various sensors to animate the user’s avatar.
  • AR and MR refer to a type of XR that includes some mixture of the real environment and virtual content.
  • a user may hold a tablet that includes a camera that captures images of the user’s real environment.
  • the tablet may have a display that displays the images of the real environment mixed with images of virtual objects.
  • AR or MR can also be presented to a user through an HMD.
  • An HMD can have an opaque display, or can use a see-through display, which allows the user to see the real environment through the display, while displaying virtual content overlaid on the real environment.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Dans un processus d'inscription d'axe visuel imperceptible, une ligne de texte ou un autre contenu est affiché à un emplacement vertical connu et à une profondeur virtuelle que l'utilisateur peut ensuite lire. Cette ligne de texte peut être un contenu que l'utilisateur doit lire en tant que partie du processus d'inscription normal. Lorsque l'utilisateur lit la ligne de texte, des caméras de suivi de l'œil peuvent capturer des images de l'œil. Ces données peuvent être ensuite utilisées pour estimer un plan de stimulus. L'erreur entre le plan de stimulus estimé et le plan de stimulus de réalité de terrain (l'emplacement réel de la ligne de texte dans l'espace virtuel) peut ensuite être utilisée pour estimer l'angle kappa.
PCT/US2023/085526 2022-12-22 2023-12-21 Inscription d'axe visuel WO2024138044A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202263476924P 2022-12-22 2022-12-22
US63/476,924 2022-12-22
US18/534,398 2023-12-08
US18/534,398 US20240211039A1 (en) 2022-12-22 2023-12-08 Visual Axis Enrollment

Publications (1)

Publication Number Publication Date
WO2024138044A1 true WO2024138044A1 (fr) 2024-06-27

Family

ID=89845313

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2023/085526 WO2024138044A1 (fr) 2022-12-22 2023-12-21 Inscription d'axe visuel

Country Status (1)

Country Link
WO (1) WO2024138044A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016073131A1 (fr) * 2014-11-06 2016-05-12 Intel Corporation Étalonnage amélioré pour des systèmes de suivi oculaire
US20170263007A1 (en) * 2016-03-11 2017-09-14 Oculus Vr, Llc Eye tracking system with single point calibration
US20210011549A1 (en) * 2019-03-29 2021-01-14 Tobii Ab Updating a cornea model

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016073131A1 (fr) * 2014-11-06 2016-05-12 Intel Corporation Étalonnage amélioré pour des systèmes de suivi oculaire
US20170263007A1 (en) * 2016-03-11 2017-09-14 Oculus Vr, Llc Eye tracking system with single point calibration
US20210011549A1 (en) * 2019-03-29 2021-01-14 Tobii Ab Updating a cornea model

Similar Documents

Publication Publication Date Title
US10877556B2 (en) Eye tracking system
US11360557B2 (en) Eye tracking system
US11330241B2 (en) Focusing for virtual and augmented reality systems
US11217021B2 (en) Display system having sensors
US11442537B2 (en) Glint-assisted gaze tracker
US11792531B2 (en) Gaze-based exposure
US20240053823A1 (en) Eye Tracking System
US20240272709A1 (en) Eye model enrollment
US20230421914A1 (en) Gaze-Based Exposure
US20230315201A1 (en) Stray light mitigation in optical systems
US20240211039A1 (en) Visual Axis Enrollment
US20240211038A1 (en) Gesture-Initiated Eye Enrollment
US20240104967A1 (en) Synthetic Gaze Enrollment
EP4343497A1 (fr) Direction et origine du regard corrigées
WO2024138044A1 (fr) Inscription d'axe visuel
WO2024138025A1 (fr) Inscription oculaire initiée par un geste
WO2024064378A1 (fr) Inscription de regard synthétique
US20240104958A1 (en) User Eye Model Match Detection
US20240004190A1 (en) Eye Imaging System
US20240105046A1 (en) Lens Distance Test for Head-Mounted Display Devices
US20240211569A1 (en) Biometric Multi-Representation Eye Authentication
CN117761893A (zh) 经校正凝视方向和原点
WO2024064376A1 (fr) Détection de correspondance de modèle d'œil d'utilisateur
WO2024138036A1 (fr) Authentification d'œil à représentations multiples biométriques

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23848512

Country of ref document: EP

Kind code of ref document: A1